Your circuit breaker stops at the service layer. Slow SQL needs one too.
Your circuit breaker stops at the service layer. Slow SQL needs one too. A single slow query can take down an entire service in seconds. This post starts with a production cascade failure and walks through how. . .
Your circuit breaker stops at the service layer. Slow SQL needs one too. A single slow query can take down an entire service in seconds. This post starts with a production cascade failure and walks through how I built a Spring Boot Starter that does circuit breakingat the MyBatis interceptor layer, keyed by SQL type + SQL fingerprint — plus a few design trade-offs worth talking about. The SDK is open source, on Maven Central, and takes one dependency + a bit of YAML to wire in. 🔗Source:GitHub·Gitee(a Star ⭐ helps if you find it useful)1. A short war story: how one slow query took down a serviceOne evening at peak traffic, the order service started timing out everywhere and alerts went off. The root cause was mundane: a list query had picked up a new filter condition that wasn't covered by an index, turned into a full table scan, and took 30+ seconds per call. The thing is,it doesn't stop at "slow once":One slow query holds a DB connection hostage for 30 seconds. Under load, more of the same request keep coming in, eating connections one by one. The connection pool drains → everyotherquery (including perfectly healthy ones) can't get a connection, queues up, times out. Upstream threads all block waiting for connections → endpoints time out → callers retry → it gets worse. One slow query ends up dragging downthe whole database, and the whole service. In the post-mortem the first reaction was:"Don't we already have a circuit breaker?"2. Why the usual circuit breakers aren't enoughResilience4j, Hystrix, Sentinel — all mature. But they operate at theendpoint / RPC / method-calllevel. They can tell you"this endpoint's failure rate is high, trip it", but theydon't understand SQL:They don't knowwhichquery is slow — they can only trip the whole endpoint, even though 90% of the SQL behind it might be perfectly healthy. They aggregate byendpoint, but the actual fault lives at the level ofone class of SQL. Within a single endpoint,select * from order where . . . being slow says nothing about whether theinsertis fine. They can't catch SQL that doesn't sit on an obvious boundary (scheduled jobs, Mapper calls buried deep in a call chain). What I actually wanted was a breaker whose granularity lands exactly at"SQL type + SQL shape":When oneclassof SQL goes bad, fast-fail only that class — leave the rest alone. And the trip has to happenbefore the request is actually sent to the DB, so the connection pool stays protected. The MyBatis / MyBatis-PlusInterceptor(plugin) is exactly the right seam: every CRUD statement passes through it, you can read the SQL, measure the execution time, and interceptbeforeit runs. So I built this. 3. What it looks like: two steps to wire inThe punchline first; the rest is just the "why". Integration iszero-touchfor your business code:1. Add the dependency(Spring Boot 2. x; for 3. x use-spring-boot3-starter):io. github. showingdata. starter. frameworksql-circuit-breaker-spring-boot-starter2. 1. 5Enter fullscreen modeExit fullscreen mode2. Configure the YAML(per SQL type; all four are required):sql-circuit-breaker:enabled:trueselect:timeout-ms:10000# SELECT timeout thresholdfailure-threshold:3# consecutive timeouts that trip the breakercircuit-open-ms:60000# how long the breaker stays open (60s)cache-max-size:10000# max entries in the breaker-state cacheinsert:{timeout-ms:5000,failure-threshold:1,circuit-open-ms:30000,cache-max-size:5000}update:{timeout-ms:5000,failure-threshold:1,circuit-open-ms:30000,cache-max-size:5000}delete:{timeout-ms:5000,failure-threshold:1,circuit-open-ms:30000,cache-max-size:5000}Enter fullscreen modeExit fullscreen modeRestart and you're done —not a single line of business code changes. From then on, when a class of SQL times out enough times in a row, it trips: while open, that class of SQLfast-fails locally and never reaches the DB, giving the database room to breathe; it recovers automatically when the window expires. Why split SELECT and DML config? Because their risk profiles are completely different. DML holds locks and has a wide blast radius — often "one timeout should trip it". SELECT comes in many shapes and is more tolerant — you can afford "three in a row before tripping". A single one-size-fits-all threshold just doesn't make sense. 4. The design, and the parts worth talking about4. 1 The matching unit: SQL fingerprint, not the full SQLIf you key on the full SQL text, thenwhere user_id = 123andwhere user_id = 456become two different statements with separate counters — which is wrong: they share the same shape, and they're slow because of thequery pattern, not the specific parameter. So the matching unit is theSQL fingerprint: normalize the parameters away, keep the structure. -- raw (two calls, different params)SELECT*FROMorderWHEREuser_id=123ANDstatus=1SELECT*FROMorderWHEREuser_id=456ANDstatus=2-- fingerprint (identical)select*fromorderwhereuser_id=?andstatus=?Enter fullscreen modeExit fullscreen modeThe rule is simple: replace parameter placeholders (?/#{xxx}) with?, collapse whitespace, lowercase, take the MD5 as the key. That wayone trip protects the entire class of SQL, instead of the count getting "diluted" across parameter values and never reaching the threshold. The final breaker key is:datasource_id : sql_type : fingerprint_md5 e. g. default:SELECT:a3f2c1. . . Enter fullscreen modeExit fullscreen modeThe datasource prefix is for multi-datasource setups (more on that later), the middle is the SQL type, the tail is the fingerprint MD5 (to avoid absurdly long keys). 4. 2 State machine: two states, no half-openMany circuit breakers are three-state: CLOSED → OPEN → HALF_OPEN (probing). Ideliberately kept it to two:consecutive timeouts >= failureThreshold CLOSED ──────────────────────────────────────────→ OPEN ↑ │ └──────── auto-reset when circuitOpenMs elapses ───┘Enter fullscreen modeExit fullscreen modeWhy drop half-open?Half-open means managing "probe permits" (let a few requests through to test the waters) and reconciling those permits under concurrency — non-trivial complexity. And the SQL breaker'sfailure-thresholdis tiny to begin with (default 3, DML even 1). After the open window expires, it just fully reopens;even if the fault isn't fixed, it re-trips within a couple of failures. The cost of "re-trip fast" is entirely acceptable here. Two states are simple to reason about and easy on ops: "either it's open or it isn't", no mysterious middle state. One engineering detail: while OPEN, requests that werealready in flight before the tripand only time out late donotadd to the count or refresh the window — so the open window stays exactlycircuit-open-msand never gets silently extended by stragglers. 4. 3 A misconfiguration I designed away: cache "expire-after-access" isn't configurableEach SQL fingerprint's breaker state lives in a Guava Cache (four independent caches, one per SQL type). Each cache has two eviction policies:PolicySourcePurposeLRU size capcache-max-size(per type)Hard memory ceiling, prevents unbounded growthexpire-after-accessderived fromcircuit-open-ms(20× and at least 5 min),notconfigurableCleans up long-idle SQLThe point is that second row:I deliberately don't let you configure expire-after-access. Because it has a hard constraint with the open window: the access-expiry must besignificantly larger thancircuit-open-ms. Otherwise you get a subtle bug — a query trips and is sitting in OPEN, but during that window no new requests come in (it's fast-failing, and callers may have backed off), soits state gets evicted by access-expiry prematurely; the next request finds no state, treats it as brand-new CLOSED, and lets it through… and your protection just quietly weakened. By deriving it fromcircuit-open-ms, the "two values configured backwards" mistake becomes impossible. The memory ceiling is still enforced bycache-max-size; access-expiry only handles idle cleanup. If a constraint can eliminate a misconfiguration, don't leave it as a config knob. 4. 4 It trips on timeouts only, not on exceptionsThis is a deliberate boundary: exceptions thrown by SQL execution — connection errors, syntax errors, constraint violations — arenever countedtoward the breaker. Only "execution time exceeded the threshold" counts as a failure. Why? Because the breaker protects against one specific failure mode:slow SQL exhausting the connection pool. A syntax error is a code bug; a constraint violation is a data problem. Neither holds connections hostage or drags down the DB, and counting them would only causefalse trips(a temporarily-erroring query gets tripped, which actually masks the real business problem). Single responsibility keeps it predictable. 4. 5 A performance detail under load: the exception doesn't fill in its stack traceTheSqlCircuitBreakerExceptionthrown on fast-fail overridesfillInStackTrace()to skip stack capture entirely. Because during a high-concurrency trip, this exception may be thrown thousands of times per second, andfillInStackTraceis one of the more expensive things the JVM does (it walks the entire call stack). Skipping it saves a lot of CPU/memory. But that buys you agotcha you must know about(next section). 4. 6 Heads up: you can't catch this breaker exceptionFollowing from the above — because the exception carries no stack trace,andMyBatisre-wrapsit in aMyBatisSystemExceptionon the way out:// ❌ Writing this in your Service / Controller won't catch it!try{orderMapper. queryByUser(param);}catch(SqlCircuitBreakerExceptione){// the thrown type is MyBatisSystemException; instanceof doesn't match. . . }Enter fullscreen modeExit fullscreen modeThe right way is to catch it centrally in a global exception handler — and, importantly,log using the wrapperMyBatisSystemException, becauseitsstack trace contains the full business call chain (Controller → Service → Mapper):@RestControllerAdvicepublicclassGlobalExceptionHandler{@ExceptionHandler(MyBatisSystemException. class)publicResponseEntityhandle(MyBatisSystemExceptionex){SqlCircuitBreakerExceptioncb=findCircuitBreaker(ex);// walk the cause chainif(cb!=null){// key point: log with the wrapper `ex` — it has the business line numberslog. error("[SqlCircuitBreaker] fast-fail | key={} | business stack below",cb. getCircuitKey(),ex);returnResponseEntity. status(503). body(. . . );}. . . }}Enter fullscreen modeExit fullscreen modeThis way you get both the "no stack fill" performanceandthe ability to pinpoint which Service/Controller triggered it. Best of both worlds — as long as you know the mechanism exists. 4. 7 Config priority: from one-size-fits-all to fine-grainedFour layers of config, higher overrides lower:ThreadLocal (programmatic) > method annotation > interface annotation > global YAMLEnter fullscreen modeExit fullscreen modeGlobal YAMLis the only place you can configureper SQL type(fine-grained). Annotations / ThreadLocalare coarse overrides — they apply uniformly to all SQL types under the annotated Mapper / method. Typical usage:@SqlCircuitBreaker(timeoutMs=5000)// interface level: 5s for this whole MapperpublicinterfaceOrderMapperextendsBaseMapper{@SqlCircuitBreaker(timeoutMs=2000,circuitOpenMs=30000)// method-level overrideListcomplexQuery(QueryParamparam);@SqlCircuitBreaker(disableCircuitBreaker=true)// admin query, skip the breakerListadminQuery(AdminParamparam);}Enter fullscreen modeExit fullscreen modeThreadLocal fits the "loosen/disable just for this request" case — e. g. a scheduled data-repair job that youknowwill be slow but don't want to trip the breaker:try{SqlCircuitBreakerContext. disableCircuitBreaker();orderMapper. batchFixData(ids);}finally{SqlCircuitBreakerContext. clear();// must clear, or thread-pool reuse pollutes the next request}Enter fullscreen modeExit fullscreen modeA trade-off here: the interceptordoes notauto-clear. We want a Service to set it once and have it apply across the several Mapper calls it makes; if the interceptor cleared after the first SQL, it'd be lost from the second onward. The cost is puttingclear()on the caller (in afinally), in exchange for "set once, applies to the whole block" semantics. 5. Production essential: observabilityThe scariest thing about a circuit breaker is "it's quietly working, but you don't know". Withspring-boot-actuatoron the classpath, the SDK auto-exposes 5 Micrometer metrics withzero extra config:MetricTypeMeaningsql. circuit. breaker. intercept. totalCountertotal SQL interceptedsql. circuit. breaker. timeoutCountertimeoutssql. circuit. breaker. openCountertrips (CLOSED→OPEN)sql. circuit. breaker. fast. failCounterfast-failssql. circuit. breaker. open. countGaugebreakers currently OPEN (real-time)The most useful one is that last Gauge. A single alert rule is enough:# any breaker currently open → alert immediately; back to zero means everything auto-recovered sql_circuit_breaker_open_count > 0Enter fullscreen modeExit fullscreen modeThere's also atrap that only shows up at scale:timeout/open/fast. failcarry amapper_idlabel by default (handy for pinpointing a specific Mapper). But that label explodes your time series — per service it's roughly(# Mapper methods) × 4 (types) × 3 (metrics). With a few hundred Mappers across multiple replicas, Prometheus chokes (and series-billed backends cost you real money). So there's a switch to drop themapper_idlabel, collapsing the series count from N×12 to a fixed 12; you then locate the specific Mapper via logs instead:sql-circuit-breaker:metrics:include-mapper-id:false# turn off when you're at scale or sensitive to series cardinalityEnter fullscreen modeExit fullscreen modeWhen you build infrastructure-type components, you have to think ahead to "what happens once this is at scale". Labels-on-by-default makes the out-of-the-box experience nice, but you must give people an escape hatch for when it grows. 6. Other "production-grade" details I bolted onA few, all forged by production:Notifications fire once: implement theMessageCenterClientinterface to push breaker events to Slack / Teams / whatever. But notifications fireonly when the breaker first opens, not on the fast-fail path — otherwise a high-concurrency trip would spam thousands of messages a second. Multi-datasource isolation: the breaker key includes a datasource identifier, so a slow query in DB-A won't trip DB-B. With a runtime-routing framework, implement aDataSourceKeyResolverthat returns the current datasource key; single-datasource needs zero config. SELECT . . . FOR UPDATEmisclassification: MyBatis types SQL by its XML tag, soSELECT . . . FOR UPDATEis treated as a SELECT (the loose threshold) — even though it holds locks and behaves more like DML. Tighten such methods with a dedicated annotation. Config validated at startup: all four SQL-type blocks are required and illegal values (e. g. timeout-ms <= 0) fail the boot — surfacing errors at startup rather than when some query hits it in production. Unified log prefix[SqlCircuitBreaker]: easy to filter in ELK, with a ready-made example for a dedicated logback appender (don't forgetadditivity="false", or you haven't actually isolated anything). 7. Its boundaries: know what it isnotBeing honest about boundaries matters more than hyping features:It's an "after-the-fact, statistical" breaker — it does not interrupt in-flight SQL. A query already sent to the DB and running won't be killed by the breaker (that's the job of JDBC / driver / pool timeouts). The breaker stops thesubsequentrequests of the same class from piling on. State lives in each instance's memory, not shared across instances. In a multi-replica deployment each counts on its own, so treat the thresholds as "per-instance". Under uneven traffic you can lower them so a single instance converges faster. Timeouts only, not other exceptions(covered above — deliberate). These aren't shortcomings, they'reexplicit design boundaries— a component that's clear about what itwon'tdo is one that won't get misused under the wrong expectations. 8. Wrapping upBack to the original question: a single slow query shouldn't have the power to take down an entire service. The idea behind this SDK is actually plain —push circuit breaking down to the MyBatis interceptor layer, use the SQL fingerprint as the matching unit, configure per SQL type, and fast-fail before the request ever reaches the DB. It fills exactly the "SQL layer" that Resilience4j / Hystrix-style frameworks can't reach. But what really decides whether a component survives in production is rarely the core algorithm — it's the small trade-offs:why two states and not half-open;why access-expiry is derived instead of configurable;why it trips on timeouts only, not exceptions;why the exception skips its stack trace, and the "can't catch it" gotcha that follows;why metrics need an escape hatch to drop a label. A good piece of infrastructure bakes the "traps you've hit" and the "boundaries you've thought through" into its defaults and constraints, so its users hit fewer of them. Hope this is useful next time you're building something similar. The SDK is open source and on Maven Central, with both Spring Boot 2. x and 3. x support, running in production across several systems. One dependency + a bit of YAML to integrate. Try it, break it, file issues. 📦Repo:GitHub:https://github. com/showingdata/sql-circuit-breakerGitee:https://gitee. com/LanyXP/sql-circuit-breaker(If this helped, a Star ⭐ helps more devs who've been burned by slow SQL find it. )