The MCP circuit breaker previously had no path back to the closed
state: once _server_error_counts[srv] reached _CIRCUIT_BREAKER_THRESHOLD
the gate short-circuited every subsequent call, so the only reset
path (on successful call) was unreachable. A single transient
3-failure blip (bad network, server restart, expired token) permanently
disabled every tool on that MCP server for the rest of the agent
session.
Introduce a classic closed/open/half-open state machine:
- Track a per-server breaker-open timestamp in _server_breaker_opened_at
alongside the existing failure count.
- Add _CIRCUIT_BREAKER_COOLDOWN_SEC (60s). Once the count reaches
threshold, calls short-circuit for the cooldown window.
- After the cooldown elapses, the *next* call falls through as a
half-open probe that actually hits the session. Success resets the
breaker via _reset_server_error; failure re-bumps the count via
_bump_server_error, which re-stamps the open timestamp and re-arms
the cooldown.
The error message now includes the live failure count and an
"Auto-retry available in ~Ns" hint so the model knows the breaker
will self-heal rather than giving up on the tool for the whole
session.
Covers tests 1 (half-opens after cooldown) and 2 (reopens on probe
failure); test 3 (cleared on reconnect) still fails pending fix#2.
Ben
8cc3cebca2