Reviewer regression (reproduced): _STATE_DB_DISPLAY_ROW_BACKSTOP (newest-50k SQL
cap) was applied UNCONDITIONALLY to the GET /api/session display path's state.db
read. But merge_session_messages_append_only needs the rows at/around the
session's truncation_boundary (the preserved prefix) to reconcile correctly. A
newest-N-only cap drops those boundary rows for a >N-row session and corrupts
the merge (silently losing the preserved prefix), and also prevents older-page
(msg_before) requests from ever reaching them. So a pathological-but-real >50k-
row compressed session got a corrupted merge / unreachable older history rather
than a clean bound.
Fix: gate the backstop on the SAME conditions
_state_db_since_timestamp_for_limited_display uses to decide a read is
boundary-free — not msg_before paging, and no truncation_watermark /
truncation_boundary. Those cases stay on the full (uncapped) read; the backstop
is now a defensive net for the common uncompressed initial-tail path only.
Extracted into _state_db_backstop_limit_for_display for direct test coverage.
The get_state_db_session_messages(limit=) signature + newest-rows subquery are
unchanged — only the display-path APPLICATION of the cap is gated.
Test: tests/test_state_db_backstop_boundary_gate.py — backstop applies with no
boundary prefix; SKIPPED when truncation_boundary / truncation_watermark set or
msg_before paging; empty-string markers treated as unset (matching the existing
helper). Existing backstop + reconciliation suites (53 tests) unchanged.
Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent).
Reviewer regression (reproduced): the broadcast drop-oldest loop did
put_nowait -> on Full, get_nowait -> on Empty, break. But Empty here means a
concurrent consumer (the SSE handler thread) DRAINED the queue between our Full
and get_nowait — the queue now has space, which is exactly the condition to
retry the put. `break` silently discarded `item`, and when `item` was a terminal
frame (stream_end/error/cancel) the subscriber never received it, leaving the
client attached indefinitely (spinner-forever). The "# cap 0 defensive guard;
never reached" comment was the flawed assumption: any concurrent drain makes
the Empty branch reachable at any cap.
Fix: on queue.Empty, `continue` (retry the put into the now-empty queue) in
both the broadcast path and the replay path (for consistency; the replay path
runs under self._lock with a fresh queue so it's unreachable in practice, but
the same race-safe rule applies). No infinite-loop risk: each iteration either
succeeds a put (break) or succeeds a get (progress); the continue only fires on
Empty, after which the next put has space.
Test: test_terminal_frame_survives_concurrent_drain_during_drop_oldest
deterministically simulates the interleaving with a custom queue whose first
post-Full get_nowait raises Empty (as if a concurrent consumer drained it),
then runs BOTH the pre-fix `break` logic (drops the terminal frame — proving
the test bites) and the post-fix `continue` logic (delivers it).
Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent).
Codex gate on #6143 found a SILENT regression the reuse-path fix introduced:
_replace_session_db_in_kwargs early-returned the OLD handle when the rebuild
failed, but master assigned None. If the old handle was already CLOSED, both
credential self-heal callers (streaming.py:9032/10253) then built a fresh agent
around a closed SessionDB -> every persist/search fails with 'NoneType has no
attribute execute' while the chat keeps going. Guard on _session_db_is_open:
keep a live handle (subagents hold it by ref), else degrade to None (clean lazy
reinit, master behaviour). +2 regression tests.
Co-authored-by: carlotestor <carlotestor@users.noreply.github.com>
Memory: get_state_db_session_messages had no SQL LIMIT — it full-scanned the
state.db messages table and fetchall()'d into a Python list. The display path
(GET /api/session) already has a careful since_timestamp optimization that
bounds the common uncompressed tail load, but it deliberately bails (returns
None) on compressed sessions (truncation_watermark/boundary) and msg_before
paging — in those cases the read used to full-scan with no cap. A
pathological/huge state.db could then materialize unbounded rows into memory.
Add an opt-in `limit` parameter to get_state_db_session_messages (applied as a
SQL LIMIT that keeps the NEWEST rows via a DESC subquery re-sorted ASC). It is
a BACKSTOP, not a semantic window: the display window counts visible rows
post-reconciliation, so a true window LIMIT here would corrupt the
sidecar/state.db merge (see _state_db_since_timestamp_for_limited_display,
which deliberately does NOT SQL-LIMIT raw rows for exactly that reason).
Thread a generous _STATE_DB_DISPLAY_ROW_BACKSTOP (50000) from the GET /api/session
display caller only. The full-history model-context callers (reconciliation at
models.py:3437, context reconstruction at :8576, new-turn context at
streaming.py:8443) do NOT pass a limit — their semantics require the full
history, unchanged.
Test: tests/test_state_db_read_backstop.py — no-limit returns full history
oldest-first (model-context contract unchanged); limit=N keeps the newest N
re-sorted oldest-first; limit > row count returns all; limit<=0 clamps to 1
(newest); invalid limit falls back to unbounded; limit composes with
since_timestamp. Existing state.db active-filter + reconciliation suites (54
tests) unchanged.
Contract Routing
- Task type: memory-bound fix (unbounded state.db row materialization on the
display path's bail cases).
- Touched areas: state.db message reader (api/models.py) + GET /api/session
display caller (api/routes.py). NOT the model-context callers.
- Relevant public docs: AGENTS.md, CONTRIBUTING.md, docs/CONTRACTS.md,
docs/rfcs/webui-run-state-consistency-contract.md (Model context layer).
- State layer mutated: none (read-only). The model-context read path is
UNCHANGED (no limit passed) — Invariant 1 (visible current turns enter model
context) and the reconciliation contract are preserved. Only the display path
gets the defensive cap, which does not affect what the agent receives.
- Scope boundaries: this is narrower than a naive "add LIMIT everywhere" —
_state_db_since_timestamp_for_limited_display's docstring explains why a
semantic window LIMIT would corrupt the merge; this cap is purely a
pathological-input safety net above any legitimate session size.
Release note: GET /api/session's state.db read now has a defensive row backstop,
capping memory when the bounded since_timestamp optimization bails on
compressed/paged sessions.
Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent).
- Log failed closes in _adopt_session_db_for_cached_agent with
logger.debug(..., exc_info=True) instead of bare pass, matching
_replace_session_db_in_kwargs so a failed close (EMFILE risk) is
diagnosable.
- Drop the fragile exec/source-slice fallback in the test helper import;
real import or pytest.skip. Source-level pins still cover reverts
without importing streaming.
Background subagents (delegate_tool) capture parent_agent._session_db by
reference. The cached-agent reuse path created a fresh SessionDB every
stream request and unconditionally closed the previous handle (PR #1421
EMFILE fix). A server-side wakeup / new turn for the same parent session
could therefore close the shared handle while children were still
flushing messages, producing:
Session DB append_message failed: 'NoneType' object has no attribute 'execute'
Reproduce (observed in production):
1. Parent WebUI session runs delegate_task with N background subagents
2. Parent turn tears down / idle-hook redelivers a deferred wakeup
3. New stream for the parent adopts a fresh SessionDB and closes the old one
4. Live subagents' append_message hits conn=None
Fix: _adopt_session_db_for_cached_agent reuses a still-open handle (and
closes the unused new one — still EMFILE-safe) and only replaces when
the existing handle is missing or already closed. Same guard applied to
_replace_session_db_in_kwargs (credential self-heal). LRU eviction still
closes at true session boundaries.
Tests updated to pin the adopt helper and the subagent-safe semantics.
Memory: each connected tab's subscriber queue was queue.Queue() with no
maxsize. A slow/backpressured/backgrounded tab accumulated every coalesced
token frame for the WHOLE turn (producer = agent token stream) — unbounded
per-tab growth, OOM risk with many tabs x long agentic turns. The offline
buffer was already capped (#4633), but once a subscriber attached the
live-broadcast path flowed into an unbounded per-subscriber queue.
Cap per-subscriber queues at _SUBSCRIBER_QUEUE_MAXSIZE (== _OFFLINE_BUFFER_MAXLEN)
and drop the OLDEST frame on queue.Full in both the broadcast (put_nowait) and
offline-buffer replay (subscribe_with_snapshot) paths, so a slow tab keeps the
most recent tail (older frames stay recoverable via the run journal by
last_event_id). Mirrors SessionChannel.emit's drop-on-full contract.
The cap intentionally equals the offline buffer bound, not SessionChannel's
smaller 16: StreamChannel carries the live token stream and has a #4633
reconnect-replay contract (a resubscribing tab must receive the full retained
offline tail), so capping below the buffer would force every flaky-network
reconnect through the run journal. Matching the buffer bound preserves that
contract while bounding live-broadcast memory to the same worst-case #4633
already accepts. The SSE write deadline (SSE_WRITE_DEADLINE_SECONDS) breaks a
stuck socket within ~20s, so the overflow window is short; this also bounds it
by frame count.
Adds a cumulative subscriber_dropped_events diagnostic counter (ops visibility).
Test: tests/test_stream_subscriber_queue_cap.py — bounded queue never exceeds
cap, drop-oldest keeps newest tail, draining vs non-draining subscribers, and
the #4633 reconnect-replay contract still delivers the full offline tail.
Existing offline-buffer/gap-recovery/multitab tests unchanged.
Contract Routing
- Task type: memory-bound fix (unbounded per-subscriber queue).
- Touched areas: live chat SSE broadcast (StreamChannel).
- Relevant public docs: AGENTS.md, CONTRIBUTING.md, docs/CONTRACTS.md,
docs/rfcs/webui-run-state-consistency-contract.md (Live stream/SSE layer),
docs/rfcs/session-sse-contract-v1.md (distinct: that RFC governs the PROPOSED
per-session endpoint /api/sessions/{id}/events, NOT StreamChannel, which is
the /api/chat/stream live token broadcast; no text in that RFC is changed).
- State layer mutated: Live stream / SSE layer — specifically the per-subscriber
queue inside StreamChannel (api/config.py). NOT the durable run journal, NOT
the transcript, NOT replay.
- Invariant preserved: the #4633 reconnect-replay contract — a resubscribing
tab must receive the full retained offline tail. The cap is set EQUAL to
_OFFLINE_BUFFER_MAXLEN so replay delivery is unchanged; only the previously-
unbounded live-broadcast-to-slow-tab growth is bounded. Regression:
test_reconnect_replay_still_delivers_full_offline_tail (and the pre-existing
test_stream_offline_buffer_cap.py suite) pass unchanged.
- Scope boundaries: does not touch replay idempotency (Invariant 5), the turn
journal, compression, or sidebar metadata. Slow-tab frames dropped at the cap
stay recoverable via the run journal by last_event_id.
Release note: StreamChannel per-subscriber SSE queues are now bounded
(drop-oldest), capping memory for slow/backgrounded tabs during long turns.
Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent). No AI-specific tool
use mattered beyond the repo's standard read/edit/test workflow.
Memory: the CLI/cron session projection cache was a plain dict with TTL storage
but NO size cap and only lazy, same-key eviction. Each value is a copy.deepcopy()
of the full CLI/cron session list (the expensive projection behind #4842/#4672).
The cache key folds in a state.db content fingerprint that advances on every
streamed message, so the next poll builds a NEW key; the previous key's heavy
deepcopy was never looked up again, so its lazy-expiry path never ran. A
long-lived process under churn could accumulate orphaned heavy deepcopies until
the next structural clear_cli_sessions_cache().
Make it an OrderedDict capped at _CLI_SESSIONS_CACHE_MAX_ENTRIES (8) with
drop-oldest on write (while len > MAX: popitem(last=False)) and move_to_end on
fresh hits, mirroring the _CLAUDE_CODE_PARSE_CACHE / _SIDECAR_METADATA_CACHE LRU
pattern in the same file. TTL remains the primary freshness control; the cap is
the backstop the plain dict lacked. clear_cli_sessions_cache() keeps .clear().
Test: tests/test_cli_sessions_cache_cap.py — cache stays at len == MAX after
MAX+5 distinct keys (oldest 5 evicted), a fresh hit marks the entry
most-recently-used so it survives over colder entries, and the cache is an
OrderedDict (the regression: it was an unbounded plain dict). Existing
streaming-cache-TTL/cron-session tests unchanged.
Contract Routing
- Task type: memory-bound fix (unbounded projection cache).
- Touched areas: CLI/cron sidebar projection cache (api/models.py
_CLI_SESSIONS_CACHE).
- Relevant public docs: AGENTS.md, CONTRIBUTING.md, docs/CONTRACTS.md,
docs/rfcs/webui-run-state-consistency-contract.md (Sidebar/session metadata
layer).
- State layer mutated: Sidebar/session metadata layer — specifically the
in-memory projection cache that backs the CLI/cron sidebar rows. The cache is
a read-derivation of state.db content; it is NOT a source of truth. Dropped
entries are rebuilt on the next poll (TTL/invalidation already drive rebuilds).
- Invariant preserved — Invariant 4 (Maintenance is not activity): the cache is a
derived projection; evicting an entry cannot move a session in the sidebar or
change a row's content, because the next poll recomputes the projection from
state.db. TTL remains the primary freshness control; the LRU cap only bounds
how many STALE orphaned deepcopies linger between invalidations. Regression:
test_cli_sessions_cache_cap.py plus the existing streaming-cache-TTL and
cron-session tests (which exercise the real invalidate/rebuild path).
- Scope boundaries: does not change sidebar ordering, unread state, updated_at,
or the projection result. The cap is a memory backstop; correctness still
flows from the existing TTL + clear-on-mutation invalidation.
Release note: the CLI/cron sidebar projection cache is now LRU-bounded, capping
worst-case memory for long-lived processes under heavy streaming churn.
Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent). No AI-specific tool
use mattered beyond the repo's standard read/edit/test workflow.
Centralize custom-provider model-ID extraction into _configured_model_ids()
+ _configured_model_options(), used by both catalog construction and runtime
routing so the picker and resolver can't drift. Fixes list-shaped
custom_providers[].models allowlists silently staying on the default provider
instead of routing to their own (#6121). Existing mapping / list-of-strings
configs unaffected.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Reconcile the gateway pending-mirror in _session_attention_summary()
before counting sidebar approval attention, so a parent session's red
approval dot clears once the underlying gateway approval has drained.
Live approvals stay visible; clarify fallback unchanged. WebUI read-path
mitigation; child-session context-propagation root cause stays in #6100.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex re-gate found the same derived-title leak on the sibling /api/sessions/search
endpoint (3 branches redacted only 'title'). Extract _redact_sidebar_title_fields()
shared helper, apply it in both _sidebar_session_response_item and all 3 search
branches, read api_redact_enabled once per search response (mirrors #4662). +helper
test + source-guard that every search branch calls the redactor.
Codex gate found the #6056 derived display_title (from raw user-message content)
bypassed sidebar redaction — the serializer redacted only 'title', so a credential
in a delegated goal leaked to the sidebar even with api_redact_enabled=True.
Redact display_title + _state_db_title + parent_title on the same path; +regression
test (proven to fail on the unfixed serializer).
Codex re-gate asked to distinguish a confirmed-absent sidecar from an
indeterminate/unreadable one before entering the abandoned-shell grace path.
Add _session_sidecar_exists() (True/False/None) and only grace-evict when the
sidecar is confirmed absent (False); a present-or-indeterminate sidecar keeps
the session resident. Age-window bound retained (fully closes the leak, no
data-loss edge); an SSE-mounted-lease refinement is noted as optional follow-up.
Codex gate found the eviction-reorder made never-saved empty shells immortal
(abandoned /api/session/new tabs accumulate unbounded past sessions_cache_max).
Add a grace-window bound: protect a never-persisted empty shell only while fresh
(<30min old) OR while it carries a composer draft; a stale, empty, draftless,
never-saved shell is treated as abandoned and becomes evictable again. A real
compose persists a draft on the first keystroke (disk-backed + reloadable), so
the brick fix is preserved while the leak is closed. +2 regression tests.
Two independent defects that chain into "new conversations cannot be started":
every POST /api/session/draft and /api/chat/start 404s a few seconds after
POST /api/session/new returned 200.
1. Search inputs are parsed as username fields.
index.html contains no <form> at all, but does contain password inputs (the
Settings "Access Password" panel). Chromium therefore groups the document's
unowned fields into one synthetic password form and picks the first text input
in the DOM — #sessionSearch, the sidebar conversation filter — as that form's
username field, then autofills the saved account name into it on load.
autocomplete="off" was already present; it is ignored for credential
heuristics.
The autofill fires oninput -> filterSessions() -> GET /api/sessions/search
?content=1&depth=5: a full content search the user never asked for, repeated
on every page load.
Fix: give the four search inputs type="search" plus the per-manager opt-outs
(data-1p-ignore / data-lpignore / data-bwignore / data-form-type), and give the
Settings password inputs a real <form> owner with proper autocomplete tokens so
Chromium scopes the credential form to them instead of to the whole document.
The form uses display:contents, so layout is unchanged. CSS suppresses WebKit's
native search clear button, since these fields ship their own
(#sessionSearchClear).
2. A never-persisted session is treated as evictable.
new_session() intentionally keeps a session in RAM until its first message
(#1171), so the SESSIONS cache is its only copy. _session_is_evictable()
(#4765) short-circuited on zero messages, reasoning that an empty shell "is
recreated on next access". It is not: get_session() has no recreate path and
raises KeyError, so both routes 404 and the session can never be started.
The content search from (1) pulls every hit through get_session(), which blows
past sessions_cache_max (default 300) and evicts the session being composed.
That is why this reproduces within seconds on an install with ~1700 sessions,
and why it grew worse as the session count rose.
Fix: require proof of persistence before evicting, for empty sessions too —
which is what the function's own docstring already promised ("Its full state is
already persisted to the JSON sidecar"). The zero-message branch was the single
path that bypassed it. #4765's memory bound is unaffected: sessions loaded from
disk remain evictable, and its existing tests still pass.
Trade-off: an unsaved empty session now stays resident for the process
lifetime (one per "New Conversation" click, an empty object each). That is a
deliberate application of the function's stated invariant — "slightly more RAM
for a session we are unsure about is strictly better than evicting an unsaved
session". Bounding it by age instead would also work if maintainers prefer.
Validation:
pytest tests/test_issue4765_sessions_lru_eviction.py -> 9 passed
The added regression test fails on the unpatched predicate (verified by
reverting api/models.py alone) and passes with the fix.
Note: (1) fixes what triggers the churn, (2) fixes what turns that churn into
an unstartable session. Either fix alone leaves a real defect in place, so
they are submitted together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gate review follow-ups for the #6022 no-leak rule: every non-deliberate
newSession/session-new call site passes worktree:false; only deliberate
New Chat inherits the config default.
- Strict boolean config read: only YAML true enables the default;
quoted strings/ints/lists/dicts/null fall to the safe no-worktree
default instead of truthiness-coercing.
- Presence-based explicit detection: 'worktree' in body — an explicit
worktree:null is an opt-out, never a fall-through to config.
- Guarded remaining auto-mint call sites with explicit worktree:false:
profile switch (panels.js), promptWorkspacePath + switchToWorkspace
blank-page binds (panels.js), promptNewFile + promptNewFolder
(ui.js), terminal auto-session (commands.js).
- Tests: worktree:null vs config-true (factory must not be called),
strict-boolean malformed-shape matrix, static regression gates for
every auto-mint site + a bare-body sweep so future auto-mint paths
can't silently omit the key; updated the cmdTerminal preflight
assertion for the new call shape.
/api/session/new treated an absent 'worktree' key as false, so the
agent's config-level 'worktree: true' default was never consulted for
WebUI sessions even though the WebUI reuses the agent's own worktree
helper. Both callers should agree on same-agent/same-repo isolation.
Three-value model:
- body 'worktree' absent -> agent config default applies (profile-aware
via get_config_for_profile_home, so a user with worktree:true in one
profile but not another gets per-profile behavior)
- explicit true/false -> honored verbatim; explicit always wins
Non-git workspaces: a config-default request degrades to a plain
session and surfaces 'worktree_skipped' in the payload; an explicit
worktree:true keeps the hard 400.
Frontend (same change, load-bearing): newSession() forwards explicit
true/false and omits the key otherwise; the boot-time auto-bind
(_maybeBindFreshDefaultWorkspaceSession) and the onboarding session
send worktree:false explicitly so a config default can never mint a
worktree + branch from merely opening the UI. Deliberate 'New Chat'
sends nothing and inherits the default.
Tests: route matrix (config on/off x body absent/true/false), non-git
fallback vs explicit 400, explicit-false-beats-config-true, config
helper (ambient/named-profile/fail-soft), static UI assertions that
auto-bind and onboarding opt out and New Chat paths don't pin a value.
Fixes#6022
In the local (non-gateway) approval path, _resolve_approval_legacy grouped
"once" together with "session" and called approve_session() for both.
approve_session() records the pattern in the session-wide allowlist that the
guard later consults, so a single "Approve once" click silently approved every
later matching tool call for the whole session -- a destructive tool could then
run with no approval card.
Only "session"/"always" persist now. "once" falls through: the
resolve_gateway_approval() call below still unblocks the parked agent thread for
every choice, so the current tool call proceeds, but nothing is written to
_session_approved and the next matching guarded call re-prompts.
Adds tests/test_issue6017_approve_once_no_persist.py, asserting is_approved is
False after "once" and True after "session".
WebUI-created worktrees are locked at creation by the agent
(cli._setup_worktree locks with --reason 'hermes pid=...'), but
remove_worktree_for_session never unlocked, so git refused every removal
with "use 'remove -f -f' to override or unlock first" — force=true
included, since a single --force cannot override a lock.
Mirror the agent's own _cleanup_worktree ordering: fail-soft
'git worktree unlock' right before the remove, after all
dirty/untracked/unpushed/stream/terminal guards have passed. Those
guards remain the real safety layer; force keeps meaning 'override
dirty working state', never 'override a foreign lock'.
Tests: locked worktree now removes cleanly without force; unlock is
fail-soft when the worktree was never locked; existing mocked
call-order assertions updated for the leading unlock call.
Fixes#6023
nesquena-hermes