gooneyryan/hermes-webui
1
0
Fork 0
mirror of https://github.com/nesquena/hermes-webui.git synced 2026-07-16 12:40:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Prune orphan native-WebUI zero-message sessions from sidebar (#4985)

Browse Source 
Sidebar rows whose backing state.db session has zero messages (a '+'-click
that opened a row but the first turn never committed, or a sidebar nav that
opened then closed before any message landed) now get pruned instead of
lingering forever. The existing #3238/#4591 orphan-prune path only handled
CLI/API-server sidecars; this adds a parallel pass for native-WebUI rows.

- api/models.py: add agent_session_zero_message_sids() batch helper that
  mirrors agent_session_rows_existing()'s safe-degrade contract (returns
  frozenset() on any error so a transient failure never causes data loss).
  Add a small tombstone file at SESSION_DIR/_pruned_webui_orphans.json
  (capped at the last 500 sids) so the next poll's
  recover_missing_index_sidecars does not re-add a pruned orphan to the
  sidebar index — the per-poll fsync'd index write + state.db probe loop
  the original PR would have hit on every poll for every orphan. The
  helper, the load, and the full-scan fallback all honor the tombstone.
  new_session() and import_cli_session() clear any matching tombstone
  entry so a fresh session with the same id isn't shadowed.
- api/routes.py: extract the webui zero-message prune into a private
  _prune_orphaned_webui_zero_message_sessions() helper, then call it
  from BOTH branches of _build_session_list_cache_payload. Established
  installs pin show_cli_sessions=False (per api/config.py:7637-7648)
  and are exactly the long-time users who accumulated the #4985 404
  orphans — the original PR only ran the prune in the if branch and
  silently skipped the else branch (maintainer review
  IC_kwDOR1LuPM8AAAABHsyFGg). The helper also honors the tombstone
  defensively so the prune can't thrash on a row the index still
  carries from a prior poll.
- tests/test_issue4985_orphaned_webui_zero_message.py: 39 regression
  tests — the original 25 (7 helper-direct + 12 monkeypatch predicate
  tests + 6 real-pipeline end-to-end tests) plus 14 follow-up tests
  driven by the MUST-FIX / tombstone / r6 signal-swap reviews:
    * test_helper_prunes_orphan_keeps_safety_retained_rows — direct call
      to the extracted helper with a row set covering every shape the
      gate is responsible for.
    * test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_false
      — explicit regression for the established-install case the
      original PR missed.
    * test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
      — companion positive control.
    * test_tombstone_persists_across_polls — drives two pipeline builds,
      asserts prune fires once total and the tombstoned sid is NOT
      re-added to SESSION_INDEX_FILE.
    * test_tombstone_does_not_block_new_session_with_same_id — defensive
      cover for the new_session() / import_cli_session() tombstone clear.
    * test_tombstone_trimmed_to_last_N_entries — verifies the cap.
    * test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
      — companion positive control (show_cli_sessions=True path).
    * test_tombstone_self_heals_when_message_added — confirms
      Session.save()'s belt-and-suspenders tombstone clear fires when
      a sidecar commit lands with len(messages) > 0.
    * test_tombstone_loader_is_fail_open_on_corrupt_file[non-json|version-mismatch|non-dict-root]
      — 3-parameter regression for the fail-open corrupt-file contract
      (#5023 lesson).
    * test_sidecar_only_webui_session_is_retained — sidecar JSON carries
      real messages but state.db.messages is empty (mirroring lag); the
      r6 loaded-sidecar probe retains the row.
    * test_sidecar_only_webui_session_self_heals_tombstone — the r4.5
      self-heal path combined with the r6 sidecar-only retention rule.
    * test_truly_empty_sidecar_with_title_still_pruned — companion
      negative test that the r6 fix does NOT over-retain empty sidecars
      with non-Untitled titles.
    * test_sidecar_only_webui_session_retained_when_show_cli_sessions_false
      — established-install path also respects the r6 retention rule.
- CHANGELOG.md: Unreleased entry under Fixed.

fix: target post-#1171 survivors + hoist + tombstone + r6 sidecar-load probe

The r5 sidecar-only retention signal keyed off the row's cached
``message_count`` — which is stale-positive on the very phantom rows
#4985 exists to prune (sidecar ``messages`` empty but cached count > 0).
The r6 signal swaps that cached-count probe for a ``Session.load(sid)
.messages`` probe, gated to ``state.db``-empty candidates only so the
common live-row path stays a no-op. Three retain tests now write real
populated ``messages`` arrays (not just ``[]``) so the loaded-sidecar
check has something to find; the stale-count phantom test is left as
the canonical r6 regression (its ``messages=[]`` + ``message_count=5``
sidecar is exactly the shape the r6 loaded-sidecar check correctly
classifies as orphan). Maintainer review 4584722701.
This commit is contained in:
nanw
2026-06-26 20:53:37 +08:00
parent ffef6fa4ec
commit e8f1bb10da
4 changed files with 2800 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4
View File
@@ -3,6 +3,10 @@
## [Unreleased]
### Fixed
- **Orphan native-WebUI sessions (sidebar rows that survive the upstream `all_sessions()` #1171 keep-filter but whose actual `state.db.messages` is empty) are now pruned instead of lingering.** When a WebUI chat flow created a `state.db.sessions` row but no persisted turn (boot-time `+` click, profile switch that resets the active id, sidebar nav that opens then closes before the first message commits), the row stayed in the sidebar forever; clicking it hit `GET /api/session` → 404 and "Session not available in web UI." indefinitely. The existing #3238/#4591 orphan prune only handled CLI/API-server sidecars. The sidebar cache now also probes `state.db.messages` for native-WebUI rows that *survived* `all_sessions()` #1171 — i.e. rows that are titled OR carry a positive but stale `message_count` — and prunes via `prune_session_from_index`. Three new retain tests cover worktree, pending, and streaming cases. The prune is hoisted out of the `if show_cli_sessions:` branch so established installs (where settings pin `show_cli_sessions=False`) get the same prune; a tombstone file `_pruned_webui_orphans.json` keeps `recover_missing_index_sidecars` from re-adding a pruned orphan to the sidebar index on the next poll (capping at the lexicographically-first 500 sids so the file stays bounded on long-running installs). New regression covers the `show_cli_sessions=False` pipeline. The tombstone self-heals: a tombstoned sid whose `state.db.messages` later gains real content is automatically cleared (in the prune helper's post-probe branch + as a belt-and-suspenders clear in `Session.save()` whenever a sidecar commit lands with `len(messages) > 0`), so the row re-surfaces in the sidebar instead of being hidden forever — strictly better than the previous "blind-drop on every poll" behavior. The tombstone load-modify-write/unlink is serialized by a dedicated `threading.Lock` so concurrent sidebar polls and `Session.save` calls cannot lose each other's updates. A new fail-open regression covers corrupt tombstone files (non-JSON, version mismatch, non-dict root) — the loader degrades to `frozenset()` instead of raising, so a transient crash or hand-edit can never accidentally admit a row that should stay tombstoned (#5023 lesson). Sidecar-only sessions (sidecar JSON carries real messages via `message_count > 0` but `state.db.messages` is empty — common when an agent hasn't mirrored a transcript yet) are **retained**: the per-sid decision now consults the row's sidecar-authoritative `message_count` (preserved by `Session.load_metadata_only()`) and skips prune + self-heals any tombstone, so a real conversation is never hidden from the sidebar. The r5 cached-count signal was stale-positive on the very phantom rows #4985 exists to prune (sidecar `messages` empty but cached count > 0), so the r5 retain branch kept the phantom and re-opened the bug; the r6 signal swaps the `message_count` probe for a loaded-sidecar `Session.load(sid).messages` probe, gated to `state.db`-empty candidates only so the common live-row path stays a no-op. (#4985)
## [v0.51.677] — 2026-06-26 — Release YG (cron run logs render as literal text, not mangled markdown)
### Fixed
api/models.py
+341 -1
View File
@@ -112,6 +112,19 @@ _SESSION_INDEX_REBUILD_LOCK = threading.Lock()
_SESSION_INDEX_REBUILD_THREAD = None
_SESSION_INDEX_REBUILD_THREAD_TARGET: tuple[Path, Path] | None = None
# Serializes ``_record_webui_zero_message_orphan_tombstone`` /
# ``_clear_webui_zero_message_orphan_tombstone`` so two concurrent sidebar
# polls (or a poll racing ``Session.save`` / ``new_session`` /
# ``import_cli_session``) cannot lose each other's load-modify-write/unlink.
# Without this lock each operation rewrites the entire tombstone file from
# scratch, so a concurrent recorder and clearer can land last-writer-wins and
# silently drop each other's update — defeating the self-healing invariant
# that ``Session.save`` clears the tombstone the same poll that re-prunes
# would otherwise re-add the row for. ``threading.Lock`` is sufficient (the
# WebUI sidebar polling path is single-process) but must wrap the WHOLE
# load-modify-write/unlink sequence in both helpers.
_WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_LOCK = threading.Lock()
# Path-safety contract for session IDs. Accept alphanumerics, underscore, and
# hyphen so API/gateway-issued ids (``api-*``, ``reachy-voice-*``) round-trip
# through filesystem load/save/delete/worktree paths without traversal risk.
@@ -432,6 +445,200 @@ def prune_session_from_index(session_id: str) -> None:
_write_session_index(updates=None)
# ---------------------------------------------------------------------------
# #4985 webui zero-message orphan tombstone
# ---------------------------------------------------------------------------
# ``prune_session_from_index()`` only removes a row from SESSION_INDEX_FILE —
# the on-disk sidecar at ``SESSION_DIR / f"{sid}.json"`` is intentionally
# kept (it may hold legitimate WebUI-owned metadata a future code path wants
# to recover). On the next ``/api/sessions`` poll, ``all_sessions()``'s
# ``recover_missing_index_sidecars`` pass (``missing_persisted_ids``) sees
# the orphaned sidecar, re-loads it via ``Session.load_metadata_only()``,
# and writes it back to SESSION_INDEX_FILE — undoing the prune.
#
# For #4985 zero-message webui orphans, that round-trip would also re-add
# the row to the sidebar (it survives #1171 because it is titled or has a
# stale positive message_count), so the next prune fires again. N orphans
# therefore cost 2N fsync'd index writes + N state.db probes per poll,
# forever. The fix is a small, dedicated tombstone set written alongside
# the prune: any sid in the tombstone is skipped by
# ``recover_missing_index_sidecars`` (no re-add to index) and is therefore
# never re-presented to the prune batch.
#
# The file lives in SESSION_DIR (sibling of _index.json) so it is
# profile-local and survives across processes, and is intentionally NOT
# itself listed as a session sidecar — it is excluded from
# ``_persisted_session_ids_snapshot()`` via the same ``name.startswith('_')``
# convention (its name starts with ``.``, a dot — but we add a dedicated
# check below for paranoia). Bounded size keeps the file from growing
# without limit on long-running installs.
# ---------------------------------------------------------------------------
WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_CAP = 500
WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_VERSION = 1
def _webui_zero_message_orphan_tombstone_file() -> "Path":
"""Return the current tombstone file path.
Resolved at call time (not module load) so tests that monkeypatch
``SESSION_DIR`` (e.g. ``_real_pipeline``) get a per-test path without
having to also rewrite the module-level constant. Mirrors how
``SESSION_INDEX_FILE`` is computed but resolved at call time so the
real path tracks the live ``SESSION_DIR``.
"""
return SESSION_DIR / "_pruned_webui_orphans.json"
def _load_webui_zero_message_orphan_tombstone() -> frozenset[str]:
"""Return sids we've explicitly pruned as webui zero-message orphans.
Degrades to ``frozenset()`` on any read error, missing file, version
mismatch, or schema mismatch so the recovery path never accidentally
admits a row that should stay tombstoned.
"""
p = _webui_zero_message_orphan_tombstone_file()
if not p.exists():
return frozenset()
try:
raw = json.loads(p.read_text(encoding='utf-8'))
except Exception:
logger.debug(
"Failed to load webui zero-message orphan tombstone",
exc_info=True,
)
return frozenset()
if not isinstance(raw, dict):
return frozenset()
try:
if int(raw.get("version", 0)) != WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_VERSION:
return frozenset()
except (TypeError, ValueError):
return frozenset()
ids = raw.get("ids", [])
if not isinstance(ids, list):
return frozenset()
return frozenset(
str(sid).strip() for sid in ids if str(sid or "").strip()
)
def _save_webui_zero_message_orphan_tombstone(ids) -> None:
"""Persist the tombstone set with a bounded size cap (lexicographically-first N entries).
Sorts + dedupes so the on-disk file is deterministic and diff-friendly.
Atomic write via ``.tmp.<pid>.<tid>`` + ``os.replace`` mirrors
``_write_session_index`` and ``Session.save`` so a crash mid-write does
not leave a half-written tombstone file.
Note on eviction order: ``sorted_ids[:WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_CAP]``
keeps the lexicographically-FIRST ``N`` sids (sorted ascending), not the
last-pruned ``N``. Session ids are random UUIDs (``uuid.uuid4().hex[:12]``),
so the eviction is effectively random across installs; the cap exists to
keep the file bounded on long-running installs, not to implement FIFO
pruning. If true FIFO is ever needed, switch the slice to ``[-N:]`` and
keep an insertion-ordered data structure.
"""
try:
sorted_ids = sorted(set(
str(sid).strip() for sid in (ids or []) if str(sid or "").strip()
))
except TypeError:
return
if len(sorted_ids) > WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_CAP:
sorted_ids = sorted_ids[-WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_CAP:]
payload = {
"version": WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_VERSION,
"ids": sorted_ids,
}
p = _webui_zero_message_orphan_tombstone_file()
_tmp = None
try:
SESSION_DIR.mkdir(parents=True, exist_ok=True)
_tmp = p.with_suffix(
f'.tmp.{os.getpid()}.{threading.current_thread().ident}'
)
with open(_tmp, 'w', encoding='utf-8') as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
f.flush()
os.fsync(f.fileno())
os.replace(_tmp, p)
except Exception:
logger.debug(
"Failed to save webui zero-message orphan tombstone",
exc_info=True,
)
if _tmp is not None:
try:
_tmp.unlink(missing_ok=True)
except Exception:
pass
def _record_webui_zero_message_orphan_tombstone(sid: str) -> None:
"""Add ``sid`` to the tombstone.
No-op if already present (avoids re-sorting and re-fsync'ing on every
redundant prune). Called from the ``#4985`` prune helper in
``api.routes`` immediately after ``prune_session_from_index``.
Wraps the entire load-modify-write in ``_WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_LOCK``
so two concurrent sidebar polls (or a poll racing ``Session.save`` /
``new_session`` / ``import_cli_session``) cannot lose each other's
writes. Without the lock each operation rewrites the entire tombstone
file from scratch, so a concurrent recorder and clearer can land
last-writer-wins and silently drop each other's update — defeating the
self-healing invariant that ``Session.save`` clears the tombstone the
same poll that the prune helper re-prunes would otherwise re-add the
row for.
"""
sid = str(sid or "").strip()
if not sid:
return
with _WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_LOCK:
current = set(_load_webui_zero_message_orphan_tombstone())
if sid in current:
return
current.add(sid)
_save_webui_zero_message_orphan_tombstone(current)
def _clear_webui_zero_message_orphan_tombstone(sid: str) -> None:
"""Remove ``sid`` from the tombstone.
Called when a new Session is created with an explicit sid (e.g.
``new_session()`` / ``import_cli_session()``) and belt-and-suspenders
whenever ``Session.save`` writes a real conversation (a save with
``len(messages) > 0`` proves the row is alive, so the tombstone entry
must drop). Safe to call with an unknown sid (no-op). If the tombstone
becomes empty as a result, the file is removed entirely so an empty
poll-time load stays free.
Wraps the entire load-modify-write/unlink in
``_WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_LOCK`` so concurrent
recorders/clearers cannot lose each other's writes — see the docstring
on ``_record_webui_zero_message_orphan_tombstone``.
"""
sid = str(sid or "").strip()
if not sid:
return
with _WEBUI_ZERO_MESSAGE_ORPHAN_TOMBSTONE_LOCK:
current = set(_load_webui_zero_message_orphan_tombstone())
if sid not in current:
return
current.discard(sid)
if current:
_save_webui_zero_message_orphan_tombstone(current)
return
try:
_webui_zero_message_orphan_tombstone_file().unlink(missing_ok=True)
except Exception:
logger.debug(
"Failed to remove empty webui zero-message orphan tombstone",
exc_info=True,
)
def _active_stream_ids():
with STREAMS_LOCK:
active_ids = set(STREAMS.keys())
@@ -930,6 +1137,29 @@ class Session:
if not skip_index:
_write_session_index(updates=[self])
# #4985 belt-and-suspenders self-heal: a successful save with at
# least one real message on the sidecar is unconditional proof the
# row is alive (the #4985 "zero-message orphan" only ever exists
# when ``len(self.messages) == 0``). Clear the tombstone so the
# next ``/api/sessions`` poll does not need the prune helper to
# run before the row re-appears — useful when the message-commit
# happens on a poll that does not yet see state.db.messages rows
# (e.g. the WebUI's own sidecar commit lands before the agent's
# state.db append, or the helper is skipped via a different code
# path). Wrapped because a tombstone failure must never block a
# save. The helper's self-healing branch in
# ``_prune_orphaned_webui_zero_message_sessions`` is the primary
# fix; this is the belt.
if self.messages:
try:
_clear_webui_zero_message_orphan_tombstone(self.session_id)
except Exception:
logger.debug(
"Failed to clear webui zero-message orphan tombstone for %s",
self.session_id,
exc_info=True,
)
@classmethod
def load(cls, sid):
# Validate session ID format to prevent path traversal. API/gateway
@@ -2665,6 +2895,19 @@ def new_session(workspace=None, model=None, profile=None, model_provider=None, p
worktree_created_at=wt.get('created_at') if wt else None,
enabled_toolsets=enabled_toolsets,
)
# #4985: defensive — auto-generated uuids don't collide with the
# tombstone, but if a future caller ever passes an explicit id that
# was previously pruned, clear the entry so the new session isn't
# shadowed on the next poll. Wrapped because a tombstone failure
# must never block new-session creation.
try:
_clear_webui_zero_message_orphan_tombstone(s.session_id)
except Exception:
logger.debug(
"Failed to clear webui zero-message orphan tombstone for %s",
s.session_id,
exc_info=True,
)
with LOCK:
SESSIONS[s.session_id] = s
SESSIONS.move_to_end(s.session_id)
@@ -3251,6 +3494,70 @@ def agent_session_rows_existing(
return frozenset(wanted)
def agent_session_zero_message_sids(
session_ids: list[str] | set[str] | frozenset[str],
*,
profile=None,
) -> frozenset[str]:
"""Return session ids confirmed to have zero rows in the agent ``messages`` table.
Used by the sidebar orphan-prune path (#4985) to detect native-WebUI sessions
whose backing agent row exists but was never written to (boot-time ``+`` click,
profile switch that resets the active id, sidebar nav that opens a session then
closes the tab before the first message commits). Such rows linger in the
sidebar forever because the WebUI delete affordance is not exposed for them,
and the existing #3238/#4591 orphan prune explicitly excludes webui sources.
Mirrors ``agent_session_rows_existing``'s batched chunked probe, safe-degrade
contract (returns ``frozenset()`` on any error so a transient failure NEVER
causes a stale-prune data loss), and ``messages`` table absence handling.
"""
wanted = {str(sid).strip() for sid in (session_ids or []) if str(sid or "").strip()}
if not wanted:
return frozenset()
try:
import sqlite3
except ImportError:
return frozenset()
db_path = _agent_state_db_path(profile=profile)
if db_path is None:
return frozenset()
try:
with closing(sqlite3.connect(str(db_path))) as conn:
cur = conn.cursor()
cur.execute("PRAGMA table_info(sessions)")
sessions_cols = {str(row[1]) for row in cur.fetchall()}
if 'id' not in sessions_cols:
return frozenset()
cur.execute("PRAGMA table_info(messages)")
messages_cols = {str(row[1]) for row in cur.fetchall()}
if 'session_id' not in messages_cols:
return frozenset()
zero_message: set[str] = set()
ids = list(wanted)
chunk_size = 500
for i in range(0, len(ids), chunk_size):
chunk = ids[i:i + chunk_size]
placeholders = ','.join('?' * len(chunk))
cur.execute(
f"SELECT s.id FROM sessions s "
f"WHERE s.id IN ({placeholders}) "
f"AND NOT EXISTS ("
f" SELECT 1 FROM messages m WHERE m.session_id = s.id"
f")",
chunk,
)
zero_message.update(str(row[0]).strip() for row in cur.fetchall())
return frozenset(zero_message)
except Exception:
logger.debug(
"agent_session_zero_message_sids probe failed for %d ids",
len(wanted),
exc_info=True,
)
return frozenset()
def agent_session_row_exists(session_id: str, *, profile=None) -> bool:
"""Return True if ``session_id`` still has a backing row in the agent state.db.
@@ -3577,6 +3884,17 @@ def all_sessions(diag=None, *, include_lineage_metadata: bool = True):
str(sid) for sid in persisted_ids
if sid and str(sid) not in indexed_ids
)
# #4985: the tombstone is intentionally NOT a blind-drop filter
# on missing_persisted_ids. A tombstoned sid whose sidecar is
# still on disk is recovered into the index here so the
# post-recovery prune helper (``_prune_orphaned_webui_zero_message_sessions``
# below) gets a chance to self-heal: if the row's state.db.messages
# is still empty the helper leaves the tombstone in place (no
# redundant re-prune); if state.db.messages now has rows the
# helper clears the tombstone and the row stays visible. A
# blind-drop here would be strictly worse than the orphan it
# suppresses — it would silently swallow a legitimately-resurfaced
# row forever, even after the user actually sent messages.
recovered_sidecars = []
if missing_persisted_ids:
_diag_stage(diag, "all_sessions.recover_missing_index_sidecars")
@@ -3653,6 +3971,15 @@ def all_sessions(diag=None, *, include_lineage_metadata: bool = True):
# Full scan fallback
_diag_stage(diag, "all_sessions.full_scan")
out = []
# #4985: the tombstone is intentionally NOT a blind-drop filter on the
# full-scan fallback either. A tombstoned sid whose sidecar is still
# on disk must be loaded here so the post-recovery prune helper
# (``_prune_orphaned_webui_zero_message_sessions`` in api/routes) gets a
# chance to self-heal: if state.db.messages is still empty the helper
# leaves the tombstone in place; if state.db.messages now has rows the
# helper clears the tombstone and the row stays visible. A blind-drop
# here would be strictly worse than the orphan it suppresses — silently
# swallowing a legitimately-resurfaced row forever.
for p in SESSION_DIR.glob('*.json'):
if p.name.startswith('_'): continue
try:
@@ -3674,7 +4001,7 @@ def all_sessions(diag=None, *, include_lineage_metadata: bool = True):
and not s.active_stream_id
and not s.pending_user_message
and not getattr(s, 'worktree_path', None)
)]
)] # fmt: skip
if include_lineage_metadata:
_diag_stage(diag, "all_sessions.lineage_metadata")
_enrich_sidebar_lineage_metadata(result)
@@ -3890,6 +4217,19 @@ def import_cli_session(
updated_at=updated_at,
parent_session_id=parent_session_id,
)
# #4985: import_cli_session uses an explicit sid (the CLI sidecar's id).
# If that sid was previously tombstoned as a webui zero-message orphan,
# clear the tombstone entry so the freshly-imported session is visible
# on the next poll. Wrapped because a tombstone failure must never block
# an import.
try:
_clear_webui_zero_message_orphan_tombstone(s.session_id)
except Exception:
logger.debug(
"Failed to clear webui zero-message orphan tombstone for %s",
s.session_id,
exc_info=True,
)
s.save(touch_updated_at=False)
return s
api/routes.py
+277 -2
View File
@@ -2008,6 +2008,221 @@ def _session_list_cache_done(key: tuple, event: threading.Event | None) -> None:
event.set()
def _prune_orphaned_webui_zero_message_sessions(rows, *, diag_stage=None):
"""#4985 second-pass orphan prune for native-WebUI rows whose ``state.db.messages`` is empty.
Takes the post-``#3238`` ``webui_sessions`` list (i.e. rows that already
survived the #3238/#4591 CLI/API-server prune) and returns a NEW list with
any row whose backing ``state.db.messages`` table is empty removed.
Removed sids are also persisted to the tombstone via
``_record_webui_zero_message_orphan_tombstone`` so
``recover_missing_index_sidecars`` does not re-add them to the sidebar
index on the next poll (avoids the cache-thrash loop where every poll
does one fsync'd index write + one state.db probe per orphan, forever).
Invariants preserved:
- Rows with ``active_stream_id`` / ``has_pending_user_message`` /
``worktree_path`` set are NEVER pruned — the inflight / worktree-bound
/ pending safety contract from ``IC_kwDOR1LuPM8AAAABHrkF1Q``.
- Rows whose ``state.db.messages`` is empty AND that survived the
upstream ``all_sessions()`` ``#1171`` keep-filter (i.e. titled OR has
positive ``message_count``) ARE pruned — the post-#1171-survivor
shape #4985 actually describes (a row that lingers VISIBLY in the
sidebar because of a stale positive ``message_count`` or a title set
before the first turn committed).
This helper is intentionally extracted out of the ``if show_cli_sessions:``
branch so the prune fires in BOTH branches of
``_build_session_list_cache_payload``. Established installs have
``settings.show_cli_sessions`` pinned to ``False`` (per
``api/config.py:7637-7648``) and those are exactly the long-time users
who have accumulated the #4985 404 orphans — without hoisting, the
``else:`` branch silently skipped the prune and the sidebar kept
dangling rows that 404 on click (review
``IC_kwDOR1LuPM8AAAABHsyFGg``).
"""
if not rows:
return list(rows) if rows is not None else []
_diag = diag_stage if callable(diag_stage) else (lambda *_a, **_k: None)
# #4985 self-healing: the tombstone is NOT a blind-drop filter at the
# top of the helper. A row whose sid is in the tombstone is allowed
# into the gate predicate like any other row — and the post-probe
# logic below explicitly distinguishes four cases:
#
# 1. probe says NOT empty AND sid IS tombstoned → SELF-HEAL: the row
# has actually gained messages, so clear the tombstone and keep
# the row (do NOT add to missing_webui_orphan_ids). This is the
# primary fix for review IC_kwDOR1LuPM8AAAABHvY-dw.
# 2. probe says empty AND sid IS tombstoned → tombstone persists
# (orphan shape unchanged), but the row is excluded from the
# returned list so the tombstone continues to suppress it on
# this poll too. Do NOT redundantly prune+tombstone (would
# cycle).
# 3. probe says empty AND sid is NOT tombstoned → new orphan: prune
# from index, record tombstone, diag_stage.
# 4. probe says NOT empty AND sid is NOT tombstoned → row has
# messages, retain (gate already passes anyway).
#
# A blind-drop at the top (the previous behavior) is strictly worse
# than the orphan it suppresses — it would silently swallow a
# legitimately-resurfaced row forever, even after the user actually
# sent messages. The self-healing case is what makes the tombstone a
# recoverable "this sid is currently empty" signal rather than a
# permanent hide-list.
if not rows:
return []
# Gate predicate mirrors the inline block that lived here before the
# helper extract. The (title!='Untitled' OR count>0) clause is what makes
# this gate actually reach a row #1171 kept — without it, the gate is a
# no-op because ``all_sessions()`` at ``api/models.py:3892-3898`` (and
# its full-scan fallback at 3946-3952) has already stripped every
# (Untitled ∧ count==0 ∧ ¬active_stream_id ∧ ¬has_pending_user_message ∧
# ¬worktree_path) row before our prune block runs.
_webui_orphan_probe_rows = [
s for s in rows
if _session_source_is_webui(s)
and not s.get("active_stream_id")
and not s.get("has_pending_user_message")
and not s.get("worktree_path")
and (
s.get("title", "Untitled") != "Untitled"
or _numeric_count(s.get("message_count")) > 0
)
]
if not _webui_orphan_probe_rows:
return list(rows)
rows_by_profile_webui: dict[object, list[dict]] = defaultdict(list)
for row in _webui_orphan_probe_rows:
rows_by_profile_webui[row.get("profile")].append(row)
_tombstoned = _load_webui_zero_message_orphan_tombstone()
self_healed_ids: set[str] = set()
missing_webui_orphan_ids: set[str] = set()
still_hidden_ids: set[str] = set()
for profile_key, profile_rows in rows_by_profile_webui.items():
probe_ids = [
str(row.get("session_id")).strip()
for row in profile_rows
if str(row.get("session_id") or "").strip()
]
zero_message_sids = agent_session_zero_message_sids(
probe_ids,
profile=profile_key if isinstance(profile_key, str) and profile_key else None,
)
# Iterate over the actual rows (not just probe_ids) so each sid
# decision can probe the sidecar for real ``messages``. The r5
# signal keyed off the row's cached ``message_count`` (which is
# stale-positive on the very phantom rows #4985 exists to prune:
# sidecar ``messages`` empty but cached count > 0), so the r5
# retain branch kept the phantom and re-opened the bug (maintainer
# review 4584722701, supersedes the r5 cached-count signal). The
# r6 signal probes ``Session.load(sid).messages`` directly — but
# ONLY for ``state.db``-empty candidates (the small set; the
# common live-row path takes the ``else`` branch and pays
# nothing). Full ``Session.load`` is intentional (vs
# ``load_metadata_only`` which zeroes the messages array at
# ``api/models.py:1210``).
for row in profile_rows:
sid = str(row.get("session_id") or "").strip()
if not sid:
continue
is_empty = sid in zero_message_sids
is_tombstoned = sid in _tombstoned
if is_empty:
# ``state.db.messages`` is empty. Probe the sidecar JSON
# for real messages — the cached ``message_count`` alone
# is stale-positive on phantom rows (sidecar ``messages``
# empty but cached count > 0) and would retain the very
# phantom this feature exists to prune (maintainer review
# 4584722701, supersedes the r5 cached-count signal).
# Full ``Session.load`` is intentional (vs
# ``load_metadata_only`` which zeros the messages array
# at ``api/models.py:1210``); the common live-row path
# pays nothing because it skips the load via the
# ``else`` branch below.
try:
from api.models import Session as _Session
_loaded = _Session.load(sid)
sidecar_has_messages = bool(
_loaded is not None and len(_loaded.messages or []) > 0
)
except Exception:
logger.debug(
"Failed to load sidecar for webui orphan decision %s; "
"treating as empty for prune purposes",
sid,
exc_info=True,
)
sidecar_has_messages = False
else:
# ``state.db.messages`` is non-empty — the conversation is real.
sidecar_has_messages = True
if sidecar_has_messages:
# Real transcript (state.db OR loaded sidecar). Retain; if
# tombstoned, self-heal so it stops thrashing on recovery.
if is_tombstoned:
self_healed_ids.add(sid)
continue
if not is_empty and is_tombstoned:
# Case 1: SELF-HEAL — clear tombstone, keep row.
self_healed_ids.add(sid)
elif is_empty and is_tombstoned:
# Case 2: still-empty tombstoned row stays hidden this
# poll (do not add to missing_webui_orphan_ids — would
# cycle through prune_session_from_index + record).
still_hidden_ids.add(sid)
elif is_empty and not is_tombstoned:
# Case 3: new orphan.
missing_webui_orphan_ids.add(sid)
# Case 4 (not empty + not tombstoned): row has messages, retain.
if self_healed_ids:
for _sid in self_healed_ids:
try:
_clear_webui_zero_message_orphan_tombstone(_sid)
logger.debug(
"self-heal: cleared webui zero-message orphan tombstone "
"for %s (state.db.messages now non-empty)",
_sid,
)
except Exception:
logger.debug(
"Failed to clear webui zero-message orphan tombstone for %s",
_sid,
exc_info=True,
)
_diag("self_heal_webui_zero_message_orphan")
if missing_webui_orphan_ids:
for _sid in missing_webui_orphan_ids:
try:
prune_session_from_index(_sid)
_diag("prune_orphaned_webui_zero_message")
except Exception:
logger.debug(
"Failed to prune orphaned webui zero-message row %s",
_sid,
exc_info=True,
)
# Tombstone the sid in a SECOND step so a tombstone-write failure
# never blocks the prune itself (the prune still removes the row
# from the sidebar; only the re-prune avoidance would degrade).
try:
_record_webui_zero_message_orphan_tombstone(_sid)
except Exception:
logger.debug(
"Failed to tombstone webui zero-message orphan %s",
_sid,
exc_info=True,
)
# Return rows excluding both the freshly-pruned orphans AND the
# tombstoned rows that the probe confirmed are still empty (case 2).
# Self-healed rows (case 1) and live rows (case 4) stay in the result.
_hidden = missing_webui_orphan_ids | still_hidden_ids
return [
s for s in rows
if str(s.get("session_id") or "").strip() not in _hidden
]
def _build_session_list_cache_payload(
active_profile: str | None,
all_profiles: bool,
@@ -2090,7 +2305,17 @@ def _build_session_list_cache_payload(
# CLI_VISIBLE_SESSION_LIMIT (20) — an existing session can fall
# out of that window and look deleted. Native WebUI sessions
# (source == "webui") that merely have a CLI ancestor are never
# pruned.
# pruned by this path.
#
# #4985: parallel pass for native-WebUI rows that have a backing
# agent row in state.db but zero messages (a `+`-click that opened a
# row but the first turn never committed, or a sidebar nav that
# opened then closed before any message landed). The same #3238
# helper doesn't catch these because source == "webui" is excluded
# above, and the WebUI delete affordance isn't exposed for them,
# so they would otherwise linger forever. Inflight first-turn
# safety is preserved by gating on `active_stream_id` (after
# _reconcile_stale_stream_state has cleared stale stream ids).
_orphan_probe_rows = []
_kept_after_orphan_prune = []
for s in webui_sessions:
@@ -2137,7 +2362,41 @@ def _build_session_list_cache_payload(
diag_stage("prune_orphaned_agent_sidecar")
continue
_kept_after_orphan_prune.append(s)
webui_sessions = _kept_after_orphan_prune
# #4985 second pass — probe state.db.messages for native-WebUI rows
# that *survived* the upstream all_sessions() #1171 keep-filter (so
# the row is TITLED or has a POSITIVE message_count, meaning it IS
# shown in the sidebar — and the 404 click reported in #4985 happens),
# BUT whose actual state.db.messages table is empty (the ground-truth
# probe). This is the orphan shape #4985 actually describes: a row
# that lingers VISIBLY in the sidebar because of a stale positive
# message_count or a title set before the first turn committed.
#
# The (title!='Untitled' OR count>0) clause is the part that makes
# this gate actually reach a row #1171 kept. Without it, the gate is
# a no-op because all_sessions() at api/models.py:3892-3898 and
# 3946-3952 has already stripped every (Untitled ∧ count==0 ∧
# ¬active_stream_id ∧ ¬has_pending_user_message ∧ ¬worktree_path)
# row before this point — making the earlier 6-condition gate a
# no-op against the real pipeline (review IC_kwDOR1LuPM8AAAABHrkF1Q).
#
# Implementation lives in ``_prune_orphaned_webui_zero_message_sessions``
# above so the prune runs in BOTH branches of this function
# (``if show_cli_sessions:`` AND ``else:``). Established installs
# have ``settings.show_cli_sessions`` pinned to False (per
# api/config.py:7637-7648) and those are exactly the long-time
# users who accumulated the #4985 404 orphans — without hoisting,
# the ``else:`` branch silently skipped the prune
# (review IC_kwDOR1LuPM8AAAABHsyFGg).
#
# Inflight / worktree / pending safety: same as before — any row
# still carrying active_stream_id / has_pending_user_message /
# worktree_path is never pruned, even if its messages table is
# momentarily empty. _reconcile_stale_stream_state_for_session_rows
# at line 2224 has already cleared stale stream ids above this point.
webui_sessions = _prune_orphaned_webui_zero_message_sessions(
_kept_after_orphan_prune,
diag_stage=diag_stage,
)
for s in webui_sessions:
meta = cli_by_id.get(s.get("session_id"))
if not meta:
@@ -2165,6 +2424,18 @@ def _build_session_list_cache_payload(
else:
diag_stage("filter_webui_sessions")
webui_sessions = [s for s in webui_sessions if not _is_cli_session_for_settings(s)]
# #4985 second pass — see _prune_orphaned_webui_zero_message_sessions
# for the gate predicate and the post-#1171-survivor rationale. The
# prune MUST run here too: established installs have
# ``settings.show_cli_sessions`` pinned to False
# (api/config.py:7637-7648) and those are exactly the long-time
# users who accumulated the 404 orphans — review
# IC_kwDOR1LuPM8AAAABHsyFGg. Without this call the else branch
# silently skipped the prune and the sidebar kept dangling rows.
webui_sessions = _prune_orphaned_webui_zero_message_sessions(
webui_sessions,
diag_stage=diag_stage,
)
deduped_cli = []
diag_stage("sort_sessions")
merged = webui_sessions + deduped_cli
@@ -6734,6 +7005,10 @@ from api.models import (
_hide_from_default_sidebar,
prune_session_from_index,
agent_session_rows_existing,
agent_session_zero_message_sids,
_load_webui_zero_message_orphan_tombstone,
_record_webui_zero_message_orphan_tombstone,
_clear_webui_zero_message_orphan_tombstone,
ensure_cron_project,
is_cron_session,
is_safe_session_id,
tests/test_issue4985_orphaned_webui_zero_message.py
+2178
View File
File diff suppressed because it is too large Load Diff