mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-21 03:39:54 +00:00
teknium1
f9ad7400e3
The freeform /goal judge was capped at max_tokens=200, which reliably
truncated the JSON verdict on reasoning-heavy models (deepseek-v4-pro,
qwq, etc.) — the model burns tokens on hidden reasoning before emitting
visible content, and the first /goal turn's prompt is larger than later
turns, blowing past 200. Symptom: agent.log shows
`judge reply was not JSON: '{"done": true, "reason": "The agent successfully'`
followed by repeated `judge returned empty response` lines, then the
goal pauses with a misleading 'judge model isn't returning the required
JSON verdict' message.
Diagnosed live by @helix4u — empirically verified that raising the
budget on an unmodified worktree makes the failures go away on the
exact configs users were hitting on Nous Plus subscription paths.
Changes:
- DEFAULT_JUDGE_MAX_TOKENS = 4096 (up from 200)
- New auxiliary.goal_judge.max_tokens config knob for tuning in
specifically constrained setups
- _goal_judge_max_tokens() resolves the value with fail-open semantics
(non-int / non-positive / load failure → default). load_config() is
mtime-cached so per-turn lookup is cheap.
Scoped narrowly to the verified root cause — does not introduce a
submit_verdict tool-call schema (see #26162 / #23671 for that direction;
they can land separately if we want them).
Tests: tests/hermes_cli/test_goals.py + tests/cli/test_cli_goal_interrupt.py
+ tests/gateway/test_goal_verdict_send.py — 62/62 passing.
E2E verified: config override honored (8192), missing/garbage/zero
values fall back to 4096, no-auxiliary-section falls back to 4096.
Co-authored-by: helix4u <4317663+helix4u@users.noreply.github.com>
Credits:
- @helix4u (Gille) — diagnosed the max_tokens=200 truncation via live
testing on an unmodified worktree, drafted the original fix shape
in #26162.
- @AhmetArif0 — flagged the freeform judge fragility in #23671 from
the tool-call angle.
- @0xharryriddle (HarryRiddle.eth) — reported the issue from a Nous
Plus subscription setup in #23876 with full debug reports.
Closes #23876
Supersedes #26162, #23671, #23881