gooneyryan/hermes-webui
1
0
Fork 0
mirror of https://github.com/nesquena/hermes-webui.git synced 2026-05-25 19:20:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f60db40133cf348232bea1024aaadca84dfbe97b
hermes-webui/tests/test_provider_management.py
T
nesquena-hermes 0e9bd651a4 fix: TTS toggle CSS specificity collision (#1409) + Ollama env var bleed (#1410) 
Two unrelated UX/Settings bugs, both small surgical fixes with regression
tests.

Issue #1409 — TTS toggle has no effect
=======================================
Reported via Discord: ticking Settings → Voice → "Text-to-Speech for
responses" did nothing. The speaker icon never appeared on assistant
messages despite the checkbox saving to localStorage correctly.

Root cause (CSS specificity collision):
  static/panels.js _applyTtsEnabled() set
    btn.style.display = enabled ? '' : 'none'
  on every .msg-tts-btn. The '' branch removes the inline override, after
  which the .msg-tts-btn { display:none; } rule from style.css re-hides the
  button. Both branches left the icon hidden, so the toggle has been
  silently broken since #499 first shipped the TTS feature.

Fix (body-class toggle, Option B from the issue):
  - panels.js: _applyTtsEnabled now toggles body.classList('tts-enabled')
  - style.css: new compound selector
      body.tts-enabled .msg-tts-btn { display:inline-flex; align-items:center; }
  - default-hidden rule (.msg-tts-btn{display:none;}) preserved so the icon
    stays hidden by default (CSS-only state)
  - boot.js paths that already call _applyTtsEnabled(localStorage…) work
    unchanged — the new function applies state at the body level instead of
    inline-styling individual buttons, so the rule survives renderMd()
    re-renders without re-querying every button

Verified end-to-end against live server: getComputedStyle on a probe
.msg-tts-btn returns display:flex when body has tts-enabled, display:none
when it doesn't. Two regression tests in TestIssue1409TtsToggleBodyClass
explicitly check for the body-class shape and forbid the broken inline-style
pattern.

Issue #1410 — Ollama (local) shows "API key configured" when only
              Ollama Cloud key is set
=================================================================
Reported via Discord: configuring Ollama Cloud lit up the local Ollama card
too. Both providers were mapped to OLLAMA_API_KEY in api/providers.py
_PROVIDER_ENV_VAR.

Root cause:
  api/providers.py:47-48
    "ollama":       "OLLAMA_API_KEY",
    "ollama-cloud": "OLLAMA_API_KEY",
  _provider_has_key("ollama") found the value the user set for Ollama Cloud
  and returned True. But the runtime code path in
  hermes_cli/runtime_provider.py only consumes OLLAMA_API_KEY when the base
  URL hostname is ollama.com (Ollama Cloud) — local Ollama is keyless by
  default and reaches a custom base URL with no auth. The WebUI was
  reporting "configured" for a key local Ollama doesn't even read.

Fix (Option A from the issue body, preferred):
  - Drop bare "ollama" from _PROVIDER_ENV_VAR with an inline comment
    explaining why
  - _provider_has_key("ollama") falls through to the config.yaml branch,
    which already supports providers.ollama.api_key for local users who
    genuinely need to set a token
  - ollama-cloud retains its OLLAMA_API_KEY mapping unchanged

Verified end-to-end against live server with OLLAMA_API_KEY=sk-cloud-key-test
in env: GET /api/providers reports has_key=True only for ollama-cloud, and
has_key=False for bare ollama. Two regression tests in
TestIssue1410OllamaEnvVarBleed cover the bleed-prevention case AND the
"local user with config.yaml api_key still reports configured" case to
guard against over-correction.

Tests
-----
3572 passed, 2 skipped, 3 xpassed (was 3567 — added 5 new regression tests).

Closes #1409
Closes #1410

Reported by @AvidFuturist (Discord, May 1 2026)
2026-05-01 17:14:51 +00:00

464 lines
18 KiB
Python
Raw Blame History

"""Tests for /api/providers CRUD endpoints (provider key management).
Closes #586 — allow provider key update from the WebUI.
Part of #604 — multi-provider model picker support.
"""
import json
import sys
import types
import urllib.error
import urllib.request
import api.config as config
import api.profiles as profiles
from tests._pytest_port import BASE
# ── HTTP helpers ──────────────────────────────────────────────────────────
def _get(path):
"""GET helper — returns parsed JSON."""
with urllib.request.urlopen(BASE + path, timeout=10) as r:
return json.loads(r.read())
def _post(path, body=None):
"""POST helper — returns (parsed_json, status_code)."""
data = json.dumps(body or {}).encode()
req = urllib.request.Request(
BASE + path, data=data, headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=10) as r:
return json.loads(r.read()), r.status
except urllib.error.HTTPError as e:
body_text = e.read().decode("utf-8", errors="replace")
try:
return json.loads(body_text), e.code
except Exception:
return {"error": body_text}, e.code
def _install_fake_hermes_cli(monkeypatch):
"""Stub hermes_cli modules so tests are deterministic and offline."""
fake_pkg = types.ModuleType("hermes_cli")
fake_pkg.__path__ = []
fake_models = types.ModuleType("hermes_cli.models")
fake_models.list_available_providers = lambda: []
fake_models.provider_model_ids = lambda pid: []
fake_auth = types.ModuleType("hermes_cli.auth")
fake_auth.get_auth_status = lambda _pid: {}
monkeypatch.setitem(sys.modules, "hermes_cli", fake_pkg)
monkeypatch.setitem(sys.modules, "hermes_cli.models", fake_models)
monkeypatch.setitem(sys.modules, "hermes_cli.auth", fake_auth)
monkeypatch.delitem(sys.modules, "agent.credential_pool", raising=False)
monkeypatch.delitem(sys.modules, "agent", raising=False)
# Flush the 60-second TTL model cache so no prior test's result bleeds in.
try:
from api.config import invalidate_models_cache
invalidate_models_cache()
except Exception:
pass
# ── Unit tests (api/providers.py functions directly) ──────────────────────
class TestGetProviders:
"""Unit tests for get_providers() function."""
def test_returns_list_of_known_providers(self, monkeypatch, tmp_path):
"""GET /api/providers should return a list of all known providers."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import get_providers
try:
result = get_providers()
assert "providers" in result
assert "active_provider" in result
assert isinstance(result["providers"], list)
# Should include at least the built-in providers
provider_ids = {p["id"] for p in result["providers"]}
assert "anthropic" in provider_ids
assert "openai" in provider_ids
assert "openrouter" in provider_ids
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_provider_entries_have_required_fields(self, monkeypatch, tmp_path):
"""Each provider entry should have id, display_name, has_key, configurable."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import get_providers
try:
result = get_providers()
for p in result["providers"]:
assert "id" in p, f"Missing 'id' in provider entry"
assert "display_name" in p, f"Missing 'display_name' for {p['id']}"
assert "has_key" in p, f"Missing 'has_key' for {p['id']}"
assert "configurable" in p, f"Missing 'configurable' for {p['id']}"
assert "key_source" in p, f"Missing 'key_source' for {p['id']}"
assert isinstance(p["has_key"], bool)
assert isinstance(p["configurable"], bool)
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_oauth_providers_not_configurable(self, monkeypatch, tmp_path):
"""OAuth providers (copilot, nous, openai-codex) should not be configurable."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import get_providers
try:
result = get_providers()
for p in result["providers"]:
if p["id"] in ("copilot", "nous", "openai-codex"):
assert p["configurable"] is False, f"{p['id']} should not be configurable"
# ollama-cloud is now configurable (uses OLLAMA_API_KEY)
if p["id"] == "ollama-cloud":
assert p["configurable"] is True, "ollama-cloud should be configurable"
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
class TestSetProviderKey:
"""Unit tests for set_provider_key() function."""
def test_set_key_writes_to_env_file(self, monkeypatch, tmp_path):
"""Setting a key should write the env var to ~/.hermes/.env."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
# Also pin HERMES_HOME so code that reads it directly gets tmp_path,
# not the conftest session TEST_STATE_DIR that bleeds into the main process.
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import set_provider_key
try:
result = set_provider_key("anthropic", "sk-ant-test-key-12345678")
assert result["ok"] is True
assert result["provider"] == "anthropic"
assert result["action"] == "updated"
# Verify .env file was written
env_path = tmp_path / ".env"
assert env_path.exists(), f".env not written to {env_path}; HERMES_HOME={__import__('os').environ.get('HERMES_HOME')!r}"
content = env_path.read_text()
assert "ANTHROPIC_API_KEY=sk-ant-test-key-12345678" in content
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_remove_key_deletes_from_env_file(self, monkeypatch, tmp_path):
"""Removing a key should delete the env var from .env."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import set_provider_key
try:
# First set a key
set_provider_key("anthropic", "sk-ant-test-key-12345678")
# Then remove it
result = set_provider_key("anthropic", None)
assert result["ok"] is True
assert result["action"] == "removed"
# Verify .env file no longer has the key
env_path = tmp_path / ".env"
content = env_path.read_text() if env_path.exists() else ""
assert "ANTHROPIC_API_KEY" not in content
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_oauth_provider_rejected(self, monkeypatch, tmp_path):
"""Setting a key for an OAuth provider should fail."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import set_provider_key
try:
result = set_provider_key("copilot", "some-key")
assert result["ok"] is False
assert "OAuth" in result["error"]
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_short_key_rejected(self, monkeypatch, tmp_path):
"""API keys shorter than 8 chars should be rejected."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import set_provider_key
try:
result = set_provider_key("anthropic", "short")
assert result["ok"] is False
assert "too short" in result["error"]
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_empty_provider_id_rejected(self, monkeypatch, tmp_path):
"""Empty provider ID should be rejected."""
from api.providers import set_provider_key
result = set_provider_key("", "some-key")
assert result["ok"] is False
assert "required" in result["error"]
def test_newline_in_key_rejected(self, monkeypatch, tmp_path):
"""API keys with newlines should be rejected."""
from api.providers import set_provider_key
result = set_provider_key("anthropic", "sk-ant-key\nINJECTED=evil")
assert result["ok"] is False
assert "newline" in result["error"]
class TestRemoveProviderKey:
"""Unit tests for remove_provider_key() wrapper."""
def test_remove_provider_key_calls_set_with_none(self, monkeypatch, tmp_path):
"""remove_provider_key should delegate to set_provider_key(id, None)."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import remove_provider_key
try:
result = remove_provider_key("anthropic")
assert result["ok"] is True
assert result["action"] == "removed"
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
# ── Integration tests (via HTTP endpoints) ───────────────────────────────
class TestProvidersEndpoints:
"""Integration tests for /api/providers HTTP endpoints."""
def test_get_providers_returns_200(self):
"""GET /api/providers should return 200 with provider list."""
result = _get("/api/providers")
assert "providers" in result
assert isinstance(result["providers"], list)
def test_post_provider_set_key(self):
"""POST /api/providers with provider + api_key should set the key."""
body, status = _post("/api/providers", {
"provider": "anthropic",
"api_key": "sk-ant-integration-test-key-12345678",
})
assert status == 200
assert body.get("ok") is True
assert body.get("provider") == "anthropic"
def test_post_provider_remove_key(self):
"""POST /api/providers with provider but no api_key should remove the key."""
body, status = _post("/api/providers", {
"provider": "anthropic",
"api_key": None,
})
assert status == 200
assert body.get("ok") is True
assert body.get("action") == "removed"
def test_post_provider_delete(self):
"""POST /api/providers/delete should remove the key."""
body, status = _post("/api/providers/delete", {
"provider": "anthropic",
})
assert status == 200
assert body.get("ok") is True
def test_post_provider_missing_id(self):
"""POST /api/providers without provider should return 400."""
body, status = _post("/api/providers", {"api_key": "some-key"})
assert status == 400
assert "required" in body.get("error", "").lower()
def test_post_provider_delete_missing_id(self):
"""POST /api/providers/delete without provider should return 400."""
body, status = _post("/api/providers/delete", {})
assert status == 400
class TestIssue1410OllamaEnvVarBleed:
"""Regression: Ollama Cloud key must not flip local Ollama to has_key=True.
Both providers used to share OLLAMA_API_KEY in _PROVIDER_ENV_VAR. After
a user added a key for Ollama Cloud, the local Ollama card also lit up
"API key configured" — incorrect because the runtime in
hermes_cli/runtime_provider.py only consumes OLLAMA_API_KEY when the
base URL hostname is ollama.com. Local Ollama is keyless by default.
Fix: drop bare "ollama" from _PROVIDER_ENV_VAR so the env-var check is
only applied to ollama-cloud. Local Ollama users who genuinely need a
key can still set providers.ollama.api_key in config.yaml.
"""
def test_ollama_local_not_configured_when_only_cloud_env_var_set(
self, monkeypatch, tmp_path,
):
"""OLLAMA_API_KEY in env should mark ollama-cloud configured but not bare ollama."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
monkeypatch.setenv("OLLAMA_API_KEY", "sk-cloud-key-xyz")
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import get_providers
try:
result = get_providers()
by_id = {p["id"]: p for p in result["providers"]}
assert "ollama-cloud" in by_id, "ollama-cloud should appear in provider list"
assert "ollama" in by_id, "ollama (local) should appear in provider list"
assert by_id["ollama-cloud"]["has_key"] is True, \
"ollama-cloud should be has_key=True when OLLAMA_API_KEY is set"
assert by_id["ollama"]["has_key"] is False, (
"ollama (local) must NOT be has_key=True when only the cloud env "
"var is set — local Ollama is keyless and shares no env var with "
"Ollama Cloud (#1410)."
)
# ollama-cloud should be configurable, but local ollama should not
# (it has no env var mapping — keys go through providers.ollama.api_key
# in config.yaml if the user explicitly opts in).
assert by_id["ollama-cloud"]["configurable"] is True
assert by_id["ollama"]["configurable"] is False
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
def test_ollama_local_still_configured_via_config_yaml(
self, monkeypatch, tmp_path,
):
"""providers.ollama.api_key in config.yaml should still mark local ollama configured."""
_install_fake_hermes_cli(monkeypatch)
monkeypatch.setattr(profiles, "get_active_hermes_home", lambda: tmp_path)
# Important: clear the env var so the only signal is config.yaml.
monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
old_cfg = dict(config.cfg)
old_mtime = config._cfg_mtime
config.cfg.clear()
config.cfg["model"] = {}
config.cfg["providers"] = {"ollama": {"api_key": "local-token-abc"}}
try:
config._cfg_mtime = config.Path(config._get_config_path()).stat().st_mtime
except Exception:
config._cfg_mtime = 0.0
from api.providers import get_providers
try:
result = get_providers()
by_id = {p["id"]: p for p in result["providers"]}
assert by_id["ollama"]["has_key"] is True, (
"Local Ollama users with providers.ollama.api_key in config.yaml "
"should still report configured (#1410 fix must not regress this)."
)
# And ollama-cloud should NOT be configured by ollama's config entry.
assert by_id["ollama-cloud"]["has_key"] is False
finally:
config.cfg.clear()
config.cfg.update(old_cfg)
config._cfg_mtime = old_mtime
Reference in New Issue View Git Blame Copy Permalink