Hermes Agent

Hermes Agent is an open-source AI agent framework by Nous Research that runs in your terminal, messaging platforms, and IDEs. It belongs to the same category as Claude Code (Anthropic), Codex (OpenAI), and OpenClaw — autonomous coding and task-execution agents that use tool calling to interact with your system. Hermes works with any LLM provider (OpenRouter, Anthropic, OpenAI, DeepSeek, local models, and 15+ others) and runs on Linux, macOS, and WSL.

What makes Hermes different:

• Self-improving through skills — Hermes learns from experience by saving reusable procedures as skills. When it solves a complex problem, discovers a workflow, or gets corrected, it can persist that knowledge as a skill document that loads into future sessions. Skills accumulate over time, making the agent better at your specific tasks and environment.
• Persistent memory across sessions — remembers who you are, your preferences, environment details, and lessons learned. Pluggable memory backends (built-in, Honcho, Mem0, and more) let you choose how memory works.
• Multi-platform gateway — the same agent runs on Telegram, Discord, Slack, WhatsApp, Signal, Matrix, Email, and 10+ other platforms with full tool access, not just chat.
• Provider-agnostic — swap models and providers mid-workflow without changing anything else. Credential pools rotate across multiple API keys automatically.
• Profiles — run multiple independent Hermes instances with isolated configs, sessions, skills, and memory.
• Extensible — plugins, MCP servers, custom tools, webhook triggers, cron scheduling, and the full Python ecosystem.

People use Hermes for software development, research, system administration, data analysis, content creation, home automation, and anything else that benefits from an AI agent with persistent context and full system access.

This skill helps you work with Hermes Agent effectively — setting it up, configuring features, spawning additional agent instances, troubleshooting issues, finding the right commands and settings, and understanding how the system works when you need to extend or contribute to it.

Docs: https://hermes-agent.nousresearch.com/docs/

Quick Start

# Install
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

# Interactive chat (default)
hermes

# Single query
hermes chat -q "What is the capital of France?"

# Setup wizard
hermes setup

# Change model/provider
hermes model

# Check health
hermes doctor

---

CLI Reference

Global Flags

hermes [flags] [command]

  --version, -V             Show version
  --resume, -r SESSION      Resume session by ID or title
  --continue, -c [NAME]     Resume by name, or most recent session
  --worktree, -w            Isolated git worktree mode (parallel agents)
  --skills, -s SKILL        Preload skills (comma-separate or repeat)
  --profile, -p NAME        Use a named profile
  --yolo                    Skip dangerous command approval
  --pass-session-id         Include session ID in system prompt

No subcommand defaults to chat.

Chat

hermes chat [flags]
  -q, --query TEXT          Single query, non-interactive
  -m, --model MODEL         Model (e.g. anthropic/claude-sonnet-4)
  -t, --toolsets LIST       Comma-separated toolsets
  --provider PROVIDER       Force provider (openrouter, anthropic, nous, etc.)
  -v, --verbose             Verbose output
  -Q, --quiet               Suppress banner, spinner, tool previews
  --checkpoints             Enable filesystem checkpoints (/rollback)
  --source TAG              Session source tag (default: cli)

Configuration

hermes setup [section]      Interactive wizard (model|terminal|gateway|tools|agent)
hermes model                Interactive model/provider picker
hermes config               View current config
hermes config edit          Open config.yaml in $EDITOR
hermes config set KEY VAL   Set a config value
hermes config path          Print config.yaml path
hermes config env-path      Print .env path
hermes config check         Check for missing/outdated config
hermes config migrate       Update config with new options
hermes auth                 Interactive credential manager
hermes auth add PROVIDER    Add OAuth or API-key credential (e.g. nous, openai-codex, qwen-oauth)
hermes auth list            List stored credentials
hermes auth remove PROVIDER Remove a stored credential
hermes doctor [--fix]       Check dependencies and config
hermes status [--all]       Show component status

Tools & Skills

hermes tools                Interactive tool enable/disable (curses UI)
hermes tools list           Show all tools and status
hermes tools enable NAME    Enable a toolset
hermes tools disable NAME   Disable a toolset

hermes skills list          List installed skills
hermes skills search QUERY  Search the skills hub
hermes skills install ID    Install a skill (ID can be a hub identifier OR a direct https://…/SKILL.md URL; pass --name to override when frontmatter has no name)
hermes skills inspect ID    Preview without installing
hermes skills config        Enable/disable skills per platform
hermes skills check         Check for updates
hermes skills update        Update outdated skills
hermes skills uninstall N   Remove a hub skill
hermes skills publish PATH  Publish to registry
hermes skills browse        Browse all available skills
hermes skills tap add REPO  Add a GitHub repo as skill source

MCP Servers

hermes mcp serve            Run Hermes as an MCP server
hermes mcp add NAME         Add an MCP server (--url or --command)
hermes mcp remove NAME      Remove an MCP server
hermes mcp list             List configured servers
hermes mcp test NAME        Test connection
hermes mcp configure NAME   Toggle tool selection

How the built-in MCP client connects servers (stdio/HTTP), auto-discovers
their tools, and exposes them as first-class tools, plus catalog install
(hermes mcp install <name>): skill_view(name="hermes-agent", file_path="references/native-mcp.md").

Gateway (Messaging Platforms)

hermes gateway run          Start gateway foreground
hermes gateway install      Install as background service
hermes gateway start/stop   Control the service
hermes gateway restart      Restart the service
hermes gateway status       Check status
hermes gateway setup        Configure platforms

Supported platforms: Telegram, Discord, Slack, WhatsApp, Signal, Email, SMS, Matrix, Mattermost, Home Assistant, DingTalk, Feishu, WeCom, BlueBubbles (iMessage), Weixin (WeChat), API Server, Webhooks. Open WebUI connects via the API Server adapter.

Platform docs: https://hermes-agent.nousresearch.com/docs/user-guide/messaging/

Sessions

hermes sessions list        List recent sessions
hermes sessions browse      Interactive picker
hermes sessions export OUT  Export to JSONL
hermes sessions rename ID T Rename a session
hermes sessions delete ID   Delete a session
hermes sessions prune       Clean up old sessions (--older-than N days)
hermes sessions stats       Session store statistics

Cron Jobs

hermes cron list            List jobs (--all for disabled)
hermes cron create SCHED    Create: '30m', 'every 2h', '0 9 * * *'
hermes cron edit ID         Edit schedule, prompt, delivery
hermes cron pause/resume ID Control job state
hermes cron run ID          Trigger on next tick
hermes cron remove ID       Delete a job
hermes cron status          Scheduler status

Pitfall: don't guess cron IDs from session memory. When the user says "stop that cron" / "停掉 X" / "那个 cron 怎么还在跑", the first action is cronjob_manage action=list (or hermes cron list --all) to find the actual job_id — never retrieve a stale ID from earlier in the conversation. Job IDs are not stable across the lifetime of a session the way a chat message is. Two specific failure modes from guessing:

• The job doesn't exist → cronjob_manage remove returns "not found" and the cron keeps running, so the user is lied to about the fix taking effect
• The wrong job is removed → a different recurring task the user actually wanted dies silently

After remove, run list again to verify the job is gone before reporting success.

Pitfall: when the user reports a notification/message that "definitely came from you" — check your own crons first, don't deny. If the user says "我刚才收到一条 X 是你发的吗" / "你怎么又发 X 给我", the worst response is "不是我发的，我没推过任何东西" before checking. The agent often controls recurring push channels (cron jobs, kanban dispatchers, scheduled heartbeats) that produce exactly that kind of message. The right move: cronjob_manage action=list (and a quick hermes status/tail of cron output dirs) BEFORE answering. If a cron you own produced it, say so and offer to stop/pause it. If nothing you own produced it, then say "不是我的" with evidence. Denying first and being walked back to "就是你的" wastes a turn and erodes trust.

Webhooks

hermes webhook subscribe N  Create route at /webhooks/<name>
hermes webhook list         List subscriptions
hermes webhook remove NAME  Remove a subscription
hermes webhook test NAME    Send a test POST

Full setup, route config, payload templating, and event-driven agent-run
patterns: skill_view(name="hermes-agent", file_path="references/webhooks.md").

Profiles

hermes profile list         List all profiles
hermes profile create NAME  Create (--clone, --clone-all, --clone-from)
hermes profile use NAME     Set sticky default
hermes profile delete NAME  Delete a profile
hermes profile show NAME    Show details
hermes profile alias NAME   Manage wrapper scripts
hermes profile rename A B   Rename a profile
hermes profile export NAME  Export to tar.gz
hermes profile import FILE  Import from archive

Credential Pools

hermes auth add             Interactive credential wizard
hermes auth list [PROVIDER] List pooled credentials
hermes auth remove P INDEX  Remove by provider + index
hermes auth reset PROVIDER  Clear exhaustion status

Other

hermes insights [--days N]  Usage analytics
hermes update               Update to latest version
hermes pairing list/approve/revoke  DM authorization
hermes plugins list/install/remove  Plugin management
hermes honcho setup/status  Honcho memory integration (requires honcho plugin)
hermes memory setup/status/off  Memory provider config
hermes completion bash|zsh  Shell completions
hermes acp                  ACP server (IDE integration)
hermes claw migrate         Migrate from OpenClaw
hermes uninstall            Uninstall Hermes

---

Slash Commands (In-Session)

Type these during an interactive chat session. New commands land fairly
often; if something below looks stale, run /help in-session for the
authoritative list or see the live slash commands reference.
The registry of record is hermes_cli/commands.py — every consumer
(autocomplete, Telegram menu, Slack mapping, /help) derives from it.

Session Control

/new (/reset)        Fresh session
/clear               Clear screen + new session (CLI)
/retry               Resend last message
/undo                Remove last exchange
/title [name]        Name the session
/compress            Manually compress context
/stop                Kill background processes
/rollback [N]        Restore filesystem checkpoint
/snapshot [sub]      Create or restore state snapshots of Hermes config/state (CLI)
/background <prompt> Run prompt in background
/queue <prompt>      Queue for next turn
/steer <prompt>      Inject a message after the next tool call without interrupting
/agents (/tasks)     Show active agents and running tasks
/resume [name]       Resume a named session
/goal [text|sub]     Set a standing goal Hermes works on across turns until achieved
                     (subcommands: status, pause, resume, clear)
/redraw              Force a full UI repaint (CLI)

Configuration

/config              Show config (CLI)
/model [name]        Show or change model
/personality [name]  Set personality
/reasoning [level]   Set reasoning (none|minimal|low|medium|high|xhigh|show|hide)
/verbose             Cycle: off → new → all → verbose
/voice [on|off|tts]  Voice mode
/yolo                Toggle approval bypass
/busy [sub]          Control what Enter does while Hermes is working (CLI)
                     (subcommands: queue, steer, interrupt, status)
/indicator [style]   Pick the TUI busy-indicator style (CLI)
                     (styles: kaomoji, emoji, unicode, ascii)
/footer [on|off]     Toggle gateway runtime-metadata footer on final replies
/skin [name]         Change theme (CLI)
/statusbar           Toggle status bar (CLI)

Tools & Skills

/tools               Manage tools (CLI)
/toolsets            List toolsets (CLI)
/skills              Search/install skills (CLI)
/skill <name>        Load a skill into session
/reload-skills       Re-scan ~/.hermes/skills/ for added/removed skills
/reload              Reload .env variables into the running session (CLI)
/reload-mcp          Reload MCP servers
/cron                Manage cron jobs (CLI)
/curator [sub]       Background skill maintenance (status, run, pin, archive, …)
/kanban [sub]        Multi-profile collaboration board (tasks, links, comments)
/plugins             List plugins (CLI)

Gateway

/approve             Approve a pending command (gateway)
/deny                Deny a pending command (gateway)
/restart             Restart gateway (gateway)
/sethome             Set current chat as home channel (gateway)
/update              Update Hermes to latest (gateway)
/topic [sub]         Enable or inspect Telegram DM topic sessions (gateway)
/platforms (/gateway) Show platform connection status (gateway)

Utility

/branch (/fork)      Branch the current session
/fast                Toggle priority/fast processing
/browser             Open CDP browser connection
/history             Show conversation history (CLI)
/save                Save conversation to file (CLI)
/copy [N]            Copy the last assistant response to clipboard (CLI)
/paste               Attach clipboard image (CLI)
/image               Attach local image file (CLI)

Info

/help                Show commands
/commands [page]     Browse all commands (gateway)
/usage               Token usage
/insights [days]     Usage analytics
/gquota              Show Google Gemini Code Assist quota usage (CLI)
/status              Session info (gateway)
/profile             Active profile info
/debug               Upload debug report (system info + logs) and get shareable links

Exit

/quit (/exit, /q)    Exit CLI

---

Key Paths & Config

~/.hermes/config.yaml       Main configuration
~/.hermes/.env              API keys and secrets
$HERMES_HOME/skills/        Installed skills
~/.hermes/sessions/         Gateway routing index, request dumps, *.jsonl transcripts (and optional per-session JSON snapshots when sessions.write_json_snapshots: true)
~/.hermes/state.db          Canonical session store (SQLite + FTS5)
~/.hermes/logs/             Gateway and error logs
~/.hermes/auth.json         OAuth tokens and credential pools
~/.hermes/hermes-agent/     Source code (if git-installed)

Profiles use ~/.hermes/profiles/<name>/ with the same layout.

Config Sections

Edit with hermes config edit or hermes config set section.key value.

Section	Key options
model	default, provider, base_url, api_key, context_length
agent	max_turns (90), tool_use_enforcement
terminal	backend (local/docker/ssh/modal), cwd, timeout (180)
compression	enabled, threshold (0.50), target_ratio (0.20)
display	skin, tool_progress, show_reasoning, show_cost
stt	enabled, provider (local/groq/openai/mistral)
tts	provider (edge/elevenlabs/openai/minimax/mistral/neutts)
memory	memory_enabled, user_profile_enabled, provider
security	tirith_enabled, website_blocklist
delegation	model, provider, base_url, api_key, max_iterations (50), reasoning_effort
checkpoints	enabled, max_snapshots (50)

Full config reference: https://hermes-agent.nousresearch.com/docs/user-guide/configuration

Providers

20+ providers supported. Set via hermes model or hermes setup.

Provider	Auth	Key env var
OpenRouter	API key	OPENROUTER_API_KEY
Anthropic	API key	ANTHROPIC_API_KEY
Nous Portal	OAuth	hermes auth
OpenAI Codex	OAuth	hermes auth
GitHub Copilot	Token	COPILOT_GITHUB_TOKEN
Google Gemini	API key	GOOGLE_API_KEY or GEMINI_API_KEY
DeepSeek	API key	DEEPSEEK_API_KEY
xAI / Grok	API key	XAI_API_KEY
Hugging Face	Token	HF_TOKEN
Z.AI / GLM	API key	GLM_API_KEY
MiniMax	API key	MINIMAX_API_KEY
MiniMax CN	API key	MINIMAX_CN_API_KEY
Kimi / Moonshot	API key	KIMI_API_KEY
Alibaba / DashScope	API key	DASHSCOPE_API_KEY
Xiaomi MiMo	API key	XIAOMI_API_KEY
Kilo Code	API key	KILOCODE_API_KEY
OpenCode Zen	API key	OPENCODE_ZEN_API_KEY
OpenCode Go	API key	OPENCODE_GO_API_KEY
Qwen OAuth	OAuth	hermes auth add qwen-oauth
Custom endpoint	Config	model.base_url + model.api_key in config.yaml
GitHub Copilot ACP	External	COPILOT_CLI_PATH or Copilot CLI

User docs: https://hermes-agent.nousresearch.com/docs/integrations/providers

Provider-specific quirks (the OpenCode Go relay's no-auth /models
endpoint, the proc-environ key extraction trick, the direct-curl 403
pitfall): skill_view(name="hermes-agent", file_path="references/opencode-go-probe.md"). A copy-pasteable probe
lives at scripts/opencode_go_probe.py.

For wiring a local OpenAI-compatible relay (codebuddy2api, LiteLLM proxy, vLLM, LM Studio) as a named provider that shows in /model and accepts --provider, the plugin path is the right approach: skill_view(name="hermes-agent", file_path="references/adding-custom-providers.md") — covers the two-layer config (plugin + env var), the verified codebuddy plugin as a worked example, the end-to-end verification command, and when to use a plugin vs. a providers: config-only entry.

When the user runs multiple relay containers (e.g. the Python codebuddy2api and the Go Buddy-2API-Go from different projects on different ports), or a relay uses an OAuth login flow that periodically re-auths, see skill_view(name="hermes-agent", file_path="references/codebuddy2api-fleet.md") — the port-discovery workflow, the Go-version's /admin/login + /admin/account/oauth/start flow, the "which container am I on" probes, and the failure-mode table.

Self-Provisioning: Don't Trust the Initial Config

A user-stated preference that came up while working with this skill: do
not treat the stock config.yaml (or whatever shipped with the install)
as the final state. Models, providers, and credentials change
underneath; the agent is expected to probe what's actually available,
register everything it can reach, and configure fallbacks itself rather
than assume the defaults reflect the current environment.

Concrete steps when the user says "自己照顾自己 / figure it out yourself"
or similar:

1. Discover the real provider set. curl -s https://opencode.ai/zen/go/v1/models (OpenCode Go) and analogous
   /models endpoints for other providers — these often work without
   auth and return the full list of registered model IDs. Don't assume
   the model.name in config.yaml is the only option.
2. Locate the live credentials in the running gateway, not your
   shell. Hermes's terminal sandbox does not export the gateway's
   OPENCODE_GO_API_KEY (or any provider key) into the tool's child
   env. Read it from /proc/<gateway-pid>/environ instead — the
   gateway runs as the hermes user and its env is world-readable.
   The PID lives in ~/.hermes/gateway.pid or is the child of
   /opt/hermes/.venv/bin/hermes gateway run (find it with
   pgrep -f "hermes gateway run").
3. The provider's chat/completions endpoint is NOT the same as
   Hermes's internal path. A Bearer token that works for Hermes may
   return 403 on direct curl — the relay adds extra headers (session
   id, forwarded-for) that only the in-process client supplies. To
   probe what works, route the call through Hermes (e.g. /model <name> in a chat session) rather than curl-ing the upstream
   directly.
4. Register everything you can reach in providers: +
   fallback_providers: in config.yaml. Default the order by
   latency, not by name. Re-probe on a schedule (cron
   */30 * * * * hermes doctor --check provider,network,disk,telegram
   is the existing tool) and prune entries that go red.
5. Use hermes model (interactive picker) to verify the active model
   matches what you think it is. The persona's claimed model and the
   runtime model can disagree — the ~/.hermes/config.yaml model.name
   field is the source of truth, not the SOUL.md / README that may have
   been written by a different session.

Anti-pattern: writing SOUL.md or README with a model identity ("I am
claude-opus-4.6", "I run on three-model failover") without ever
checking config.yaml and ps to confirm. The agent WILL get
corrected for this.

Adding a custom provider — first move is to read a bundled plugin, not to guess

When the user says "add X as a provider" / "把 X 接到模型库" / "X 反代作为备用模型" where X is a local OpenAI-compatible relay (codebuddy2api, LiteLLM, vLLM, LM Studio, etc.), the failure mode is to start writing YAML and configs before reading the source. Do not guess; the bundled plugins under /opt/hermes/plugins/model-providers/<name>/ are the canonical template (each is ~50-150 lines). Deep dive: references/adding-custom-providers.md. The short version:

1. List what's bundled first. ls /opt/hermes/plugins/model-providers/. If the relay matches a built-in protocol (any of the bundled plugins), the existing custom provider's aliases (ollama, vllm, llamacpp, lmstudio, local) are enough — see the "when to use plugin vs config entry" decision in the reference. If the user just wants the existing custom provider pointed at a different base_url, edit providers.<slug>: in config.yaml, do NOT write a new plugin.
2. If a new provider name is needed, copy an existing plugin as the template. cat /opt/hermes/plugins/model-providers/deepseek/__init__.py (or any __init__.py of ~80 lines that calls register_provider(ProviderProfile(...)) with env_vars=... and base_url=...) is the canonical shape. Two files: __init__.py (registers the profile) and plugin.yaml (manifest). Place at $HERMES_HOME/plugins/model-providers/<slug>/ — NOT under /opt/hermes/... (root-owned, PermissionError).
3. The provider type is the wire protocol, not the container name. "codebuddy" / "litellm" / "vllm-server" are container names. The provider type for any OpenAI chat-completions relay is the bundled custom provider — so the plugin name is whatever you want to show in the picker (e.g. codebuddy is fine as a slug; the provider field is what register_provider exposes). Writing provider: codebuddy2api in a fallback_providers entry doesn't work because no built-in has that slug.
4. Verify before claiming done. A single hermes -z "Reply with exactly: PONG" --provider <slug> --model <m> --usage-file /tmp/u.json proves both that the plugin registered (no usage file = plugin missing) AND that the key resolved (401 in usage = key missing in .env). This is the only hermes-internal probe that exercises the full path; direct-curl to the relay proves only the relay itself, not the Hermes integration.
5. Read back the on-disk state. After any hermes config set, grep the relevant config.yaml section — hermes config set returns exit 0 on success and exit 0 on some silent-failure modes, so the exit code is not evidence. For providers:, also call get_compatible_custom_providers(load_config()) from a Python REPL — a list-form providers block (providers: [{...}] instead of providers: {slug: {...}}) writes successfully and is then silently dropped (see pitfall below). The custom providers: 0 个 print is the failure signal.

providers: vs fallback_providers: are different mechanisms (don't conflate them)

A user-visible symptom: "I added my new codebuddy/custom relay under
fallback_providers: in config.yaml, then ran /model in chat and
the new model isn't in the picker." The two keys are NOT
interchangeable:

Config key	What it does	Where it shows up
providers: (v12+ format, list of {name, api, key_env, model, ...})	Adds a selectable provider/model to the picker	/model interactive picker AND credential rotation
custom_providers: (legacy list)	Same as providers: for back-compat	/model picker (deduped with providers:)
fallback_providers: (list of {provider, model, base_url?, key_env?})	Auto-failover when primary model fails (rate-limit / 5xx / connection)	Hidden from /model; only fires on primary error

Same YAML shape, totally different routing. The table above is the
complete set; treat them as orthogonal axes. Common misroute: putting a
relay in both providers: AND fallback_providers: because the
user says "I want it as a backup." The result is a /model picker entry
the user didn't ask for, plus auto-failover. If the user only wants
failover, put it in fallback_providers: only and add the provider slug
to model_catalog.excluded_providers if it shows up via auto-discovery;
this gives the silent-failover shape without polluting the picker. To
make a custom relay manually selectable in /model, you must add
it to providers: (OR to the legacy custom_providers: list —
get_compatible_custom_providers()
dedupes both). fallback_providers: is a backup-only mechanism; the
user has to break the primary before fallback fires. The hermes fallback add CLI writes the fallback_providers: key, not
providers: — by design, not a bug.

Required fields per entry differ too:

• providers: entry: name, api (= base_url), key_env, model — name is the display label in the picker.
• fallback_providers: entry: provider (= provider slug, e.g. custom), model, optional base_url + key_env.

If the user says "I want to pick X manually," put it in providers:.
If the user says "use X only when the primary is down," put it in
fallback_providers:. The same relay can live in BOTH if it serves
both purposes (manual select + auto-failover) — they're independent
lists.

Verify after writing:

hermes fallback list   # shows fallback_providers chain
# TUI only — /model picker cannot be invoked from a non-TTY subprocess:
#   hermes -z "/model"  → "没有 /model 命令可切" (one-shot mode has no picker)
#   hermes model        → "requires an interactive terminal"
# So the only way to confirm /model visibility is from a real hermes TUI session.

"Why didn't fallback fire?" — read the Fallback skip: lines before touching config

When the primary 429s (quota / rate limit) and the turn dies instead of switching, the chain is usually configured correctly — a candidate was skipped. grep -n 'Fallback skip' /opt/data/logs/agent.log | tail gives the reason. Three shapes, three different fixes:

Log line	Real meaning	Fix
credential pool is exhausted (every entry in cooldown)	Misleading wording. _candidate_pool_exhausted fires when next_available_at() is None, which includes a pool row that exists but carries no secret — not only a timed cooldown.	Remove the dead pool row (below)
resolves to the same backend as the current one (<base_url>)	Backend identity compared along the axis the failure invalidated — see the scope table below; a different model on the same URL is NOT skipped for a 429	For credential/endpoint failures, add an entry on a different provider+key; for model-scope failures a sibling model is enough
was rejected as unentitled / previously marked unavailable	Entitlement rejection or an in-session suppression.	Pick another provider

The dead-pool-row trap. A config-only provider slug (any relay added via providers: without a plugins/model-providers/<slug>/ plugin) is not in PROVIDER_REGISTRY. If a pool row for it is source: env:<VAR> (env-sourced rows never store the secret; they re-hydrate on every load), hydration needs _seed_from_env, which early-returns on a failed registry lookup — so access_token stays empty, _available_entries drops the row, has_available() is False, next_available_at() is None, and the candidate is skipped on every attempt forever. hermes auth list shows that row with no warning; the emptiness is only visible in Python:

cd /opt/hermes && source .venv/bin/activate && python3 -c "
from agent.credential_pool import load_pool
p = load_pool('workbuddy-hub')
for e in p.entries(): print(e.id, e.source, 'key_len=', len(e.runtime_api_key))
print('has_available:', p.has_available(model='deepseek-v4.1-flash'), 'next:', p.next_available_at(model='deepseek-v4.1-flash'))"

Fix (back up first — pool state is /opt/data/auth.json, key credential_pool):

cp /opt/data/auth.json /opt/data/cache/scratch/auth.json.bak-$(date +%Y%m%d-%H%M%S)
hermes auth remove <provider> <credential-id>

With the row gone, has_credentials() is False → the skip check passes → the fallback call uses providers.<slug>.api_key from config.yaml. Side effect to state out loud: hermes auth remove also strips that var from .env and records it under suppressed_sources, so it won't be re-seeded. That is harmless when the key lives inline in providers: — check before assuming.

Verify the predicate flipped and the route resolves, instead of guessing from an exit code:

python3 -c "
from agent.chat_completion_helpers import _candidate_pool_exhausted
from agent.auxiliary_client import resolve_provider_client
class A: _credential_pool=None; provider='minimax-cn'; model='MiniMax-M3'
print('skip?', _candidate_pool_exhausted(A(), 'workbuddy-hub', 'deepseek-v4.1-flash'))  # want False
c,_ = resolve_provider_client('workbuddy-hub','deepseek-v4.1-flash'); print(c.base_url, str(c.api_key)[:6])"

The pool row is loaded from disk per attempt, so this fix needs no gateway restart. Note also that a one-entry chain pointing at the primary's own endpoint is no backup at all — say that plainly instead of letting the user believe failover exists.

Same-hub multi-model chains: what they buy you, and what they don't

Fallback candidates are compared against the failed route with agent/backend_identity.py::should_skip_candidate, and the comparison axis depends on the failure class:

Failure	Scope	Predicate	Same provider, different model, same base_url → skipped?
429 / timeout / 5xx / bad response	MODEL (default)	same_deployment() — provider and model must match	No — the sibling model is a real fallback
401 / 402 (key dead, account unpaid)	CREDENTIAL	same_credential_surface() — same provider label	Yes — every entry on that hub is skipped at once
DNS / connection refused	ENDPOINT	same_endpoint() — same base_url	Yes

So a chain of several models behind ONE relay (one api_key, one base_url) is genuine model-level failover for quota and overload, and worthless for the relay itself dying. Describe it that way to the user, in those words — "protects against a single model 429ing, not against the hub's key expiring or the account running dry" — rather than calling it failover without qualification. If the user needs survival of the relay, the chain needs one entry on a different provider with its own credential.

Compose the order by intent, not by name: strongest sibling first (same family, larger), then cheapest high-throughput model, then a different vendor on the same hub (avoids one vendor's bad hour), then a free/zero-credit backstop last. Cheap models with huge max_output_tokens make the best middle entries.

A chain entry is not real until a chat call returns content through it. The relay's /v1/models disagrees with reality in both directions — it omits models that answer fine, and lists models that 400 (e.g. a model that rejects the reasoning_effort parameter). Probe every primary and fallback candidate:

cd /opt/hermes && source .venv/bin/activate && python3 - <<'EOF'
from agent.auxiliary_client import resolve_provider_client
for m in ["<model-id>", ...]:
    try:
        c, _ = resolve_provider_client("<provider-slug>", m)
        r = c.chat.completions.create(model=m, messages=[{"role": "user", "content": "reply with just: ok"}], max_tokens=300)
        print(m, "OK", repr((r.choices[0].message.content or "")[:40]))
    except Exception as e:
        print(m, "FAIL", type(e).__name__, str(e)[:120])
EOF

Also assert the skip predicates per candidate, so a config edit can't silently re-arm the dead-pool trap: _candidate_pool_exhausted(...) and should_skip_candidate(candidate, current) must both be False for the current primary.

providers: must be a DICT (key=slug), not a list — list form silently returns 0 entries

A near-miss that wastes 5+ minutes: the v12+ providers: block must be
{slug: {...}, ...}, NOT [{...}, ...] with - name: inside. The
get_compatible_custom_providers() function in
hermes_cli/config_providers.py:332 calls
providers_dict_to_custom_providers(providers_dict) which does
if not isinstance(providers_dict, dict): return [] — a list-form
providers block is silently dropped, the picker shows the entry is
"missing" with no error message, and hermes fallback list still
shows the chain correctly (because fallback_providers: is a separate
list). The asymmetry is what makes the failure confusing.

Worse: hermes config set providers '[{...}]' (or any JSON-array
literal) parses cleanly, writes to disk, returns exit 0. There is no
schema validation on the CLI side. The first time you discover the
shape is wrong is when /model doesn't show the entry. The only
reliable read-back is the Python one below — grep alone is not
enough because both shapes are valid YAML.

Wrong (silently fails):

providers:
  - name: codebuddy
    api: http://192.168.31.201:8787/v1
    key_env: CODEBUDDY2API_KEY
    model: minimax-m3
    models: [deepseek-v4.1-flash, ...]

Right:

providers:
  codebuddy:                # key IS the slug
    enabled: true
    api: http://192.168.31.201:8787/v1
    key_env: CODEBUDDY2API_KEY
    model: minimax-m3
    models: [deepseek-v4.1-flash, ...]

Verify the dict form was actually accepted (the only reliable way from
a non-TTY subprocess):

import sys; sys.path.insert(0, "/opt/hermes/hermes_cli")
from hermes_cli.config import load_config, get_compatible_custom_providers
provs = get_compatible_custom_providers(load_config())
# provs == [] → list form was written; rewrite as dict
# provs == [expected entry] → confirmed

This call also flushes plugin discovery (it goes through
sync_plugin_provider_registry() under the hood), so an empty result
can mean either (a) the dict shape is wrong, OR (b) the plugin at
$HERMES_HOME/plugins/model-providers/<slug>/ was not picked up.
Check both before concluding.

Unknown keys (e.g. enabled: true if the provider doesn't declare it
in _KNOWN_PROVIDER_KEYS) emit a warning to the gateway log — harmless
because the default is enabled, but worth knowing if you see
"unknown config keys ignored" lines after a config save.

Local relay container name ≠ provider type

A local OpenAI-compatible reverse-proxy container has TWO names: the
container name (e.g. codebuddy2api, litellm, vllm-server) and
the provider type the relay speaks (always custom / OpenAI
chat-completions for these). The providers: entry's provider field
takes the provider type, not the container name. Writing
provider: codebuddy in a fallback chain, --provider codebuddy on
the CLI, or naming the plugin codebuddy works only if you've
also registered codebuddy as a provider name in the registry —
either by adding a plugins/model-providers/codebuddy/ plugin
(register_provider is the only path that adds a new provider name
to PROVIDER_REGISTRY), or by reusing an existing built-in (custom,
ollama, vllm, llamacpp are aliases on the bundled custom
provider).

The diagnostic symptom: --provider <name> returns silently with no
output, no usage file, exit 0. The correct move is to look at
$HERMES_HOME/plugins/model-providers/<name>/ and confirm a plugin
exists that calls register_provider. If not, write one — the
container name is irrelevant to the provider name; pick whatever slug
makes sense for the picker UI.

discover_models: true includes models that 401 upstream — curate models: by hand

hermes model will populate the picker with whatever /v1/models
returns from the relay, but a third of those will 401 on actual
chat-completions (the relay's /v1/models is a static catalog, not a
live probe). End result: the user sees a model in the picker, picks
it, and gets a runtime 401. Two ways to avoid:

1. Set an explicit models: [...] list of only the models you
   hand-verified return 200 to a real chat call. List order is the
   picker order.
2. Leave discover_models: true and tell the user which models are
   known-broken so they don't waste a turn trying each one.

For codebuddy2api (Tencent WorkBuddy/CodeBuddy reverse proxy) the
verified-working set is small: deepseek-v4-pro,
deepseek-v3-2-volc, deepseek-v4.1-flash, minimax-m3, glm-5.2,
kimi-k2.6, kimi-k2.5, hunyuan-chat. Everything else in
/v1/models returned a wrapped 401 (APISIX upstream rejection) or a
reasoning-effort-not-supported error.

For workbuddy2api:7863 (the DGZSbot Go port of the same upstream, current deploy 2026-10), the verified-working set is different from codebuddy2api even though both proxy the same upstream. Hand-verified via real chat/completions calls (200 + non-empty content), not just /v1/models enumeration: cn:deepseek-v4.1-flash, cn:glm-5.3-flash, cn:kimi-k3-1, cn:minimax-m3, cn:glm-5.3. Not working despite /v1/models returning them: cn:kimi-k2-thinking (503 no_healthy_account), cn:glm-4.6 (503 no_healthy_account). The two sets don't generalize across relay implementations — always hand-verify the listed models: on the specific container before declaring done, even if the same model is "verified" on a sibling container.

WorkBuddy/CodeBuddy reverse proxy requires cn: / global: prefix on model name

The workbuddy2api Sliverkiss/DGZSbot relay (current deploy: 7863
core + 8787 GUI + 7864 manager) and the codebuddy2api
maiphucgiang/EchoPing07 relays all proxy Tencent's WorkBuddy or
CodeBuddy upstream, where every chat call must carry a realm
prefix — cn:deepseek-v4.1-flash for the CN domain
(copilot.tencent.com / www.codebuddy.cn) or global:kimi-k2.6
for the international domain (www.workbuddy.ai). A bare
deepseek-v4.1-flash (no prefix) is rejected by the upstream with
400 code=11102 service info not found. When configuring
providers.<slug>.models: and providers.<slug>.model: for any
WorkBuddy-family relay, always include the prefix — both in the
default model: and in every entry of the models: list. The relay
itself documents this in its /v1/models output (every listed ID
already has the prefix), so copy verbatim from there rather than
guessing. Verified-working set on the current
workbuddy2api:7863 relay: cn:deepseek-v4.1-flash,
cn:glm-5.3-flash, cn:kimi-k2-thinking, cn:minimax-m3,
cn:glm-4.6. The relay's /v1/models returns 47+ IDs but only a
handful of those are confirmed working for actual chat — hand-verify
before listing.

hermes -z is a session override, not a config persist

A subtle trap: hermes -z "..." --model X --provider Y proves the
runtime can reach X via Y for this one invocation, but it does NOT
prove model.provider / model.default in config.yaml was changed.
The CLI override sits in the agent's runtime state, not on disk; a
container restart or new session reverts to whatever's in config.yaml.
This produces the failure shape "agent said the model is X, user
reboots the container, the model is back to whatever config.yaml
says, agent gets re-corrected."

Rule: when claiming "model switched to X", verify on disk:

grep -nE "^  (provider|default|base_url|key_env|providers|fallback_providers):" \
  ~/.hermes/config.yaml

The --usage-file flag in hermes -z shows the runtime model for
that one call — useful, but easy to mistake for proof of persistence.
It isn't.

The same shape applies to hermes -m (one-shot model override) and
--provider flags: runtime-only, not persistent. Use hermes config set (followed by the read-back above) to actually change defaults.

hermes -z startup scans every provider's models: list — slow relays hang the whole CLI

hermes -z loads every configured provider, hits each /v1/models to enumerate, and waits up to its per-call timeout before continuing. If a relay (codebuddy2api, Buddy-2API-Go, custom Ollama-via-LAN) returns slowly — even a 5-second response is enough — every hermes -z "..." call will block on that provider's scan, even when the prompt has nothing to do with it. Symptoms: hermes -z exceeds 30 seconds and emits nothing; the gateway log shows nothing because the CLI is a separate process. Two practical rules: (1) keep models: lists small and hand-verified (3-5 entries) rather than discover_models: true with 30+ — the relay's /v1/models is a static catalog that often disagrees with what the relay actually serves at /v1/chat/completions; (2) when a relay is in maintenance or its model list is 401-rotating, comment out the providers: entry rather than letting hermes -z time out. The fix is not "wait longer" — it's "remove the slow provider from config.yaml and retry."

"自我修复" / "你自己改" / "你直接修好" is NOT a license to invent new config

A recurring failure mode this user has flagged repeatedly: the agent hears a self-repair directive and immediately starts adding things — new providers: entries, new fallback_providers: chains, new plugins under $HERMES_HOME/plugins/, new env vars. Self-repair means "pick the highest-confidence fix and apply it" — it does NOT mean "be creative." When the user says "微信端自己检查" / "自我修复" after a breakage, the right move is: (1) re-read the existing skill for the failing subsystem to find a known fix; (2) revert any unverified recent changes; (3) only if no known fix exists, ask. Adding a new provider/plugin as a "self-repair" to a "model not working" complaint almost never fixes the actual issue and frequently cascades — new models: lists slow the next hermes -z, new env vars conflict with old ones, and the user ends up with config drift that the next session has to untangle. The hermes-gateway-platform-diagnostics skill states the principle for one platform; the broader version for hermes config in general is: self-repair starts with reading the existing skill + reverting unverified changes, not with new structures.

"Stop guessing" — when to present options vs. when to act

This user has corrected this pattern repeatedly across sessions ("你又瞎猜啊" / "你怎么没听懂" / "等你说"). Two failure shapes:

1. The agent picks a model/plugin/provider name and runs with it. The user says "加一个模型" — the agent guesses cn:glm-5.3-flash from memory of which models returned 200 in past probes, without asking. If the user actually wanted cn:kimi-k3-1, the agent has wasted a turn and may have to undo the wrong add. The correction is always "我说的不是这个."
2. The agent decides a config intent without confirming. "切到 X 吧" → agent edits model.default, restarts gateway, and the user wanted a runtime /model switch instead, not a persistent change. Or: "清理一下无效的" → agent deletes 4 of 5 models and the user wanted to keep 3 of them.

Decision rule. When the user's request has more than one plausible interpretation AND no explicit "自己改 / self-repair" directive AND no clearly-destructive default to revert to, stop and ask. One short question with 2-3 options is faster than guessing wrong and having to undo. The signal "等你说" or "回我" or any equivalent from the user means the user has noticed the guess-loop and is asking the agent to break it. Honor the request immediately — do not produce a 4th guess to "be helpful." The user's explicit options-question form (clarify tool) is the right shape; the free-text "wait, what model do you want?" is fine too.

Anti-anti-pattern. Do not turn this into total paralysis. When the user has given an unambiguous identifier ("用 cn:kimi-k3-1" / "切到 MiniMax-M3" / "删 nous"), act. The rule is about resolving ambiguity with a one-question check, not about questioning every instruction. The user's frustration with "瞎猜" is specifically about inventing identifiers the user never named; it is NOT about acting on identifiers the user did name.

hermes config set does not always persist — always read back to verify

A successful hermes config set KEY VALUE exit code is not proof the
change landed in ~/.hermes/config.yaml. The CLI can return 0 while the
write was lost, especially when the gateway is under load (memory
pressure, IO contention, slow disk). Verified case: after a multi-step
config-change sequence intended to switch the default provider from
deepseek → minimax-cn, the CLI reported success on every step, but
the on-disk config.yaml still showed provider: deepseek. The
session-time model override (visible in the system prompt) was
disconnected from the persisted config — when the container restarted,
the "switch" silently reverted.

Rule: after any hermes config set (or hermes config unset),
read the file back before trusting the change:

grep -nE "^  (provider|default|base_url|key_env|fallback_providers):" \
  ~/.hermes/config.yaml

hermes config unset <key> works for whole top-level keys (providers,
fallback_providers, custom_providers, telemetry, etc.) and is the
cleanest way to remove a section. The earlier "unset is less reliable"
framing was overcautious; verified working for top-level maps and lists.
hermes config set KEY VAL is the right alternative for nested
keys where unset does not match (e.g. model.default: "" to clear a
scalar).

hermes config set list-shape differs by key — comma string for some, JSON literal for others. Two list-shaped keys with two different value-parsing rules, neither documented in --help. Empirical behavior:

Key	Accepted value form	Example
providers.<slug>.models	JSON list literal (comma-string parses as a single 1-element list, breaking the picker)	hermes config set providers.buddy2api.models '["cn:deepseek-v4.1-flash","cn:glm-5.3-flash","cn:kimi-k3-1"]'
fallback_providers	JSON literal (bare string rejected)	hermes config set fallback_providers '[{"provider":"buddy2api","model":"cn:deepseek-v4.1-flash"}]'
custom_providers (legacy)	Same as fallback_providers — JSON literal	'[{...}]'

If you feed fallback_providers a bare string (buddy2api instead of [{"provider":"buddy2api",...}]), the CLI returns a clean error: ✗ Cannot set 'fallback_providers': it must be a list, got a string — nothing was written. For providers.<slug>.models the failure mode is silent: a comma-separated string like "a,b,c" is accepted, written, and the picker then shows the slug as buddy2api (1) (one element, the whole string) instead of (3). The CLI returns exit 0 — no error, no warning, no signal. The only fix is to re-set with a JSON list literal. The asymmetry between the two list keys is the confusing part: there's no consistent "list of dicts" vs "list of strings" rule, just two CLI parsers that happen to be implemented differently. The shape for fallback_providers and providers.<slug>.models is the harder one to discover; remember it.

hermes config set splits on DOTS — any key segment containing a . (which every model id in this fleet does: deepseek-v4.1-flash, minimax-m3.1-flash, glm-5.3) silently nests into the wrong shape. Setting model_overrides.workbuddy-hub.deepseek-v4.1-flash.context_window 1000000 writes:

model_overrides:
  workbuddy-hub:
    deepseek-v4:
      1-flash:
        context_window: 1000000

Exit 0, correct-looking confirmation line, completely dead config — the resolver looks up the literal id deepseek-v4.1-flash and finds nothing. Rule: never address a model id through the dotted path. Write the parent map as one JSON object value instead, and unset the mangled intermediate key first:

source /opt/hermes/.venv/bin/activate   # bare `hermes` is not on PATH in this container
hermes config unset 'model_overrides.workbuddy-hub.deepseek-v4'
hermes config set model_overrides.workbuddy-hub '{"deepseek-v4.1-flash": {"context_window": 1000000}}'

The JSON-object form is accepted for map-shaped keys generally (same parser family that takes JSON literals for the list keys above), so it is the safe default whenever a key segment contains a dot.

model_overrides: — the canonical per-model context/capability patch

model_overrides: { <provider>: { <model_id>: {...} } } in config.yaml is the one place to correct what Hermes believes about a model. Canonical key space (the only fields consumers read): context_window, supports_tools, supports_vision, supports_reasoning, model_family. An explicit <provider>.<model_id> entry always wins over the models.dev catalog; <provider>._default and top-level _default are fill-gap only (they apply to models the catalog does NOT know and never displace catalog data). Provider keys accept either the Hermes id or the models.dev id.

Use it when the user hands over a client-side spec (contextWindow, maxTokens, reasoningEfforts) for a relay model and asks for the numbers to match on the Hermes side — the relay's /v1/models is the authority for what to put in (context_length, max_output_tokens, reasoning_efforts).

Verify the resolution actually happens, not just the file write:

cd /opt/hermes && source .venv/bin/activate && python3 -c "
from hermes_cli.config import load_config_readonly
from agent import models_dev as M
print(M._explicit_model_override('workbuddy-hub','deepseek-v4.1-flash', config=load_config_readonly()))
print(M._override_context_window('workbuddy-hub','deepseek-v4.1-flash'))"
# expect {'context_window': 1000000} then 1000000

_override_context_window(provider, model) takes no config= kwarg (it reads the live config itself); _explicit_model_override does. Both live in agent/models_dev.py; the config loader is hermes_cli.config.load_config_readonly, not anything in hermes_yaml.py.

State-change caveat to tell the user: overrides are read at process start, so the running session keeps the old window — it applies to new sessions / after a gateway restart.

Give every relay model that lands in model.default or fallback_providers: a context_window entry in the same pass. Unknown relay models fall back to a 200k default, and compression thresholds are computed from that number — so a 1M-context model silently compresses at 200k until the override exists. Put the whole provider map in one JSON set (hermes config set model_overrides.<provider> '{...}') rather than one dotted path per model, which mangles ids containing dots.

"Clean up the /model picker" — model_catalog.excluded_providers is the right tool

The /model picker shows entries from THREE sources mixed together: the active primary (from model.default), the configured providers: (each slug with its models: list), and the static model catalog (every provider that ships with hermes — Nous Portal with 68 models, DeepSeek with 2, etc.). Catalog entries appear even when the user has no credential and no active auth for that provider — hermes auth status nous returning logged out does NOT remove the entry from the picker. The user sees a long list of phantom providers that look selectable but immediately 401 when picked. When the user says "面板太乱" / "清理无效的" / "only show what works" / "hide X", the canonical fix is model_catalog.excluded_providers.

Format: JSON list of provider slugs (the _<LABEL_OVERRIDES> key in hermes_cli/providers.py, NOT the display label). Common mappings: nous → "Nous Portal", deepseek → "DeepSeek", openai-codex → "ChatGPT or Codex Subscription", <custom> → whatever slug you put in providers:.

# Hide three catalog providers the user never logs into
hermes config set model_catalog.excluded_providers '["nous", "deepseek", "buddy2api"]'

# Verify (the file shape is a real YAML list, not a comma-string)
hermes config get model_catalog.excluded_providers
# Expected:
#   - nous
#   - deepseek
#   - buddy2api

Does NOT affect fallback_providers chain. fallback_config.py does not read excluded_providers — a provider can be hidden from the picker AND still serve as auto-failover when the primary fails. This is the right shape when the user says "I want to use X as a backup but not see it in the picker." If the user wants the opposite (manual select, no failover), put it in providers: without a fallback_providers entry.

Does NOT affect hermes auth status or credential resolution. Excluded providers still authenticate if their keys exist; only the picker hides them. If the goal is "I don't have a key for X and don't want to see it", exclude it. If the goal is "I have a key but I want to force-disable the provider", the right move is hermes auth remove <slug> instead — exclusion is for picker noise, auth removal is for the credential itself.

excluded_providers does NOT auto-invalidate the picker cache — the entry still shows until you clear provider_models_cache.json. When the user reports "I added it to excluded_providers but the picker still shows Nous Portal / Buddy 2API", the diagnosis is cache, not config. The picker reads from /opt/data/provider_models_cache.json (and /opt/data/models_dev_cache.json), a per-provider snapshot of /v1/models calls. Editing excluded_providers updates config.yaml but the picker re-renders from the cached file until the gateway does a forced refresh — and the refresh is lazy (only on model-list endpoint hit, not on every /model invocation). The fix in one step:

rm -f /opt/data/provider_models_cache.json /opt/data/models_dev_cache.json /opt/data/models_dev_cache.etag
# Then the user reopens /model (or sends a message) and the gateway re-probes.

After the cache is gone, the next hermes model invocation (interactive only — hermes model --refresh from subprocess errors with "requires an interactive terminal") rebuilds the file. Take a backup first if config drift matters: cp /opt/data/provider_models_cache.json /opt/data/provider_models_cache.json.bak. The cache file is rebuilt lazily, not eagerly — there's no hermes command to force a rebuild from a non-TTY subprocess; the rebuild happens on the next gateway-driven /v1/models probe. The same cache holds models_dev info per provider (Nous Portal 68, DeepSeek 2, etc.) — that count is what shows in the picker, NOT the real /v1/models from the relay. A user wondering "why does the picker show 47 for Buddy 2API when /v1/models only returns 5" is seeing the stale cache, not a misconfig.

excluded_providers only filters catalog auto-discovered entries, NOT user-configured providers.<slug>: entries. The /model picker merges three sources: catalog auto-load (Nous, DeepSeek, etc. from models_dev), user-configured providers.<slug>: (each slug with its models: list), and the active model.default (with the ✓ mark). excluded_providers removes the first source only. A user with providers.buddy2api: configured will STILL see buddy2api (5) in the picker even if buddy2api is in excluded_providers — the entry is sourced from the user's own config block, not the catalog. Verified shape: hidden catalog entry + visible user-configured entry for the same slug is the normal result, not a bug. To fully remove a custom provider from the picker, you must delete the providers.<slug>: block from config.yaml (and clear the cache as above). excluded_providers is for catalog noise reduction, not for "I don't want to see this provider I configured" — those are orthogonal concerns.

Sessions inherit gateway_runtime from their parent at reset — a reset of an old session will silently bring back dead base_url values. Verified failure shape: a session 20261018_153733_xxx was reset from a parent 20260926_042244_xxx that had base_url: http://192.168.31.201:10082/v1 (a port that died weeks earlier). The reset session inherited the dead URL; the user's TG message went to https://api.minimaxi.com/... (config default) → failed → triggered fallback to the inherited 10082 base_url → 502. The session is not "stale on disk" in the obvious sense — the user sees model: MiniMax-M3 provider: minimax-cn in state.db for the new session, but model_config.gateway_runtime.base_url is the OLD 10082 URL. Two correction paths:

1. Quick surgical fix (single session, single column): direct UPDATE sessions SET model_config=... WHERE id=... against /opt/data/state.db. Read with SELECT id, model, model_config FROM sessions WHERE last_activity_at > <day-ago> AND source IN ('telegram','weixin') ORDER BY last_activity_at DESC to find the active session row, then write a clean model_config JSON. Confirm the change with a re-SELECT.
2. Bulk archive of dead-base_url sessions: UPDATE sessions SET archived=1 WHERE model_config LIKE '%10082%' AND archived=0 (or any other dead port substring) — preserves the rows for inspection but removes them from active routing.

To verify before/after: SELECT id, json_extract(model_config, '$.gateway_runtime.base_url') AS base_url FROM sessions WHERE source IN ('telegram','weixin') AND archived=0. If a "current" session shows a base_url pointing at a port that ss -tlnp says isn't listening, that's the bug. The request_dump_*.json files in /opt/data/sessions/ are the time-correlated evidence — they contain the URL of the actual outbound request, which lets you connect "user reported 502 at HH:MM" to the exact session row that produced it.

Pre-cleanup verification before excluding (the "检测无效就删" workflow the user keeps asking for): don't exclude a provider just because it looks unused in the picker. Walk these three checks:

1. hermes auth status <slug> — logged in means the user is actively using it; do not exclude.
2. grep -E "<slug>:" /opt/data/config.yaml — if providers.<slug>: or fallback_providers: references it, exclusion hides the picker entry but keeps the chain live (verify intent).
3. hermes config get providers.<slug>.models — if the list is non-empty, the user is using those models; the picker needs to show them.

The "exclude" action is reversible: hermes config unset model_catalog.excluded_providers clears the whole list. Take a backup of the file first if the user is sensitive to config drift: cp /opt/data/config.yaml /opt/data/config.yaml.bak-$(date +%Y%m%dT%H%M%SZ).

/opt/data/.env values that show as *** in cat are display-masked placeholders, NOT the real key — applies to every provider, not just minimax. A fresh agent reading cat /opt/data/.env sees MINIMAX_CN_API_KEY=*** and CODEBUDDY2API_KEY=*** and concludes "the secrets are redacted." They're not — those are static display masks written to the file as the value itself. This pattern holds for any *_API_KEY=*** line in .env regardless of provider; do not assume a specific key is "set" or "missing" from the cat output alone. The real secret is fetched at runtime by the gateway's token provider (auth.json for OAuth, or the source config like the workbuddy2api config.json on the host). env from a child shell ALSO shows *** because the gateway doesn't inject the real key into the child env. Two ways to find the real key:

1. For WorkBuddy/CodeBuddy-class relays: read the source of truth on the host (e.g. ssh fnos 'cat /vol1/1000/Docker/workbuddy2api/workbuddy2api-source/config.json' | python3 -c "import json,sys;print(json.load(sys.stdin)['api_key'])").
2. For OAuth-backed providers (minimax-cn, minimax-oauth, etc.): read from the running gateway process: cat /proc/$(pgrep -f "hermes gateway run" | head -1)/environ | tr '\0' '\n' | grep MINIMAX_CN_API_KEY.

The first wrong inference is "the user already set the key, I'll trust the .env value." The second is "the gateway injects the real key into my env, so I can probe it." Both are wrong; the real key is in either the host's relay config or the gateway's live process env, not in the on-disk .env.

hermes config edit opens $EDITOR (requires TTY) for multi-line edits
the CLI can't express.

write_file / patch are NOT fallback paths — they refuse to touch
/opt/data/config.yaml with a hard "Refusing to write to Hermes config
file" error. The tool wrapper treats the Hermes config as
security-sensitive regardless of file permissions; the only escape is
the hermes config CLI. If the file genuinely drifts from what the CLI
shows, restart the gateway and re-read — do not bypass via the editor.

Don't claim a config change is done based on the CLI's exit code
alone. The same rule applies to hermes auth logout <provider> —
verify with hermes auth list that the provider is actually gone.

Toolsets

Enable/disable via hermes tools (interactive) or hermes tools enable/disable NAME.

Toolset	What it provides
web	Web search and content extraction
search	Web search only (subset of web)
browser	Browser automation (Browserbase, Camofox, or local Chromium)
terminal	Shell commands and process management
file	File read/write/search/patch
code_execution	Sandboxed Python execution
vision	Image analysis
image_gen	AI image generation
video	Video analysis and generation
tts	Text-to-speech
skills	Skill browsing and management
memory	Persistent cross-session memory
session_search	Search past conversations
delegation	Subagent task delegation
cronjob	Scheduled task management
clarify	Ask user clarifying questions
messaging	Cross-platform message sending
todo	In-session task planning and tracking
kanban	Multi-agent work-queue tools (gated to workers)
debugging	Extra introspection/debug tools (off by default)
safe	Minimal, low-risk toolset for locked-down sessions
spotify	Spotify playback and playlist control
homeassistant	Smart home control (off by default)
discord	Discord integration tools
discord_admin	Discord admin/moderation tools
feishu_doc	Feishu (Lark) document tools
feishu_drive	Feishu (Lark) drive tools
yuanbao	Yuanbao integration tools
rl	Reinforcement learning tools (off by default)
moa	Mixture of Agents (off by default)

Full enumeration lives in toolsets.py as the TOOLSETS dict; _HERMES_CORE_TOOLS is the default bundle most platforms inherit from.

Tool changes take effect on /reset (new session). They do NOT apply mid-conversation to preserve prompt caching.

Memory IS shared across platforms; session history is NOT (don't conflate)

A frequent point of confusion that produces wrong answers to the user:
"you can't see what I said on the other platform" is half true. The
two persistence layers are independent:

Layer	File	Cross-platform?	Visible when?
Memory	/opt/data/memories/MEMORY.md, /opt/data/memories/USER.md (or Honcho/Mem0 backend)	Yes — loaded into every session's system prompt regardless of platform	Every session, every platform
Session history	state.db (sessions + messages tables, key session_id)	No — each platform's session rows are isolated	Only inside that session's own session_search calls

Concrete shape: a TG chat and a WeChat chat each have a distinct
session_id in state.db, but both load the same MEMORY.md at boot.
If the user says "I told you yesterday on WeChat that 满满 has
urticaria" and the memory doesn't have that, the correct answer is
"memory has no record" — not "I can't see WeChat sessions" (which is
also true, but irrelevant). The memory layer is the shared one, so
it's where cross-platform context lives.

Memory tool buffer vs. file: the memory tool's add action
returns an "exceeds limit" error when the in-memory budget is full
(typical cap ~1,375 chars in the default profile), but the on-disk
file is editable directly with write_file or patch regardless of
the tool's buffer state. When the tool refuses, fall back to direct
file write — don't conclude "memory is permanently full." When the
file is actually large, prune stale entries first (single batch
operation in memory action=patch with old_text matches) so the
next add fits.

The memory file is git-tracked — not "stuck in tool context." On
this user's install, /opt/data/memories/MEMORY.md and USER.md are
a real git repository rooted at /opt/data/ (not /opt/data/memories/),
remote Russ534/Myopenclawbot on GitHub. Three implications:

1. When memory tool refuses to add, the on-disk file may still
   be small and writable. Run wc -c /opt/data/memories/MEMORY.md
   and git -C /opt/data log --oneline -5 to see the real state.
   Don't conclude "memory is permanently full" from a tool-buffer
   error alone.
2. The cross-platform sync happens through git, not through the
   memory tool. After distilling, the workflow is
   write_file/patch → git add → git commit → git push origin main
   (force-with-lease if needed). The memory tool's add never touches
   the remote — only disk writes that get committed do.
3. A failed git push is the failure to investigate, not "memory
   is broken." Common cause: rebase-merge state in
   /opt/data/.git/rebase-merge/ left over from a partial
   git pull --rebase. Diagnose with
   ls /opt/data/.git/rebase-merge/ 2>&1 and
   git -C /opt/data status. If the working tree is dirty with
   untracked skills/ and .gitignore, the rebase will refuse to
   --abort because reset would clobber them — accept the
   wanted-side (typically git add the untracked hermes-installed
   files so they're considered resolved) and continue.

See fnnas-admin/references/hermes-github-backup.md for the full
git-sync architecture, and weekly-review-planning for the mandated
weekly cadence (user explicitly asked: "记得每周存 memory, 跨端聊的
信息 TG 端的我都不知道").

When the user says "I told you on WeChat X", the right move is
grep the memory files first; if absent, ask them to repeat it
(knowing it WILL now persist across platforms once you write it).
Do not invent a fake WeChat history.

---

Security & Privacy Toggles

Common "why is Hermes doing X to my output / tool calls / commands?" toggles — and the exact commands to change them. Most of these need a fresh session (/reset in chat, or start a new hermes invocation) because they're read once at startup.

Secret redaction in tool output

Secret redaction is on by default — tool output (terminal stdout, read_file, web content, subagent summaries, etc.) is scanned for strings that look like API keys, tokens, and secrets before it enters the conversation context and logs. Leave it enabled for normal use:

hermes config set security.redact_secrets true       # keep enabled globally

Restart required. security.redact_secrets is snapshotted at import time — toggling it mid-session (e.g. via export HERMES_REDACT_SECRETS=false from a tool call) will NOT take effect for the running process. Tell the user to change it in config from a terminal, then start a new session. This is deliberate — it prevents an LLM from flipping the toggle on itself mid-task.

Disable only when you deliberately need raw credential-like strings for debugging or redactor development:

hermes config set security.redact_secrets false

Redacted strings in tool output look like real values to Python — len() reports the masked length, not the original. When curl -sS returns {"data":{"token":"eyJhbG...nbf4"}} and the redactor substitutes a 5-15 char placeholder, json.loads(...)["data"]["token"] returns that placeholder — Python sees it as the literal string eyJhbG...nbf4, ~13 chars long. Calling len() on it confirms the wrong length but looks like a real measurement. Consequence: bearer-token auth on the next API call fails with 401, the agent blames the credentials, and wastes 3+ turns before realizing the value it has is a redacted stub. Confirmed 2026-10-19 oci-helper session: token appeared as eyJhbG...W3gE (len=13) on the Hermes side, but the same flow on fnOS-side shell showed len=165 and worked first try. Workaround: never let a credential round-trip through json.loads on the Hermes side. Drive the auth + every API call from a script that runs on the remote host (fnos / target server), so the token only exists in the remote shell's environment variable, never in a string that Hermes's Python kernel prints. Pattern:

ssh fnos-root bash /tmp/run.sh
# /tmp/run.sh on fnos: login, capture token to $TOK, do all subsequent calls,
# never echo $TOK. Use $TOK only as -H "Authorization: Bearer $TOK" header value.

If the credential MUST cross Hermes (e.g., agent-side REST client), treat any string that looks like ***...*** / <token>...</token> / truncated mid-character as definitely redacted and refuse to use it. The detection cue is the literal ... substring inside what should be a base64/hex blob.

PII redaction in gateway messages

Separate from secret redaction. When enabled, the gateway hashes user IDs and strips phone numbers from the session context before it reaches the model:

hermes config set privacy.redact_pii true    # enable
hermes config set privacy.redact_pii false   # disable (default)

Command approval prompts

By default (approvals.mode: manual), Hermes prompts the user before running shell commands flagged as destructive (rm -rf, git reset --hard, etc.). The modes are:

• manual — always prompt (default)
• smart — use an auxiliary LLM to auto-approve low-risk commands, prompt on high-risk
• off — skip all approval prompts (equivalent to --yolo)

hermes config set approvals.mode smart       # recommended middle ground
hermes config set approvals.mode off         # bypass everything (not recommended)

Per-invocation bypass without changing config:

• hermes --yolo …
• export HERMES_YOLO_MODE=1

Note: YOLO / approvals.mode: off does NOT turn off secret redaction. They are independent.

Shell hooks allowlist

Some shell-hook integrations require explicit allowlisting before they fire. Managed via ~/.hermes/shell-hooks-allowlist.json — prompted interactively the first time a hook wants to run.

Disabling the web/browser/image-gen tools

To keep the model away from network or media tools entirely, open hermes tools and toggle per-platform. Takes effect on next session (/reset). See the Tools & Skills section above.

---

Voice & Transcription

STT (Voice → Text)

Voice messages from messaging platforms are auto-transcribed.

Provider priority (auto-detected):

1. Local faster-whisper — free, no API key: pip install faster-whisper
2. Groq Whisper — free tier: set GROQ_API_KEY
3. OpenAI Whisper — paid: set VOICE_TOOLS_OPENAI_KEY
4. Mistral Voxtral — set MISTRAL_API_KEY

Config:

stt:
  enabled: true
  provider: local        # local, groq, openai, mistral
  local:
    model: base          # tiny, base, small, medium, large-v3

TTS (Text → Voice)

Provider	Env var	Free?
Edge TTS	None	Yes (default)
ElevenLabs	ELEVENLABS_API_KEY	Free tier
OpenAI	VOICE_TOOLS_OPENAI_KEY	Paid
MiniMax	MINIMAX_API_KEY	Paid
Mistral (Voxtral)	MISTRAL_API_KEY	Paid
NeuTTS (local)	None (pip install neutts[all] + espeak-ng)	Free

Voice commands: /voice on (voice-to-voice), /voice tts (always voice), /voice off.

---

Spawning Additional Hermes Instances

Run additional Hermes processes as fully independent subprocesses — separate sessions, tools, and environments.

When to Use This vs delegate_task

	delegate_task	Spawning hermes process
Isolation	Separate conversation, shared process	Fully independent process
Duration	Minutes (bounded by parent loop)	Hours/days
Tool access	Subset of parent's tools	Full tool access
Interactive	No	Yes (PTY mode)
Use case	Quick parallel subtasks	Long autonomous missions

One-Shot Mode

terminal(command="hermes chat -q 'Research GRPO papers and write summary to ~/research/grpo.md'", timeout=300)

# Background for long tasks:
terminal(command="hermes chat -q 'Set up CI/CD for ~/myapp'", background=true)

Interactive PTY Mode (via tmux)

Hermes uses prompt_toolkit, which requires a real terminal. Use tmux for interactive spawning:

# Start
terminal(command="tmux new-session -d -s agent1 -x 120 -y 40 'hermes'", timeout=10)

# Wait for startup, then send a message
terminal(command="sleep 8 && tmux send-keys -t agent1 'Build a FastAPI auth service' Enter", timeout=15)

# Read output
terminal(command="sleep 20 && tmux capture-pane -t agent1 -p", timeout=5)

# Send follow-up
terminal(command="tmux send-keys -t agent1 'Add rate limiting middleware' Enter", timeout=5)

# Exit
terminal(command="tmux send-keys -t agent1 '/exit' Enter && sleep 2 && tmux kill-session -t agent1", timeout=10)

Multi-Agent Coordination

# Agent A: backend
terminal(command="tmux new-session -d -s backend -x 120 -y 40 'hermes -w'", timeout=10)
terminal(command="sleep 8 && tmux send-keys -t backend 'Build REST API for user management' Enter", timeout=15)

# Agent B: frontend
terminal(command="tmux new-session -d -s frontend -x 120 -y 40 'hermes -w'", timeout=10)
terminal(command="sleep 8 && tmux send-keys -t frontend 'Build React dashboard for user management' Enter", timeout=15)

# Check progress, relay context between them
terminal(command="tmux capture-pane -t backend -p | tail -30", timeout=5)
terminal(command="tmux send-keys -t frontend 'Here is the API schema from the backend agent: ...' Enter", timeout=5)

Session Resume

# Resume most recent session
terminal(command="tmux new-session -d -s resumed 'hermes --continue'", timeout=10)

# Resume specific session
terminal(command="tmux new-session -d -s resumed 'hermes --resume 20260225_143052_a1b2c3'", timeout=10)

Tips

• Prefer delegate_task for quick subtasks — less overhead than spawning a full process
• Use -w (worktree mode) when spawning agents that edit code — prevents git conflicts
• Set timeouts for one-shot mode — complex tasks can take 5-10 minutes
• Use hermes chat -q for fire-and-forget — no PTY needed
• Use tmux for interactive sessions — raw PTY mode has \r vs \n issues with prompt_toolkit
• For scheduled tasks, use the cronjob tool instead of spawning — handles delivery and retry

---

Durable & Background Systems

Four systems run alongside the main conversation loop. Quick reference
here; full developer notes live in AGENTS.md, user-facing docs under
website/docs/user-guide/features/.

Delegation (delegate_task)

Synchronous subagent spawn — the parent waits for the child's summary
before continuing its own loop. Isolated context + terminal session.

• Single: delegate_task(goal, context, toolsets).
• Batch: delegate_task(tasks=[{goal, ...}, ...]) runs children in
  parallel, capped by delegation.max_concurrent_children (default 3).
• Roles: leaf (default; cannot re-delegate) vs orchestrator
  (can spawn its own workers, bounded by delegation.max_spawn_depth).
• Not durable. If the parent is interrupted, the child is
  cancelled. For work that must outlive the turn, use cronjob or
  terminal(background=True, notify_on_complete=True).

Config: delegation.* in config.yaml.

Cron (scheduled jobs)

Durable scheduler — cron/jobs.py + cron/scheduler.py. Drive it via
the cronjob tool, the hermes cron CLI (list, add, edit,
pause, resume, run, remove), or the /cron slash command.

• Schedules: duration ("30m", "2h"), "every" phrase
  ("every monday 9am"), 5-field cron ("0 9 * * *"), or ISO timestamp.
• Per-job knobs: skills, model/provider override, script
  (pre-run data collection; no_agent=True makes the script the whole
  job), context_from (chain job A's output into job B), workdir
  (run in a specific dir with its AGENTS.md / CLAUDE.md loaded),
  multi-platform delivery.
• Invariants: 3-minute hard interrupt per run, .tick.lock file
  prevents duplicate ticks across processes, cron sessions pass
  skip_memory=True by default, and cron deliveries are framed with a
  header/footer instead of being mirrored into the target gateway
  session (keeps role alternation intact).

User docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/cron

Curator (skill lifecycle)

Background maintenance for agent-created skills. Tracks usage, marks
idle skills stale, archives stale ones, keeps a pre-run tar.gz backup
so nothing is lost.

• CLI: hermes curator <verb> — status, run, pause, resume,
  pin, unpin, archive, restore, prune, backup, rollback.
• Slash: /curator <subcommand> mirrors the CLI.
• Scope: only touches skills with created_by: "agent" provenance.
  Bundled + hub-installed skills are off-limits. Never deletes —
  max destructive action is archive. Pinned skills are exempt from
  every auto-transition and every LLM review pass.
• Telemetry: sidecar at ~/.hermes/skills/.usage.json holds
  per-skill use_count, view_count, patch_count,
  last_activity_at, state, pinned.

Config: curator.* (enabled, interval_hours, min_idle_hours,
stale_after_days, archive_after_days, backup.*).
User docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/curator

Kanban (multi-agent work queue)

Durable SQLite board for multi-profile / multi-worker collaboration.
Users drive it via hermes kanban <verb>; dispatcher-spawned workers
see a focused kanban_* toolset gated by HERMES_KANBAN_TASK, and
orchestrator profiles can opt into the broader kanban toolset. Normal
sessions still have zero kanban_* schema footprint unless configured.

• CLI verbs (common): init, create, list (alias ls),
  show, assign, link, unlink, comment, complete, block,
  unblock, archive, tail. Less common: watch, stats, runs,
  log, dispatch, daemon, gc.
• Worker/orchestrator toolset: kanban_show, kanban_complete,
  kanban_block, kanban_heartbeat, kanban_comment, kanban_create,
  kanban_link; profiles that explicitly enable the kanban toolset
  outside a dispatcher-spawned task also get kanban_list and
  kanban_unblock for board routing.
• Dispatcher runs inside the gateway by default
  (kanban.dispatch_in_gateway: true) — reclaims stale claims,
  promotes ready tasks, atomically claims, spawns assigned profiles.
  Auto-blocks a task after failure_limit consecutive spawn failures
  (default 2; configurable via kanban.failure_limit or per-task
  max_retries).
• Isolation: board is the hard boundary (workers get
  HERMES_KANBAN_BOARD pinned in env); tenant is a soft namespace
  within a board for workspace-path + memory-key isolation.

User docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban

---

Windows-Specific Quirks

Hermes runs natively on Windows (PowerShell, cmd, Windows Terminal, git-bash
mintty, VS Code integrated terminal). Most of it just works, but a handful
of differences between Win32 and POSIX have bitten us — document new ones
here as you hit them so the next person (or the next session) doesn't
rediscover them from scratch.

Input / Keybindings

Alt+Enter doesn't insert a newline. Windows Terminal intercepts Alt+Enter
at the terminal layer to toggle fullscreen — the keystroke never reaches
prompt_toolkit. Use Ctrl+Enter instead. Windows Terminal delivers
Ctrl+Enter as LF (c-j), distinct from plain Enter (c-m / CR), and the
CLI binds c-j to newline insertion on win32 only (see
_bind_prompt_submit_keys + the Windows-only c-j binding in cli.py).
Side effect: the raw Ctrl+J keystroke also inserts a newline on Windows —
unavoidable, because Windows Terminal collapses Ctrl+Enter and Ctrl+J to
the same keycode at the Win32 console API layer. No conflicting binding
existed for Ctrl+J on Windows, so this is a harmless side effect.

mintty / git-bash behaves the same (fullscreen on Alt+Enter) unless you
disable Alt+Fn shortcuts in Options → Keys. Easier to just use Ctrl+Enter.

Diagnosing keybindings. Run python scripts/keystroke_diagnostic.py
(repo root) to see exactly how prompt_toolkit identifies each keystroke
in the current terminal. Answers questions like "does Shift+Enter come
through as a distinct key?" (almost never — most terminals collapse it
to plain Enter) or "what byte sequence is my terminal sending for
Ctrl+Enter?" This is how the Ctrl+Enter = c-j fact was established.

Config / Files

HTTP 400 "No models provided" on first run. config.yaml was saved
with a UTF-8 BOM (common when Windows apps write it). Re-save as UTF-8
without BOM. hermes config edit writes without BOM; manual edits in
Notepad are the usual culprit.

execute_code / Sandbox

WinError 10106 ("The requested service provider could not be loaded
or initialized") from the sandbox child process — it can't create an
AF_INET socket, so the loopback-TCP RPC fallback fails before
connect(). Root cause is usually not a broken Winsock LSP; it's
Hermes's own env scrubber dropping SYSTEMROOT / WINDIR / COMSPEC
from the child env. Python's socket module needs SYSTEMROOT to locate
mswsock.dll. Fixed via the _WINDOWS_ESSENTIAL_ENV_VARS allowlist in
tools/code_execution_tool.py. If you still hit it, echo os.environ
inside an execute_code block to confirm SYSTEMROOT is set. Full
diagnostic recipe in references/execute-code-sandbox-env-windows.md.

Testing / Contributing

scripts/run_tests.sh doesn't work as-is on Windows — it looks for
POSIX venv layouts (.venv/bin/activate). The Hermes-installed venv at
venv/Scripts/ has no pip or pytest either (stripped for install size).
Workaround: install pytest + pytest-xdist + pyyaml into a system Python
3.11 user site, then invoke pytest directly with PYTHONPATH set:

"/c/Program Files/Python311/python" -m pip install --user pytest pytest-xdist pyyaml
export PYTHONPATH="$(pwd)"
"/c/Program Files/Python311/python" -m pytest tests/foo/test_bar.py -v --tb=short -n 0

Use -n 0, not -n 4 — pyproject.toml's default addopts already
includes -n, and the wrapper's CI-parity guarantees don't apply off POSIX.

POSIX-only tests need skip guards. Common markers already in the codebase:

• Symlinks — elevated privileges on Windows
• 0o600 file modes — POSIX mode bits not enforced on NTFS by default
• signal.SIGALRM — Unix-only (see tests/conftest.py::_enforce_test_timeout)
• Winsock / Windows-specific regressions — @pytest.mark.skipif(sys.platform != "win32", ...)

Use the existing skip-pattern style (sys.platform == "win32" or
sys.platform.startswith("win")) to stay consistent with the rest of the
suite.

Path / Filesystem

Line endings. Git may warn LF will be replaced by CRLF the next time Git touches it. Cosmetic — the repo's .gitattributes normalizes. Don't
let editors auto-convert committed POSIX-newline files to CRLF.

Forward slashes work almost everywhere. C:/Users/... is accepted by
every Hermes tool and most Windows APIs. Prefer forward slashes in code
and logs — avoids shell-escaping backslashes in bash.

---

Troubleshooting

Voice not working

1. Check stt.enabled: true in config.yaml
2. Verify provider: pip install faster-whisper or set API key
3. In gateway: /restart. In CLI: exit and relaunch.

Tool not available

1. hermes tools — check if toolset is enabled for your platform
2. Some tools need env vars (check .env)
3. /reset after enabling tools

Model/provider issues

1. hermes doctor — check config and dependencies
2. hermes auth — re-authenticate OAuth providers (or hermes auth add <provider>)
3. Check .env has the right API key
4. Copilot 403: gh auth login tokens do NOT work for Copilot API. You must use the Copilot-specific OAuth device code flow via hermes model → GitHub Copilot.

Changes not taking effect

• Tools/skills: /reset starts a new session with updated toolset
• Config changes: In gateway: /restart. In CLI: exit and relaunch.
• Code changes: Restart the CLI or gateway process

Skills not showing

1. hermes skills list — verify installed
2. hermes skills config — check platform enablement
3. Load explicitly: /skill name or hermes -s name

Gateway issues

Check logs first:

grep -i "failed to send\|error" ~/.hermes/logs/gateway.log | tail -20

Common gateway problems:

• Gateway dies on SSH logout: Enable linger: sudo loginctl enable-linger $USER
• Gateway dies on WSL2 close: WSL2 requires systemd=true in /etc/wsl.conf for systemd services to work. Without it, gateway falls back to nohup (dies when session closes).
• Gateway crash loop: Reset the failed state: systemctl --user reset-failed hermes-gateway

Platform-specific issues

• Discord bot silent: Must enable Message Content Intent in Bot → Privileged Gateway Intents.
• Slack bot only works in DMs: Must subscribe to message.channels event. Without it, the bot ignores public channels.
• Windows-specific issues (Alt+Enter newline, WinError 10106, UTF-8 BOM config, test suite, line endings): see the dedicated Windows-Specific Quirks section above.

Auxiliary models not working

If auxiliary tasks (vision, compression, session_search) fail silently, the auto provider can't find a backend. Either set OPENROUTER_API_KEY or GOOGLE_API_KEY, or explicitly configure each auxiliary task's provider:

hermes config set auxiliary.vision.provider <your_provider>
hermes config set auxiliary.vision.model <model_name>

---

Where to Find Things

Looking for...	Location
Config options	hermes config edit or Configuration docs
Available tools	hermes tools list or Tools reference
Slash commands	/help in session or Slash commands reference
Skills catalog	hermes skills browse or Skills catalog
Provider setup	hermes model or Providers guide
Platform setup	hermes gateway setup or Messaging docs
MCP servers	hermes mcp list or MCP guide
Profiles	hermes profile list or Profiles docs
Cron jobs	hermes cron list or Cron docs
Memory	hermes memory status or Memory docs
Env variables	hermes config env-path or Env vars reference
CLI commands	hermes --help or CLI reference
Gateway logs	~/.hermes/logs/gateway.log
Session files	hermes sessions browse (reads state.db)
Source code	~/.hermes/hermes-agent/

---

Contributor Quick Reference

For occasional contributors and PR authors. Full developer docs: https://hermes-agent.nousresearch.com/docs/developer-guide/

Project Layout

hermes-agent/
├── run_agent.py          # AIAgent — core conversation loop
├── model_tools.py        # Tool discovery and dispatch
├── toolsets.py           # Toolset definitions
├── cli.py                # Interactive CLI (HermesCLI)
├── hermes_state.py       # SQLite session store
├── agent/                # Prompt builder, context compression, memory, model routing, credential pooling, skill dispatch
├── hermes_cli/           # CLI subcommands, config, setup, commands
│   ├── commands.py       # Slash command registry (CommandDef)
│   ├── config.py         # DEFAULT_CONFIG, env var definitions
│   └── main.py           # CLI entry point and argparse
├── tools/                # One file per tool
│   └── registry.py       # Central tool registry
├── gateway/              # Messaging gateway
│   └── platforms/        # Platform adapters (telegram, discord, etc.)
├── cron/                 # Job scheduler
├── tests/                # ~3000 pytest tests
└── website/              # Docusaurus docs site

Config: ~/.hermes/config.yaml (settings), ~/.hermes/.env (API keys).

Adding a Tool (3 files)

1. Create tools/your_tool.py:

import json, os
from tools.registry import registry

def check_requirements() -> bool:
    return bool(os.getenv("EXAMPLE_API_KEY"))

def example_tool(param: str, task_id: str = None) -> str:
    return json.dumps({"success": True, "data": "..."})

registry.register(
    name="example_tool",
    toolset="example",
    schema={"name": "example_tool", "description": "...", "parameters": {...}},
    handler=lambda args, **kw: example_tool(
        param=args.get("param", ""), task_id=kw.get("task_id")),
    check_fn=check_requirements,
    requires_env=["EXAMPLE_API_KEY"],
)

2. Add to toolsets.py → _HERMES_CORE_TOOLS list.

Auto-discovery: any tools/*.py file with a top-level registry.register() call is imported automatically — no manual list needed.

All handlers must return JSON strings. Use get_hermes_home() for paths, never hardcode ~/.hermes.

Adding a Slash Command

1. Add CommandDef to COMMAND_REGISTRY in hermes_cli/commands.py
2. Add handler in cli.py → process_command()
3. (Optional) Add gateway handler in gateway/run.py

All consumers (help text, autocomplete, Telegram menu, Slack mapping) derive from the central registry automatically.

Agent Loop (High Level)

run_conversation():
  1. Build system prompt
  2. Loop while iterations < max:
     a. Call LLM (OpenAI-format messages + tool schemas)
     b. If tool_calls → dispatch each via handle_function_call() → append results → continue
     c. If text response → return
  3. Context compression triggers automatically near token limit

Testing

python -m pytest tests/ -o 'addopts=' -q   # Full suite
python -m pytest tests/tools/ -q            # Specific area

• Tests auto-redirect HERMES_HOME to temp dirs — never touch real ~/.hermes/
• Run full suite before pushing any change
• Use -o 'addopts=' to clear any baked-in pytest flags

Windows contributors: scripts/run_tests.sh currently looks for POSIX venvs (.venv/bin/activate / venv/bin/activate) and will error out on Windows where the layout is venv/Scripts/activate + python.exe. The Hermes-installed venv at venv/Scripts/ also has no pip or pytest — it's stripped for end-user install size. Workaround: install pytest + pytest-xdist + pyyaml into a system Python 3.11 user site (/c/Program Files/Python311/python -m pip install --user pytest pytest-xdist pyyaml), then run tests directly:

export PYTHONPATH="$(pwd)"
"/c/Program Files/Python311/python" -m pytest tests/tools/test_foo.py -v --tb=short -n 0

Use -n 0 (not -n 4) because pyproject.toml's default addopts already includes -n, and the wrapper's CI-parity story doesn't apply off-POSIX.

Cross-platform test guards: tests that use POSIX-only syscalls need a skip marker. Common ones already in the codebase:

• Symlink creation → @pytest.mark.skipif(sys.platform == "win32", reason="Symlinks require elevated privileges on Windows") (see tests/cron/test_cron_script.py)
• POSIX file modes (0o600, etc.) → @pytest.mark.skipif(sys.platform.startswith("win"), reason="POSIX mode bits not enforced on Windows") (see tests/hermes_cli/test_auth_toctou_file_modes.py)
• signal.SIGALRM → Unix-only (see tests/conftest.py::_enforce_test_timeout)
• Live Winsock / Windows-specific regression tests → @pytest.mark.skipif(sys.platform != "win32", reason="Windows-specific regression")

Monkeypatching sys.platform is not enough when the code under test also calls platform.system() / platform.release() / platform.mac_ver(). Those functions re-read the real OS independently, so a test that sets sys.platform = "linux" on a Windows runner will still see platform.system() == "Windows" and route through the Windows branch. Patch all three together:

monkeypatch.setattr(sys, "platform", "linux")
monkeypatch.setattr(platform, "system", lambda: "Linux")
monkeypatch.setattr(platform, "release", lambda: "6.8.0-generic")

See tests/agent/test_prompt_builder.py::TestEnvironmentHints for a worked example.

Extending the system prompt's execution-environment block

Factual guidance about the host OS, user home, cwd, terminal backend, and shell (bash vs. PowerShell on Windows) is emitted from agent/prompt_builder.py::build_environment_hints(). This is also where the WSL hint and per-backend probe logic live. The convention:

• Local terminal backend → emit host info (OS, $HOME, cwd) + Windows-specific notes (hostname ≠ username, terminal uses bash not PowerShell).
• Remote terminal backend (anything in _REMOTE_TERMINAL_BACKENDS: docker, singularity, modal, daytona, ssh, managed_modal) → suppress host info entirely and describe only the backend. A live uname/whoami/pwd probe runs inside the backend via tools.environments.get_environment(...).execute(...), cached per process in _BACKEND_PROBE_CACHE, with a static fallback if the probe times out.
• Key fact for prompt authoring: when TERMINAL_ENV != "local", every file tool (read_file, write_file, patch, search_files) runs inside the backend container, not on the host. The system prompt must never describe the host in that case — the agent can't touch it.

Full design notes, the exact emitted strings, and testing pitfalls:
references/prompt-builder-environment-hints.md.

Refactor-safety pattern (POSIX-equivalence guard): when you extract inline logic into a helper that adds Windows/platform-specific behavior, keep a _legacy_<name> oracle function in the test file that's a verbatim copy of the old code, then parametrize-diff against it. Example: tests/tools/test_code_execution_windows_env.py::TestPosixEquivalence. This locks in the invariant that POSIX behavior is bit-for-bit identical and makes any future drift fail loudly with a clear diff.

Commit Conventions

type: concise subject line

Optional body.

Types: fix:, feat:, refactor:, docs:, chore:

Key Rules

• Never break prompt caching — don't change context, tools, or system prompt mid-conversation
• Message role alternation — never two assistant or two user messages in a row
• Use get_hermes_home() from hermes_constants for all paths (profile-safe)
• Config values go in config.yaml, secrets go in .env
• New tools need a check_fn so they only appear when requirements are met