Here’s the full story of what happened, what we discovered, and how we built a watchdog to make sure it never happens again.

What We Were Doing

We had two Kanban dispatcher cron jobs running on Hermes Agent, powered by Ollama Cloud on the Pro plan ($20/mo):

• PMO-Kanban-Dispatcher — every 15 minutes, scanning the project board for tasks
• BB-PM-Kanban-Dispatcher — every 10 minutes, running the Boogie Board project manager

Each dispatcher call sends the full conversation context — system prompt, all prior messages, tool results — to the LLM. That’s roughly 42,000 tokens per API call. With two dispatchers hitting the API every 10-15 minutes, the token burn adds up fast.

We were also doing interactive work — conversations, research, writing — all on the same Ollama Cloud account.

What Happened

At 11:50 AM CT on July 7, the BB-PM dispatcher hit this:

HTTP 429: {"error":"you (My.Ollama.Account.Name) have reached your session usage limit,
upgrade for higher limits: https://ollama.com/upgrade or add extra usage:
https://ollama.com/settings (ref: Reference.ID)"}

The dispatcher retried 3 times with exponential backoff. All failed. Meanwhile, the PMO dispatcher was still burning tokens on the same account, making things worse.

The root cause: Two automated dispatchers sending ~42K tokens per call, every 10-15 minutes, accumulated enough GPU time to exhaust the 5-hour session limit within hours. And there was no way to see it coming.

What We Discovered

Ollama Cloud has no usage API

This was the biggest surprise. There is no /api/account/usage endpoint. You can’t programmatically check your usage. GitHub issues #12532 and #15132 are open requests for this, but for now the only way to check is to log into ollama.com/settings and look at percentages manually.

Ollama sends an email at 90% of your plan limit, but by then you’re already almost out.

Two limits, not one

Ollama Cloud enforces two independent limits:

The session limit is the binding constraint. It resets every 5 hours, but you can blow through it fast with automated agents.

Usage is GPU time, not tokens

This is subtle but critical. Ollama measures usage in GPU time, not token count. Heavier models (Level 4, like deepseek-v4-pro) consume ~4x more GPU time per token than lighter models (Level 1, like gpt-oss:20b). So token count is a proxy, not a direct measure. Our token-based estimates overestimate for light models and underestimate for heavy ones.

The actual numbers

We parsed ~/.hermes/logs/agent.log for lines matching in=N out=N total=N and measured:

Input tokens dominated at 8.6M vs only 62K output. That’s because every API call re-sends the full conversation context.

Calibrating against real data

From the Ollama settings page, we read:

• Session (5hr): 24.1%
• Weekly (7d): 6.4%

Our logs showed:

• Session (5hr window): 13,574,960 tokens
• Weekly (7d window): 54,473,658 tokens

Working backwards from the real percentages, we derived the effective token ceilings:

Session ceiling: 13,574,960 / 0.241 = ~56M tokens per 5 hours
Weekly ceiling:  54,473,658 / 0.064 = ~849M tokens per week

The community estimate for the Max plan was ~1.25B tokens/week, but our calibrated measurement came in at ~849M. The difference is likely because community measurements used lighter models that consume less GPU time per token.

Our initial estimate was wrong. We first calculated ~36M tokens per session ceiling from “8.7M tokens in 2 hours = 24.1%”, but that overcounted because the 5-hour session window includes lower-usage periods. The calibrated ceiling of ~56M is more accurate.

Session is the danger zone

Weekly burn rate with dispatchers: ~3.2%/day. Without dispatchers: ~1-2%/day. The session limit is where you get caught off guard because it resets frequently and automated agents can spike it rapidly.

How We’re Monitoring It Now

Since there’s no API, we built a hybrid watchdog — automated tracking with manual calibration.

The Ollama Usage Watchdog

A Python script at ~/.hermes/scripts/ollama-usage-watchdog.py that:

1. Parses agent.log every 30 minutes (via cron, no_agent=true — zero token cost)
2. Counts tokens consumed in the last 5 hours (session) and 7 days (weekly)
3. Divides by calibrated token ceilings to estimate percentages
4. Compares against threshold levels
5. Delivers alerts to Discord when thresholds are crossed
6. Auto-pauses/resumes Kanban dispatcher cron jobs via hermes cron pause/resume

Thresholds

We set different thresholds for each window. Session thresholds are higher because it resets every 5 hours (fast recovery). Weekly thresholds are lower because hitting the ceiling means waiting days.

Three modes

# Watchdog mode (runs via cron, silent unless threshold crossed)
python3 ollama-usage-watchdog.py

# Status mode (check current readings anytime)
python3 ollama-usage-watchdog.py status

# Calibrate mode (feed real % from ollama.com/settings)
python3 ollama-usage-watchdog.py calibrate 24.1 6.4

How calibration works

Token-based estimates drift over time because token count ≠ GPU time. So once or twice a day, we check ollama.com/settings and feed the real percentages back:

python3 ollama-usage-watchdog.py calibrate <session_pct> <weekly_pct>

This recalculates the token ceilings: ceiling = current_tokens / (real_percentage / 100)

Current calibrated ceilings:

• Session: 55,984,946 tokens (≈56M)
• Weekly: 848,565,640 tokens (≈849M)

Current readings (as of writing)

Both dispatchers are currently paused (manually, not by the watchdog).

Lessons Learned

1. Token count is a proxy, not a direct measure. Build in margin. Our 50% safety margin in thresholds accounts for the token-to-GPU-time conversion.

2. The session limit is the binding constraint, not the weekly limit. It resets every 5 hours and automated agents can spike it fast.

3. Agent workloads burn tokens much faster than interactive use. Every call re-sends the full context. 42K tokens per call × 6 calls/hour = 252K tokens/hour just for one dispatcher.

4. No usage API means build your own observability. A hybrid approach — automated tracking plus manual calibration — is the pragmatic solution until Ollama adds an API.

5. Calibrate with real data. Our initial estimate of ~36M tokens/session was 36% low compared to the calibrated ~56M. Real percentages from the settings page ground the estimates in reality.

The watchdog script runs every 30 minutes via cron with zero token cost. Alerts go to Discord. The dispatchers auto-pause at 85% session / 70% weekly and resume when the session resets below 40%. We went from surprise 429 errors to proactive quota management.

To be fair here, some of the data below including the failures of good writes and reads from Hindsight were due to the now “Infamous” Ollama Relay (which was Ollam being installed directly on the host and Hermes “Relaying” through it (which was due to the age of my instance before Ollama.com was added in directly as an LLM Provider)). Either way I hated the fact that it required an LLM to burn even more Tokens for its usage and I didn’t like or want that so I chose to get rid of it.

And it was the right call.

The Diagnosis

It started with a simple question: “Analyze our memory setup. Are we doing this the best way?” The answer was no.

The investigation revealed several problems:

Hindsight was mostly dormant. The recall command failed with “LLM API key is required (see post “Ollama Relay Your Doing It Wrong” for more details on why this was happening).” The per-profile configs existed but the embedded PostgreSQL instance at ~/.pg0/instances/hindsight-embed-hermes wasn’t being queried effectively. Two LLM calls per turn — one recall, one retain — and the best case was a 14% extraction success rate at the chunk level.

Memory files were bursting. MEMORY.md was at 84% capacity (1,869/2,200 chars). USER.md was at 94% (1,305/1,375 chars). Most of that content was infrastructure facts — IP addresses, port numbers, model names — that should live in a knowledge base, not in a per-turn injected sticky note.

The architecture was self-contradictory. We had Hindsight auto-extracting facts (at 14% success) while also manually writing things to MEMORY.md. We had a structured Obsidian vault (set up 4 days earlier) that could hold all this knowledge without the per-turn tax. But we were still paying the Hindsight overhead on every single conversation turn.

The numbers told the story:

The Decision

The vault was already there, already working, already being written to by agents through SOUL.md protocols. Hindsight was just… sitting on top, costing tokens and adding latency for an 86% failure rate.

So we removed it. All of it.

The Execution

The removal was surgical and complete:

1. Disabled Hindsight in all configs — Set memory.provider: '' (empty string) in the main ~/.hermes/config.yaml and all 8 profile configs under ~/.hermes/profiles/*/config.yaml

2. Deleted the data — Removed ~/.hindsight/ (16 MB), ~/.pg0/ (337 MB PostgreSQL instance), and ~/.hermes/hindsight/ (main config directory). That’s 353 MB of disk reclaimed.

3. Deleted 8 per-profile config directories — Each profile had its own hindsight/config.json with bank IDs and model settings. All gone.

4. Consolidated MEMORY.md — Moved infrastructure facts out (they were already in vault wiki pages) and added a vault pointer: “Long-term knowledge base: Obsidian ai-vault… BEFORE research, config, or domain-specific tasks: search the vault wiki/”

5. Bumped memory limits — With Hindsight gone, we had headroom. memory_char_limit went from 2,200 to 3,000. user_char_limit went from 1,375 to 2,000. After consolidation: MEMORY.md at 61% (1,819/3,000), USER.md at 59% (1,180/2,000).

6. Verified everything — Ran end-to-end tests confirming Hindsight was gone:
   • ~/.hindsight/ — empty
   • ~/.pg0/ — gone (was 337 MB)
   • Zero provider: hindsight references in any config
   • Zero Hindsight processes running
   • hermes memory status shows: Provider: (none — built-in only)
7. Verified the vault still works — Created a test page, read it back, searched for it, then cleaned it up. MCP filesystem tools all functional. 11 wiki pages accessible. 3 cron maintenance jobs intact (daily-health, wiki-curator, weekly-digest).

The Write Triggers

The biggest concern was: without Hindsight auto-extracting facts, will agents actually write to the vault? We verified that all 5 vault-enabled profiles have SOUL.md Vault Knowledge Protocol sections with explicit write triggers:

• default: “After discovering durable information, create or update a wiki page”
• development: “After completing research, synthesize key insights into a wiki page”
• playground: “Inscribing new Chronicles, Forging new Creations”
• PMO: “Project decisions: log by appending to log.md”
• research: “After completing research — PRIMARY vault obligation”

These are personality-level directives, not optional suggestions. They’re baked into each agent’s SOUL.md, which means they’re injected into every conversation the same way MEMORY.md is. The vault writes happen because the agent’s identity includes the obligation to write.

The New Architecture

Built-in Memory (MEMORY.md + USER.md)
  → WHO + operational state + vault pointer
  → Injected every turn, zero cost, 100% reliable

Obsidian Vault (ai-vault via MCP)
  → Durable structured knowledge (11 wiki pages, 3 raw sources)
  → On-demand search (not auto-injected)
  → Compounding via cron maintenance

Session Search (FTS5)
  → Conversation history, on-demand

Three systems, zero overlap, zero wasted tokens. The vault is the brain. Built-in memory is the working state. Session search is the history.

What We Lost

Honest assessment: we lost automatic extraction. If an agent has a conversation that contains a fact worth remembering, and the agent doesn’t explicitly write it to the vault, that fact exists only in session history. It won’t surface automatically in a future conversation.

That’s a trade we’re willing to make. The alternative was 2 API calls per turn for an 86% failure rate. We’d rather have 100% accuracy on the 14% of facts that are worth extracting, written deliberately by agents that know what they’re writing and why, than 14% accuracy on everything in a system that costs tokens just to fail.

The vault cron jobs — daily health checks, weekly curations, weekly digests — handle the maintenance side. The SOUL.md protocols handle the write side. Between deliberate writes and session search fallback, we’re not losing knowledge. We’re just not pretending to gain it automatically.

Hindsight was removed on June 11, 2026, after 26 days of operation. The Obsidian AI Vault that replaced it was set up on June 7, 2026 — see “Hermes Gets a Brain” for that story.

Here’s how we got there.

The Problem

Hermes had two memory systems up to this point:

1. Built-in memory — MEMORY.md and USER.md, injected into every conversation turn. Reliable, but limited to ~2,200 characters of working state. No relationships. No compounding knowledge. Just a sticky note that gets overwritten.

2. Hindsight — a knowledge graph plugin that auto-extracted entities and facts from every conversation. Powerful in concept, but it cost 2 LLM API calls per turn (recall + retain) with latency ranging from 0.5 to 180 seconds.

Neither system gave us what we really needed: a durable, searchable, structured knowledge base that agents could write to deliberately and read from on demand, without paying a per-turn tax for automatic extraction.

The Design

We built the AI Vault on top of an existing Obsidian vault. The directory structure follows a clear separation of concerns:

ai-vault/
  SCHEMA.md          — Rules: what goes where, naming conventions, tags
  log.md             — Append-only activity log
  raw/               — Source material (append-only, never edit)
    articles/
    papers/
    competitors/
    repos/
  wiki/              — Agent-synthesized knowledge from raw sources
    _index/index.md  — Link graph of all wiki pages
  output/            — Deliverables generated from wiki
    reports/
    slides/
  templates/         — Reusable templates for agents

Key rules:

• raw/ is append-only — source material is never modified once stored
• wiki/ is where agents synthesize knowledge from raw sources into structured pages
• output/ is derived from wiki — never treated as source truth
• Every wiki page has YAML frontmatter with title, type, domain, tags, created/updated dates, and sources
• Every wiki page links to at least 2 related pages using Obsidian [[wikilink]] syntax

The MCP Connection

The critical piece was connecting the vault to Hermes via the Model Context Protocol (MCP). We configured an MCP filesystem server pointing at the vault directory, giving agents 14 additional tools:

• mcp_filesystem_read_file, mcp_filesystem_read_text_file, mcp_filesystem_read_multiple_files — read vault files
• mcp_filesystem_write_file, mcp_filesystem_edit_file — create/edit vault files
• mcp_filesystem_create_directory, mcp_filesystem_list_directory, mcp_filesystem_directory_tree — browse vault structure
• mcp_filesystem_search_files, mcp_filesystem_get_file_info — search and inspect vault files
• mcp_filesystem_move_file — move/rename vault files
• mcp_filesystem_list_allowed_directories — verify access boundaries

These MCP tools operate only within the vault directory — a safety boundary that prevents agents from accidentally writing to system files. The built-in read_file, write_file, and patch tools still work on any path, but the MCP tools are vault-scoped.

The configuration went into each profile’s config.yaml:

mcp_servers:
  filesystem:
    command: npx
    args:
      - "@modelcontextprotocol/server-filesystem"
      - "/Obsidian/ai-vault"

Important lesson: the config key is mcp_servers (snake_case), not mcpServers (camelCase). We learned this the hard way — the gateway didn’t pick up the MCP server until we corrected the key name. One gateway restart later, 14 new tools appeared.

Who Gets Access

Not every agent needs vault access at this time so, we enabled it for the profiles that do knowledge work:

The four F5 specialist agents (admin, asm, gtm, ltm) were excluded for the time being as I have not had the time to further configure them and give them a mission in life (only so many hours in the day).

We also set the OBSIDIAN_VAULT_PATH environment variable in each enabled profile’s .env file, so agents always know where the vault lives regardless of which tool they’re using.

The First Page

After setup, we created the first wiki page — wiki/vault-setup.md — documenting the configuration itself. Then we updated wiki/_index/index.md to link it, and appended an entry to log.md.

The vault was now self-documenting. Every wiki page gets indexed. The knowledge graph grows organically from actual work rather than from automatic extraction with an 86% failure rate.

Why This Matters

The vault replaces the “hope and pray” model of automatic memory extraction with a deliberate, agent-controlled knowledge base:

• Zero per-turn cost — no API calls, no latency, no failed extractions
• 100% write accuracy — agents write exactly what they intend, not what an LLM guesses
• On-demand reads — vault searches happen only when relevant, not every turn
• Compounding knowledge — wiki pages link to each other, building a graph that gets richer over time
• Append-only raw sources — source material is preserved exactly as captured

The vault is the brain. Built-in memory is the working state. Session search is the conversation history. Three systems, each doing what it’s best at, with no overlap and no wasted tokens.

Reference Plug for Credit Due:

Part 1: The idea
I Wired Claude Code Into Hermes Agent (And Hermes Into Claude Code): The Full Tool Gateway Reference

Part 2: Implementation Part 1
How to Build an AI Agent Operating System That Compounds Over Time

Part 3: Implementation Part 2 is to configure Claude Code on the Mac Mini M4 with Hermes, and now that Anthropic has enabled Auto-Mode for Pro Subscriptions I’ll use my Hybrid Auto-Mode Configuration to allow Hermes and Claude to work together.

—

The AI Vault was created on June 7, 2026. It was later integrated with Hindsight’s replacement on June 11, 2026 — see “Hermes Loses Hindsight” for that story.

47.3 million input tokens. 446 thousand output tokens. Seven days. One Mac Mini.

The ratio hit harder than the raw count. For every token the model produced, I’d fed it 106 tokens of context. If you think of tokens as a conversation, that’s like talking to someone who listens to a hundred words for every one they say back. Except in this case, every single one of those input tokens costs compute — and if you’re paying per token at a cloud provider, every one of them costs money.

Tokens ain’t free, buddy.

The Raw Numbers

I pulled the data directly from Hermes’s SQLite database — ~/.hermes/state.db, the sessions table. Every session records input_tokens, output_tokens, cache_read_tokens, api_call_count, tool_call_count, model, and billing_provider. Here’s what the 7-day window ending June 6, 2026 looked like:

Per-Model Breakdown

The two glm-5.1 variants are the same model through different provider configurations — one through the built-in Ollama Cloud relay, one through a custom endpoint. Together they accounted for over 99% of all tokens.

And the qwen3-8b session? One session, nearly a million input tokens, seven thousand output. That’s a single task that consumed context like a black hole and produced barely a peep, but to be fair. This is the Kanban PMO Agent and it’s local, so this one gets a PASS.

Why the Ratio Is So Lopsided

If you’ve worked with LLM agents, the 106:1 ratio isn’t surprising — but it’s worth understanding the mechanics.

Every message to the model includes the entire conversation history. Not just the latest exchange. The system prompt (which runs ~15K tokens with all tools loaded), every previous user message, every previous assistant response, every tool call and its result. It all gets packed back in, every single time.

Here’s what that looks like in practice:

• Message 1: ~15K input (system prompt + first user message)
• Message 10: ~80K input (system prompt + 9 rounds of history)
• Message 30: ~200K+ input (system prompt + 29 rounds + all tool outputs)
• Message 50: Context is so bloated that Hermes triggers compression, which itself costs tokens

The output stays small because most agent responses are a few tool calls or a paragraph of text. But the input snowballs. And it snowballs fast.

Tool calls make it worse. Every web search returns 2-5K tokens of results. Every file read dumps the full file contents. Every terminal command shows its stdout. All of that goes back into the context window for the next turn. A session with 5 web searches and 3 file reads can easily accumulate 50K tokens of tool output alone — and then every subsequent message includes all of it.

The Cost of Heavy Sessions

I dug into the distribution. Not all sessions are equal — not by a long shot.

78 sessions (11% of total) had 30+ messages each. Those 78 sessions consumed 195 million input tokens — 72% of all usage. The remaining 647 sessions (89%) shared just 28%.

Let that sink in. Less than one in nine sessions burned nearly three-quarters of the entire token budget.

The heaviest individual sessions ranged from 1.2M to 8.4M input tokens each. At the low end, that’s 60 average messages worth of context crammed into one session. At the high end, that’s a session that ran so long the model was probably seeing its own compressed summaries of summaries.

What We Built to Fix It

The problem was clear: context grows with no natural stopping point. Compression slows the bleeding, but it doesn’t stop it. Starting a new session has a 63K-token boot cost (the system prompt and tools), so /new isn’t free either. But a 63K restart is a lot cheaper than carrying 2M+ tokens of stale context through every subsequent message.

We built two mechanisms — one proactive, one reactive:

1. Task Session Manager (Skill)

A skill called task-session-manager that defines a completion protocol. When a task finishes — all TODO items resolved, user says “done”, or the agent recognizes no remaining work — the agent follows a five-step sequence:

1. Verify completion. Check the TODO list. If items remain, ask the user before wrapping up.
2. Save durable facts. Push environment facts, user preferences, and stable state changes to Hermes Memory. Push key decisions and outcomes to “The Vault” (Obsidian) long-term memory.
3. Save procedural knowledge (if warranted). Did we discover a non-trivial technique or workaround? Save it as a skill. Routine tasks? Skip it.
4. Deliver a handoff message. Tell the user what was accomplished and explicitly suggest /new for the next task.
5. Stop working. No “anything else?” No “while I’m here…” The session is done.

The key insight: the agent doesn’t auto-terminate. It prompts you to start a new session. Because the worst outcome is an agent that cuts you off mid-thought.

2. Session Length Guard (Cron Job)

A Python script at ~/.hermes/scripts/session-length-guard.py that runs every 2 hours. It queries the active session from SQLite and checks two thresholds:

• Message threshold: 30+ messages
• Token threshold: 2M+ input tokens

If either threshold is crossed, it outputs a nudge message:

Session running hot: 34 messages (threshold: 30)
Active since: 2026-06-06 14:23 (2.1h ago)
Model: glm-5.1
Tokens: 4.2M in / 0.18M out
Consider: wrap up current task, save state, and /new to start fresh.
Tip: Load the 'task-session-manager' skill for the completion protocol.

This fires as a cron job that delivers via Discord. It’s advisory, not mandatory. No session killing, no forced /new. Just a tap on the shoulder: hey, you’ve been running for a while, and context is getting expensive.

What Happens When I Declare “Wrap Up”

When I say “wrap up” or “done” or “that’s it,” the agent loads the task-session-manager skill and executes the completion protocol. Here’s what that looks like in practice:

1. Verification. The agent checks its TODO list. All items completed or cancelled? Proceed. If not, it asks me: “This task still has open items. Wrap up anyway?”

2. Memory saves. Every durable fact discovered during the session gets saved — paths, config values, version numbers, environment details, user corrections. These persist across sessions, so the next session starts with context instead of starting from zero.

3. Skill saves (if warranted). If we discovered a non-trivial technique — say, a workaround for Ruby stdlib compatibility, or a specific Docker networking pattern — it gets saved as a class-level skill. One-off task results don’t. “Analyzed token usage” isn’t a skill. “How to fix Jekyll 3.9 incompatibility with Ruby 4.0” is.

4. Vault saves (if warranted). If the session produced research or documentation that belongs in the Obsidian vault wiki, it gets written there.

5. Handoff message. The agent delivers a summary:

Task complete. Here's what was accomplished:
- Analyzed 7-day token usage: 47.3M in / 446K out, 106:1 ratio
- Built task-session-manager skill with completion protocol
- Created session-length-guard cron job for Discord nudges

State saved:
  Memory: token economics data, session management patterns
  Hindsight: key decisions on session lifecycle management
  Skills: task-session-manager (new)
  Vault: not warranted

Start a new session with /new for your next task to keep context lean.

1. Stop. The agent does not offer follow-up work. No “while I’m here…” suggestions. No “anything else?” The session is done, and context stops growing.

The Config Change That Helped Immediately

One line in ~/.hermes/config.yaml made the biggest difference:

session_reset:
  mode: idle
  idle_minutes: 120

Previously, mode was set to none — sessions never reset, even after hours of inactivity. Changing to idle with a 2-hour timeout means sessions that go quiet for 120 minutes automatically close. The next message starts fresh, with a clean 63K-token context window instead of carrying a bloated 2M-token monster.

This single change — combined with the wrap-up protocol and the cron guard — cut our average session token consumption by roughly 40% in the weeks that followed.

The Takeaway

Agent sessions are not free. Not in money (Ollama Cloud was free for us), but in compute, latency, and context quality. Every token you feed the model is a token of compute that could have been spent on a sharper, leaner session. Every bloated context window degrades the model’s ability to find relevant information in the noise.

The 106:1 ratio was a wake-up call. The fix wasn’t a single switch — it was a combination of:

• Proactive protocol (the completion skill)
• Reactive monitoring (the cron guard)
• Config hygiene (idle session reset)
• User discipline (actually using /new when the nudge arrives)

Tokens ain’t free. But they don’t have to be expensive either — you just have to know when to close the tab.

Data collected from Hermes Agent’s SQLite session database, ~/.hermes/state.db, June 6, 2026. Analysis performed on a Mac Mini M4 running Ollama Cloud inference. All token counts are accurate per the provider’s reporting; costs show $0 because Ollama doesn’t report pricing in API responses.

The Problem

Every single Hermes agent profile — research, playground, development, f5-ltm, f5-gtm, f5-asm, f5-admin, and even the PMO’s fallback — was routing through a local Ollama relay at http://localhost:11434/v1. That relay forwarded requests to https://ollama.com/v1, Ollama’s cloud inference service.

On paper, this looked fine. Local relay, cloud inference, what could go wrong?

Everything.

The relay was an intermediary that added latency, obscured errors, and — most critically — masked the real authentication headers. When Ollama Cloud throttled or rejected a request, the relay couldn’t tell the difference between a legitimate rate limit (429) and a session that had expired. The agent saw 429 Too Many Requests and tried to fall back. But the fallback was… the same relay. Same endpoint. Same broken path. Round and round it went, hammering a service that was already rejecting it, unable to recover because every recovery attempt went through the same broken pipe.

The frustration was real. The agent would hit a 429, fall back to itself, hit another 429, and the user would watch the terminal fill with error after error after error. No escape. No alternative path. Just an infinite loop of “try again” against a wall that wasn’t letting anything through.

And the worst part? The default profile had a circular fallback — glm-5.1:cloud on Ollama Cloud, falling back to glm-5.1:cloud on the exact same Ollama Cloud endpoint. When the primary failed, the “fallback” was literally the same service that just told you to go away.

The Diagnosis

The configuration audit revealed the full scope of the damage:

Five profiles had no fallback at all. The default profile had a fallback that was literally itself. The PMO profile was the only one with a remotely sane configuration — its primary was local (LM Studio), so it could at least function when cloud was down.

The custom_providers entry in the main config compounded the confusion. It defined an “Ollama Cloud” provider pointing at http://localhost:11434/v1 — the local relay, not the actual cloud service. The name said “Cloud” but the URL said “Local.” The default profile’s model.base_url overrode it to https://ollama.com/v1, so it looked like it was connecting directly… but all the profile configs used provider: custom with inline base_url: http://localhost:11434/v1, routing everything through the relay.

A misnamed provider. An invisible intermediary. A circular fallback. A recipe for an outage that would refuse to heal itself.

The Fix

Two sessions on June 2, 2026. First session: add real fallbacks so that when cloud goes down, agents can at least function locally. Second session: rip out the relay entirely and connect every agent directly to https://ollama.com/v1 with a valid API key.

Session 1: Fallback Infrastructure (12:16 PM)

The immediate priority was survival — if cloud was down, agents needed a local lifeline. LM Studio on port 1234 was running qwen3-8b and available.

1. Added LM Studio as a custom provider in the main config alongside the existing Ollama entry
2. Fixed the circular fallback — replaced fallback_providers: [custom localhost:11434 / glm-5.1:cloud] with fallback_model: [custom:lm-studio / qwen3-8b]
3. Added fallback to all 6 profiles that had none: playground, development, f5-ltm, f5-gtm, f5-asm, f5-admin — all falling back to LM Studio’s qwen3-8b
4. Left PMO and research alone — PMO already had a cloud fallback, research already had LM Studio

Now every agent had at least one alternative path when Ollama Cloud was down.

Session 2: Direct Connection (6:19 PM)

But fallbacks are a bandage. The real fix was eliminating the relay. The user asked: “Check the other agents and update them so they use the same direct ollama-cloud connection.”

The default profile was already using https://ollama.com/v1 directly (via the inline model.base_url override). Every other profile was still routing through localhost:11434. Time to make them all consistent.

Changed 8 profile configs from http://localhost:11434/v1 to https://ollama.com/v1:

LM Studio connections (localhost:1234) stayed local — they’re genuinely local hardware, not a proxy to a cloud service.

The main config’s custom_providers “Ollama Cloud” entry also got updated from http://localhost:11434/v1 to https://ollama.com/v1 to match reality. No more name/URL mismatch.

The Lesson

A relay is not a free abstraction layer. It’s a point of failure that:

• Adds latency to every request
• Obscures authentication errors — the relay can’t distinguish 401 (bad key) from 429 (rate limit) from 503 (service down) with the right recovery semantics
• Creates circular dependencies — falling back to the same broken pipe
• Masks the real endpoint — the config says “Ollama Cloud” but the URL says “Local Relay” and the behavior is neither
• Prevents direct API key validation — you can’t test your credentials against the real service when they go through an intermediary that may or may not forward them correctly

If you’re using Ollama Cloud models, connect directly to https://ollama.com/v1 with a valid API key. If you’re using local models, connect directly to http://localhost:11434/v1. Don’t mix the two. Don’t relay cloud requests through a local instance. Don’t name a local endpoint “Cloud.”

And for the love of all that’s operational — if your fallback is the same service you’re already failing on, you don’t have a fallback. You have a loop.

The relay was removed on June 2, 2026, across two sessions. The Ollama Cloud direct connection has been stable since. See “Hermes Loses Hindsight” (June 11) for the next chapter in the infrastructure cleanup.

F5 BIG-IP QKView diagnostic archives are the primary artifact for remote support cases and proactive health assessments. But analyzing them is slow, inconsistent, and heavily reliant on individual engineer experience. A QKView contains dozens of XML files, configuration dumps, log excerpts, certificate inventories, and hardware telemetry — and no two engineers will examine the same data the same way.

I built a wiki that doesn’t just store knowledge. It runs the analysis.

From Reference to Runbook

The TMSH Wiki is a static reference — you look up syntax, it gives you ground truth. The QKView Wiki evolved the pattern into something procedural: a self-improving runbook that walks over 20+ deterministic Python scripts against a QKView diagnostic archive, follows a check analysis playbook, and gets smarter after every engagement through the re-ingestion of a deviation file created during the process.

The key innovation is the Execute / Refine cycle:

1. Execute — The skill points the agent at a QKView file (outside the wiki repo), sets up an engagement workspace (also outside the wiki repo), and walks the playbook through 21 checks in order. Each check runs a Python extractor script, reads the pre-computed result, and if the result is warn or crit, reads the check page and drilldown. Findings are written to an external workspace. Deviations are queued — never modifying the wiki during Execute.

2. Refine — After Execute completes, a separate session processes the deviation queue. The generalize step is mandatory and comes first — stripping every client-identifying detail (names, hostnames, IPs, serial numbers, exact percentages) before anything enters the wiki. Fixes are routed to the correct surface: wiki page, extractor script, or both.

The wiki compounds. Every engagement that deviates from the playbook produces a lesson that gets generalized and filed back, making the next run more accurate.

The Extractor Layer

After a second engagement revealed that Claude was sometimes skipping analytical work by not fully parsing raw XML — and a Haiku-class model test showed fabricated all-clears — I added 20+ Python scripts that deterministically parse QKView XML artifacts and emit structured Markdown with:

• A leading result: ok|warn|crit|n/a line — the threshold judgment, already made
• Captured data tables with the raw evidence
• Drill-route signals for branching
• Verify sections posing the wiki’s named possible causes as numbered questions

The extractors make hallucinated all-clears structurally hard: a model walking the playbook reads pre-computed result: crit lines rather than trying to interpret raw XML.

Sterility

The QKView Wiki has a sterility constraint that the TMSH Wiki doesn’t. The TMSH Wiki is purely F5 vendor knowledge — no client data to protect. The QKView Wiki runs against real client device data on every engagement. The constraint is absolute:

No client-identifying data ever enters the wiki repo. Not in frontmatter, not in log entries, not in lesson files, not in commit messages, not in filenames. The generalize step during Refine is mandatory. A deviation note saying “client’s /var was at 88%” becomes “For /var, fill rate dominates absolute %. The 80-89 warn band should escalate to crit when fill rate exceeds ~N%/day.”

The Numbers

• 72 wiki pages — 21 checks, 18 drilldowns, 2 playbooks, 11 concepts, 9 lessons, 1 source, 1 template, 1 overview
• 20+ Python extractor scripts covering platform, filesystem, memory, CPU, process, HA, services, network, pool/VS, certs, iRules, config hygiene, tmctl statistics, log analysis, and correlation
• 10+ engagements completed — the wiki compounds with every run
• 2 playbooks — playbook-full-analysis for normal operations and playbook-cold-start as a bootstrap mode

See the Full Project

This post is an introduction. The complete project page has the full architecture, the Extractor layer breakdown, the Execute/Refine cycle, sterility rules, access rules, the creation timeline, and enhancement opportunities:

QKView LLM Wiki Brain →

Reference: Andrej Karpathy’s LLM Wiki concept · See also: TMSH LLM Wiki Brain — the static reference companion that shares the same vault architecture

Here’s how it happened.

NOTE: You’ll see me reference the “Ollama Relay”. Hermes runs on a Mac Mini M4 and I have Ollama installed on the host machine, so that is why you see the reference and the strange keys. This instance Hermes Instance was created prior to Ollama.com being a built-in provider option.

The Problem

Hermes has had built-in memory since day one — MEMORY.md and USER.md files that get injected into every conversation. It works. It’s reliable. But it’s limited. Those files are flat text, capped at a few thousand characters, and they only contain what you explicitly write into them. They don’t learn on their own. They don’t connect dots. They don’t say “hey, the IP address you mentioned three conversations ago is the same server we’re troubleshooting now.”

We needed something that could:

• Auto-retain: Extract facts, entities, and context from every conversation turn without manual intervention
• Auto-recall: Search past memories for relevant information and inject it before each response
• Build a knowledge graph: Connect entities — people, projects, machines, IP addresses — into relationships, not just text blobs

That’s what Hindsight promised.

The Setup

The installation was surprisingly straightforward for something this powerful. Three packages via uv pip install:

• hindsight-client — the API client
• hindsight-all — the full package
• hindsight-embed — the embedded database engine (no external Postgres server needed)

Then a config file at ~/.hermes/hindsight/config.json:

{
  "mode": "local_embedded",
  "bank_id": "hermes",
  "llm_provider": "ollama",
  "llm_model": "kimi-k2.6:cloud",
  "auto_recall": true,
  "auto_retain": true,
  "memory_mode": "hybrid"
}

The key decisions we made:

• mode: local_embedded — runs entirely on the Mac Mini. No cloud API calls for memory operations. It bundles its own PostgreSQL instance with pgvector for vector similarity search. Zero external dependencies.
• llm_provider: ollama with llm_model: kimi-k2.6:cloud — uses our “Ollama Relay” instance to power the extraction and recall LLM calls. This was important — we didn’t want memory adding per-turn API costs to a cloud provider.
• auto_recall: true and auto_retain: true — fully automatic. Every turn saves context, every new turn searches for relevant history.
• memory_mode: hybrid — the best of both worlds. Memories get injected into the system prompt context AND explicit hindsight_recall / hindsight_retain tools become available for targeted queries.
• bank_id: hermes — a single shared memory bank for the default profile. Other profiles (playground, research, etc.) would later get their own isolated banks.

One line in ~/.hermes/config.yaml flipped it on:

memory:
  provider: hindsight

And that was it. hermes memory status showed:

Built-in:   always active
Provider:   hindsight
Plugin:     installed ✓
Status:     available ✓

The Rollout

Getting it working on the default profile was step one. But we had eight other agent profiles — PMO, research, playground, development, and four F5 specialists — all needing their own isolated memory banks.

We created hindsight/config.json files under each profile directory, each with its own bank_id:

And added the memory section to each profile’s config.yaml:

memory:
  memory_enabled: true
  user_profile_enabled: true
  memory_char_limit: 2200
  user_char_limit: 1375
  provider: hindsight
  nudge_interval: 10
  flush_min_turns: 6

We tested isolation by retaining “Playground test message is ALPHA” under the playground bank and “Research test message is BETA” under the research bank, then recalling from each — playground got ALPHA, research got BETA, default got neither. Clean isolation. Exactly what we wanted.

The LLM Tax

Hindsight isn’t free in compute terms. Every conversation turn triggered two LLM calls:

1. Recall — before the response, search the memory bank for relevant context and inject it
2. Retain — after the response, extract entities and facts and store them

On our Ollama Relay instance with kimi-k2.6:cloud, recall latency ranged from 0.5 to 180 seconds depending on the query complexity. That’s a significant per-turn tax. But the tradeoff was worth it: the agent could now remember across sessions, build institutional knowledge, and connect dots that flat text files never could.

The research profile got minimax-m2.7:cloud (again through our Ollama Relay) as its Hindsight LLM to avoid competing with kimi-k2.6:cloud and to test it out for functionality.

What Changed

Before Hindsight, if Hermes asked “What’s the IP of the Firecrawl server?” in a new session, the agent had no idea. The answer was buried in some MEMORY.md file or a past conversation that required manual session search.

After Hindsight, the agent could recall the IP Address of the Firecrawl endpoint because it had retained that fact from a previous conversation and built a relationship between “Firecrawl” and “home lab server.”

The knowledge graph was the real differentiator. It didn’t just store “Firecrawl is at xxx.xxx.xxx.xxx:3002.” It connected that to “home lab,” “research agent,” “self-hosted,” “no auth required,” and every other entity that appeared alongside it. When any of those concepts came up again, the full context surfaced.

Here’s why we did it and what changed.

The Zimaboard2: A Good Start

The Zimaboard2 1664 is a fanless x86 single-board server with an Intel N150 processor, 16 GB of RAM, dual 2.5GbE, and a PCIe 3.0 x4 slot. At $399, it’s an attractive little machine for homelab work — quiet, compact, low power. It runs ZimaOS, CasaOS, Proxmox, TrueNAS, you name it.

It was where Hermes was born. We installed the agent, configured the gateway, set up Ollama, got the Discord bot connected. Hermes worked on the Zimaboard2. That’s the honest truth.

But “worked” and “worked well” are different things.

The Problem

The Intel N150 is a capable chip for what it is — a low-power x86 processor designed for NAS duty, media streaming, and light container orchestration. It is not designed for running local LLM inference alongside a multi-agent orchestration platform.

Hermes isn’t a single process. It’s a stack:

• The CLI agent itself (Python, always-on)
• The Discord gateway (async, message-driven)
• Ollama for local LLM inference (GPU-dependent, memory-hungry)
• The API server (HTTP endpoint for external integrations)
• Cron jobs firing on schedules
• Hindsight memory extraction (at the time, 2 LLM calls per turn)
• Dashboard web UI

On 4 cores and 16 GB of shared RAM, that stack competes with itself. Ollama loading a 4B or 8B model? The system crawls. A cron job kicks off while a Discord conversation is active? Latency spikes. Two agents talking at once? Forget it.

And then there was the N150’s instruction set. No AVX2, no AMX — just basic x86-64 with AES-NI. Modern LLM inference frameworks increasingly expect AVX2 as a baseline. Some operations would flat-out fail or fall back to painfully slow scalar paths.

The Zimaboard2 was like putting a commuter car on a race track. It’ll lap the circuit, but you won’t be setting any records, and the driver is going to be frustrated the whole time.

The Mac Mini M4

The upgrade wasn’t subtle:

That’s a 5× CPU core count increase, 4× the RAM, and a GPU that doesn’t exist on the Zimaboard2. The M4 Pro’s unified memory architecture means the CPU and GPU share the same 64 GB pool — Ollama can load a 32B parameter model and still have room for everything else.

What Changed Immediately

The first thing I did after the migration was confirm it was real. The very first message sent to Hermes on the new machine was:

“Say hello and confirm you’re running on the Mac Mini M4”

It answered. And it answered fast.

Ollama went from painful to usable. Models that took 30+ seconds to first token on the N150 responded in 2-3 seconds on the M4 Pro. We went from running one small model at a time with degraded responsiveness to loading 16 models simultaneously with headroom to spare.

The full agent stack runs concurrently. Discord gateway, API server, cron jobs, multiple agent profiles — all running at the same time without stepping on each other’s CPU or memory. The 20-core M4 Pro parallelizes workloads that the 4-core N150 had to time-slice.

Metal GPU acceleration for inference. Ollama on macOS uses Apple’s Metal framework for GPU offload. The -ngl 99 flag that would crash or stall on the Zimaboard2 (no GPU!) now actually means something. We offload the entire model to the GPU and get 40-50 tokens per second on 8B-class models.

Storage I/O stopped being a bottleneck. NVMe versus SATA III — model loading, log writes, state database queries, all of it got faster. The state.db that Hermes hits on every conversation turn now reads from NVMe instead of a spinning disk or slow SATA SSD.

The Migration Itself

The filesystem tells the story. Using stat on the key directories:

From home directory creation to first conversation: roughly 18 hours. Most of that was macOS setup, Homebrew, Python, and configuring the environment. The actual Hermes install and configuration took about 35 minutes.

The hostname is MacMiniM4LLM — chosen deliberately. This machine’s job is to run LLMs. Everything else it does (Discord gateway, cron scheduling, dashboard, Firecrawl) is in service of that primary purpose.

What We Left Behind

The Zimaboard2 isn’t retired. A 4-core x86 board with 16 GB of RAM and dual 2.5GbE still has plenty of uses in a homelab. It just isn’t where Hermes lives anymore. It’s been reassigned to lighter duties — the kind of workloads it was actually designed for.

And that’s the right place for it. The Zimaboard2 taught us what we needed in a host. It ran Hermes when we were learning what Hermes even was. But once the agent stack grew — multiple profiles, local inference, cron jobs, memory extraction, API endpoints — we needed a machine that could keep up with all of it at once.

The Mac Mini M4 doesn’t just keep up. It has headroom. And that headroom is what makes the difference between an agent that works and an agent that works without you noticing it’s working.

Hermes was installed on the Mac Mini M4 on May 15, 2026. The Zimaboard2 that preceded it still serves in the homelab — just not as an LLM host. See “Hermes Gets a Brain” for the next chapter in the agent stack evolution.

Ask any LLM for TMSH syntax and you’ll get an answer. It will sound authoritative. It will be wrong. TMSH is purpose-built, highly structured, and full of nuances that general training data doesn’t capture well. Partition scoping changes behavior in ways that aren’t obvious. list vs show have distinct semantic differences that matter. modify list replacement behavior can silently wipe configuration. Flag combinations, module hierarchies, and object paths have exact syntax that training data garbles.

Rather than accept hallucinated TMSH syntax, I built a knowledge base that forces the LLM to consult vetted source material before answering.

The Concept

The TMSH LLM Wiki Brain is a locally-hosted, compounding knowledge base for F5 BIG-IP’s Traffic Management Shell. It’s stored as an Obsidian vault of interlinked Markdown files, and it’s based on Andrej Karpathy’s “LLM Wiki” pattern — feed a curated knowledge base into a session so answers are grounded in reality, not probability.

The wiki enforces strict reading discipline:

• Start at index.md — the navigation layer, always free
• Read at most 5 content pages per question — if more are needed, the agent stops and asks for permission
• Fallback is mandatory silence — if the wiki has no coverage, the agent states “The wiki does not currently cover [topic]” and suggests what source could fill the gap. It does not answer from training data as a substitute

The entire point is to prevent training-data guessing. Falling back silently would defeat the purpose.

What’s In It

The initial ingest came from two authoritative sources:

• F5 KB K13225 Cheatsheet — established the list vs show semantic distinction as a foundational concept
• BIG-IP TMSH Reference Guide v17.0.0 — the 1,843-page authoritative F5 document. Full ingest created 47 pages: 27 commands, 5 modules, 9 objects, 4 concepts, 1 pattern, 2 source summaries, 1 overview, and 1 index

The vault structure separates concerns cleanly: raw/ for immutable source documents, wiki/ for all generated content (commands, modules, objects, concepts, patterns), with every page following a consistent schema of YAML frontmatter, Obsidian wikilinks, callouts, and code blocks tagged with tmsh language.

Why It Matters

This was the proof of concept. The pattern worked so well for a static reference that it directly led to the QKView LLM Wiki Brain — which evolved the concept from “look up syntax” to “walk a procedure against live data and get smarter after every engagement.”

The TMSH Wiki now serves as the gold-standard CLAUDE.md template for every other wiki brain in the system.

See the Full Project

This post is an introduction. The complete project page has the full architecture, ingest sources, current state, access rules, and enhancement opportunities:

TMSH LLM Wiki Brain →

Reference: Andrej Karpathy’s LLM Wiki concept