LLM cost control — defense in depth

This is the canonical map of how MemQL bounds LLM spend so that a runaway loop cannot burn unbounded money regardless of which path triggers it (epic memql#1141).

#Why "defense in depth"

Earlier protections were all rate limiters that self-heal: the per-fingerprint loop breaker resets after its cooldown, and the rate ceiling drains its window and re-admits. That is correct for a transient spike, but it means a genuinely stuck loop — one that varies its request body and paces just under the rate ceiling — trickles spend forever (you'd see 429 ... rate ceiling ... blocked repeating indefinitely while money keeps leaving). Every per-path fix (per-turn breaker, produceArtifact re-delegation cap, …) patched one loop; the runaway reappeared via another.

The fix is layered. The lower layers are graceful (throttle, converge, park); the bottom layer is a hard kill-switch that latches and never drains. Because the kill-switch lives at the single chokepoint every chat/messages call passes through and is path-agnostic (it counts calls, not callers), a brand-new runaway path nobody anticipated is bounded automatically.

#The layers

Layer 0 is the backstop behind every other layer: even when a higher layer is generous or a loop's terminal condition is loose, the cumulative kill-switch caps total spend.

#The selection seam is a layer -- and it is NOT the enforcement one

Since epic memql#5127 every call to a model goes through one router seam: a call declares a level (fast / strong / reasoning / embeddings) instead of a model, a rule maps the call's metadata to a policy, and the policy is an ordered chain that reaches the fleet first, a signed-in subscription app second and a metered vendor last. So the shipped default now chooses the cheapest sufficient door before any of the layers above has to refuse an expensive one. That is genuine cost control and it belongs in this document -- but it is a different kind from everything in the table.

A kill switch a policy can author around is not a kill switch. The rules system is authored: a DSL bundle may add rules, and an owner may add one at runtime through routingRuleActivate. So the split is explicit and enforced by where the code lives:

The two meet at exactly one place, unchanged by this epic: the federation hop asks the cost ceiling before it is taken, and only when a local door preceded it in the chain. A chain that starts at a vendor is a decision somebody made rather than a fallback, and refusing it there would break every deliberately-paid policy. See Layer 4b.

One call was added, and it is bounded by construction. The work spine's failure path now runs the deterministic symptom table first and reaches the classifySymptom prompt -- at level fast, the cheapest tier -- only when the table has no opinion. A rules-classified failure costs zero provider calls; a novel one costs exactly one. Both figures are asserted by a proving scenario whose negative control is the classifier being reached at all, because a counter that never rises on any path reads as zero forever.

Every resolution writes one v1:router:call row saying which rule matched, which policy it named, which door served, what it cost, and every entry the walk passed over with the reason. Read them with routerDecisionsRecent. A cost control nobody can audit is an assertion; see AI routing.

#Where the chokepoint is (and why it moved)

Every layer below is described in terms of a single point every LLM call passes. Until local models landed that point was the HTTP transport, and the statement "every chat/messages completion leaves the process through one guardedTransport" was complete — the guard was path-agnostic precisely because there was exactly one way out of the process.

A fleet call has no *http.Client at all. It leaves over the WorkerService stream to a model on one of the user's own machines (epic memql#4676), so it would have passed no gate whatever. A runaway loop on a free model is still a runaway loop: it burns a laptop's battery, occupies the machine its owner is trying to work on, and converges on nothing.

So there are now two entry points into the same guard, not two guards:

Both run the same four checks in the same order — latch, loop breaker, rate ceiling, cumulative accounting — and both read and write the same counters. That shared state is what stops the two paths from disagreeing about how many calls a runaway has made. TestTheFleetProviderItselfPassesTheGuard holds that a call placed through the registered fleet provider passes the guard.

One thing differs, and only this: the dollar figure is zero. Nobody was billed for a local call, so charging it would park work over money that was never spent — and the more someone leans on hardware they already own, the sooner their plans would stop, which is exactly backwards. The call tallies are unchanged, process-wide and per-scope. This is the same split memql#4362 made for subscription spend, arriving for a second reason; see recordAndMaybeLatchCost and TokenState.SpentLocal.

The fleet fingerprint (FleetCallFingerprint) keys on the model and the exact conversation, deliberately not on the plan id, task id or purpose: those vary across a genuine loop, and folding one in would make every repetition look novel — defeating the cheap catch the breaker exists to be.

#Layer 0 — the kill-switch (ai_guard.go)

guardedHTTPClient wraps the *http.Client of all four chat provider builds (OpenAI + Anthropic, stream + non-stream), so every chat/messages completion leaves the process through one guardedTransport. There it is checked, in order: (0) kill-switch latch → (1) loop breaker → (1) rate ceiling → (3) cumulative accounting. Crossing a cumulative cap latches the breaker open permanently: every further call returns a terminal 402 (non-retryable in both vendor SDKs, so it surfaces as a clear terminal error to the calling loop) and makes no vendor request. It does not drain until the process restarts (or the guard is reset).

The $ figure is a deliberately conservative upper bound estimated from the request only (size + max_tokens, priced at an Opus-tier rate) — the response usage is never visible at the RoundTripper. Over-estimating is the safe direction: it latches sooner.

Process caps default to 0 (unlimited) — a since-boot cumulative latch has no single value safe for both a tiny local run and a long-lived production node, so the process backstop is opt-in. The on-by-default conservative protection is the per-scope latch (Layer 4). Local repro: MEMQL_LLM_MAX_TOTAL_CALLS=20.

#What the guard deliberately does NOT count: the federation token exchange

Both AI vendors are reached by workload identity federation (memql#4333 for Anthropic, epic memql#5088 for OpenAI), and that is the ONLY way either is reached -- there is no manually entered vendor API key anywhere in the product. So on any federated cluster one more request leaves the process through this same guardedTransport: the token exchange that trades the pod's projected Kubernetes token for a bearer of at most an hour. It is outside the guard's fingerprint by design — isGuardedLLMPath matches only /messages and /chat/completions, and a test holds that neither vendor's exchange path is one of them.

That is not an oversight to be tidied up later, and it must not be "fixed": the exchange spends no tokens and has no completion to price, so costing it would inflate a budget meant to bound spend. Worse, it is byte-identical every time and happens on a schedule, which is precisely the shape the loop breaker exists to stop — guarding it would let a routine hourly refresh trip the breaker and take the cluster's model access down, in the name of cost control that saved nothing.

It is not unobserved, though. It has its own counter, memql_ai_federation_exchanges_total{vendor,outcome=ok|denied|error}, and a denial logs the vendor's own error body. Alert on denied: the last good bearer keeps working until it expires, so a broken credential does not surface as an outage for up to an hour after the change that caused it. Runbooks: ../operate/auth/anthropic-federation.md and ../operate/auth/openai-federation.md.

#Layer 2 — automation budget

A misfiring automation (one that re-fires on its own failure, or a plan-level loop that re-creates a plan each cycle) is a runaway multiplier: each execution can drive fresh LLM calls. Storm detection used to be log-only; now a process-global, cross-executor budget (total, per-automation and per-row executions/window) skips the execution once a storm blows past it, and the cluster guard's fail-open path is bounded (it admits a capped number of unguarded executions per window during a DB outage, then fails closed).

#Layer 3 — loop terminal-condition audit

Every LLM-driving loop already carries a per-turn terminal condition; the residual gap is cross-turn cumulative spend, which Layers 0 and 4 close.

Verdict: no loop lacks a per-turn terminal condition. The agent / engine tool loops have no native cross-turn cap and rely on Layer 0 (process) and Layer 4 (per plan-lineage / work run) for cumulative bounding.

#Layer 4 — subscription spend, counted but not charged (memql#4362)

A Task delegated to a local app (epic memql#4358) spends real tokens that MemQL was not billed for. That breaks a single spend counter in a way worth being explicit about, because the two existing controls want opposite answers:

So the spend is split rather than summed:

• v1:work:run.spent.tokensSubscription accumulates covered tokens, separately from tokens. component/work/budget.go's CheckCeilings compares only the metered tokens against the ceiling.
• v1:router:call.billing (metered | subscription | unknown) and executionSurface carry it on the ledger, so a cost reader can separate "what we spent" from "what ran somewhere we do not pay for" without a join.

unknown is a real answer, not a rounding. An app that reports no usage gets unknown and zeroes — never "free", never "metered by omission". The number this whole layer exists to produce (what the subscription covered) is only trustworthy if silence stays visible as silence.

An executor that reports NO billing is treated as metered. Unattributed spend counts against the ceiling rather than vanishing into the covered bucket, where it would be invisible to the one control that stops runaway cost. That is the fail-safe direction, and it is why ExecutorResult.EffectiveBilling() defaults the empty string to metered rather than to unknown.

#Layer 4b — the ceiling gates the FEDERATION HOP (memql#5096)

The shipped policies changed shape in epic memql#5096: every one begins fleet:*, then app:*, then the vendor entries. That makes falling back to a paid provider a distinct, nameable moment in the chain walk — and it is the moment the cumulative ceiling is consulted.

Three properties are worth stating because each is a mistake somebody would otherwise make:

• It is a read. Nothing has been spent at the moment it is asked, and a probe that incremented a tally would make merely CONSIDERING the cloud count against the ceiling — so a run that ended up on a local model anyway would still have moved the process closer to its cap.
• It asks about the scopes the call would be charged to, read off the same context the charge would use. Asking only the process-wide budget would let a per-scope latch be discovered one call too late.
• A chain that STARTS at a vendor is untouched. The ceiling governs falling back to paid inference, not choosing it; refusing there would break every cloud-quality policy in the tree.

#A ceiling refusal is not a park

They look alike from a distance — both stop the work — and they are different answers with different resumes:

Both park the run as a v1:work:approval of kind inferenceUnavailable, naming every door tried and why each did not open. That approval is where a person says "use a paid provider for this run", and it replaces the retired per-plan cloudApproved metric — a field on a concept (v1:planner:plan) that went with memql#5000.

#Environment reference

#Layer 0 — kill-switch

#Layer 1 — rate ceiling + loop breaker

#Layer 2 — automation budget

#Layer 3 — loop caps

#Layer 4 — per-scope budget

The per-scope latch is on by default (unlike the process caps): a scope is one plan-lineage or one work run, so a runaway within one scope is unambiguous and latching it kills just that loop — other plans are unaffected and no restart is needed. Caps sit above the 120-iteration per-turn cap so a single deep turn never trips them. Scopes are stamped via ContextWithBudgetScope at the streaming + non-streaming agent loops, the work run's execution context (app/work_execution_context.go) and the compile pass (integrations/work/compile.go).

#Reproducing a runaway safely (local)

Set tight caps in the cluster env, then trigger any loop and confirm spend is hard-bounded and the loop terminates with a clear signal:

MEMQL_LLM_MAX_TOTAL_CALLS=3          # Layer 0 hard stop after 3 calls
MEMQL_TOOL_LOOP_MAX_REPEAT_FAILURES=1
MEMQL_MAX_AUTOMATION_EXECUTIONS_PER_WINDOW=20

The kill-switch alert LLM KILL-SWITCH LATCHED is the terminal signal; after it, every LLM call returns a 402 and no vendor request is made.

#AI-call caching (cost reduction, not just bounding)

The layers above stop a runaway. Caching is the orthogonal lever: it stops re-paying the LLM for an answer we already have. Two cache layers sit on the ai() / InvokeAI path, checked in order:

1. Exact-hash cache (component/memql/ai_cache.go, 5.8). Keys on sha256(templateId|provider|rendered-prompt). Cheap, byte-exact — so "thanks!" and "thanks" miss each other. On by default (MEMQL_SI_CACHE_DEFAULT_ENABLED, ceiling MEMQL_SI_CACHE_MAX_SECONDS).

2. Semantic (vector) cache (component/memql/ai_semantic_cache.go, 5.9). On an exact-hash miss, for an enabled classification namespace only, embeds the rendered prompt and looks up the nearest neighbour in the semantic_ai_cache pgvector table scoped to that namespace. Returns the cached result only when cosine similarity is at or above the namespace's threshold; otherwise it falls through to a fresh LLM call and stores the result back. This is what collapses "thanks" / "thx" / "thank you" onto one cache row.

ai() call
  └─ exact-hash cache  (byte-exact)            → hit? return
       └─ semantic cache (enabled namespace)    → similarity ≥ threshold? return
            └─ fresh LLM call → store into BOTH caches

#The wrong-answer guard (read before enabling a namespace)

A near-duplicate prompt does not in general imply an identical correct answer: "cancel the meeting" and "confirm the meeting" are lexically and semantically close yet demand opposite answers. So the semantic cache is:

• opt-in per namespace. A classification namespace is a stable input→label call type (failure symptom, complexity triage, domain-shape). The enablement registry (ai_semantic_cache_registry.go) is empty by default — shipping the layer changes no AI behaviour. NEVER enable it for free-form generation (agent prose replies): two prompts that "look similar" there can demand entirely different correct outputs.
• threshold-gated. Per-namespace cosine floor, conservative default 0.95. A neighbour below the floor is rejected and counted (SemanticCacheStats.BelowThreshold — the guard firing).

#Enabling a namespace (the path)

1. Confirm the call is a classification / structured-output call with a stable input→label mapping.
2. Run the false-positive eval (ai_semantic_cache_eval_test.go) over that namespace's near-duplicate pairs (should hit) and lexically-close-but-opposite pairs (must not hit). Pick the lowest threshold whose false-positive rate stays under the agreed bound (0% over the curated opposite corpus at 0.95).
3. Register it in defaultSemanticNamespaces() or set MEMQL_SI_SEMANTIC_CACHE_NAMESPACES=<ns>:<threshold>[:<ttlSeconds>] (comma- separated). Start above the eval-chosen threshold and tune down only with telemetry.

The strongest first-enablement candidate is the work spine's symptom classification (work.classify_symptom, the classifySymptom prompt) — structured output, one of five labels plus a reason, high redundancy across runs. It ships not enabled: turning it on changes a live healing decision path and is a deliberate, eval-gated, owner-signed-off follow-up, not a risk taken to satisfy acceptance criteria.