docsOperate

Local apps as execution surfaces

Delegating a planner Task to an app the user already pays for, running on a machine they own, with MemQL's tools reachable from inside that app over MCP. Epic memql#4358; design record docs/superpowers/specs/2026-08-22-local-apps-as-execution-surfaces-design.md.

The engine drives exactly two apps today — Claude Code (claude-code) and Codex (codex) — and that set is closed in the engine on purpose. See Why the app set is closed.


The shape of it

text
planner Task the user's machine
executionSurface=containerExecutor
executorBackend=cockpit-app:claude-code
│
▼
cockpit-app backend ──── consent gates ────► refused, with a reason
(agent node) naming which gate
│
│ AppSessionStart {credential, mcpEndpoint, workspace, prompt}
▼
WorkerService.Stream ═══════════════════════► memql (the Cockpit)
│ │
│ ◄══ AppSessionChunk (stdout/stderr/event) ══ │ runs `claude -p`
│ ═══ AppSessionControl (cancel / renew) ════► │
│ ◄══ AppSessionEnd {exitCode, usage, ...} ═══ │
▼ │
v1:worker:appSession │ MCP over HTTPS
v1:router:call (billing=subscription) ▼
mcp.<domain>/mcp ◄── the app,
acting as the user

The engine cannot dial the machine. It is behind NAT. The stream the cockpit opened outward is the only channel, so it is the transport, and MCP is the back-channel the app uses to reach back in.

The cockpit installs as the memql command (memql#4553). The engine builds a bin/memql too, but that one only ever runs inside a pod; these two never share a PATH.


Why the app set is closed

claude-code and codex are the only ids the engine will open a session for. A cockpit may report others — they are stored on v1:worker:registration.apps and are never given a routing label, so they are visible in the console and unroutable.

This is the direction that fails safe. A cockpit ships on its own cadence; if an unknown id could be driven, a newer cockpit reporting a third app would make the engine attempt a protocol it does not have, at dispatch time, inside somebody's plan. Growing the set is a value change in component/worker/apps.go — not a wire change.

Why a machine must be BOTH allowed and signed in

An app:<id> routing label is derived only from an entry that is:

allowedthe machine's own policy.yaml apps.allow lists it
signedInthe app's auth state, as the cockpit can observe it

Either one false means no label, which means the router cannot select that machine. The alternative — selecting on presence and discovering the refusal on the far side — commits a plan to a machine that then rejects it, and the resulting failure names the wrong thing.

Signing into an app takes effect on the next heartbeat, not the next reconnect: an inventory change is applied to the live registry immediately and persisted outside the 60-second lastSeenAt throttle, because it is a routing change. Persisting it there is also what lets a planner node answer "is a machine with this app online" — that node holds no worker streams at all, so the row and its derived labels are the only thing it can read.

Which machine, and on which replica

Selection is the Fleet router (epic memql#4349), asked for the app:<id> label as a requirement. It applies the owner's routing policy, orders by their chosen strategy, and knows which replica holds each machine's stream. Nothing in this feature picks between machines itself — a second selector would disagree with the first, and the plan would commit to a machine that then refused.

A session runs only on the replica holding that machine's stream. The tool-dispatch path can forward across nodes (WorkerForward, memql#4352); the app-session envelope cannot yet. A machine attached to a different agent replica is therefore skipped during selection rather than failing the run — which makes it a routing outcome instead of an error, and is why the refusal says "on a stream this replica holds".

A cockpit that reports no apps at all is normal — that is what a build older than the app protocol does, and what a machine with neither binary on PATH does. It is not an error state.


Sessions, not dispatches

A ToolDispatch carries one timeout and returns one result. A headless claude -p can run for an hour and emits output the whole way, so an app run is a session:

MessageDirectionCarries
AppSessionStartserver → workerapp, kind, prompt, inputs, workspace, credential, mcpEndpoint, limits, responseSchema, level
AppSessionChunkworker → serverstream (stdout / stderr / event), data, seq
AppSessionControlserver → workercancel or renew_credential
AppSessionEndworker → serverexitCode, usage, appSessionRef, producedArtifactIds, result, model, effort

The level goes out; the model comes back. level is one of the closed four (fast, strong, reasoning, embeddings) and says how much intelligence the run needs rather than which model serves it. THE COCKPIT owns the translation into the app's own knobs -- Claude Code's --model and --effort, Codex's model flag and reasoning-effort key -- from one table per app the machine's owner may override in policy.yaml, because the knob names are the app's and only the machine knows which app, at which version, is installed. An EMPTY level is what every session had before the field: the app runs at its own defaults. embeddings is always refused -- an embedding must come from the embedder of the index it is written into, and no app exposes one.

model and effort on the end are what the app REPORTED, never what was asked for. Empty means it did not say, which is recorded as unknown: a report copied from the request would record as measured something nobody measured. Claude Code's headless output states no effort at all, so an empty effort is the ordinary case.

Three kinds:

  • run — headless and autonomous. The engine reads the output.
  • open — launch the app for the HUMAN with the workspace and prompt loaded. Ends when the window closes, or immediately with a failure if the app cannot be opened.
  • attach — stream a run the human started, named by app_session_ref.

A tool-needing call resolved to an app door now opens a run session (epic memql#5391, design D7). It used to be passed over: MemQL drives a tool loop and an app is an agent that drives itself, so the app could not serve the TURN -- which meant that on a cluster whose only open door is a signed-in Claude Code, Ask stayed dark. The app does not serve the turn; it takes the whole STEP, drives its own loop, reaches MemQL's tools back over MCP, and answers once. The step records a childRunId naming the subrun, and a call carrying no step is REFUSED at resolution rather than falling through to the vendor behind it. run is still the only kind anything initiates today. The protocol carries all three and the runner accepts all three; open and attach have no engine-side caller yet, because a planner Task is autonomous by definition — they are for a console hand-off and a resume, neither of which exists. Said plainly here rather than implied, because the same section of CLAUDE.md spent two years describing a coding agent nothing ran. The cockpit half implements all three (memql-cockpit#347 / #350); the engine-side initiator is the missing piece.

Chunk seq is monotonic, and out-of-order or duplicate chunks are dropped. A transcript is a record; silently interleaving a replayed chunk corrupts it in a way no later reader can detect.

A session is RECORDED as rows of the work spine (epic memql#5396). The stream on each chunk decides where it goes: an event is a completed action the cockpit normalized -- a command, a file read or write, a fetch, a call back into MemQL -- and becomes one v1:work:step plus one v1:work:observation in the session's recording run, which the row names as sessionRunId. stdout and stderr are the model's prose and become one content-addressed v1:library:file at end, named as transcriptFileId. The row used to carry a flattened 256 KiB transcript string with the stream and the sequence discarded; transcript and transcriptBytes are retired.

Three things an operator reads off that:

  • recordedSteps and droppedActions. A hole in the action sequence is recorded as a note observation on the run AND counted on the row, because anything lifted from an incomplete recording is incomplete and a silent gap lets that happen unnoticed. A refused repeat is not counted: the first copy was recorded.
  • File contents are content-addressed. Two identical writes are one Library file referenced twice, so a later branch from a recorded step costs no copy. A content above the per-file cap is referenced BY DIGEST with contentOmitted recorded, and a Library failure never fails the session.
  • An app's MCP calls are recorded by the MCP node, not by the replica running the session -- the harness sees an opaque result and the session's replica never sees the call at all. submit and next_task are excluded: they are the session's protocol rather than its work.

A node with no recorder wired runs the session and records nothing, exactly as it did before; sessionRunId is then empty, which is not the same answer as a session that did nothing.

A caller context that dies cancels the run on the machine. Otherwise a plan that was cancelled leaves a headless agent working on somebody's laptop. A worker disconnect ends every live session with a named error rather than leaving callers parked until their own deadlines expire.


The back-channel credential

At session start the engine mints a class="app_session" JWT and sends it with mcpEndpoint = https://mcp.<domain>/mcp. The cockpit writes the app's MCP configuration naming both, runs the app, and deletes that file at end.

Its sub is the owning user's id. That single choice is the security story: the app reads over MCP as that user, so row authz applies to it exactly as it does to their browser. The machine-credential interceptor pins the surface to read/query and stamps role=system rather than owner, so the credential reaches no credential mutation and no cluster-owner gate.

Why it has a class of its own (memql#4857)

It was minted as class="service_account" until then, and the collapse cost a capability the design had already granted on paper.

Every other machine class names a machine: a node token's subject is a binary, a service account's is a CI job. This one's subject is a person. component/server/http_access.go refuses to resolve an HTTP actor for a machine class, and it is right to — the Library's byte routes stamp ownership from actor.userId, so resolving one for a binary would invent somebody to own the bytes. With one class name covering both kinds of subject, that rule could not be stated for one and not the other, and the cockpit's Library.Pull and Library.Push 401'd against the user's own rows.

With the class separated, the rule is sayable and narrow:

Surfaceservice_accountapp_session
gRPC MemqlService.Streamread/query pinthe same pin
POST /artifacts, GET /artifacts/{id}/content, the chunked session familyrefusedadmitted, as the user it names
POST /sites/{id}/bundlesadmittedrefused

Over HTTP the token carries no role claim, so the actor resolves to reader plus the real user id. That is enough for the byte routes — they gate on the actor resolving to a USER and never on a role — and short of every admin gate. Nothing else about the credential widened.

It is not revocable, and this doc will not pretend otherwise

The verify path is JWKS-only and DB-free — that is what lets it work on every node without a lookup — so there is no row to strike. Revoking one issued token means rotating the cluster's signing key, which invalidates every other token in the cluster. That is not a per-session operation.

Three things stand in for revocation:

  1. The lifetime is short and hard-capped at 8 hours by the identity service, whatever the delegation policy asks for.
  2. The cockpit deletes the MCP configuration file at session end, which is where the bearer actually lives.
  3. A run that outlives its credential is handed a replacement (AppSessionControl{renew_credential}) rather than a longer-lived bearer up front, so no single bearer is ever long-lived.

Residual exposure: the unexpired remainder of one short-lived, user-scoped, read-surface-pinned bearer. That is the trade the DB-free verify path buys.

The mint widens the bootstrap secret

POST /node/bootstrap with tokenClass="app_session" is the mint surface — the request field and the resulting class claim now agree, which they did not before memql#4857. Before this path existed, that secret bought only the machine principal node; it now also buys a user-scoped one. Narrowing it:

  • the named user must exist (a forged subject would verify and then act as a user nobody can point at in an audit);
  • the TTL is clamped at the identity service;
  • the surface stays read/query-pinned;
  • the session id is the token label, so a leaked bearer traces back to the run it was handed to.

An operator who does not want this path at all leaves MEMQL_NODE_BOOTSTRAP_TOKEN unset, which darkens the whole endpoint exactly as it does today.


An app run edits files and runs commands on somebody's own computer. It gets exactly the gates workerHost gets — the backend calls the same preDispatchCheck function, in the same package, rather than a copy:

  1. Per-task approval — the Task must carry a PlanId from an approved scope-elevation Plan.
  2. The kill switch — v1:identity:user.preferences.computerUseEnabled.
  3. Standing scope — the agent's agentAuthorization.computerUseScope must be full; an app run does shell exec and file writes.
  4. The safety classifier, fail-closed on this surface.
  5. Plus apps.allow on the machine itself, which the cockpit enforces and the routing label reflects.

A refusal names which gate refused it.


Delegation is a preference with a fallback

v1:worker:delegationPolicy, one row per user. MemQL OS → Fleet → Apps edits the caller's delegation preferences and shows app sessions. It reads the policy on demand, reports when it was read, and saves the complete form. Opening the editor does not enable delegation. Fleet → Machines also shows apps reported by a machine.

FieldMeaning
preferSubscriptionAppsthe master switch. False (the default, and the state of an absent row) means never delegate
eligibleKindstask kinds that may be delegated. An empty list allows nothing — opting in does not opt every kind in with it
appOrderwhich apps to try, in order. An app not listed is never selected even on a machine that has it
maxConcurrentSessionslive sessions across every machine. 0 reads as the default of 1, never as "none"
workspaceRootwhere per-run directories are created. The cockpit still gets to veto a path outside its own roots
credentialLifetimeSecondsdefault 4h, clamped to 8h at the mint

If no machine with an allowed, signed-in app is online, the task runs in-process. A plan never waits for a laptop to wake up: a delegation design that can block turns "my laptop was asleep" into "my plan hung", which is worse than the cost it was trying to save.

task.delegationReason records the outcome on both branches:

ReasonMeans
delegateda machine was found
delegation_not_enabledthe master switch is off
kind_not_eligiblethe kind is not in eligibleKinds
no_apps_configuredappOrder is empty
no_machine_with_app_onlinenothing online has an allowed, signed-in app from appOrder
max_concurrent_sessions_reachedthe cap
task_has_no_ownera machine-touching backend cannot run unattributed

The reasons are checked cheapest-first so the recorded one points at the real cause: somebody with delegation switched off is told that, rather than "no machine online" — which would send them to check a laptop that was never going to be consulted.


Accounting

Every session writes one v1:router:call row, including failures: a run that burned an hour of somebody's subscription and then crashed still spent it, and a cost surface that records only successes understates what work cost.

  • billing ∈ metered | subscription | unknown
  • executionSurface = cockpit-app:<appId>
  • door = app for a chat turn an app answered, session for a whole STEP an app was handed and drove its own loop through (they are different answers, so they are different values)
  • servedModel / servedEffort = what the app reported, empty when it said nothing
  • plan.tokenSpentSubscription accumulates the covered spend separately from plan.tokenSpent

The dollar ceiling excludes subscription tokens; the loop caps include the call. The two caps want opposite answers, so they are counted in different places. Counting covered tokens against the ceiling would park a plan over money nobody was charged — and the more a user leans on the subscription they already pay for, the sooner their plans would stop, which is backwards. A cap blind to those calls, meanwhile, is a hole the cheapest path walks straight through.

Everything a session produced carries producedBy. {app, model, effort, sessionId} is stamped on each v1:library:artifact the session pushed -- the transcript included -- and on the backing v1:library:file row, by the session runner at end. model and effort are the app's own report and may be empty; sessionId is the v1:worker:appSession row the run is readable at. A lifted construct's source reads this stamp, which is what lets its provenance say "recorded from claude-code, model X, effort Y, session Z".

It is ABSENT on anything a person uploaded, and the absence is the answer. A stamp naming no app would read as "an app produced this and reported nothing", which is a different fact from "nobody delegated this". The upload route physically cannot write one: its params struct has no field for it, and the two stamping mutations are @serverOnly. Caller-scoping is not the alternative there -- the party who would write a false stamp is the row's own OWNER, and actor.userId answers whose row it is rather than whether the claim on it is true.

Cost is never synthesised. MemQL knows its own providers' per-token prices, not the prices inside somebody's subscription. totalCost carries what the app reported (claude --output-format json gives total_cost_usd) and nothing when it reported nothing. billing falls to unknown whenever the app's usage report or the machine's subscription signal is silent — the number the owner asked for is only worth having if silence stays visible as silence.


Operating it

shell
# Is the machine reporting apps at all?
psql "$MEMQL_DATABASE_DSN" -c \
"select payload->>'name', payload->'apps' from memory_nodes
where concept='v1:worker:registration' order by created_at desc limit 5;"
 
# Which sessions are live right now?
psql "$MEMQL_DATABASE_DSN" -c \
"select id, payload->>'app', payload->>'status', payload->>'billing'
from memory_nodes where concept='v1:worker:appSession'
and payload->>'status' in ('starting','running');"
 
# Did the agent node install the executor?
kubectl logs -n memql deploy/agent | grep 'cockpit-app container executor installed'

Symptoms

What you seeWhat it means
cockpit-app: the backend is registered but not wired on this nodethe Task reached a node with no WorkerService. Only an agent node serves one
cockpit-app: no credential minter configuredMEMQL_IDENTITY_VERIFIER_BASE_URL or MEMQL_NODE_BOOTSTRAP_TOKEN is unset on the agent. The run is REFUSED rather than started with a blank bearer — an app with no credential reaches nothing over MCP and reports that as "MemQL's tools are broken"
no machine online with claude-code allowed and signed incheck Fleet -> Machines page; the badge says which half is missing
executorBackend "..." is not registered at task creationthe name was validated against RegisteredExecutors(). With an empty registry the message says so
A session stuck in startingthe node holding it died. The row is the record; a live session cannot survive its node

Environment

No new variables. The path reuses MEMQL_IDENTITY_VERIFIER_BASE_URL, MEMQL_NODE_BOOTSTRAP_TOKEN and MEMQL_MCP_PUBLIC_URL.


The cockpit half

Filed in memql-cockpit against memql#4364: app detection (claude / codex on PATH, versions, auth state), apps.allow in policy.yaml, the session runner (process supervision, streaming, cancel, resume by app session id), the MCP config writer and its cleanup, the Library pull/push with the session credential, and the open kind per platform.