MemQL Deployment Console -- Operator Guide

The Deployment Console is the role-gated UI (see the role matrix) for driving the deployment-v2 machinery -- "what is deployed, is it healthy, and how do I deploy / promote / roll back" -- from a UI instead of a terminal.

MemQL ships one installation shape (epic memql#3943). Every RPC on this surface operates on THIS installation and takes no environment: an operator who wants a second environment installs a second instance, which has its own console at its own address.

It is a read + action surface over the existing machinery. It does not reimplement any of it: Git stays the single source of truth, Argo CD reconciles the cluster to the digest-pinned overlay, Argo Rollouts drives progressive delivery, and that one overlay pins the release as {engine version, bundle digest, client digest}. This guide is the console-driven workflow; the underlying mechanics live in the deployment-v2 docs cross-linked at the bottom and are not duplicated here.

#The surface

The VS Code Deployments panel is the only deploy-control surface. MemQL OS has none, and the Cockpit's Topology view and its D menu went with the Cockpit's terminal UI on 2026-08-25 (memql#4550); the Cockpit is now the machine worker runtime and cluster CLI. Everything below about the SERVICE, its gates, its refusals and its audit trail holds for any caller.

How a browser client would reach the deploy surface, recorded because the forward it needs is still built and still tested. DeployControlService runs shell scripts against an on-disk overlay checkout, so it exists only on the identity node. A hosted console is served by the edge and dials the origin that served it (a deliberate property -- see memql-os.md) -- but "the origin that served it" names where the BUNDLE came from, not where its gRPC/WS stream terminates: that stream still reaches a bff (through the edge's own /_memql/* proxy, memql#3712), the same node type it always did. Until memql#3380 that mismatch was the whole story: history rendered (it is ordinary v1:cluster:deployment concept rows on the normal query surface) and live state plus every control came back UNIMPLEMENTED. A bff now forwards the deploy RPCs to the identity node over NodeService.Stream, the way the AI surfaces forward to their workers. The caller travels with the call as a typed, verified authority assertion and the receiving node resolves the actor from that and nothing else -- not from the forwarding node's own credential -- so the role matrix below is the same one whether you dialled identity directly or went through a bff. In particular rollback stays owner-only across the hop. The server-rendered /admin/deployments page that covered the gap is retired with the fix (memql#3380). /admin/ now answers 410 Gone and points at the console.

The panel's deploy-control actions call the role-gated deploy-control API (MemQL DeployControlService); none of them shells out to kubectl / argocd / git. The same Deployments view also manages a local cluster on this machine -- install, repair, rebuild, uninstall -- and those flows do run git and the install graph's scripts, which use kubectl. They are not deploy control and never reach this service.

The panel HIDES an action the caller's role cannot take (editors/vscode/src/deploy/actions.ts), so an admin is not offered a rollback that would come back PermissionDenied. That is a courtesy, not a control: the gate is the service's, and it applies identically however the RPC arrived.

#Two transports, one service (memql#3311)

DeployControlService is a unary gRPC service mounted on the same listener as MemqlService. A native gRPC client (the Go SDK, grpcurl) dials it directly. A browser cannot -- and neither can anything else reaching MemQL through the /memql/ws WebSocket bridge, which tunnels MemqlService.Stream and nothing else.

So every deploy RPC is also reachable on the stream, as a DeployControlMsg envelope whose request oneof carries the service's own request messages verbatim (the reply is DeployControlResult). The TS SDK exposes it as @znasllc-io/memql-sdk-core/deploy; that is how the VS Code extension drives the console.

This is a transport, not a second implementation. The stream handler calls the identical service methods the unary path calls, so the role gate and the audit write are one code path -- the parity test in component/deploycontrol/stream_parity_test.go drives the real service through both transports, for every RPC against every role, and fails if either path answers differently. Denials arrive on the stream as an ordinary QueryError carrying the gRPC status code verbatim, so a caller below the role floor sees PermissionDenied on either transport. Actions return their audit event id identically -- and so do refusals (memql#3334), by different mechanisms that carry the same value: a RefusalInfo status detail on the unary path, the DeployControlResult.audit_event_id field on the streamed one, because a multiplexed stream has no per-message status to hang a detail off.

Deployment history is deliberately not bridged: v1:cluster:deployment rows are ordinary concept rows, read with a normal query.

#Reaching the API from a WebSocket client (memql#3311)

DeployControlService is a separate unary gRPC service mounted on the same listener. That is dialable from Go (sdk/go's DeployControlClient) and from grpcurl, but not from a browser and not from any WebSocket client -- so the VS Code extension and the MemQL OS, which both speak /memql/ws, could not reach the deploy surface at all. (Identity's own server-rendered pages only sidestepped this by not being browser clients.)

Every RPC is therefore also reachable over MemqlService.Stream as a single bridged envelope pair, DeployControlMsg / DeployControlResult, whose inner oneof carries the DeployControlService request messages verbatim. sdk/ts exposes it as DeployControlClient (@znasllc-io/memql-sdk-core/deploy) with the same eight methods as the Go client.

Two properties an operator should know:

• The gate is the same gate, not a second copy of it. The stream handler stamps the caller's identity and invokes the same service methods the unary path serves, so the role matrix below and the audit event hold identically on both transports -- and a parity test asserts a denied role gets PermissionDenied from both, for every RPC.
• Errors ride inside the reply. A multiplexed stream has no per-message status channel, so a bridged refusal comes back as error_code (the canonical gRPC code, e.g. 7 = PERMISSION_DENIED) rather than as a stream error. sdk/ts turns that back into a thrown DeployControlError carrying the code.

Deployment history is deliberately not bridged: v1:cluster:deployment rows stay readable as ordinary concept rows through the normal query surface.

#The role matrix

The deploy-control API enforces three role floors, server-side, on every surface. This table is authoritative: it is read off the gate helpers in component/deploycontrol/service.go (authorize / authorizeDeploy / authorizeOwner), and the parity test in component/deploycontrol/stream_parity_test.go fails the build if the streamed path and the unary path ever answer a role differently.

writer and reader get nothing on this surface. Every RPC refuses them with PermissionDenied and writes a blocked audit event. So does an unauthenticated caller, with Unauthenticated. The role model itself is the cluster-wide one described in access-model.md.

#A refusal gives you its audit id (memql#3334)

A denial is an audited event, and the caller who was refused is told which one. The refusal carries the correlation id of the blocked v1:identity:auditEvent the gate wrote, so an operator who hit a floor has a reference to quote rather than a timestamp to argue from.

An empty id means no event was written, not a lost one. Only the role gate audits. An Unavailable never reached the service at all, and an InvalidArgument is a caller error the gate does not record -- so neither leaves a trail, and neither has an id to show. Note that an InvalidArgument from this service now means the caller cleared the role floor; a caller who did not never sees one (see below).

Why the refused caller sees it rather than only an admin: the id is an opaque token that dereferences nothing (the audit log is admin-gated), each attempt mints a fresh one, and the refusal message beside it already names both the required role and the caller's own -- so it discloses nothing new while making a real support conversation possible. This is the same call the identity-admin console makes for its own refusals (memql#3324). The reasoning is recorded in component/deploycontrol/refusal.go.

On the permitted path nothing changed: the id stays where it has always been, on ActionResult.audit_event_id.

#The gate runs before argument validation (memql#3457, memql#3505)

Every RPC on this surface -- all six actions and both reads -- checks the role floor first and validates its arguments only afterwards. They used to do it the other way round, and that left one hole in "every denied attempt is audited": a caller below the floor who sent a bad argument was rejected by the parser before the gate ever ran, so the attempt was never recorded.

This changes an error code you may observe. Nothing about the request or reply shape changed -- same fields, same audit_event_id, same RefusalInfo detail -- only which code comes back in one combination:

If you script against this surface, a below-floor caller sending a malformed request now gets PERMISSION_DENIED rather than INVALID_ARGUMENT. That is the honest answer -- the caller was never going to be allowed to run the action, whatever they typed -- and it is the safer one: an unauthorized caller no longer learns anything about the argument parser from probing it. The audit event records the argument as sent, invalid values included, because the shape of a probe is the interesting part of it.

Below the floor is a per-action fact. memql#3457 shipped the re-order for the write RPCs; memql#3505 finished the surface with the reads. Where it counts most:

• RollbackDeployment is owner-only, so admin sits below that floor and no other. Until #3505 an empty to_deployment_id was the one probe of this surface an admin could make without leaving a trail.
• GetDeploymentStatus and SuggestNextVersion now take no argument at all (their only one was the environment, removed by epic memql#3943), so there is nothing left to probe them with -- the gate is the only thing they answer to. Repair (memql#4209) is the same shape: its request is empty, because a repair operates on this installation and names no version.

A caller who cleared the floor sees exactly what they saw before, on every RPC: same code, same message, and still no audit event, because the gate records denials and not admissions.

#Reading status is owner/admin, deliberately (memql#3332)

An earlier version of this matrix said any role could view. That was never what the code did. GetDeploymentStatus has been gated to owner/admin since the service shipped (#728), and memql#3311 kept that gate when it bridged the service onto MemqlService.Stream rather than loosening a read gate on a deploy-control surface. memql#3332 resolved the discrepancy in favour of the code: the shipped gate is the contract, and this table now states it.

The read is admin-tier because of what it returns -- the version and image digests in force in an environment, Argo CD sync and health, the live Rollouts, and the deploy gate's legs. That is the cluster's release posture, not a status badge.

#Enforcement

• The VS Code Deployments panel: resolves your cluster role and hides an action you cannot take. That is a courtesy, not the control: the call still reaches the identity node and is gated there.
• API: every deploy-control RPC enforces its floor server-side, so the gate holds for a direct API caller -- below the floor you get PermissionDenied. This is one enforcement point, not one per surface: the WebSocket bridge (memql#3311) calls the same service methods, so it cannot admit anyone the unary service refuses, and the mesh forward (memql#3380) runs that same deploycontrol.Dispatch fan-out on the receiving node rather than a copy of it.
• Across the mesh: a forwarded call carries the ORIGINATING caller as a typed ForwardedAuthority (the memql#3205 contract), verified fail-closed on arrival. The receiving node never derives an actor from the forwarding node's own class="node" credential -- if it did, every caller who could reach a bff would arrive as the bff. Pinned by TestDeployControlForwardIgnoresTheForwardingNodesOwnCredential and by the per-role matrix in component/grpc/deploy_control_forward_test.go.

#What a non-admin sees: a deliberately split surface

The panel reads from two different places, and only one of them is deploy-control. This is by design, and it is why a developer/writer/reader sees a partly-populated page rather than an error page or an empty one:

Concept rows go through the normal query surface and never touch the deploy-control gate; the deploy actions do. So a developer opening the VS Code Deployments panel gets history and per-tier composition as usual, and in place of the deploy actions the line "Your role can't view deployment status." -- the engine's own refusal, which names the role required, is under Details. That is the designed behaviour of the surface, not a failure of it -- nothing is broken, and re-reading will not fix it.

The panel draws the deploy actions only when the status read answers (editors/vscode/src/deploy/pipelineState.ts), so it offers a developer none -- not even Prepare and Deploy, which the service admits at the developer floor. The upgrade button is the exception: it is gated on the role alone (editors/vscode/src/deploy/upgrade.ts), so a developer offered a newer release can cut and ship it in one confirmed step. Otherwise a developer cuts and deploys through DeployControlClient (cutVersion / deploy in @znasllc-io/memql-sdk-core/deploy, CutVersion / Deploy in sdk/go), or asks a cluster owner or admin for the role.

#Reading the console

The panel reads the cluster this editor is connected to. There is no env to scope by -- MemQL ships one installation shape (epic memql#3943); an operator who wants a second environment installs a second instance with its own console.

For a remote cluster it shows (editors/vscode/src/webview/deploymentScreens.ts):

• What it runs -- the version, beside the cluster's name, and where MemQL OS answers.
• The pipeline state, one of three and none of them an error (editors/vscode/src/deploy/pipelineState.ts):
  • Present -- the status read answered. The actions your role admits are drawn, each naming what it acts on before you press it (see Performing actions): Deploy and Roll back on the action bar, a Promote and an Abort on the row of each Rollout part-way through (under Rollouts), and the versions the cluster can be prepared at (under Next version). An action with nothing to act on is not drawn.
  • Deployments aren't set up for this cluster -- the read failed for a reason other than the role gate, most often because the cluster has no deploy pipeline at all. The engine's message is under Details, as sent.
  • Your role can't view deployment status -- the role gate refused the read. The engine's refusal is under Details, as sent.
• History -- one row per v1:cluster:deployment record: what kind of run, when, and how it ended. A row opens that deployment's page, which lists its per-tier v1:cluster:deploymentNodeSpec rows as "Services": each declares a version, replica count and digest rather than records a step that ran.
• Details (collapsed) -- the records behind the page (the release running, the one prepared, the one Roll back returns to), the last action's audit id, and the engine's own words for a read that failed.

The panel uses the status read to decide the pipeline state, and takes one thing from it: the name and phase of each Rollout, which Promote and Abort act on. It draws nothing else the RPC returns, and an SDK caller reads all of it (getDeploymentStatus() in @znasllc-io/memql-sdk-core/deploy, GetDeploymentStatus in sdk/go):

• Version + digests -- the deployed release version and the per-component image @sha256: digests, read from the committed cloud overlay (deploy/k8s/overlays/cloud under the deploy repo, MEMQL_DEPLOY_REPO_ROOT). That one overlay pins the release directly as {engine version, bundle digest, client digest} -- there is no separate per-version lockfile.
• Argo CD -- the memql Application's sync status (Synced / OutOfSync), health (Healthy / Progressing / Degraded), last sync, and drift (live-vs-desired).
• Rollouts -- per Rollout: BFF blue/green active vs preview color; engine canary current step / set-weight; and the latest AnalysisRun result (pass / fail).
• Gate -- the most recent deploy-gate AnalysisRun legs (the /readyz schema probe, the service_account-JWT authenticated query, SLO metrics, and the headless-browser tier) with pass/fail and a timestamp.

Reads are not audited per call.

#Performing actions

Each action is gated by the role matrix, audited, and none of them bypass Git or the reconciler:

The panel's four deploy-control actions are catalogued in editors/vscode/src/deploy/actions.ts, and editors/vscode/src/deploy/controls.ts turns each into the controls it draws, deciding every argument a request carries when the page is built; where each is drawn is editors/vscode/src/deploy/instanceActions.ts's. A press sends exactly what its control named; one whose target has moved since the page was read sends nothing and says so. Rollback and Repair have no panel control: call them through DeployControlClient -- Rollback / Repair in sdk/go, rollback() / repair() in @znasllc-io/memql-sdk-core/deploy.

A refusal the panel shows is the identity node's own PermissionDenied, carried back verbatim through the bff that forwarded the call; an UNAVAILABLE means that bff could not reach an identity peer, which is a cluster problem rather than a permissions one.

Notes:

• Confirmation. In the panel, Roll back asks you to type the target deployment's id back, and Abort the Rollout's name; a mismatch is rejected and the RPC is never sent. The confirmation is the panel's alone: the service takes no confirmation argument, so an SDK caller -- including every call to Rollback and Repair, which only an SDK caller reaches -- is not asked for one.
• Audit. Every action (and every denied attempt) writes a v1:identity:auditEvent (category admin, action deployment_console_<verb>, with the actor, target version / digest / rollout, and outcome). The console surfaces the audit-event id back to you on success (SUCCESS: <action> (audit <id>)); failures show ERROR: <message>, and a refusal shows the id of the blocked attempt alongside it. The VS Code panel says what happened in a sentence, writes that line to the MemQL output channel, and keeps the audit id under Details -- see A refusal gives you its audit id. No emojis.
• Nothing here pushes. Deploy and Roll back write deployment records and observe ArgoCD; neither edits the overlay. The deploy is the overlay change itself, landed on main through the normal review path and reconciled by ArgoCD. Rollback leaves its revert commit in the deploy checkout for review. Rollout promote/abort act on the live Rollout directly.

#Where audit events land

All console writes and denials append to the identity audit log (v1:identity:auditEvent), readable through the concept browser view. Deploy and rollback in particular are auditable after the fact: actor, target version / digest, and outcome.

A denial appears there with outcome = blocked and the correlation id the refused caller was shown, so an id quoted in a support thread resolves to the exact attempt. Resolving it is an admin read -- the caller holds a reference to their own denial, not access to the trail.

#When to drop to the terminal (break-glass)

The console is the day-to-day path. Drop to the terminal for anything the console does not cover -- suspending Argo auto-sync, emergency direct kubectl changes, or DR. Those procedures are owned by the deployment-v2 runbooks:

• Argo CD break-glass (suspend / resume auto-sync): deploy/argocd/README.md
• Rollouts promote / abort / watch, from an operator machine with the argo-rollouts kubectl plugin installed: kubectl argo rollouts promote|abort <name> -n memql and kubectl argo rollouts get rollout <name> -n memql --watch, against the controller deploy/rollouts/install installs. The console's Rollout action runs the same promote / abort verb only on the exec substrate (component/deploycontrol/executor.go); on an identity pod it refuses and points here (component/deploycontrol/k8sapi.go)
• Overlay digest-pin + promotion mechanics: deploy-bundle-runbook.md
• Disaster recovery: docs/internal/ops/dr-runbook.md

#Automation-driven deploy path (#2115)

The gRPC Deploy / RollbackDeployment actions are automation-driven kick-offs. The synchronous Go apply they once ran (select driver -> scripts/release/promote.sh -> inline in_progress -> succeeded | failed transition) was retired in #2115 step 6: it is no longer a code path, and there is no opt-in/opt-out flag.

How a deploy now flows:

1. The deploy pack (examples/deploypack) is always anchored on the identity binary (app/anchor_deploypack.go, a bootstrap phase after env layering and before the engine loads its DSL tree), so the driveDeploymentInProgress (E2.3) / recordReconciledState (E2.4) automations are loaded where the Deploy Console writes deployment records.
2. Deploy validates the record's provider and transitions it to in_progress, then returns. RollbackDeployment creates the new rollback record at in_progress (carrying the historical digest + previousDeploymentId), then returns.
3. The in_progress CDC edge triggers driveDeploymentInProgress, which observes ArgoCD's reconciled state (deployObserveReconciledState, examples/deploypack/dsl/logic.memql) and owns the terminal transition.

#Async response contract

The action RPCs are async: ok=true means "accepted and kicked off", NOT "deploy succeeded". The reply carries details.async="true" and details.status="in_progress" (Deploy) plus details.newDeploymentId (rollback). The terminal status is read back from the deployment concept (or GetDeploymentStatus) once the automation resolves it -- the console already polls deployment status, so it reflects succeeded / failed when the rollout settles.

Terminal-status parity: an automation-driven rollback lands in rolled_back (the driveDeploymentInProgress automation branches on previousDeploymentId, #2168); a forward deploy lands in succeeded when ArgoCD reports synced AND healthy, and in failed when it does not.

What the automation OBSERVES rather than runs (epic memql#3943). It used to fire an effect on the in_progress edge: pin one environment's overlay from another's digests. With one installation there is nothing to pin FROM -- the deploy IS the merge, which ArgoCD reconciles -- so the automation reads ArgoCD's reconciled state through observeReconciledState and resolves the record from that. This is strictly more evidence than the effect path had: it asserted success from a script's exit code.

The MEMQL_DEPLOY_AUTOMATION_DRIVEN env flag that gated this during the owner-supervised cutover (steps 1-5) is removed -- the path is unconditional. The cutover was verified live on 2026-06-24; step 6 (this retirement) lands the cleanup.

#Repair (memql#4209)

Repair is the vision's third verb beside update and restore, and the one a browser could not have until it existed on the wire: a cluster must not be shelled into from a browser console, so the console rendered an honest "not exposed from the cluster yet" line until the RPC landed.

#What a repair does

A repair re-converges this installation onto its committed overlay. The identity node asks ArgoCD to hard-refresh the installation's Application (re-fetch the manifests, discarding the repo-server's cached render) and then to run an explicit sync with prune, and it watches the Application until it reports synced AND healthy. Drifted or hand-edited resources are re-applied, deleted ones are recreated, and tracked resources Git no longer describes are pruned. Nothing changes version: a repair that installs a different version is an upgrade wearing a repair's name (memql#3605), so the request carries none.

The effect is the command deploy/argocd/README.md documents for an operator-triggered sync (kubectl -n argocd patch app memql --type merge -p '{"operation":{"sync":{}}}'), issued through the deploy-control Executor on the identity node and stamped with the repair's record id, never a direct kubectl apply.

It also clears the previous operation's selection first, and a repair is silently partial without that (memql#4871). ArgoCD's application controller persists a new operation by merge-patching status.operationState with omitempty: a sync that names no resources and no syncOptions omits both keys, and a merge patch that omits a key leaves the existing value. So after ANY selective sync on the Application -- one naming particular resources -- every later repair inherited that selection, reconciled only those resources, and reported success. Observed live on 2026-09-02: a repair that should have applied a new Role, RoleBinding and Ingress annotation ran as a ten-Deployment sync instead, twice, and no status.history entry was recorded because ArgoCD writes none for a selective sync -- so the timeline looked untouched while several bumps had in fact been applied.

Sending "resources": [] does not help; the empty slice is dropped by the same omitempty on the round trip. The fields have to be cleared on the status side, which the repair now does in the same atomic merge patch:

"status": {"operationState": {"operation": {"sync": {
    "prune": false, "resources": null, "syncOptions": null }}}}

An operator running the patch by hand needs the same reset. If you are diagnosing a sync that touched fewer resources than the render, read status.operationState.operation.sync before believing the sync was complete.

#Per provider

The provider is read off the graph -- the v1:cluster:cluster row, then the newest v1:cluster:deployment record -- never off an environment variable (epic memql#3943). The local cluster and the cloud reconcile through the same ArgoCD Application (environment parity), so both providers have the repair above; what differs is only what it undoes:

An installation whose graph says nothing about its provider takes the same default every other deployment record takes (azure).

#The owner floor, and what a refusal looks like

Repair is owner only -- not even admin, the RollbackDeployment floor rather than Deploy's: it mutates a running cluster with no Git revert trail. The gate runs first (there is no argument to validate after it), on every transport -- unary, the DeployControlMsg stream bridge, and the bff-to-identity mesh forward, where the originating human's role decides (component/grpc/deploy_control_forward_test.go pins that a forwarded admin is refused and audited as that admin, and a forwarded owner is admitted).

Every refusal carries its audit id back, by the mechanism that fits the refusal:

Both SDKs carry the id with the outcome either way.

#Honest progress: the repair record

ok = true means accepted and kicked off, not "repaired". The RPC writes a v1:cluster:deployment record at in_progress before the kick-off -- its notes start with repair: and it carries the version in force -- so the deployment timeline already shows the repair from its first instant (marked repair). A kick-off that fails lands the record at failed in the same call. Otherwise a watcher on the identity node resolves it from what it observes on the Application:

Until the controller has picked the repair's own operation up, the Application's previous Succeeded is not read as this repair's verdict -- the stamped operation id is what tells them apart. Poll the record (the timeline is live on the concept) or GetDeploymentStatus for the Argo sync / health it reports; there is no progress stream.

One repair runs at a time per identity node, and a sync already running on the Application -- whoever started it -- refuses a new repair until it finishes.

#References

• Automation-driven deploy cutover: znasllc-io/memql#2115 (Epic 2 follow-up; meta #2060).
• Deployment Console epic: znasllc-io/memql#724 (children #725-#729 + cockpit#144/#145).
• Deployment-v2 design + epic: docs/internal/design/deployment-v2.md, #697.
• Supervised live cutovers: #712.
• Cluster role model: docs/public/operate/auth/access-model.md.
• Deploy-control stream bridge: znasllc-io/memql#3311.
• Repair verb: znasllc-io/memql#4209 (the console gap it closes: #4193).
• Role-matrix reconciliation (the read gate is owner/admin, the code is authoritative): znasllc-io/memql#3332.
• Machine identity the gate uses: docs/public/operate/auth/service-account-jwt.md, #691.