Service-account JWTs (machine identity)

A service-account JWT is the machine identity for automation and synthetic checks — most importantly the in-cluster deploy gate (the Argo Rollouts AnalysisTemplate, deployment-v2 Phase 3). It authenticates to MemQL's MemqlService.Stream on the BFF / mesh surface, where a PAT cannot. Resolves #691; part of epic #697.

#Why this exists (the #691 problem)

The deep-smoke gate needs to run a real authenticated query against the BFF (/memql/ws → MemqlService.Stream → agent) to prove the authenticated app path actually works. The obvious credential — a PAT (mql_pat_…) — does not work there:

  PAT verification is wired ONLY on the identity node.
  Every other node's per-node verifier is built with a nil PATVerifier:
 
      app/config.go:  verifier.New(cfg, cache, /* PATVerifier */ nil, logger)
                                                 └── deliberate: keeps token
                                                     verification DB-free on
                                                     the hot path.
 
  So:  PAT ──► identity node      = OK (DB lookup of the PAT hash)
       PAT ──► BFF / agent…       = 401  "PAT path not wired on this node"

A JWT, by contrast, is verified by the JWKS path that every node already runs (signature + claims, no DB). So the fix is a JWT-shaped machine credential — not wiring PAT verification onto every API node (which would add a DB round-trip to the hot auth path and broaden where a long-lived static user credential is honored). PATs stay the human-CLI credential; service-account JWTs are the machine credential.

#The credential-class family

MemQL stamps a class claim on identity-issued JWTs; each class is admitted by a dedicated interceptor that pins the surface it may use. Service-account joins node (#105) as a machine class.

                         identity service (Ed25519 signing key, JWKS published)
                                          │ mints
   ┌───────────────┬───────────────┬──────┴───────────────┐
   │ class=user    │ class=node    │ class=service_account│
   │ (humans)      │ (#105)        │ (#691, THIS doc)     │
   ▼               ▼               ▼
 full app        NodeService.    MemqlService.Stream
 surface         Stream only     pinned to the
 (per-row authz) (mesh)          read/query surface
 
   PAT (mql_pat_) ─ NOT a JWT ─► verifies ONLY on the identity node (DB lookup)
 
   Verify path for ALL JWT classes = the per-node JWKS verifier (NO DB):
        Authorization: Bearer <eyJ…>  ──►  verifier.VerifyBearer()  ──►  class

#Token shape

A service-account JWT is a regular identity-issued EdDSA JWT with:

Signed with the same EdDSA key as user JWTs, so the per-node verifier validates it through the same JWKS endpoint. No v1:identity:identity row is persisted (unlike node): the verify path is DB-free and the token is short-lived, so revoke = expiry / signing-key rotation.

#Minting

memql service-account-token mint (on the identity binary, subcommand_service_account_token.go, build tag identity).

  operator / CI                       identity binary (memql, -tags identity)
       │                                          │
       │  memql service-account-token mint \      │
       │     --label deploy-gate-staging \        │
       │     --subject system:deploy-gate \       │
       │     [--ttl 1h] [--out /path]             │
       ├─────────────────────────────────────────►│
       │                          1. ApplyLocalOverride(".")  (/.env if present)
       │                          2. app.Build + start config+db+engine+identity
       │                             (servers SKIPPED — no port bind; safe to
       │                              `docker exec` into the running identity pod)
       │                          3. svc.Issuer().IssueServiceAccountAccessToken(
       │                                 {Subject, Label, TTL})  — signs EdDSA
       │                          4. print JWT to stdout (or --out, mode 0600)
       │◄─────────────────────────────────────────┤
       │  eyJhbGciOiJFZERTQSIsImtpZCI6…            │  (stderr carries the logs;
       │                                          │   stdout is JUST the token so
       │                                          │   `TOKEN=$(… mint …)` is clean)

Flags:

The mint is unauthenticated by design — the operator is docker exec-ing the identity binary, which already holds the signing key. It binds no ports (selectMintDependencies drops the server deps), so it is safe to run inside the live identity pod.

#Verification + surface pinning (BFF / mesh)

On every API-serving node the interceptor chain (app/transport.go) now ends:

  base (JWKS verifier) → sessionRevocation → operatorAware
       → serviceAccount → panicRecovery → handler

The service-account interceptor (component/grpc/service_account_stream_interceptor.go):

   Authorization: Bearer <jwt>
            │
            ▼
   ┌──────────────────────────────────────────────────────────┐
   │ reserved prefix (mql_pat_ / mql_wkr_)?  ── yes ──► base    │  (not a JWT)
   │ scheme != Bearer / empty / v==nil?      ── yes ──► base    │
   └──────────────────────────────┬───────────────────────────┘
                                   │ no
                                   ▼
                 v.VerifyBearer(token)   (JWKS — NO DB)
                                   │
              class == "service_account" && Source == JWT ?
                  │ no ──► base (user JWT for another surface)
                  │ yes
                  ▼
        admit + stamp system actor (role=system, class, sub, label)
                  │
                  ▼  per inbound MemqlClientMessage:
        ┌─────────────────────────────────────────────────────────┐
        │ isServiceAccountPayload(payload)?                         │
        │   ALLOW: ClientHello, Ack, Unsubscribe, CancelRequest,    │
        │          ExecuteQuery, Subscribe, ConceptsList,           │
        │          ConceptsSubscribe, MyAccess, AgentGenerateTurn   │
        │   DENY (PermissionDenied): IdentityCreate/Update,         │
        │          CreateWorkerToken, DelegationCreate, RotateAuth, │
        │          Revoke*, … every credential/ admin mutation +    │
        │          other classes' message types                     │
        └─────────────────────────────────────────────────────────┘

Two independent containments: (1) the message-type allowlist (a leaked synthetic credential can run reads + one agent turn, never a credential mutation), and (2) per-row authz still applies to whatever ExecuteQuery runs. The allowlist — not the actor role — is the primary blast-radius bound.

#Use by the deploy gate (deployment-v2 Phase 3)

   Argo Rollout step ──► AnalysisTemplate (Job, IN-CLUSTER)
        initContainer `mint` ──► best-effort self-mint a FRESH token (#2179)
            execs `service-account-token mint` in the identity pod,
            writes /gate/svc-jwt; on ANY failure exits 0 (no file)
        │
        ▼  token resolution (#2179):  MEMQL_SVC_JWT_FILE (/gate/svc-jwt, fresh)
        │                              > MEMQL_SVC_JWT (static Secret fallback)
        ▼  authenticated WS/gRPC, IN-CLUSTER (no public host, no firewall dep)
   bff:50058 (MemqlService.Stream)
        │  serviceAccount interceptor admits class=service_account, pins surface
        ▼
   ExecuteQuery / AgentGenerateTurn ──► agent   (proves the app path)
        │
        ▼  assert result + SLO metrics → pass/FAIL → Rollout promote/auto-abort

Because the check runs inside the mesh against service DNS, it is immune to the public-host routing (#680) and rolling-convergence (#682) failure modes the old shell smoke hit, and needs no firewall exception.

Self-minting gate + static-secret fallback (#2179). The gate Job no longer depends on the static deploy-gate-jwt Secret being fresh. A best-effort mint init container finds the running identity pod and execs service-account-token mint --ttl 3h, writing the fresh token to a shared emptyDir at /gate/svc-jwt; deploy-gate-check reads it via MEMQL_SVC_JWT_FILE (precedence: --jwt flag > --jwt-file/$MEMQL_SVC_JWT_FILE file, if present and non-empty > $MEMQL_SVC_JWT). On any mint failure (no identity pod, mint error, non-JWT output) the init container exits 0 without writing the file, and the gate falls back to the static MEMQL_SVC_JWT secret. This is strictly safer than the static-only path: best case a fresh mint, worst case identical to before. The init container reuses the deploy-gate-jwt-minter ServiceAccount's RBAC (get/list pods, create pods/exec, read the Secret). The static secret going stale — minted, then rejected Unauthenticated — was the #2179 root cause that aborted every bff rollout on the gate.

#Provisioning into the cluster

The gate self-mints a fresh token at gate time (#2179, above); the static deploy-gate-jwt Secret (consumed as MEMQL_SVC_JWT) is the fallback for when the self-mint can't run. Seed/refresh it the same way:

TOKEN="$(kubectl -n memql exec deploy/identity -- \
          memql service-account-token mint --label deploy-gate-staging --ttl 6h)"
kubectl -n memql create secret generic deploy-gate-jwt \
  --from-literal=MEMQL_SVC_JWT="$TOKEN" --dry-run=client -o yaml | kubectl apply -f -

The identity-side deploy-gate-jwt-mint CronJob (deploy-gate-jwt.yaml, shipped with the deployment that runs the gate; this repository's deploy/k8s carries no copy) re-mints the fallback every 45m with a wide 6h TTL, so a missed/failed mint can't strand the gate before the next run. The token never lives in git and never leaves the cluster.

#Rotation & revocation

• Rotate: mint a fresh token (and refresh the Secret). The 1-hour TTL means a stale token self-expires fast.
• Revoke all at once: rotate the identity signing key (re-seal the seed onto the memql-secrets Secret and roll identity, per identity-service.md) — this invalidates every outstanding JWT of every class, so use it only for a true key-compromise.
• There is no per-token revoke list (no DB row by design). If you need individual revoke semantics, prefer a PAT on the identity surface instead.

Why this class did NOT get the row-state kill switch memql#349 added to node. That gate works by reading the credential's v1:identity:identity row, and a service-account subject is explicitly not required to name one — the verify path is JWKS-only by design. There is no row whose state could be read, so the same gate would fail open on every call. What stands in for it is the TTL: 1 hour (DefaultServiceAccountTokenTTLSeconds) against the node class's 30 days. That two-order-of-magnitude difference is the whole reason one class needed a kill switch and this one does not — a leaked service-account token is dead by lunchtime; a leaked node token was live for a month.

#Class comparison

#Code map

See also node-jwt.md (the sibling machine class), identity-service.md (signing-key rotation), and threat-model.md.