Identity Service (Operator Guide)

component/identity is the in-house authentication provider for the cluster. It runs as its own node-type binary (go build -tags identity . or make identity) and owns:

• The public auth web pages (login, registration, magic-link, legal docs, /me/* self-service).
• The OAuth-style token endpoints (/oauth/token, /auth/refresh).
• What is left of the admin web app at /admin/*: the sign-in pages, and an /admin/ root that answers 410 Gone. Six of its seven screens (users, tokens, audit, JWKS, cluster settings, and the dashboard) now live in MemQL OS (memql#3324, epic memql#4984), writes and owner/admin gate together -- see ../memql-os.md. Deployments left in memql#3380 and are driven from the VS Code Deployments panel (../deployment-console.md): DeployControlService still runs against an on-disk overlay checkout and therefore exists only on this node, but a bff forwards the deploy RPCs here over NodeService.Stream, carrying the caller's authority so the owner-only rollback gate is unchanged by the hop.
• The JWKS feed at /.well-known/jwks.json that every other node binary fetches to verify access tokens.
• The Personal Access Token (PAT) layer for CLI clients.
• The WebAuthn passkey ceremonies, the single-use enrolment link, and the RFC 8628 device authorization grant -- the paths that let someone sign in without a mailbox. Which one to reach for is sign-in-paths.md.

This document covers the operator-side narrative: what to set, what to watch, how to rotate keys.

#Topology

A cluster runs:

• One identity binary -- holds the signing key on disk, publishes JWKS, mints + verifies tokens.
• Many other binaries (bff / agent / planner / workbench / mcp) -- pull the public JWKS and verify incoming JWTs locally. They never see the private key.

CLI clients (memql-cockpit, custom tooling) authenticate against the identity binary directly using mql_pat_<...> PATs. Browser clients (the product SPA, the identity web app itself) authenticate via magic link, then carry the resulting access JWT to bff/agent/etc.

#Browser-side routing of identity XHR

Browsers send /auth/refresh, /auth/logout, /oauth/token, and /.well-known/jwks.json SAME-ORIGIN through the SPA's host (e.g. app.${DOMAIN} / os.${DOMAIN}). The edge binary proxies those four exact paths to the identity service (MEMQL_IDENTITY_VERIFIER_BASE_URL); /auth/callback is a site route and is not forwarded. Top-level magic-link redirects (the /login UI, the /authorize navigation) still go to identity.${DOMAIN} directly. runtime-config.json publishes an empty identityApiBaseUrl so fetch() stays on the site origin.

Same-origin XHR avoids a Safari / Chrome quirk where cross-origin fetch to a sibling host that shares a cert + IP can land on the wrong vhost (HTTP/2 connection coalescing). The OS shell then reports identity returned no access token (invalid_response) because the SPA fallback answered 200 HTML. A front-door nginx location = block for each of the four paths is still fine as defense in depth; it is no longer required for the edge-hosted sites.

#Endpoints

The identity binary mounts two route sets: the JSON auth-flow API (component/identity/http, Server.Mount) and the server-rendered web UI (component/identity/web). /.well-known/jwks.json is mounted by component/identity.Service directly.

#Auth-flow API

Every route above also answers OPTIONS for CORS, and every one is wrapped in the security-headers middleware.

#Web UI

#Required environment variables

Identity-tagged binary:

Other nodes (bff / agent / planner / workbench / mcp): The edge node is the one exception -- it is not an auth boundary (verifierRequired=false, app/auth_mode_edge.go) and installs no verifier at all.

Leaving MEMQL_IDENTITY_VERIFIER_BASE_URL unset on a non-identity node boots it in dev no-auth mode: the synthetic local-dev admin identity is stamped on every request. Never enable this in production.

#Cross-origin access (CORS)

Identity's auth endpoints carry the refresh-token cookie, and the CORS middleware sets Access-Control-Allow-Credentials: true on every matched origin. So an origin on this allowlist can make cookie-bearing requests to /oauth/token, /auth/refresh, /auth/logout and the rest and read the responses. Treat a change here as a change to what can reach a session, not as a header tweak.

The allowlist has two sources, and they are resolved per request rather than snapshotted at boot:

The env list is checked first, in memory. Only a miss consults the graph, so the origins a cluster needs in order to serve its own login page never depend on the database being reachable.

#There is a THIRD layer, and it is not covered by the admin gate

Identity's routes are mounted on the BaseRouter, so they also pass through component/server's own generic CORS middleware, configured by MEMQL_SERVER_ALLOWED_ORIGINS (the legacy spelling SERVER_ALLOWED_ORIGINS is still bridged at boot, memql#3892). That layer knows nothing about the two sources above.

Today it is "*" on identity (deploy/k8s/base/identity.yaml), and under a wildcard it deliberately emits Access-Control-Allow-Origin without Access-Control-Allow-Credentials -- so it exposes nothing, and a granted origin still gets its credentials header from identity's own middleware. One consequence worth knowing: a refused origin may still see an echoed ACAO from this layer, which it can do nothing with.

Do not narrow MEMQL_SERVER_ALLOWED_ORIGINS to an explicit list on identity without understanding this. The moment it stops being "*", that middleware starts emitting Access-Control-Allow-Credentials: true for every origin on its list -- granting credentialed cross-origin access entirely outside the owner/admin gate, outside the audit trail, and outside the * refusal described below. If you need per-origin control on identity, it belongs in the two sources above.

#Registering an OAuth client grants nothing

POST /register is RFC 7591 dynamic client registration. It is deliberately unauthenticated -- the endpoint exists so that clients nobody pre-configured (claude.ai's "add custom connector", say) can self-register.

It is DISABLED by default (MEMQL_IDENTITY_OAUTH_DCR_ENABLED, memql#3719). Enable it only on clusters that expose an MCP surface; everywhere else it is an unauthenticated write endpoint with no consumer. When off, POST /register answers 403 registration_disabled and persists nothing.

First-party editors never use this endpoint. The MemQL VS Code extension is carried as a compiled-in client on every identity node, so it signs in with DCR off and nothing configured -- enabling DCR neither helps nor hinders it. Turning this on is a decision about third-party MCP connectors and nothing else. See Connecting an Editor.

The reasoning, since "disabled by default" otherwise reads as mere caution: under one-cluster-per-customer most clusters route no mcp.<domain> at all, and their OAuth clients -- the OS shell, the Cockpit, the customer's own SPAs -- are known in advance and registerable through MEMQL_IDENTITY_REGISTERED_CLIENTS. Left on, the endpoint lets anyone who can reach identity create unbounded v1:identity:oauthClient rows and choose the client_name a human later approves on the consent screen. It was also one half of the CORS escalation in memql#3716 -- unauthenticated rows becoming an input to a different trust decision, which every future such decision would then have to remember.

There is no middle setting. Approval-gated registration -- create the row, let an admin bless it before it grants anything, which is what memql#3716 does for CORS -- cannot work here, because the entire point of DCR is completing the flow with no human present. So it is one binary per cluster.

Enable it on the identity node. The mcp node warns at boot when it serves the remote protocol head and cannot see the flag enabled, because the failure mode of forgetting is not an error anyone sees -- it is "claude.ai cannot add the connector", from a 403 the operator never reads.

A row created that way carries no CORS allowance and cannot give itself one. Granting is a separate, explicitly authorized act on the same row, and the reason is worth stating: if the allowlist were derived from a client's registered redirect_uris, one anonymous POST carrying redirect_uris: ["https://evil.example/cb"] would buy credentialed read access to the auth surface. Registration stays open; the trust decision is gated.

For the same reason the allowance names origins explicitly rather than being a per-client boolean that derives them from the redirect URI: loopback redirects (http://127.0.0.1:PORT/cb) are normal on this concept -- identity implements the RFC 8252 loopback-any-port exception for them -- so a derivation would silently admit http://127.0.0.1 and hand every local process on somebody's machine cookie-bearing read access. Naming the origins also lets an allowance differ legitimately from the redirect's origin, which it does when a SPA redirects to https://app.example.com/cb while also running at https://www.example.com.

#Granting and revoking

SetOAuthClientCorsOriginsRequest on IdentityAdminMsg (MemqlService.Stream), gated owner-or-admin in component/identity/adminops -- the same gate, refusal shape and audit trail as every other identity-admin write. The underlying mutation is @serverOnly, so there is no client-reachable seam around the gate.

• origins is the complete allowance, not an addition to it. An empty list revokes.
• Each entry is a scheme (http or https), a host and an optional port. A path, query, fragment, trailing slash, userinfo, wildcard host or * is refused with the offending entry named -- validated on the way in, where a person is holding the error message.
• The host must be a DNS name, an IPv4 address, or a bracketed IPv6 address. An internationalised domain must be given in its punycode (xn--) form, because that is what the browser puts in the Origin header; the raw unicode spelling is refused rather than stored as an entry that would match nothing. An underscore in a host is accepted -- RFC 1123 forbids it and real internal hostnames use it.
• A port must be in 1-65535. Entries are canonicalised to the form a browser actually sends -- scheme and host lowercased, an IPv6 literal reduced to its canonical spelling ([0:0:0:0:0:0:0:1] becomes [::1]), and the scheme's default port dropped (https://x.example:443 is stored as https://x.example) -- then de-duplicated. Matching is case-insensitive.
• A statically configured client (MEMQL_IDENTITY_REGISTERED_CLIENTS) has no row, so its origins belong in the env list. Naming one here returns NOT_FOUND; a read that fails returns INTERNAL and changes nothing, so "the client is missing" and "the database is unhappy" are never the same answer.
• Audited as oauth_client_cors_granted / oauth_client_cors_revoked under category=configuration, recording who acted, the origins granted, and the allowance that was replaced.

#Who can READ the granted list

The write is owner/admin only. The read is not: oAuthClientCORSGrants is on the ordinary query surface, so any authenticated caller can list every granted origin together with the client_id that holds it -- which also means listing the registered clients that carry allowances.

That is a deliberate trade, and it is a disclosure rather than an escalation. An origin is not a secret: identity hands it back to whichever browser asks, so an unauthenticated prober can already confirm a guessed origin one at a time through the Access-Control-Allow-Origin echo. What the query adds is enumeration of the whole set at once, to callers who already hold a credential on this cluster. The projection is narrowed to the client id and the allowance, so no redirect URIs or other registration metadata ride along.

The alternative was to mark the read @serverOnly, which would require stamping an internal call origin onto a context derived from an anonymous browser preflight. component/auth's call-origin allowlist admits this package as server-initiated, and a preflight is not that -- an allowlist entry with a wrong stated reason is worse than none. If the enumeration matters for your deployment, the fix is a role predicate on the query, not an origin stamp.

#Propagation and failure behaviour

The middleware reads the granted set live and holds it for at most 10 seconds (a fixed constant, deliberately not a knob). That window is a rate bound, not the correctness mechanism: cors() wraps ~15 routes including every unauthenticated OPTIONS preflight, so without it a flood of unknown origins would be one database query per request.

It is a live read rather than a cache invalidated on the concept's change feed because the grant may be executed on a different node than the middleware. The IdentityAdminMsg bridge is wired on every node with an engine, so an admin can grant from a bff while the CORS decision runs on identity; and the identity node is filtered out of every mesh broadcast target list, so an invalidation event could not reach it. A cache would pass a single-node test and silently fail in the 2-replica mesh.

Each read is bounded at 3 seconds, also a fixed constant. That deadline is about the lock rather than the query: the read runs holding a mutex so a window boundary cannot become a query stampede, and sync.Mutex waiters are not cancellable by their own clients -- so an unbounded read would park preflights indefinitely on an unauthenticated surface during a database outage. Three seconds sits under bun's pgdriver socket deadline (10s), which is where that bound previously lived by accident.

If the graph read fails, the granted half fails closed: no granted origin is allowed until a read succeeds, and the env list keeps working untouched. That is the deliberate direction -- refusing is recoverable on the next preflight, whereas serving a stale copy would let a database outage outlive a revoke.

One caveat on revoke latency, and it is the browser's rather than ours: the preflight response carries Access-Control-Max-Age: 600, so a browser that already completed a successful preflight may keep using it for up to ten minutes after the revoke lands. A failed preflight is not cached, so a grant is visible on the very next attempt. If a revoke has to be immediate for a specific browser, revoke the client's sessions as well rather than relying on CORS to stop it.

#Optional anti-abuse knobs

Each rejection emits an audit event with category=auth, action=magic_link_blocked, and a failureReason matching the specific defense (rate_limit / disposable_email / mx_invalid / turnstile / risk_threshold). Surface these in your log pipeline to tune thresholds.

Both issue paths run this stack (memql#4303). It wraps POST /auth/magic-link and POST /login -- the browser form -- because the controls are a property of the ROUTE rather than of the issuer, and until memql#4303 they guarded only the JSON API, which takes a client library to reach. The peek that extracts the address understands both JSON and application/x-www-form-urlencoded; without that, gating /login would have looked done (the per-IP limiter runs before the body is read) while Turnstile, the blocklist and the MX check silently saw an empty address.

#Audit actions the magic-link flow emits

magic_link_approved is the row that names the person who clicked a link they did not request. On a shared mailbox it is the only place that fact is recorded.

#Passkey, enrolment and device-code knobs

These are read directly by their handlers rather than through identity.Config, so they take effect per process with no other wiring. All are optional; the defaults are the shipped behaviour.

WARNING: every limiter in the identity tree is in-memory and therefore per-replica, as is the WebAuthn challenge store. A multi-replica deployment gets per-replica budgets, and a ceremony that lands on a different replica than the one that minted its challenge will not find it.

Two fixed values that have no env override, stated here because operators ask:

• MEMQL_IDENTITY_BASE_URL is the only source of the WebAuthn RP ID. Its hostname (port stripped) is the RP ID; the full scheme://host[:port] is the expected ceremony origin. It is never derived from the request Host header, and a base URL that yields no usable RP ID makes the passkey routes refuse rather than guess. Changing it orphans every passkey already enrolled.
• Enrolment tokens are 15 minutes by default with a 24-hour ceiling; a longer request is clamped, not refused. Set per-issue (--ttl, or the console's lifetime picker), not by env.

#Token + session lifetimes

#Refresh-token rotation grace window

The rotator persists each session's IMMEDIATELY-PREVIOUS refresh hash in previousRefreshTokenHash + previousRotatedAt and accepts that hash for 30 seconds after rotation. Covers the case where the SPA hard-refreshes mid-rotation -- the server has already minted the new pair and updated the cookie hash on disk, but the browser aborted before consuming the Set-Cookie. The new page's first /auth/refresh lands with the OLD cookie; the rotator falls back to the previous-hash lookup, accepts it inside the grace window, and rotates again. Without this, every rapid hard-reload bounced the user to /login. Window is hard-coded in component/identity/refresh/rotate.go (previousRefreshGraceWindow = 30 * time.Second).

#Email delivery

The identity service composes emails (magic link, invitation, admin notifications) and hands them to the email integration plug-in (integrations/email). Configure exactly one sender:

• Microsoft Graph (MEMQL_EMAIL_AZURE_TENANT_ID + MEMQL_EMAIL_AZURE_CLIENT_ID + MEMQL_EMAIL_AZURE_CLIENT_SECRET + MEMQL_EMAIL_SENDER, optional MEMQL_EMAIL_FROM_NAME) -- preferred, and the ONLY sender on a cloud install (memql#4218: Graph only; no SMTP, no Azure Communication Services on this path). The legacy un-prefixed names (AZURE_TENANT_ID, MAIL_SENDER, ...) are still read as a fallback.
• SMTP fallback (SMTP_HOST + SMTP_PORT + SMTP_USERNAME + SMTP_PASSWORD + SMTP_FROM_ADDR) -- development only.
• LogSender -- both unset; emails are written to the slog stream. Dev only, and refused outright on an install that is not local (below).

#Log-only mail is REFUSED off a local domain (memql#4477)

With nothing configured, the email lane used to select a LogSender that wrote the message to the pod log and returned nil. Mail did not fail silently -- it failed upward. Every layer above reported success: the setup wizard said the link was sent, the audit row recorded magic_link_issued with outcome=success, and the console's integration row said "degraded", which on a dev cluster means "as intended". The only evidence was a human not receiving mail, which is indistinguishable from a spam filter. On the first real cloud instance the cluster owner could not claim their own cluster and nothing anywhere said why.

A failure that hangs eventually gets investigated. A failure that reports success does not. So log-only is now a choice a local install may make and an error everywhere else, decided from MEMQL_DOMAIN:

Four layers changed, and each one closes a different way of not finding out:

• Boot -- the plug-in factory returns an error, which app.materializePlugins fatals on. A misconfigured cloud install fails where the operator is watching instead of a week later.
• Send -- a LogSender built anywhere else returns a permanent SendError rather than nil.
• The audit trail -- magic_link_issued is stamped outcome=failure, failureReason=delivery_failed when the send failed. The row is still written and the HTTP response is unchanged: the link is a real credential that can be re-requested, and a response that varied with delivery would tell an unauthenticated visitor which addresses are registered.
• The console -- the email integration reports unhealthy rather than degraded, because on this install nothing can be sent at all.

Break-glass: MEMQL_EMAIL_ALLOW_LOG_ONLY=true. make up DOMAIN=lab.example.com stands up a genuinely local parity cluster on a name that looks like production, and refusing to boot it would be a regression for a developer who never wanted mail. Setting this says log-only is deliberate here; the boot log says so, and the console keeps calling the lane degraded. Never set it on an install real people sign in to -- their sign-in links go to the pod log.

The break-glass recovery, worth knowing either way. Because LogSender writes the whole message, a magic link issued in log-only mode is recoverable:

kubectl logs deploy/identity -n memql | grep -A5 "log-only mode"

Two constraints: the link expires in 10 minutes, and links are device-bound (memql#4300) -- it must be opened in the browser that requested it, the one holding the memql_ml cookie. So a cluster is never actually locked out by this; it just gives no sign that it is the reason.

#The Graph sender lives on the MAILBOX tenant, not the AKS tenant

MEMQL_EMAIL_AZURE_TENANT_ID is the Entra tenant that HOSTS THE SENDER MAILBOX -- the Microsoft 365 tenant the mail domain is verified on -- and that is not necessarily the tenant the AKS subscription lives in. The sender runs the client-credentials flow against login.microsoftonline.com/<tenant-id> and then calls POST /users/<sender>/sendMail; Graph resolves <sender> in the tenant the token was issued for. On a tenant that does not own the mailbox that lookup is a 404 (or a guest object with no mailbox), and no permission granted on that tenant changes it.

The first entry-shape cluster hit exactly this (memql#4226): AKS and the Pay-As-You-Go subscription sat on one Entra tenant, the sender mailbox and the public mail domain on a different Microsoft 365 tenant. So EVERYTHING the sender needs is created on the MAILBOX tenant:

• the app registration (id-memql-mail by convention) and its client secret;
• the Mail.Send APPLICATION permission, with admin consent granted;
• the Exchange ApplicationAccessPolicy that scopes that app to the one sender mailbox -- not optional, see below;
• and therefore MEMQL_EMAIL_AZURE_TENANT_ID = that tenant's id, and MEMQL_EMAIL_AZURE_CLIENT_ID / _CLIENT_SECRET = that app's.

Find the tenant without guessing -- both answers come from the domain, not from whichever tenant your browser session happens to be signed in to:

# Which tenant does this login name belong to? (NameSpaceType Managed /
# Federated, plus the tenant's brand name)
curl -s "https://login.microsoftonline.com/getuserrealm.srf?login=noreply@<domain>&json=1"
 
# The tenant id itself: the issuer is https://sts.windows.net/<tenant-id>/
curl -s "https://login.microsoftonline.com/<domain>/.well-known/openid-configuration" | jq -r .issuer

#Mail.Send (Application) is tenant-wide until it is scoped (memql#4478)

The Entra display name for the permission is literally "Send mail as any user", and that is what it grants: the app registration can send as every mailbox in the tenant, not only as MEMQL_EMAIL_SENDER. Narrowing it needs an Exchange ApplicationAccessPolicy (or the newer RBAC for Applications), which is Exchange Online PowerShell and not reachable from az -- which is precisely why it gets skipped, and why every instance stood up without it has a tenant-wide send capability sitting in a Key Vault.

The invocation, the both-directions verification, and the az ad app credential reset --append trap that goes with it are in azure-entry-install.md. Two rules travel with them:

• One app registration per instance. A leaked secret then costs one instance rather than every instance sharing the registration.
• az ad app credential reset without --append deletes every existing secret on the registration. On the bring-up this was found on, the secret it destroyed was labelled memql-keep-it.

#Next client install: verify the mailbox before wiring Graph

1. Identify the mailbox tenant with the two commands above. Do not assume it is the subscription's tenant.
2. On THAT tenant, confirm the sender is a real, licensed mailbox: GET https://graph.microsoft.com/v1.0/users/noreply@<domain>?$select=id,userPrincipalName,userType,assignedLicenses must return the user with userType: Member and assignedLicenses non-empty. A 404 is the wrong tenant or no such user; empty assignedLicenses is no Exchange license, and the first sendMail then fails with MailboxNotEnabledForRESTAPI -- that error means "not a real mailbox", not a permission problem.
3. Create id-memql-mail, grant Mail.Send (Application) with admin consent, and apply the ApplicationAccessPolicy -- on that tenant only. Do not create the mail app on the AKS tenant, and do not recreate noreply@<domain> on the AKS directory: a same-named object on the wrong tenant sends every later diagnosis down the wrong path.
4. Seed MEMQL_EMAIL_AZURE_TENANT_ID with the MAILBOX tenant id, then the client id, the secret and MEMQL_EMAIL_SENDER -- through Key Vault + External Secrets on a cloud install, like every other secret.
5. Send one magic link and read the identity log: email: using Microsoft Graph sender at boot, then sendMail answering 202 Accepted. A 404 on /users/<sender> points at the tenant; a 401 points at the credential.

The entry install's own record of this is in azure-entry-install.md.

Branding controls (MEMQL_IDENTITY_BRAND_NAME, MEMQL_IDENTITY_BRAND_PRIMARY_COLOR, MEMQL_IDENTITY_BRAND_LOGO_DATA_URI) flow into all outbound templates.

#Key management

#Multi-replica: the signing key MUST be a shared seed (#1515)

The identity service has two signing-key modes:

• Shared-seed mode (MEMQL_IDENTITY_SIGNING_KEY_B64 set) -- every replica derives the SAME Ed25519 key + kid from the same seed (#550). JWKS is coherent across all replicas, survives restarts and DB resets, and the deployment can run replicas: >=2 on a rolling update. This is the REQUIRED posture for any multi-replica install.
• File-key mode (MEMQL_IDENTITY_SIGNING_KEY_B64 unset) -- each pod reads/generates an Ed25519 key on its OWN MEMQL_IDENTITY_KEY_DIR (which, in the cluster manifest, is the pod's ephemeral container filesystem -- there is no shared PVC). With >=2 replicas, each replica mints its own key, so /.well-known/jwks.json diverges across replicas and ~50% of token verifications (browser sessions AND mesh node tokens) fail with unknown kid, flapping login.

This caused the 2026-06-16 staging auth outage. Staging ran identity at replicas: 2 with no MEMQL_IDENTITY_SIGNING_KEY_B64 reaching the pods, so it fell into per-pod file-key mode and JWKS flapped between the two pods' keys. The fix is the shared seed plus the startup guard below. The seed is now a DECLARED key on memql-secrets in every environment (memql#3960) -- it used to reach the cloud only by riding inside the genesis secret-sealing mechanism retired in memql#3963, which is what made deleting that mechanism a latent repeat of this outage.

Fail-fast guard. Config.Validate() REFUSES to start a deployment that has no MEMQL_IDENTITY_SIGNING_KEY_B64 unless the operator explicitly opts into per-pod keys with MEMQL_IDENTITY_ALLOW_EPHEMERAL_KEY=true. This converts a silent ~50% auth-failure into a loud, copy-pasteable startup error.

Only a loopback issuer is exempt -- localhost, 127.0.0.1, ::1, 0.0.0.0, or an RFC 6761 *.localhost name. Those name one process on one machine and cannot have a second replica.

The *.local.<domain> dev wildcard used to be exempt too, and that was the hole (memql#3400). identity.memql.localhost is the local parity cluster's front door: it resolves to 127.0.0.1, but what listens there is traefik, proxying to a Service that make scale N=2 puts two identity pods behind. The guard therefore stayed silent on exactly the topology it exists for, and a local cluster served two kids while reporting the resulting rejections as "invalid or expired token". The dev wildcard remains exempt from the encryption-at-rest requirement, which is a genuine dev-vs-prod distinction; it is not exempt from this one, which is a one-process-vs-many distinction.

Generate a fresh seed:

make identity-signing-key      # head -c 32 /dev/urandom | base64

Set the result as MEMQL_IDENTITY_SIGNING_KEY_B64 on every identity replica -- as a KEY on the memql-secrets Secret every node envFroms, and by no other route (memql#3960). Locally there is nothing to do: scripts/k3d/seed-secrets.sh (make secrets, which make up runs) generates a seed when the cluster has none and reuses it verbatim on every later run.

Rotate by writing a new seed and rolling the deployment -- automatic rotation is disabled in shared-seed mode. Rotation invalidates every live browser session and every minted class="node" mesh token, which is why make secrets never rotates as a side effect of a re-run.

Use MEMQL_IDENTITY_ALLOW_EPHEMERAL_KEY=true only for a genuinely single-process deployment.

Verify it. make status reads /.well-known/jwks.json from every identity replica through the apiserver pod proxy and reports identityKeysShared. Run it after make scale N=2:

Identity litmus: JWKS keyset per identity replica (must be identical)
  identity-5f45458d79-924h5: kid(s) ln-RlCxzK8o
  identity-5f45458d79-qpcq2: kid(s) ln-RlCxzK8o
 
  PASS: all 2 identity replica(s) publish the same signing keyset.

#File-key mode (single-node / dev)

Ed25519 signing keys live in MEMQL_IDENTITY_KEY_DIR:

• jwt-current.ed25519 -- the active signing key.
• jwt-previous.ed25519 -- present only during the rotation overlap window. Retiring kid stays in JWKS so in-flight tokens still verify.

Files are 0600, the directory 0700. With MEMQL_IDENTITY_KEY_ENCRYPTION_KEY set, the private bytes are wrapped in AES-256-GCM with an Argon2id-derived key (32 MiB, t=2). With the env var unset (dev only), private bytes are plaintext.

Config.Validate() enforces "encryption-at-rest is mandatory in production": if MEMQL_IDENTITY_BASE_URL is not a localhost origin and MEMQL_IDENTITY_KEY_ENCRYPTION_KEY is empty, startup fails. Don't try to defeat the guard.

#Rotating the signing key

Read this before rotating anything. Rotation behaves completely differently in the two key modes, and the mode staging and production run is the one with no automation in it.

#Which regime am I in?

The boot log says which one you are in: jwt_signing_key_rotation_scheduler_active or jwt_signing_key_rotation_scheduler_inert. So does memql_identity_signing_key_rotation_supported (1 / 0).

In file-key mode a goroutine calls KeyManager.Rotate every MEMQL_IDENTITY_KEY_ROTATION_DAYS (default 90); the retiring key stays in JWKS for MEMQL_IDENTITY_JWKS_OVERLAP_HOURS (default 24) and is then swept. Other nodes pick up the new kid on their next background JWKS refresh (5 min) or immediately on the first token carrying an unknown kid.

#Recorded decision (memql#3381): the manual path stays

The gap is real -- in every deployed environment there is no way to rotate the signing key from inside the running system. Two options were on the table:

1. Build a rotation surface that works in env mode -- mint the successor in-process, publish both through the overlap window, retire the old one.
2. Accept the manual procedure and close the gap with observability: a key-age metric, an alert, this runbook, and honest naming so the dormant scheduler stops implying coverage.

Option 2 was chosen. The reasoning, so the next reader can re-open it on evidence rather than taste:

• Option 1 cannot be built without moving where signing material lives. Env mode exists (memql#550) so that N identity replicas derive an identical key from one seed with no single writer -- that is what let identity drop its ReadWriteOnce key PVC and scale past one replica. A replica that minted a successor on its own would sign with a key its peers neither hold nor publish, which is exactly the JWKS-incoherence failure of memql#1523 / memql#3400 (roughly half of all auth failing, silently). So in-env-mode rotation needs shared durable state for the keyset -- a keystore -- not a rotation button. That is a different, larger design.
• Writing a new seed back into the Secret is not available to the service. The seed reaches the pod through envFrom on the memql-secrets Secret, which an operator creates out of band; identity secrets are not ESO/Key-Vault-backed. For the service to update it directly it would need patch on Secrets in its own namespace -- handing the cluster's most security-sensitive pod the ability to rewrite every other secret next to it, including the database password. And it would not even work: envFrom is snapshotted at pod start, so a patched Secret changes nothing until a roll. The overlap the option promises is unreachable through the Secret.
• Rotation is genuinely rare. A 90-day cadence on an Ed25519 signing key is hygiene, not incident response. Compromise is handled by rotating the seed immediately and accepting the hard swap -- which is what you want in that case anyway.

What option 2 owes you, and now delivers: key age is observable (memql_identity_signing_key_created_timestamp_seconds), silence is earned rather than assumed (memql_identity_signing_key_age_known), the regime is legible (memql_identity_signing_key_rotation_supported), two alerts fire in deploy/k8s/monitoring/prometheusrule-auth.yaml, and the scheduler is labelled dev-only at its definition in component/identity/identity.go and at boot.

Follow-up left open: the only shape option 1 could really take is a DB-backed keyset -- signing material in Postgres, which every replica already shares, wrapped at rest, with the seed demoted to a bootstrap credential. That would restore true overlap-window rotation in a deployed cluster. It was not attempted here; it re-opens where signing material lives, its at-rest encryption, and identity's boot ordering.

#The consequence you must plan for: a hard swap is lossier than a rotation

The manual path writes a new seed and rolls. It never populates the KeyManager's previous slot, so MEMQL_IDENTITY_JWKS_OVERLAP_HOURS does not apply -- the old kid vanishes from JWKS the moment the new pods serve. The procedure is not merely unsurfaced; it is lossier than the code path it substitutes for. Every JWT signed by the old key becomes unverifiable at once. Anything holding an opaque token is unaffected.

Verifier-side recovery is fast and needs no restart: the per-node verifier force-refreshes JWKS synchronously on any unknown kid, with no negative caching, so the first request bearing a new-kid token pays one round-trip and then succeeds. Two caveats: the JWKS handler sends Cache-Control: public, max-age=300, so a caching proxy or CDN in front of identity can serve the pre-swap keyset for up to 5 minutes -- verify none is in the path; and concurrent unknown-kid requests serialize on the refresh mutex rather than coalescing, so expect a brief latency spike at the moment of the swap.

#Procedure (env-seed mode)

Do it in a maintenance window. Everything in the "No" rows above breaks at step 3.

1. Generate a new seed (32 bytes, base64-std): make identity-signing-key
2. Update it, together with today's date, on the memql-secrets Secret the identity Deployment reads:
   • MEMQL_IDENTITY_SIGNING_KEY_B64 -- the new seed
   • MEMQL_IDENTITY_SIGNING_KEY_CREATED_AT -- today, RFC3339 (e.g. 2026-08-09T00:00:00Z). Re-stamp it every time; a stale stamp silently disarms the overdue alert. Both values must be identical across every identity replica -- they are properties of the key, not of the pod.
3. Roll identity and wait for Available: kubectl -n memql rollout restart deploy/identity && kubectl -n memql rollout status deploy/identity
4. Confirm the new keyset -- one kid, and the same one everywhere: kubectl -n memql exec deploy/identity -- curl -sS https://localhost:8085/.well-known/jwks.json Cross-replica agreement is what make status checks locally and what memql_jwks_keyset_fingerprint (max == min) checks in the cluster.
5. Repair the credentials that cannot repair themselves: re-mint every service_account JWT and restart any node pinned to a pre-provisioned MEMQL_NODE_TOKEN after re-issuing it.
6. Watch memql_auth_rejects_total{reason="unknown_kid"} settle back to zero, and confirm memql_identity_signing_key_age_known is 1 with a fresh ..._created_timestamp_seconds.

#Recovery

This applies to file-key mode only -- env-seed mode has nothing on disk to decrypt. If MEMQL_IDENTITY_KEY_ENCRYPTION_KEY is rotated incorrectly (the new secret can't decrypt the old envelope), the binary fails to load the key files at startup with a clear AES-GCM error. Restore the original secret from your secret store and redeploy. Then rotate properly: stand up the new secret and let the scheduler mint a fresh key under it (or delete jwt-current.ed25519 so the next boot generates one), and only then retire the old secret. There is no "Rotate now" control to press -- the admin console's button was removed in memql#3324 because it answered with an error in every environment that runs env-seed mode.

#Staging DB reset stays auth-coherent (#1522)

A staging DB reset (a staging-db-reset.sh shipped with the deployment that runs one; this repository carries no copy) wipes the staging database back to an empty schema. A DB wipe used to leave auth HALF-BROKEN: every v1:identity:authSession row and every mesh node-token grant lives in the DB, so the wipe invalidated them all, and the cluster only recovered after a manual reseal + mesh roll. The reset is now auth-coherent by construction -- it leans on two facts and you do NOT have to do anything by hand afterward:

• The signing key is NOT in the DB. MEMQL_IDENTITY_SIGNING_KEY_B64 is a key on the memql-secrets Secret; the wipe never touches the Secret, so every replica derives the same key + kid + JWKS after the reset -- as long as the seed is present. The reset pre-flights this BEFORE wiping: it refuses if the Secret carries no seed, or if identity is running in the divergent per-pod ephemeral-key mode (MEMQL_IDENTITY_ALLOW_EPHEMERAL_KEY=true at replicas >= 2). A wipe in either state would leave auth unrecoverable, so it is blocked.
• Mesh nodes re-mint their token on auth failure (#1521). Once identity is back with the stable key, every leaf reconnects on its own.

The reset brings identity up FIRST (waited Available) so the JWKS issuer is serving before any mesh node bootstraps, then restores the rest of the workloads. Finally it verifies identity is Available and probes the in-cluster JWKS document. Confirm end-to-end with the pack's staging smoke target (checks JWKS + the login surface over the public front door). No manual reseal or mesh roll is required.

#Dev quick-start

The local cluster runs via make up -- see docs/public/overview/quickstart.md for the port-forward reference. The local overlay wires identity at http://identity:8081 (cluster DNS) with every other node verifying against that issuer; reach it from the host via kubectl port-forward -n memql svc/identity 8085:8085.

For running the binaries standalone (no cluster), set the same URL:

# Identity binary
MEMQL_IDENTITY_ENABLED=true \
MEMQL_IDENTITY_BASE_URL=https://identity.memql.localhost \
MEMQL_IDENTITY_REGISTRATION_MODE=open \
make identity-assets identity
./bin/memql-identity
 
# bff binary, points at the identity binary above
MEMQL_IDENTITY_VERIFIER_BASE_URL=http://identity:8081 \
MEMQL_IDENTITY_VERIFIER_EXPECTED_ISSUER=https://identity.memql.localhost \
make bff
./bin/memql-bff

Without MEMQL_IDENTITY_VERIFIER_BASE_URL the bff boots into no-auth dev mode; convenient for solo development that doesn't need real tokens, but the synthetic local-dev admin identity will be on every request.

#Health + observability

• GET /healthz on the identity binary returns 200 once the key manager has loaded.
• GET /.well-known/jwks.json always reflects the current PublicKeySet (current + retiring during overlap).
• Settings -> Keys shows live key metadata (kid, algorithm, use) and explains why there is no rotate control.
• Prometheus, on the identity node (component/metrics): memql_identity_signing_key_created_timestamp_seconds (key age is time() minus this), memql_identity_signing_key_age_known (0 = the age above is meaningless -- alert on it, because it means the overdue alert cannot fire), and memql_identity_signing_key_rotation_supported (0 = the in-process scheduler is inert; the manual runbook is the only path). Alerts: MemqlIdentitySigningKeyOverdue and MemqlIdentitySigningKeyAgeUnknown in deploy/k8s/monitoring/prometheusrule-auth.yaml.
• Audit events for every auth lifecycle moment land in v1:identity:auditEvent (in addition to slog). Retention is controlled by MEMQL_IDENTITY_AUDIT_LOG_RETENTION_DAYS (default 365). Routine mechanics land on a SECOND concept -- see below.

#Two logs: decisions, and mechanics

The identity service writes to two concepts, and the split is the difference between an audit trail an operator can read and one they scroll past (memql#4328).

Why the reads split that way. authActivity declares @rowAuthz(owner="actorUserId", clusterOwner) -- the owner, or a cluster owner. The escape is the owner ROLE specifically, so a read gated An owner-or-admin gate would pass for an admin and then be narrowed by the tier to that admin's own rows: a confidently wrong answer dressed as a roll-up. So the operator roll-up is cluster-owner-only and the per-user read is available to everyone. A non-owner admin uses the second one and sees exactly what it says.

Giving a person their own activity is the point of the tier, not a side effect: "a device I do not recognise refreshed my session" is a signal the account holder is best placed to spot, and it does not need an operator's help.

#The rotation cadence, and why it used to be a storm

One session_refreshed row is one refresh-token rotation: a client presented the memql_refresh cookie (or a refresh_token grant) to POST /auth/refresh or POST /oauth/token and got a new access token plus a brand-new refresh token. It is not a heartbeat. Nothing else writes it -- not JWT verification, not a WebSocket reconnect, not the verifier's five-minute JWKS refresh, not node bootstrap.

The designed cadence is once per ~70% of the access-token TTL, per open tab -- about every 10.5 minutes at the default 900-second TTL -- with a 30-second floor. The SDK computes the delay from the token's OWN lifetime:

delay = max(30s, 0.7 * (exp - iat)) - (now - receivedAt)

exp and iat are both stamped by the identity service, so their difference carries no clock skew; now and receivedAt are both the browser's, so theirs does not either. The formula only ever subtracts same-clock pairs.

It used to be 0.7 * (exp - Date.now()), which compares the identity service's clock with the browser's. A browser running ahead by a little less than the TTL saw every freshly minted token as nearly expired and rotated every few seconds, indefinitely; ahead by more than the TTL, the delay was zero and it rotated at network speed. A second amplifier sat in a since-retired browser console, whose session probe ran on every route change rather than once per cold load -- so clicking a row in the Audit Trail wrote a row in the Audit Trail. Both are fixed (memql#4326, memql#4327); the floor is what makes a pathological token harmless even if some future arithmetic goes wrong again.

If you see rotations far more often than the cadence above, the client is the place to look, not the server.

#Refresh-token reuse detection

Each rotation records the SHA-256 of the token it RETIRED on its activity row. A presented token matching neither the current hash nor an in-grace previous one is looked up against those retired hashes (memql#4329):

• Hit -- the token was issued by this cluster and has since been rotated away, so it can only have been replayed. The whole session is revoked with revokedReason=reuse_detected, a refresh_token_reuse_detected row lands on auditEvent with the presenting IP and User-Agent, the user is notified, and the caller gets 401.
• Miss -- an ordinary stale cookie. Refused as before, with no revoke and no signal. Revoking on an ambiguous signal would be a self-inflicted denial of service.
• Inside the 30-second grace window -- a legitimate mid-rotation retry (the SPA hard-refresh race). Accepted, recorded as grace_window_accept, never treated as reuse.

Detection reaches back exactly as far as the activity retention window. Past it the row has been pruned and a replay degrades to session_not_found. The default 30 days exceeds both the 14-day idle timeout and the 30-day refresh-token TTL, so a token older than the window is already dead on its own account -- which is what makes the limit safe. Lowering MEMQL_IDENTITY_AUTH_ACTIVITY_RETENTION_DAYS below either of those TTLs opens a real gap, and nothing warns about it.

#Related

• sign-in-paths.md -- magic link, passkey, enrolment link, device code and PAT: what each is for, what recovery looks like with no mail, and why a passkey does not follow you between clusters.
• access-model.md -- enforcement layers, role spectrum, the wire-side lifecycle.
• user-provisioning.md -- registration modes, magic-link flow, invitations.
• component/identity/CLAUDE.md -- per-package developer guide.