Environment Variables -- MemQL

Audience: engineers running MemQL locally or operating it in lab/prod. Companion doc: the product frontend repo's env-vars doc covers the frontend side.

#The registry (source of truth)

scripts/secrets/manifest.yaml is the authoritative registry of every environment variable MemQL reads (Epic 7 / memql#2104). One file drives three consumers so they can never drift: the seal floor, the cockpit Configuration screen, and boot-time fail-fast validation. Each entry carries component, scope (node / global / partition), kind, required (node types that need it at boot, "all" = every node), default, and description. The small non-optional set at the top of each section is the seal floor — the strict superset a developer .env must cover to seal; the full universe is registered optional: true so cataloguing every var never breaks local sealing.

A var read in code but absent from the registry — or a registry entry that appears nowhere in the repo — fails CI via make env-registry-check (go run ./cmd/envscan -check). "Nowhere in the repo" means outside the registry itself: the check excludes both copies of the manifest (the authored one and the embedded component/envregistry/manifest.yaml snapshot), because an entry's own row is not a reference to it. Missing the second copy is what left the reverse direction unsatisfiable until memql#2971 — with both in the scan every name appeared at least twice, so the staleness threshold could never select. A name also has to appear as a whole word: 87 entries are legacy aliases that are substrings of their MEMQL_-prefixed replacement, and a substring match kept each one looking referenced by the very name that replaced it. After editing the registry, regenerate the embedded snapshot with make env-registry-sync.

Two limits worth knowing before you trust a green result:

• The scan reads .go, .memql, .yaml, .yml and .env* only. A var referenced solely from a .md, a .sh, the Makefile, a Dockerfile, or TypeScript counts as stale and will red CI. Register it somewhere the scan reads, or drop it. (No entry is in that position today.)
• It excludes a fixed list of registry copies, not "any file that looks like the registry". A third copy of the manifest added anywhere would give every entry a self-reference again and silence the reverse direction, exactly as the embedded snapshot did. The check hard-fails if a listed copy goes missing, but it cannot see an unlisted one — so if you add a copy, add it to registryFiles in cmd/envscan/scan/scan.go.

#TL;DR

MemQL splits configuration into two tiers:

1. Bootstrap values -- a small set of OS environment variables the process must see before it can read anything else. Things like "where is Postgres", "what node am I", "what's the master encryption key". They arrive by ONE path (epic memql#3958): as KEYS on the memql-secrets Secret every node envFroms. Locally make up (scripts/k3d/seed-secrets.sh) writes them; in the cloud External Secrets reconciles them from Key Vault. There is no encrypted-at-rest path for these -- they live in plain env inside the pod.

   There used to be a second path: a genesis envelope, sealed under MEMQL_GENESIS_B64, that each pod decrypted in-process at boot, applying ~150 vars set-if-absent. It is gone, along with its sealing CLI and its .znas format. If a runbook tells you to seal one, it is describing a mechanism that no longer exists.

2. Concept storage -- everything else. API keys, OAuth client secrets, model defaults, feature flags, mail-sender addresses, and any tunable a tenant might want to override. These live in four MemQL concepts and are seeded with go run ./scripts/secrets seed rather than from env files.

The bootstrap envelope is intentionally tiny so that rotating an API key, changing a default model, or adding a new tenant's BYOK credential never requires a redeploy -- only a re-seed, or a direct write of the v1:platform:globalVariable row.

There is no secrets-* / variable-* family of make targets, and there never has been. This document described one for its whole life -- init, seed, list, export, one-off set -- and every citation of it sent an operator to a command that does not exist (memql#4405). The tool underneath is real and takes exactly two subcommands, seed and health; the sections below describe those. A new bad citation now fails the build (TestMakeTargetCitationsNameRealTargets).

#Naming convention

Every env var name should answer "what subsystem owns this?" at a glance. The standard shape is:

<COMPONENT>_<VENDOR_OR_DETAIL>_<FIELD>

Where COMPONENT is the subsystem that consumes the value:

Frontend (VITE_* prefix is added by Vite to mark "safe to ship to the browser"):

#Anti-patterns to avoid

• Vendor-only names without a component prefix. AZURE_TENANT_ID is opaque -- Azure could mean storage, identity, OpenAI-on-Azure, or anything else. Always pair the vendor with the subsystem (MEMQL_EMAIL_AZURE_TENANT_ID).
• Two prefixes for the same thing. We had MAIL_* and AZURE_* for the same (email) integration; merging onto EMAIL_* with the vendor as the second segment removes that ambiguity.
• The MEMQL_ prefix where it's redundant. Inside the MemQL repo, every var is "MemQL's" -- prefixing every one of them with MEMQL_ is noise. Reserve MEMQL_ for things that are about MemQL itself (master key, node identity, engine tuning), not for things MemQL happens to call.

#Migration window

When a name changes (like AZURE_* -> EMAIL_AZURE_* in 2026-04), the consumer accepts both forms during a transition window so existing installs don't break. The pattern is:

1. Update the manifest + .env.local + docs to the new name.
2. Add a fallback in the consumer (Go integration / DSL provider) that tries the legacy name if the new one is empty.
3. Remove the legacy fallback in a follow-up commit once everyone has re-seeded.

Search for Legacy*EnvKeys / "legacy fallback" in the Go code to find the active migration shims.

#Future renames (deferred)

These would tighten the naming scheme but the change radius is too wide to justify in the same commit as the doc:

• MEMQL_SI_*_API_KEY -> SI_*_API_KEY. The MEMQL_ prefix is redundant inside the MemQL repo and the dev manifest already seeds the bare form. Touches 6 provider .memql files plus the Go STT bootstrap; coordinate with manifest + user-yaml renames.

• VITE_BYPASS_AUTH -> VITE_AUTH_BYPASS, VITE_ENABLE_ADMIN -> VITE_FEATURES_ADMIN_ENABLED for stricter prefix consistency on the frontend.

If you're touching these areas anyway, fold the rename in. Don't do them as drive-by churn.

#The four concepts

Source file: dsl/platform/concepts.memql (declares all four -- globalSecret, globalVariable, partitionSecret, partitionVariable).

The _system partition is reserved for global concepts -- platform secrets and variables live there regardless of which partition the caller is in.

#Encryption

Secrets are sealed with NaCl secretbox (XSalsa20-Poly1305) under MEMQL_MASTER_KEY (32-byte hex). The cleartext is never stored; only base64(nonce || ciphertext) plus a 4-character fingerprint for UI display. See component/secret/encryption.go.

#Resolution chain (provider auth)

When a .memql provider file references a placeholder like env("MEMQL_AI_OPENAI_IDENTITY_PROVIDER_ID"), the resolver in component/memql/ai_providers.go (resolveAuthPlaceholders) walks:

1. v1:platform:globalSecret -- systemSecretResolver
2. v1:platform:globalVariable -- systemVariableResolver
3. OS env -- bootstrap-window fallback. See the note on bootstrap order below.

#Prefix elision

Provider .memql files reference MEMQL_AI_<VENDOR>_... while a name may have been seeded in the shorter form without the AI_ segment. To bridge that gap without renaming either side, every layer of the chain tries both names in priority order:

authConceptLookupNames("MEMQL_AI_OPENAI_PROJECT_ID")
  -> ["MEMQL_AI_OPENAI_PROJECT_ID", "MEMQL_OPENAI_PROJECT_ID"]

So a provider asking for the long form will pick up a value seeded under the short one automatically. The same elision applies to the OS env fallback.

Only the AI_ segment is dropped -- MEMQL_ is part of the registered name. The deprecated MEMQL_SI_ prefix elides the same way and additionally keeps its historical bare form as a last candidate, so a product DSL bundle still declaring the old prefix does not regress.

This section described the behaviour before the code had it (memql#4338). The elision was written against MEMQL_SI_, which no provider in the tree uses -- so it fired for none of them, and a key seeded under the documented seal-floor name was simply not found. TestSealFloorSecretResolvesForAnMemqlAIPlaceholder (component/memql/ai_provider_auth_lookup_test.go) is what now pins the mapping this block claims.

#Why OS env stays around

Providers are loaded eagerly during engine initialization. On a fresh make down && make up, the database starts empty, so when providers first try to resolve their auth keys the concept storage has no rows yet. The OS env fallback (populated from the k8s Secret seeded by make up in dev, or from the deploy manifest in prod) keeps providers alive through that bootstrap window until the concept seed completes.

Future work to retire the OS env fallback cleanly: either lazy per-request provider auth resolution, or a post-seed engine reload hook so providers retry concept storage once seeding finishes.

#Failure mode

A miss at every layer produces:

auth "projectId" references MEMQL_AI_OPENAI_PROJECT_ID but no value is in
concept storage or OS env. Tried name(s) MEMQL_AI_OPENAI_PROJECT_ID,
MEMQL_OPENAI_PROJECT_ID under v1:platform:globalSecret, v1:platform:globalVariable,
and the process env. Seed ANY of those names: put it in the node's environment
(locally `make secrets`; in a cluster, whichever secret store the deployment
reads), or store a v1:platform:globalSecret row under it. The last name listed
is the short form the manifest and docs/public/operate/env-vars.md use

#The one exception: a vendor's federation set is optional at every layer

Two sets of placeholders resolve to ABSENT instead of failing when no layer holds them (optionalAuthEnvNames, memql#4334, widened for both vendors in epic memql#5088):

MEMQL_AI_ANTHROPIC_FEDERATION_RULE_ID       MEMQL_AI_OPENAI_IDENTITY_PROVIDER_ID
MEMQL_AI_ANTHROPIC_ORGANIZATION_ID          MEMQL_AI_OPENAI_SERVICE_ACCOUNT_ID
MEMQL_AI_ANTHROPIC_SERVICE_ACCOUNT_ID       MEMQL_AI_OPENAI_IDENTITY_TOKEN_FILE
MEMQL_AI_ANTHROPIC_WORKSPACE_ID
MEMQL_AI_ANTHROPIC_IDENTITY_TOKEN_FILE

They are the two vendors' credentials, and this is the one case where an absence is a legitimate configuration rather than a mistake. Every local cluster has all of them unset, because k3d's OIDC issuer is private and neither vendor can federate with it; a cloud cluster may reasonably federate one vendor and not the other. Left non-optional, either state would take that vendor's providers out of the registry with a WARNING rather than an error -- which, locally, is a log line nobody reads.

So the absence is not decided here. anthropicCredential (component/memql/ai_anthropic_federation.go) and openaiCredential (component/memql/ai_openai_federation.go) are the two places that know which combination is meaningful, and they decide the same way:

There is no API-key row in either table. Manually entered vendor API keys are gone from the product -- no env name, no seeded secret, no field in MemQL OS -- so "none of it" cannot mean "fall back to a key" and a half-configured federation has nothing to fall back to. That is the point of the refusal: falling back would work in every test and stop working an hour after a cutover, on whichever node nobody was watching.

Optionality is an ALLOW-LIST of exactly these names, not a new default; every other placeholder still fails its provider with the message above. MEMQL_AI_ANTHROPIC_WORKSPACE_ID is optional even when federating -- Anthropic needs it only when the rule spans more than one workspace.

Runbooks: auth/anthropic-federation.md · auth/openai-federation.md.

For partition-scoped resolvers (DSL resolveSecret(...) / resolveVariable(...)) the chain is:

1. v1:platform:partitionSecret -- partition row
2. v1:platform:globalSecret -- global row
3. (no env fallback for this path)

#The minimal install envelope

Installing, starting, repairing, upgrading and uninstalling a MemQL cluster require no AI provider credential, and make no call to any AI vendor.

That is the guarantee, stated plainly because it used to be false in one place: the install wizard listed an AI provider and a key file among its required fields, and the install graph ran an authenticated models-list probe against the vendor before it would touch the machine. Neither was ever an engine requirement -- a provider whose key does not resolve registers as unavailable and is skipped at selection, and nothing refuses boot over it. Epic memql#4440 removed the requirement; this section is the contract it left behind.

#What the guarantee covers

Supplying a key at install still works and behaves exactly as before: it is verified before anything on the machine changes, then seeded. Supplying nothing skips the providerKey step -- reported as skipped with the reason "no key supplied -- configure AI providers in MemQL OS" -- and seeds nothing.

#What is actually required

Read it from the registry, not from here. The authoritative set is the required: axis in scripts/secrets/manifest.yaml, consumed at boot by component/envregistry/bootvalidate.go keyed on MEMQL_NODE_TYPE. To ask it directly:

go run ./cmd/envscan -check          # every read, against the registry

As a shape rather than a list to keep in step: every node type needs the row store and the address it serves on, and identity adds its own public origin (which becomes the JWT issuer, and nothing can derive it). Nothing else in the registry is required by any node type, and no AI variable is required by any of them.

MEMQL_MASTER_KEY is not on the required: axis and is still effectively needed by every real deployment: it decrypts sealed values at rest, so a node that never reads an encrypted secret can boot without it and no realistic one does. It is separate from MEMQL_OPERATOR_KEY, which authenticates (memql#3519) -- see operator-credential.md.

Everything AI-related is console-configured: the models this cluster can call, and how it authenticates to them, live at Settings -> AI providers. Workload identity federation is the ONLY path for both vendors and needs no key at rest at all -- anthropic-federation.md and openai-federation.md. There is no key field, because there is no key.

#How this stays true

Four gates, because a guarantee in prose is a guarantee until the first plausible local reason to break it:

• TestNoAIVariableIsRequiredByAnyNodeType (component/envregistry) sweeps every MEMQL_AI_* / Anthropic / OpenAI entry against every node type.
• TestNoAIVariableIsInTheSealFloor keeps them out of the other requiredness axis. This is the one the audit found broken: the two instance-wide vendor API keys carried no optional: true while every sibling did, so a developer could not seal a .env without a vendor key -- the same requirement, in the one place nobody looked. Those two entries are gone entirely now (epic memql#5088); the gate stays, because the federation names that replaced them are on the same axis.
• TestMinimalEnvelopeIsWhatTheDocSays pins the per-node-type set, so a node type gaining a required variable fails a test and has to be justified rather than discovered by an operator.
• The install-e2e lane runs keyless by default (.github/workflows/install-e2e.yml), with the keyed round trip kept beside it for the verify path. A future step that quietly demands a key breaks CI rather than an operator.

The two requiredness axes are easy to conflate and the manifest's own comment spells them out: required: drives BOOT VALIDATION (which node types refuse to start), optional: drives the SEAL FLOOR (what a developer's .env must cover). An AI variable must be outside both.

#Bootstrap envelope (set in env, not in concepts)

These are read at process startup. Putting any of them in a concept would be circular -- the process can't reach the concept without them.

#Required to start

#Required when the matching feature is enabled

#Optional with sensible defaults

#Node identity

#Transport

#Logging

#Database tuning

All optional. Defaults baked into component/database/database.go:

A migration runs only after acquiring the shared migration lock. A competing replica defers and retries later. If a connection or context deadline expires, the server may still be executing the migration: the runner retains the lock and reports the failure instead of starting overlapping attempts. An ordinary SQL error acknowledged by PostgreSQL releases the lock for retry.

To recover a retained lock, first suspend migration attempts on every writer. Inspect pg_stat_activity using the identity node's application_name; wait for or cancel the abandoned migration and verify that it has stopped. Only then remove the matching bun_migrations row from bun_migration_locks and resume attempts. Check the failed migration's transaction and idempotency before retrying. Never clear a live migration's lock or mark a migration applied without completing its work. Increasing the migration context timeout alone does not change pgdriver's socket read deadline.

#Connection pooling: hybrid endpoint split (DIRECT_DSN)

Tiger Cloud PgBouncer transaction-mode pooling decouples client connections from server backends, so a deploy surge no longer maps 1:1 to Postgres slots (epic memql#1925). Transaction-mode poolers recycle a server backend between statements, which would drop a held session-scoped resource -- session advisory locks (the cron leader, the reconciler, planner admission) and the migrator's lock. MemQL therefore runs a hybrid endpoint split:

When DIRECT_DSN is unset, DirectBunDB() falls back to the main pool, so behaviour is identical to a single-pool deployment -- local / dev without a pooler is unaffected (env-agnostic). MAX_OPEN_CONNS / MAX_IDLE_CONNS govern only the main pool; the direct pool is bounded to a small fixed size by design.

#Auth (Identity service + per-node verifier)

For the identity binary (-tags identity):

For every other binary (bff/agent/planner/workbench/mcp) -- every node type except identity itself (the JWKS authority, which does not verify against itself) and edge (not an auth boundary; it serves public bytes to anonymous site visitors):

See docs/public/operate/auth/identity-service.md for the full operator narrative (anti-abuse knobs, key rotation, email delivery).

#Feature toggles & engine tuning

#Hosted deployables: runtime settings and the request log

Registered component: platform and component: edge; all four optional with working defaults, so a cluster that sets none of them still serves every site, still carries runtime settings and still records traffic. Full context: deployables.md.

#Streaming transcription (only if the STT path is used)

#Infra metadata

#WebSocket tuning (rarely overridden)

All in component/server/memqlws/env.go:

#Logs (the log store)

Every node type persists its log lines to the log_line hypertable and the MemQL OS Logs app reads them (epic memql#4893). Runbook: Logs.

#Concept-stored config

This is the table to look at when you ask "where do I put a new API key" or "where do I change the default model".

The authoritative manifest is scripts/secrets/manifest.yaml. Every entry in the manifest is a name go run ./scripts/secrets seed will look for in the --env-file you point it at, and push into the running MemQL when it finds a value. Names absent from the env file are skipped; the manifest is the allow-list, not the source of values.

#Default global secrets (manifest)

Stored in v1:platform:globalSecret, sealed under MEMQL_MASTER_KEY.

#Default global variables (manifest)

Stored in v1:platform:globalVariable.

#Variables consumed by the product frontend

These aren't in the manifest yet -- operators add them by writing the v1:platform:globalVariable row directly (the setGlobalVariable mutation) -- but they're documented here because they live in that concept and are read by the product frontend's runtime config layer (its publicConfig whitelist):

MEMQL_DEFAULT_PROVIDER, MEMQL_DEFAULT_CHAT_PROVIDER and MEMQL_DEFAULT_STREAM_PROVIDER are deleted (epic memql#5137). Each named a vendor model in a deployment manifest, which put a routing decision somewhere no rule could see and no decision record could explain -- and since every concrete record is a paid model, each was a paid default with an operator's name on it. Chat and streaming resolve through the router at a LEVEL now; see ai-routing.md. TTS is the one left, because the four levels are chat levels and a TTS call still names a provider.

The exact name on the MemQL side has to match the entry in the frontend's publicConfig whitelist exactly. To add a new one: add it to the whitelist, then either append it to the manifest and re-seed, or call setGlobalVariable(name: "...", value: "...") directly.

#Per-tenant overrides

Anything in v1:platform:globalSecret / v1:platform:globalVariable can be overridden per-tenant by writing the same name into v1:platform:partitionSecret / v1:platform:partitionVariable with the tenant's partition stamped on the row.

The resolver always tries the partition-scoped row first and falls back to the global one. So a tenant with MEMQL_SHOPIFY_ADMIN_TOKEN in their partition's v1:platform:partitionSecret will use their own token; everyone else keeps using the platform default.

This is the BYOK ("bring your own key") path, and it does NOT extend to the AI vendors: their credential is a projected Kubernetes token belonging to the pod, not a value any tenant can supply. The DSL surface is resolveSecret("MEMQL_SHOPIFY_ADMIN_TOKEN") and resolveVariable("..."); see component/memql/sense/builtins.go for the builtin docs.

#The env file the seeder reads

go run ./scripts/secrets seed --env-file <path> is the whole concept-seeding surface. It reads a plain KEY=value env file, keeps only the names the manifest lists, encrypts each secret under MEMQL_MASTER_KEY, and upserts the row into v1:platform:globalSecret / v1:platform:globalVariable over gRPC against the running MemQL.

# Requires MEMQL_MASTER_KEY in the environment. Default endpoint is
# https://bff.memql.localhost:443; override with MEMQL_GRPC_ENDPOINT.
go run ./scripts/secrets seed --env-file .env
go run ./scripts/secrets health     # gRPC handshake check: prints "ok"

The env file is operator-local and gitignored. It is a stash of values, not a schema: the manifest decides which names mean anything, and the row ids are derived (secret-global-<slug> / var-global-<slug>) so a re-seed overwrites in place rather than producing a second row.

This section used to describe a ~/.memql/dev-secrets.yaml stash with its own masterKey: / secrets: / variables: schema, plus init / list / export / one-off-set commands to manage it. The seeder reads a .env file and has never had those subcommands; nothing in the tree reads that yaml path (memql#4405).

In a cloud install:

• MEMQL_MASTER_KEY is set explicitly on the deploy target.
• Secrets and variables are seeded once against the cluster's gRPC endpoint, after which they live in the database.

#Where the env file sits in the bootstrap chain

The bootstrap values (including MEMQL_MASTER_KEY) are seeded onto the memql-secrets Secret by make up (scripts/k3d/seed-secrets.sh) and reach the pods via envFrom, so every pod has the key in env from first boot. The env file is only used by the concept-seeding step:

1. go run ./scripts/secrets seed --env-file <path> reads it.
2. It encrypts each manifest-listed entry under the master key (resolved from the seeded MEMQL_MASTER_KEY env) and upserts the row into the right concept over gRPC against the running MemQL.

#Backing concept state up before a repave

There is no export command. make down && make up recreates the database, so anything written directly with setGlobalSecret / setGlobalVariable and never added to your env file is gone. Keep the env file as the record: seed from it, and add to it anything you want to survive a repave.

#Master-key resolution order (in process)

component/secret/encryption.go reads MEMQL_MASTER_KEY from the OS env at first encrypt/decrypt call. There is no fallback. If absent when an encrypted secret is accessed, the process logs a fatal error. The seeded k8s Secret puts the key into the env before the pod's gRPC server starts. Inside the pod, the value is just an env var.

For non-dev installs, set MEMQL_MASTER_KEY directly on the deploy target (Cloud Run env, Kubernetes secret, etc.). The yaml is never deployed.

#How the cluster wires (peer discovery)

In cluster mode (multiple node-typed binaries), each non-BFF node needs to know how to reach BFF, and BFF needs to know how to reach each worker:

• MEMQL_PARENT_ADDRESS -- set on every worker (agent, planner, workbench). Tells the worker to dial BFF for outbound event forwarding.
• MEMQL_WORKER_PEERS -- set on every node that dials peers: BFF, and planner for its agent-only narrowing, and agent for its workbench-only narrowing under MEMQL_WORKBENCH_REMOTE=1. Comma- separated type=address list. First-boot seed only; once peers register themselves into v1:cluster:node (a global concept), DB-based discovery takes over. An entry whose type is not dialable (agent, identity, planner, workbench) or is otherwise unparseable is ignored, and the node logs a WARN naming the entry -- so a typo in the seed list is visible in the boot log rather than presenting as a peer that never appears (memql#3450).
• MEMQL_WORKBENCH_REMOTE -- set on the agent to require that workbench tool calls run on a workbench node. It is an assertion, not a preference: with it set and no reachable workbench peer, a workbench call is REFUSED (no_workbench_peer) rather than run on the agent's own disk (memql#3506). Silently degrading is what hid memql#3450 for its whole life.
• MEMQL_WORKBENCH_LOCAL_FALLBACK -- the explicit opt-in that restores the old degrade-to-local behaviour for an unreachable workbench. Off by default, which is the whole safety property: a fallback reachable by the absence of configuration fires exactly when nobody meant it to. See workbench-runbook.md.

Both are bootstrap envelope vars -- they have to be in the env before the gRPC server starts.

deploy/k8s/overlays/local (over deploy/k8s/base) has full worked examples -- the engine mesh (mcp + agent + planner + workbench), identity, and the local infra (Postgres + Azurite). The product bff head and SPA live in the downstream product pack's overlay, not here.

#Adding a new entry: decision tree

Is the value sensitive?
├── Yes → secret
│   ├── Tenant-overridable (BYOK)? → v1:platform:partitionSecret (default), with v1:platform:globalSecret as the global default
│   └── Instance-only?              → v1:platform:globalSecret only
└── No → variable
    ├── Tenant-overridable?          → v1:platform:partitionVariable (default), with v1:platform:globalVariable as the global default
    └── Instance-only?              → v1:platform:globalVariable only

If the value has to be available before MemQL connects to its database (i.e. it controls how MemQL connects), it's a bootstrap envelope var, not a concept entry. There's a strong bias against adding new entries to the bootstrap envelope -- it requires a deploy-config change every time it rotates.

#Adding a global secret

1. Append a row to scripts/secrets/manifest.yaml under secrets:.
2. Put YOUR_NAME=<value> in the env file you seed from.
3. go run ./scripts/secrets seed --env-file <path>.
4. Reference it from a provider/integration via env("YOUR_NAME") in .memql or os.Getenv("YOUR_NAME") in Go (the resolver chain works for both).

#Adding a global variable

1. Append a row to scripts/secrets/manifest.yaml under variables: and seed it the same way, or write the row directly with the setGlobalVariable(name: "YOUR_NAME", value: "...") mutation.
2. The DSL resolver returns it from resolveVariable("YOUR_NAME") or the same env() chain in provider auth.

#Adding a per-tenant (partition-scoped) entry

Call setPartitionSecret / setPartitionVariable directly. The seeder writes global rows only -- the manifest carries no partition-scoped entries -- and the partition dimension itself is being retired (#56), so treat these two concepts as legacy surface rather than the place to put new config.

#Reference: file paths

#Operational notes

#Rotating a secret

Put the new value in your env file under the same name and re-seed:

# Requires MEMQL_MASTER_KEY. For a cloud install, point the same command
# at that cluster by setting MEMQL_GRPC_ENDPOINT in the calling shell.
go run ./scripts/secrets seed --env-file .env

A single row can also be written directly with the setGlobalSecret mutation, which is what the seeder itself calls.

The old row is soft-deleted (active=false); lastUsedAt / rotatedAt get stamped on the new row. The next decrypt picks the new value; nothing else has to restart.

#The one exception: the campaign unsubscribe signing key

MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET does not rotate like the keys above, because what it signs lives outside the cluster: an RFC 8058 unsubscribe link sits in the recipient's mailbox for as long as they keep the message. Replacing the value on its own invalidates every such link ever sent, which is a compliance failure rather than a degraded feature.

It is therefore a two-variable rotation (memql#3458). Set MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET_PREVIOUS to the outgoing value in the same change, and then leave it set — it is a permanent second reader key, not a migration window. An old link keeps working forever, until a second rotation retires the key that signed it; the window is counted in rotations, not days. The worker warns at boot if this deployment has already sent campaign mail while holding only one key.

Full procedure and reasoning: campaign-sending.md.

#Backing up state before a wipe

There is no export command, so the env file has to BE the backup. Before any make down && make up (or make up-refresh), make sure every value you care about is in the file you seed from -- a row written directly with setGlobalSecret / setGlobalVariable and never added there does not survive the repave.

This section used to document an export subcommand that pulled every active row back out and merged it into a local yaml. It has never existed (memql#4405).

#"Why is my provider giving 'no value' errors?"

Check the resolver chain in order:

1. Is the row in v1:platform:globalSecret / v1:platform:globalVariable? Read it back through the globalSecret / globalVariable query, e.g. in DSL globalVariable({name: "MEMQL_IDENTITY_BASE_URL"}).
2. Does the running MemQL have MEMQL_MASTER_KEY set in env?
3. Is the master key the same one that encrypted the row? If you regenerated it, the existing rows are unreadable -- re-seed with go run ./scripts/secrets seed --env-file <path> to overwrite under the new key.

#Local override without touching the env file

setGlobalSecret / setGlobalVariable write directly to the running MemQL without modifying the env file. Useful for one-off experiments. Note that the next make down && make up recreates the database, and a re-seed replaces the value with whatever is in the env file -- so put it there too if you want it to last.

#Migration history

The current shape is the result of an 8-phase env-var refactor completed 2026-04-25. Decision summary:

• Two concept trees (globalSecret / globalVariable and partitionSecret / partitionVariable) so per-tenant BYOK overrides fall back cleanly to the platform default.
• NaCl secretbox (XSalsa20-Poly1305) over AES-GCM for the encrypted half because it has a smaller surface, no nonce-reuse pitfalls when keys aren't rotated, and the Go stdlib has no native AES-GCM with built-in random nonces.
• OS-env fallback stays because providers initialize eagerly at engine boot, before the seed step has populated concept storage. The fallback keeps the BFF alive through that bootstrap window. A lazy per-request resolver or a post-seed engine-reload hook would let us retire it.

#What is set up, and what is not

An entry declared here is a slot something can fill. Whether a slot is actually filled -- per module, per node and cluster-wide -- is configuration readiness: what the marks on an app's Settings entry mean, why two replicas can disagree mid-rollout, why "not reported" is a different answer from "not set up", and the modules: block that declares it.

#Operational record retention

The cron-leader job workJournalRetentionSweep runs nightly at 03:40 UTC. It archives complete historical versions, verifies the uploaded bytes, then removes only the archived versions in bounded transactions. Failed verification preserves the records. Active parent work and user-owned goals/runs remain protected.

Ages use the latest stored version. System runs with retained model-call or observation detail, pending approvals, or recent children stay until those constraints clear. An explicit environment setting overrides a stored global variable; the first five policies honor existing global settings, including the legacy WORKER_INVOCATION_RETENTION_DAYS global variable. Invalid/nonpositive values use the documented defaults.

MEMQL_WORK_ARCHIVE_CONTAINER defaults to MEMQL_AZURE_BLOB_CONTAINER. Archives use retention/<UTC-day>/<sha256>.ndjson.gz, with the original node identity, timestamps, schema, metadata, provenance and payload. Archive lifecycle is managed separately from database retention. No default age-based deletion applies to other concepts, including catalogs, files, goals and customer records.