Packages and deployables

A package is the memql-project-shaped tree a client repo already is: DSL domains, client apps, maybe a Go pack. Hand one to a cluster and it is live at a hostname in seconds -- versioned, rollback-able, and isolated from the platform when it breaks.

A package is a source. The things it produces are deployables, and the flow that composes one, gives it an address and manages it afterwards is deployables.md. This page is the source half: the manifest, what the analysis refuses, the pipeline's order, how DSL is delivered, and the full refusal catalogue.

Design records: packages and deployables (epic memql#4794), then the Deployables program and Compose (epic memql#4885), which folded the three sections into one flow and replaced the named cluster secret with a personal credential.

#The short version

1. Pick a source: a github.com URL, a zip already in Files, or your own CI pushing bundles. Nothing deploys.
2. Analyze. The cluster fetches, analyzes offline and parks a run at the confirm gate with a report you can read.
3. Deploy. Confirm it, with the address each new app goes to. It builds, stages, rolls and publishes -- in that order, always.
4. The repo stays the source of truth: when something newer lands, the source says so, and deploying the update is one click that starts the same run again at the new version.

Two clicks on a first deploy, one after. The surface is the MemQL OS Deployables app.

#The manifest

memql-package.yaml, at the root of the tree:

formatVersion: 1
name: acme-storefront
deployables:
  - name: storefront             # stable identity within the package
    displayName: Acme Storefront  # optional UI label
    deployment:                  # optional defaults, reviewed before deploy
      slug: quiet-cedar           # <slug>.<receiving-cluster-domain>
      domains: [shop.acme.com, www.acme.com]
    path: clients/web
    kind: shopify_storefront     # spa | static | shopify_storefront
    build:                       # optional; these ARE the defaults
      command: "npm ci && npm run build"
      output: dist
    resolutionTail: fallback     # optional; omitted means the kind decides
    binding:                     # shopify_storefront only
      store: acme.myshopify.com  # the v1:shopify:store row this fronts

The binding NAMES a store; it does not describe one. store is a v1:shopify:store row's myshopify.com domain, and the deploy resolves it to a row id and writes {storeId} onto the site. The domain the edge serves, the Storefront token reference and everything else about that store live on the row, so an edit to the store reaches every storefront bound to it with no second write -- and the manifest carries no secret and no copy. The domain is what the manifest names rather than the row id because a row id is a fact about one cluster's database, while the myshopify.com domain is the same identifier everywhere and one an operator can check by eye.

A store is attached on the deployable's Store panel. A manifest naming a store the deployer may not attach -- one this cluster does not have, one they may not read, or one they may read without holding the store part -- does not refuse the deploy. A first deploy places the storefront as a draft with no store, and the run records a non-fatal deployable_store_unknown note saying so; a redeploy of a storefront that is already bound keeps the store it has. An unattached storefront can go live for design review. Shopping requires a store connected on its Store panel. Once a later deploy can attach the manifest's store, it re-points the storefront to it.

Two halves, and the asymmetry is deliberate:

• Deployables are DECLARED. A directory with a package.json could be a site, a component library or tooling, and its kind -- whether a mistyped path 404s or falls back to index.html -- is a fact no walk can recover.
• DSL domains are DISCOVERED from dsl/<domain>/, exactly as the engine's own MEMQL_DSL_PATH mount discovers them.

Deployment configuration is optional. displayName changes the displayed label without changing name, which identifies the same app on redeploy. For a Shopify storefront, the existing deployment address and binding configure Production. A second stable Testing URL is assigned automatically: test--<production-hostname> (for example, test--graceful-fjord.cluster.example.com). Both destinations use one deployable and the same build. Testing access opens through MemQL OS's Visit → Testing action.

Set the independent testing store with the optional manifest block:

    testing:
      binding:
        store: sandbox.myshopify.com

Both bindings may name the same store, including a sandbox. An unconnected destination serves design preview. Omitting testing preserves the testing store chosen in MemQL OS; declaring a different testing store rebinds only Testing on deployment. An unavailable store leaves its prior binding intact and records a deployment note. A change to the declared testing store requires review before an automatic deployment.

deployment.slug is a lowercase DNS label of 3–40 characters under the receiving cluster's domain. Omit it to let the wizard generate an editable name. Existing site addresses are preserved on redeploy. deployment.domains lists custom hostnames without schemes, paths or wildcards. The wizard prefills these values, checks availability and shows the store binding before confirmation. Domain ownership verification and DNS setup still apply.

Explicit wizard placements override manifest defaults. An explicit empty domains: [] in a placement opts out for that run; an omitted placement inherits the declaration. New domain declarations are additive on redeploy. Omitting a setting never disconnects a store, removes a domain or relocates an existing site. An existing domain on the same site is reused. Changed declarations park automatic deployments for review. Secrets and connection credentials stay in MemQL, not in this file.

The Address step's Export manifest downloads the complete analyzed manifest with the chosen address and domain defaults. This includes generated names and preserves build, asset and store declarations. Commit that file in the source repository to carry the choices into subsequent imports; export never writes to GitHub automatically.

A bff/ with a go.mod is detected, reported and deferred: it appears in the report saying where Go delivery happens today (engine images built by CI), and every other half of the package deploys around it.

#Large files supplied separately

A GitHub repository package can keep videos and other large static files in release assets instead of including them in every source archive. Declare the files on their deployable:

formatVersion: 1
name: example
deployables:
  - name: website
    path: clients/website
    kind: static
    build: {output: out}
    assets:
      - path: media/walkthrough.mp4
        source: https://github.com/example/project/releases/download/media-v1/walkthrough.mp4
        sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
        size: 13831438

Use the file's actual SHA-256 and byte count. path is relative to the final published site, not the repository or build workspace. The build must not emit a file or directory that conflicts with that path. Files that the build command needs as inputs still belong in its source or its own input workflow.

Configuration validates these declarations without downloading the assets. Review includes their import in the build plan. After confirmation, Build builds the small site, downloads missing assets, verifies their sizes and digests, and stores them in the instance's private blob storage. Cached files are scoped to the package and shared across replicas. Subsequent builds reuse verified bytes, while rechecking the package's source credential. A changed asset declaration changes the automatic deployment approval fingerprint.

Asset URLs must name a release in the same GitHub repository as the package. The existing package source connection also authorizes private release reads; never put a token or signed URL in the manifest. GitHub supplies the original distribution copy. MemQL serves the imported copy through the existing site publisher, at the declared path, with the site's normal access and HTTP range behavior. A release tag moving or an asset being replaced cannot silently change a declared file: its digest must still match.

The existing per-file, complete-output, and file-count limits also apply to assets. An unavailable, oversized, or mismatched file stops the deployment before publishing. Keep generated payloads and authoring originals outside the current source tree once their durable copies have been verified; merely adding this block while retaining large files in Git does not shrink a source archive.

Local development uses Azurite. Both the BFF and workbench need the shared blob connection and MEMQL_AZURE_BLOB_CONTAINER; the local overlay also enables container creation for the emulator. Cloud instances supply their own storage configuration.

#Upgrading to an engine that reads the language line

Every DSL domain a package ships declares the language it is written in, in its own dsl/<domain>/memql.toml (the language line):

memql = "1.0"
edition = "2026"

An engine that reads the language line refuses to boot on a domain without one, and strict boot refuses the node rather than skipping the domain. An engine from before the language line ignores the file. So a package written before the line needs the file, and needs it on the cluster before the engine that reads it.

memqlmigrate writes the file into every domain that has none. Run it on the package's dsl/ directory:

memqlmigrate --rewrite=language-line -w dsl/

Or write the two lines by hand in each dsl/<domain>/memql.toml.

Do it in this order:

1. Add memql.toml to every domain, in the package's repository.
2. Redeploy every package whose DSL is staged on the cluster. A node reads the staged copy of a domain, never the repository, so a line that exists only in the repository is a line no node sees. Staging copies every file of a domain, memql.toml included, and the engine still running ignores it.
3. Roll the engine.

If the engine rolled first and the DSL-consuming nodes crash-loop on language_line_missing, recover in three steps:

1. Set the break-glass, MEMQL_DSL_ALLOW_SKIPS=1. It is a bootstrap variable, a key on the memql-secrets Secret (env-vars.md), so the pods see it when they restart. The nodes then boot, and each refused domain is skipped whole: its constructs are missing until the next step.
2. Redeploy the packages, now carrying the file. The deploy's roll restarts the nodes onto the new staged copies.
3. Remove the break-glass and restart the nodes. While it is set, a construct that fails to load is logged and dropped instead of refusing boot, so keep it set only as long as the recovery takes.

#What the analysis checks, and what it refuses

The analysis runs offline, before anything is fetched to a workbench or written to storage, and it runs the same gates a node runs at boot. "This DSL would refuse boot" is an answer you get here, not a crashlooping pod later.

Every refusal is a stable machine-readable code (component/packages/refusal.go). The OS renders its own sentence for the ones it knows and the server's own sentence verbatim for the rest -- inventing a friendly sentence for an unknown failure is how a real fault gets mistaken for somebody's mistake.

#Fatal to the whole package

#The source and its credential

#Reported, and NOT fatal

#Raised during the run

An unknown formatVersion refuses rather than parsing the subset it recognises, and so does an unknown KEY: deployabels: parses fine and describes a package that deploys nothing while reporting success.

#Who may deploy what

One UX, gated by CONTENTS:

• A package of web apps only deploys under the caller's own authority, and the sites it creates are theirs (the memql#4344 per-user ownership model).
• A package that ships any MemQL DSL requires an actor who may author constructs -- an owner or a developer, and deliberately not an admin -- and is refused with dsl_requires_authoring at the start of the run, before any build and before anything is staged.

The authoring line is not a second vocabulary: it is exactly create x construct, the capability auth.CanAuthor reads, which the MCP define surface and the two rule-arming gates already use. Deploying a DSL domain IS authoring constructs -- a whole domain of them at once -- so it asks the same question at a coarser grain. admin is user-management authority and holds only read on construct; developer is engineering authority and holds create/update/delete/execute.

An automatic run answers the same gate. Auto-deploy resolves the package OWNER's authority (nobody is at a keyboard to take one from), so an armed source that ships DSL deploys only while its owner may author constructs. It is not exempt, and it is not permanently barred either -- which it was, for everybody including the cluster owner, until the run started resolving an authority at all.

DSL is server-side authority for the whole cluster; multi-tenant DSL sandboxing is not built. An operator running their own instance never sees this gate.

#Two consequences of asking for a capability rather than a role

A custom role holding create x construct deploys DSL, at any rank. The gate reads the installed role catalog, so it admits a role this cluster authored for itself if that role holds the pair -- rank 1 included. Only an OWNER can put it there: roleCreate bounds a new role to grants the caller themselves holds, developer holds no create x role at all, and admin holds create x role but only read x construct. So the widening is an owner saying "this role authors constructs", which is what the sentence means. It is still wider than "owner or developer", and that is worth knowing before authoring a role.

A delegation ceiling cannot narrow developer to admin, and this pair is where that shows. auth.RoleAtMost clamps a role to a ceiling only when the ceiling's principal grants are a superset -- so RoleAtMost(owner, admin) DOES clamp to admin, while RoleAtMost(developer, admin) returns developer, because admin holds create/update/delete on principal and developer holds only read. The effect on a shared terminal pinned at admin (the class="badge" grant) is that the OWNER is refused a DSL deploy and a lower-ranked DEVELOPER is admitted. Recorded rather than fixed: that session can already author constructs directly through the MCP define surface, which is the same authority at a finer grain, so the gate is not what is leaking. The ceiling mechanism has a known hole on exactly this pair.

#The order, and why it never changes

fetch -> analyze -> confirm -> build -> stage -> roll -> publish

reversed on rollback:

publish back -> pointer back -> roll

Forward, the schema has to arrive before the app that uses it. Backward, the app has to stop using the schema before the schema changes. A failure anywhere before publish leaves every site serving exactly what it was serving -- ordering is what makes a partial deploy structurally impossible rather than merely unlikely.

A package with no DSL, or whose DSL is byte-identical to what is already staged, skips stage and roll entirely. Nothing restarts and the deploy lands in seconds. The surface draws those skipped stages with their reason rather than omitting them, so a fast deploy is legible as a fast deploy -- and a person counting missing steps has no way to tell "nothing had to restart" from "it stopped here".

The confirm gate lives on the ROW, at status: "awaiting_confirm", not in a browser. Somebody who closed the window finds their run exactly where they left it, and the list marks the deployable "a deploy is waiting for you" from packageDeploymentsAwaitingConfirm.

Confirming ADVANCES that run -- packageDeploy(confirm: true, deploymentId: <the parked run>) -- rather than starting a second one, and the resumed run re-reads the snapshot it already stored (memql#4954, memql#4955). One row per attempt is what makes the timeline readable, and it is what makes a Retry's promise survive the click that acts on it.

#Private sources

A source fetches under a personal credential its owner holds: v1:platform:sourceCredential, sealed server-side and named by the package's credentialId. The token is never on the package row, in a snapshot, or in a log line, and it is resolved under the PACKAGE OWNER's actor at fetch time -- so a package naming somebody else's credential is refused credential_not_found before any request leaves the cluster.

Add one on the Source stop while composing, or in the Sources group in the Deployables app's Settings section, which lists every credential you hold with the sources fetching under it. Rotation is adding a new credential and switching the source to it (updatePackageSource); revoking is one-way, and the row is kept because it is the history of what fetched under it.

The whole shape -- the fields, the resolution rule, the cluster-owner case, and what replaced the old cluster-wide named secret -- is in deployables.md. There is no cluster-wide source credential any more, and there is no shim: a package row still carrying a secret NAME from before reads as "no credential" and is asked for one.

#Delivering DSL: the pointer, the fetcher, the roll

DSL is data, not an image -- there is no registry, no image build and no CI in this path.

1. stage writes each domain's tree to blob://packages/<domain>/<contentHash>/. Content-addressed, so re-staging an unchanged tree writes the same bytes to the same place.
2. roll rewrites one small pointer document, blob://packages/active.json (domain -> prefix), and then restarts the DSL-consuming workloads.
3. On boot, the dsl-packages init container (memql dsl-fetch) reads that pointer and copies the named trees into the shared MEMQL_DSL_PATH volume. The node then boots exactly as it does today.

Rollback of the DSL half is pointing the document back and rolling again -- the same shape as a site rollback, and safe because an old prefix's bytes are still there.

#Applying the components

Three components, and the order is load-bearing:

# The layer that LABELS -- its own kustomization, consumed as a resource.
# labels/patches run AFTER components, so a `labels:` block beside the
# `components:` block below would be applied too late and every patch would
# match nothing (memql#4933). It renders, it applies, every pod is healthy,
# and not one of them has an init container or MEMQL_DSL_PATH.
patches:
  - target: { kind: Deployment, name: "^(bff|agent|planner|workbench)$" }
    patch: |
      - op: add
        path: /metadata/labels/memql~1product-dsl
        value: "true"

# ...and the layer that consumes it.
components:
  - ../../components/dsl-mount         # the shared volume, mount and MEMQL_DSL_PATH
  - ../../components/dsl-bundle        # optional: a product's own DSL image
  - ../../components/dsl-packages      # the fetcher for staged package DSL
  - ../../components/packages-roll-rbac # permission to restart the roll targets

dsl-mount owns the shared substrate and is applied exactly once; dsl-bundle and dsl-packages each add one init container, and their order in the list is the order those containers run. Both used to build the substrate for themselves, which made them mutually exclusive -- two volumes of one name is a Deployment the API server refuses -- and made dsl-packages silently dependent on dsl-bundle having created the initContainers list first.

Two committed examples render the working shapes and are gated by deploy/k8s/components/dsl-mount/component_test.go: components/examples/dsl-packages-only and components/examples/dsl-bundle-and-packages.

Label only the nodes that load DSL. A blanket label reaches redis, which has no volumes: key, and the first patch fails the whole render.

#Two things the fetcher and the roll each need

The fetcher needs to know which blob container to read. It reads MEMQL_AZURE_BLOB_CONTAINER, which is not a secret and is not in memql-secrets; the component takes it from an optional ConfigMap named memql-storage:

apiVersion: v1
kind: ConfigMap
metadata: { name: memql-storage, namespace: memql }
data: { MEMQL_AZURE_BLOB_CONTAINER: "memql" }

Without it the fetcher has nowhere to look, and says so and refuses -- it used to answer "no package pointer" instead, which is the ordinary state of a cluster with no packages, so the node booted healthy serving none of its packages' DSL.

The roll needs permission to restart pods. It patches each workload named by MEMQL_PACKAGES_ROLL_TARGETS as the node's own ServiceAccount (memql-engine), which by design holds no Kubernetes API privilege. The packages-roll-rbac component ships a Role pinned by resourceNames plus its binding. Its names and MEMQL_PACKAGES_ROLL_TARGETS must agree -- nothing can check that for you, and a name in one and not the other is either a 403 at the last stage of a deploy that otherwise succeeded, or a permission nothing uses.

#The fetcher's exit codes are the contract

memql dsl-fetch runs as an init container, so its exit code decides whether the pod starts.

• 0 -- the trees are in place, or there is no pointer at all. A cluster that has never deployed a package has no active.json, and that is the ordinary state rather than a fault; refusing to boot over it would mean the component could never be applied before the first deploy.
• 1 -- a pointer EXISTS and could not be honoured, or this container was never told where to look. Booting anyway would bring the node up with silently-missing product DSL, which presents as a healthy cluster answering "function not found" to every call it used to serve.

An unconfigured fetcher is the second case and not the first, and it used to be read as the first (memql#4933). "Nothing found, nothing wrong" is the right reading for a long-running engine process; for a container whose only job is to fetch, it is not -- a fetcher that cannot say where to look has not found an empty cluster, it has failed to look.

#Break-glass

If a roll leaves the cluster unhappy: put the pointer back and roll again. That is one blob write plus one restart, and it is the same operation the rollback button performs.

#Update detection

Two feeds, one effect. Both write exactly latestKnownVersion and updateAvailable on the package row, and neither ever deploys anything.

• Webhook (preferred). GitHub push and release events arrive through the existing POST /inbound/{source} seam -- deny-by-default source allowlist plus per-source HMAC (memql#2957). No new HTTP route. Point a repository webhook at https://api.<domain>/inbound/github, add github to MEMQL_INBOUND_SOURCE_ALLOWLIST, and set its HMAC secret the way that runbook describes. A cluster running GitHub Connect configures ONE webhook on the app instead of one per repository, and its secret is the app's own.
• Polling. A scheduled automation every ten minutes, for clusters no webhook can reach. It reads each repo-sourced package's upstream head under that package's own credential, resolved at call time under the package owner's actor and stored nowhere. A package whose credential cannot be resolved -- revoked, or not its owner's -- is skipped with a warning rather than polled anonymously, so a private repository never reads as unchanged because the request answered 404.

An upstream that has NOT moved is not written, so the OS cue does not flicker and the mesh does not carry an event per package per poll.

#The lifecycle, the archive, and deactivation

A standalone deployable's ladder (D10, plus the delete rung of epic memql#4937):

draft -> live <-> disabled -> archived -> deleted

• Archive requires disabled first. Archiving is the end of a deployable's life, and taking it offline is the step that gives anyone still using it a chance to notice.
• Archived restores to disabled, never straight to live: going live again is its own decision.
• An archived site answers 404 and still holds its name. To the internet it is gone, not paused (an OFFLINE one answers 503). siteDelete is the rung that releases the name.
• An archived row stays listed behind the Archived filter. An archive is a place, not a void -- which is exactly what the older soft-delete was not.
• The cluster's own OS site is exempt from all of it. It is re-seeded live at every boot; a status write on one is refused whoever asks, including a cluster owner, and the OS renders no controls on them at all.

A source's app is never archived; it is deactivated (2026-09-05 design, D2). It is reproducible from the source, so turning it off means giving its address back. packageDeactivateDeployable(packageId, deployableName, confirmName) walks the app's custom-domain bindings to removing, stamps its site deleted (the write the cluster-wide uniqueness probe reads, so the address is free at that instant), disarms the source's auto-deploy when this was its last app, and puts the name on the source's disabledDeployables -- in that order, the site before the preference. It is offered from ANY state the app is in, live included; the typed confirmation is the app's manifest name, verified server-side. An app the source declares and never deployed only lands on the off-list. enablePackageDeployables is the inverse's first half; the compose flow, scoped to that app, is the second.

Archiving a source deactivates every app it produced, then archives the source (D3). packageArchive verifies the typed name, then for each site the source deployed -- live ones included; the OS names them before it asks -- releases its domains and deletes it, puts every declared app on the off-list, disarms auto-deploy when it was armed, and archives the package LAST, so a failure part-way leaves a source that is still active and whose page still says what happened. The reply names the addresses released. packageRestore does not cascade back: a restored source's apps come back inactive, to be activated and deployed at fresh addresses. There is no package_has_active_deployables refusal any more.

One source, once (D8). A write guard beside the hostname policy refuses a v1:platform:package create -- or a source edit -- naming a repository and ref another ACTIVE package already tracks, cluster-wide and normalised (scheme, www., .git, a trailing slash, case, the SSH form). An empty ref is its own value there, because the guard cannot resolve what the default branch is called; the OS's Source stop, which knows it from the probe, reads "" and main as one. Archived packages do not count, so a source can be archived and added again.

Archiving asks you to type a confirmation -- a standalone deployable's address label (or whole hostname) for siteArchive, the package's own name for packageArchive. The server verifies it, not just the form (archive_confirmation_mismatch), and a mismatch writes nothing.

#Configuration

A cap that is set and unparseable, or non-positive, falls back to its DEFAULT rather than to "no limit".

MEMQL_PACKAGES_ROLL_TARGETS is deliberately not defaulted: a roll restarts pods, so its blast radius is a decision somebody made rather than whatever happens to be running. Only clusters hosting DSL-carrying packages need it. Every name in it must also appear in the roll Role's resourceNames -- setting the variable alone leaves the roll failing on a 403, with fetch, analyze, build and stage all green behind it. See deploy/k8s/components/packages-roll-rbac.

Sealing and unsealing a source credential uses MEMQL_MASTER_KEY, which every node already has; a ciphertext this node cannot unseal is refused as source_unreadable naming that variable, because the repair is an operator's.

#Builds

A deployable whose built output is already in the source (dist/index.html present) deploys with no build at all -- no build surface, no network, nothing to configure. That covers every tree whose own CI already builds it, which is what the memql-project template produces, and it is still the fastest path. The Build stop draws itself skipped, with the reason: its built output is in the source.

Everything else builds on the workbench (epic memql#4900).

#Where a build runs, and what it can see

A package's build script is somebody else's code: npm ci runs whatever a dependency put in a postinstall hook. So the command never runs in an engine process. It runs on a workbench node, in a directory that exists for the length of one build, under an environment this cluster constructs:

Constructing the environment covers what the command is handed. It runs as uid 10001 as well, which covers what it can go and read: the engine runs as root, and a build running as root could read /proc/1/environ and take everything the pod holds. MEMQL_PACKAGES_BUILD_UID names the uid.

That is a boundary, not a jail. A build can still reach the network and spend the pod's CPU; those are bounded by the pod's own limits, by the timeout, and by the directory being destroyed when the call returns -- whether the build succeeded, failed or timed out.

#The image

The workbench node type is the only one whose image carries a Node toolchain and git (the workbench-runtime stage in the Dockerfile). Node comes from the same pinned node:22 image this repo builds its own SPAs with, so a package builds against the toolchain the platform's own bundles do.

#Which node builds

The bff serves packageDeploy, and it forwards the build to a workbench replica over NodeService.Stream -- MEMQL_WORKBENCH_REMOTE=1 plus MEMQL_WORKER_PEERS=workbench=workbench:50060, both set in deploy/k8s/components/engine-bff. With no reachable workbench peer a build is refused (no_workbench_peer), never run on the bff: running somebody else's build script in the process holding the cluster's front door is not the isolation that flag asks for.

The forward carries a system-class assertion -- this cluster's engine acting for itself -- and the workbench refuses a build request carrying any other. That is what keeps the build entry unreachable from an agent's tool loop, which re-asserts its own caller's class and can never mint a system one.

The apps of one package prefer the same replica: the first app's node is passed as the second's affinity pin, so one run reads as one place in the logs. builtOn on the deployment row records the surface and the node.

#When a build goes wrong

Every one of them leaves every site serving exactly what it was serving: the build stage is before publish in the D6 order.

#Builds on your own machine

A target can declare that its kind builds on a machine in the owner's own Fleet -- an iOS or macOS build needs Xcode, which needs macOS. The route ships as the selection mechanism and its hop test, with no registered kind that reaches it: targets.go maps every offered kind to the workbench, and a test pins that. What is real is choosing the machine (by exact label, under the owner, with no_worker_available naming every machine considered and why each was ruled out); what is deliberately refused is the dispatch, because building on somebody's laptop is a computer-use act and a deploy has no approved plan to hang that consent on. The first target that needs a Mac brings that decision with it.

#When a node is lost mid-deploy

A deploy is one call on one node. Until epic memql#4900 a node lost mid-run left its row at a non-terminal status forever, and the surface said "building" for as long as anybody looked.

A running pipeline now writes heartbeatAt every MEMQL_PACKAGES_HEARTBEAT_SECONDS, and a sweep every two minutes closes any run whose heartbeat is older than MEMQL_PACKAGES_ABANDONED_AFTER_SECONDS with the terminal status abandoned.

• Never failed. The sweep does not know whether the build was about to succeed; it knows the node stopped answering. The error it stamps names the node and when it was last heard from, and the OS says "this cluster lost the node that was running it; nothing was published".
• The threshold is six heartbeats, not two. A run that misses one had a slow moment. Six in a row is a node that is gone. The threshold is clamped up to three heartbeats if configured below that, because a cluster that sweeps its own healthy deploys presents as a broken build surface rather than as a setting.
• Retry deploys what the lost run was deploying. packageDeploy takes fromDeploymentId, and a run started that way re-analyses the snapshot the earlier run stored rather than fetching the source again -- so a Retry does not quietly deploy whatever the branch has moved to since. A run that kept no snapshot (every run from before this epic) is refused with snapshot_unavailable, and the sentence says to deploy again instead.
• And it survives the confirm. The retry parks with its report, and the click that answers the gate used to be a fresh call carrying no fromDeploymentId at all -- so a person retrying a lost deploy of commit A, after commit B had landed, got B while reading a report describing A (memql#4955). The retry's own row records both what it was retried FROM and the snapshot it ran against, and the confirmation resumes that row.

The sweep runs on the cron leader only, so each stranded row is closed once, and under the maintenance actor, because it reads every owner's runs.

#Auto-deploy

A per-source switch: deploy the update by itself when the plan is unchanged. Off by default and off for every source that has never been switched.

With it on, the update feeds start the run they have never been allowed to start -- requested by the source's owner, marked automatic on the row -- and the run confirms itself only when the new analysis plans exactly what the last successful deploy planned: the same apps by name, kind, path, build command and output directory, the same MemQL domains. Anything else parks at the confirm gate exactly as a person's deploy does, and the surface says why.

The gate is not skipped. It is answered, and only by a plan somebody already said yes to. A changed build command is the case that matters most -- it is somebody else's shell command arriving on your cluster -- and it always parks.

Scheduled repository polling runs on one elected workbench replica, independently of general maintenance leadership. If no workbench is available, polling waits; another node never attempts the build locally. Other scheduled automations keep their existing leadership. The supported local k3d stack includes its own workbench with this same role and lease. A standalone BFF, including a process with MEMQL_NODE_TYPE unset, does not poll repositories or serve as a fallback build node.

The deployment rules are:

• One live run per source. A shared database gate serializes the short read-and-open step across replicas. An existing attempt cannot be opened again, and another live attempt must finish or be canceled before a new one starts. Confirming a parked attempt continues that same run.
• A cluster-owned source cannot auto-deploy. There is nobody to run as.
• Infrastructure retries are limited. A missing workbench or a build process that could not start can retry automatically twice after the original attempt, at least ten minutes apart. Each retry opens a new row and uses the original source snapshot; it still passes the plan confirmation gate. The same budget also recovers a pre-0.23.4 bundle publication incorrectly refused for its unchanged Shopify binding, only while the site, source, owner, and store still match the recorded failure. New or changed bindings still require store access. Canceled runs, missing snapshots, source or credential failures, build errors, and timeouts require attention. After three failed attempts for one revision and policy setting, automatic attempts stop. The failure and update remain visible; repair the deployment failure and use Retry. A later source revision has its own limit.

Workbench builds are the Build epic of the Deployables program. They change what the Build stop SAYS -- progress on the surface that built it, and the log -- not where the stop is.

#Where the pieces live

#Campaign email configuration

The same memql-package.yaml can carry public email choices alongside its Deployables. The optional campaigns section contains one Azure configuration and an organization binding for each domain:

campaigns:
  azure:
    tenantId: 11111111-1111-1111-1111-111111111111
    subscriptionId: 22222222-2222-2222-2222-222222222222
    resourceGroup: client-email
    dataLocation: United States
  domains:
    - organization: example.com
      domain: example.com
      emailService: example-email
      communicationService: example-delivery
      sender:
        username: news
        displayName: Example
        replyTo: support@example.com
      dns:
        - purpose: Domain
          name: example.com
          type: TXT
          value: ms-domain-verification=provider-supplied-value
          ttl: 3600

organization is a portable label, normally the organization's domain. It is not a cluster row ID and grants no access. Each organization has its own services, domain and sender. Optional createResourceGroup: true requires resourceGroupLocation; omit them to select an existing group. DNS purposes are Domain, SPF, DKIM and DKIM2; copy the actual provider records. Merge SPF with the existing policy, rather than publishing a second SPF record. There is no MX change: campaign sending does not create a receiving mailbox.

An owner or developer can use Campaigns → Settings → Package configuration to load a file, review one package organization, and import it into the explicitly selected organization on this cluster. Import saves setup choices; it never creates Azure resources, sends email, changes DNS, transfers resource ownership, or accepts a claimed verification status. The existing wizard obtains Microsoft consent once for the cluster, provisions reviewed resources, verifies DNS live, and creates the selected sender. A working connection with different resources or a different active sender is refused until explicitly reviewed/disconnected.

Export configuration reads the selected organization's current Azure DNS records and merges its public configuration into the loaded file. All declared Deployables, builds, assets, store bindings, addresses and other organizations remain. Load the existing package first when it is not supplied by the installed instance; exporting without one starts a campaign-only package. Export downloads the file for review and commit to the instance/product repository. Deployables' address export also preserves the campaign section.

Tokens, access keys, client secrets, unsubscribe keys and “verified” flags are not manifest fields and are rejected. Credentials stay encrypted in the cluster secret store. A package file is portable configuration, not an authorization backup; another installation must obtain its own consent and check ownership. Resources tagged for a different cluster or organization are never adopted just because the file names them. The unsubscribe key and URL are installation infrastructure, described in Campaign sending.

#Installation defaults from the product repository

To supply the file during setup, the instance composes a ConfigMap named memql-package with a memql-package.yaml key. A repository-root Kustomization can generate it directly from the package file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
  - name: memql-package
    files: [memql-package.yaml]
    options:
      disableNameSuffixHash: true

Apply this Kustomization during bring-up, or generate a ConfigMap snapshot for both overlays with kubectl kustomize .. Keep that generated file beside its own Kustomization in a child directory, check it against regenerated output in CI, and compose that child directory from both overlays. Nested Kustomizations cannot reference their own ancestor root; keep the normal load restrictions instead of disabling them to reach this file. The package YAML remains the single authored source. Engine nodes mount this optional ConfigMap at /etc/memql/package; they read it afresh when setup is opened. Kubernetes projection updates may take a short time. Missing files mean no defaults; invalid files produce a setup error.

The mounted choices prefill an unconfigured cluster and a matching organization's form. They never overwrite a saved connection, so edits are reviewed through explicit import. Publishing a website package does not grant permission to provision email; email import/setup is a separate explicit action. Changes to campaign declarations also change the package plan fingerprint and require review when an automatic website deployment encounters them.