The cluster front door — seven host rules

A MemQL cluster has one door: port 443 on one L7 proxy — the k3s-bundled traefik locally, nginx in the cloud — terminating TLS and routing by hostname. Port 80 exists only to redirect. One certificate covers the whole door locally; in the cloud it is two, split by which ACME challenge can issue each half (below).

Behind that door are seven host rules, committed to deploy/k8s.

Seven is the number, and it stays seven no matter how many customers, applications or websites the cluster serves. That property is what this page is about.

The host set is DERIVED from the closed role set plus the platform's own sites (memql#3767, memql#4224) rather than maintained as a list. It grows only when a ROLE or a platform site is added, which is a design change; it never grows with customers, apps or the sites an operator creates.

Related: environment-parity.md · install-prerequisites.md · reproduce-the-cloud-locally.md

#The seven hosts

os.<domain> and vscode.<domain> are the two platform sites (frontdoor.PlatformSites): sites the platform ships itself, each a seeded v1:platform:site row served by the edge exactly like a customer's. The OS shell is MemQL OS; vscode.<domain> is the landing page for the MemQL extension for Visual Studio Code and Cursor -- a static page, seeded like the OS, with its own exact rule for the same certificate reason (memql#5518).

Every host is a single label under the domain, and that is a routing decision. An Ingress wildcard matches exactly one label, so the one *.<domain> rule routes every present and future site to the edge.

It was once a TLS decision too — one *.<domain> certificate covering every role host and every site, with the apex as the lone extra SAN — and that was never true of a running cluster (memql#4224). letsencrypt-prod solves HTTP-01 only. ACME cannot serve an HTTP-01 challenge for a wildcard, and one wildcard dnsName fails the whole order, so the Certificate sat Pending; when it was hand-edited to exact names, the edge Ingress whose tls.hosts still said *.<domain> made ingress-nginx serve its self-signed default for os.<domain> (Safari: "This Connection Is Not Private"). So the front-door certificate memql-front-door-tls names exact hosts only — api., identity., mcp., os., vscode. and the apex — and every Ingress lists under tls exactly the hosts the certificate it points at can cover.

#The wildcard has a certificate now, and it is a second one

memql#4347 reverses the wildcard half of that, and only that half. A DNS-01 solver proves control of the zone by writing a TXT record, which is a claim over every name under it, so a wildcard is issuable where HTTP-01 could not even be asked. The cloud overlays declare a letsencrypt-dns01 ClusterIssuer — the Azure DNS solver, authenticating with a workload identity rather than a stored client secret, exactly as External Secrets reaches Key Vault — and one Certificate for *.<domain> plus the apex.

Why the apex is on both. *.<domain> matches exactly one label and the apex has none, so a sites-plane certificate without it could not serve the main website. The apex rule still terminates with memql-front-door-tls; the second SAN is what lets it move without a reissue.

Why two certificates and not one. The reversal is staged on purpose: sign-in (identity.), the API (api.) and the platform sites keep terminating with a Secret the DNS-01 issuer does not touch, so a wrong zone name, a missing role assignment or an expired federation cannot reach them. Until the install-time values below are real the wildcard Certificate simply never becomes Ready, memql-wildcard-tls is never written, and ingress-nginx serves its default for *.<domain> — which is precisely what a wildcard-routed site got before this existed. Nothing that works stops working.

Why it is safe to have at all. The wildcard key serves the site hostnames the edge node already serves, from the one edge process, which already answers on every one of them. One key for names one process already owns does not widen the blast radius; it removes an object pair per site.

The wildcard rule's tls entry is a kustomize patch, not a generator change. cmd/frontdoorhosts's whole input is the role set and the domain, and the certificate regime is neither — it is chosen per install. So the overlays add /spec/tls/- to edge-front-door rather than the generator emitting it, which also keeps front-door.generated.yaml byte-identical to what make frontdoor produces.

deploy/k8s/overlays/frontdoor_hosts_test.go gates both instance overlays, and it reads the solver, never the issuer's name: the exact-host certificate is still exactly frontdoor.CertificateSANs; a wildcard SAN is permitted only from a DNS-01 issuer declared in that overlay (an issuer the render cannot inspect is read as HTTP-01, so the strict regime holds by default); each Ingress's tls.hosts equals the rule hosts something in the overlay can certify; and every requested SAN is a host some rule serves.

Each platform site keeps its own exact rule — the certificate reason for it is gone, the server-block reason is not. ingress-nginx builds a certificate-bearing server block per rule host, never per tls host, so an exact rule is what makes os.<domain> and vscode.<domain> outrank *.<domain> and get their own certificate rather than the wildcard's. The platform sites are the sites the platform ships itself: their names exist before any operator creates a row, so the generator can write each rule and SAN, and the engine seeds each v1:platform:site hostname from MEMQL_DOMAIN through the same derivation (frontdoor.PlatformSiteHost).

WARNING: the wildcard certificate is a cloud thing with install-time prerequisites. It needs an Azure DNS zone for <domain>, a managed identity holding DNS Zone Contributor on that zone, a federated credential for system:serviceaccount:cert-manager:cert-manager, and the cert-manager controller Pod labelled for the workload-identity webhook. The steps are in azure-entry-install.md, in order. Skip them and you are back to the state above — routing without a certificate — not to a broken cluster.

(The set was once the product of role × ENVIRONMENT, with a label that hyphenated into role hosts — api-staging.<domain> — and nested into site hosts. Epic memql#3943 removed the environment dimension: MemQL ships one installation shape, and a second environment is a second install with its own domain, so the product has one factor left.)

api.<domain> is the API edge, and it is named for that role rather than for the first client that happened to connect to it. Everything that speaks to the engine dials it: the Cockpit, the VS Code extension, sdk/go, sdk/ts, workers, and the browser bridge at /memql/ws. So do strangers — a webhook POSTing to /inbound/<source>, a recipient's mail client executing RFC 8058 one-click against /unsubscribe. It is the one host carrying two backends; the next section is why.

identity.<domain> is the identity service: /login, the WebAuthn ceremonies, the OAuth endpoints, /.well-known/jwks.json, /me/*, /enroll and /device. It serves TLS in-cluster under the internal CA, which is why the proxy speaks https to it and skips verification on that hop — the hop stays inside the cluster network. See auth/identity-service.md.

mcp.<domain> is the MCP Streamable HTTP protocol head. It gets a host of its own rather than a path under api. for the same per-Service reason below, and it is not proxied through the edge either: MCP clients configure a URL, they are not browsers, and an extra hop on a tool-calling path buys nothing. See mcp-connect.md.

os.<domain>, vscode.<domain>, *.<domain> and the apex all point at the edge node, which resolves the request Host header against a v1:platform:site row and serves that site's bundle. The apex is not a special case: for a customer's own cluster the bare domain is their main website, so it is a site row like every other one. A platform site is a site like any other and takes the same path; its own rule exists for the certificate's sake (above), not because the edge treats it differently.

INFO: the edge route and the edge backend both ship as of memql#3714. deploy/k8s/overlays/local/edge-front-door.yaml carries the wildcard and apex rules (os-front-door.yaml and vscode-front-door.yaml the platform sites'); deploy/k8s/base/edge.yaml carries the Deployment, Service and PodDisruptionBudget behind svc/edge. The edge node type itself — build tag edge, make edge, see build-tags.md — is now a normal member of the mesh like any other leaf node, and its release image ships from .github/workflows/build-engine-images.yml alongside every other node type (CLAUDE.md's build-server rule). Before the backend existed, the absent-Service case produced a 404, not a 503 — worth keeping in mind, because it is the same 404 an UNROUTED hostname produces, and the two are not the same failure. Traefik drops the entire router when its backend Service is absent, so the request matches no router at all and gets traefik's own 404 — it never reaches a handler that could answer 503. Traefik said so on every reconcile:

ERR Cannot create service error="service not found" ingress=edge-front-door serviceName=edge servicePort=8085

An operator seeing 404 on a hosted-site hostname could not tell a missing rule apart from an undeployed node from the status code alone — both are Go's byte-identical http.NotFound. kubectl -n memql get svc edge is the discriminator, and now that the Service is a permanent fixture of the base manifests, the case this warns about is a regression rather than the expected state of main. A live edge still answers 404 for any hostname with no matching v1:platform:site row, and 503 only for a row whose status is disabled — that part of the contract does not change.

The site-hosting runbook covering how to deploy a site, roll one back and what the bundle contract is arrives with the same work. This page deliberately says nothing more about sites than the rule that routes them.

#Exact-versus-wildcard precedence is declared, not inherited

*.<domain> also matches api.<domain>, identity.<domain>, mcp.<domain>, os.<domain> and vscode.<domain>. So whether the named hosts keep their own backends is not a detail — it is a load-bearing assumption of the seven-host design, and it is worth knowing the state of it. (For the platform sites the question is moot — their rules and the wildcard reach the same Service — which is exactly why they are not in the precedence probe's host set.)

Precedence between an exact host and a wildcard host is implementation-defined. The Ingress specification says what a wildcard host matches — exactly one DNS label, never the apex — and says nothing about which rule wins when an exact rule and a wildcard rule both match. There is no spec guarantee to lean on.

On traefik's defaults it probably points the wrong way. Traefik's Kubernetes Ingress provider compiles a wildcard host into a HostRegexp matcher, and traefik's default router priority is the length of the matcher rule. In shape, the two rules are:

Host(`api.memql.localhost`) && PathPrefix(`/`)
HostRegexp(`^[a-zA-Z0-9-]+\.memql\.localhost$`) && PathPrefix(`/`)

The wildcard's rule is the longer string, so by that default heuristic it would outrank the exact one. (The precise regex traefik emits varies by version; the relative length is the part that carries the argument, and it is not close.) Left undeclared, the consequence would not be a narrow band of unusual hostnames — it would be that api.<domain>/ goes to the site server, so the entire API would be served by the edge: every SDK, the Cockpit, the VS Code extension and every worker dials that host. ingress-nginx ranks differently again — longest path, then oldest Ingress — so it would not even be the same wrong answer everywhere.

So precedence is declared, not inherited -- but not the way it first shipped (memql#3714, revised memql#3810). api-front-door.yaml, front-door.yaml (identity) and mcp-front-door.yaml originally each carried an explicit traefik.ingress.kubernetes.io/router.priority: "100" — comfortably above the wildcard's computed default. That broke the API: the annotation is Ingress-level, so the single value applied to every router that Ingress generates, and api-front-door.yaml alone declares 22 paths whose relative ordering had depended on traefik's default (the compiled rule length). A uniform "100" erased that variance, so / — the h2c gRPC catch-all pointed at bff:50051 — outranked the 21 specific paths pointed at bff-http:8085, and every HTTP path on api.<domain> answered 415 Unsupported Media Type with Content-Type: application/grpc.

The fix moved the declaration to the other side of the relationship instead of removing it: api-front-door.yaml, front-door.yaml, mcp-front-door.yaml and os-front-door.yaml carry NO router.priority annotation at all (each file says so explicitly, in a comment naming memql#3810). Precedence over the wildcard is still declared — it has to be, or the wildcard genuinely does swallow the exact hosts — but it is declared on the wildcard itself, in edge-front-door.yaml, whose two rules (*.<domain> and the apex) carry traefik.ingress.kubernetes.io/router.priority: "1". Priority 1 loses to every rule-length default, so all four exact hosts outrank the wildcard while keeping their own paths ordered by length exactly as before — the wildcard's two same-shaped / rules have no intra-host ordering for a uniform value to destroy, which is what makes it safe to put the annotation there and nowhere else. deploy/k8s/overlays/local/render_priority_test.go gates this shape: no Ingress declaring more than one distinct path may carry a uniform priority, and edge-front-door.yaml must carry one. On ingress-nginx nothing needs declaring: nginx resolves server_name by specificity in its own core — an exact name beats a leading wildcard — so the generated cloud front door carries no priority annotation at all, and cmd/frontdoorhosts/manifest.go says not to add one (an Ingress-level priority is what flattened the api host's path ordering on traefik). The principle is the same on both: know what the controller's ranking is, and do not trust a heuristic the spec does not guarantee.

Checked by scripts/install/verify-frontdoor.sh's precedence check, reported per host for api. and identity. only — never mcp., whose Ingress routes :8090 while the /healthz the check reads is on :8085, so it can report no better than inconclusive for that host. Before svc/edge existed, a run could not establish the property at all: with the backend Service absent, traefik drops the wildcard router entirely, so there was nothing for an exact host to take precedence over, and an exact host answering from its own backend under those conditions measured the absence of the wildcard router, not precedence. Reading that result as the latter is exactly how this assumption came to be written down as a fact the first time.

Worth being clear about even with the check passing: a probe compares responses at the paths it dials, so a passing check is evidence for the design rather than proof of it at every path. The declared priority is what holds everywhere, including the paths and hosts nothing dials — which is why declaring it matters more than measuring it, and why the declaration above is not conditioned on the check passing.

#One constraint explains most of this page

An ingress controller's backend protocol is a per-Service setting, not a per-port one.

That single fact shapes the whole front door:

• It is why bff and bff-http exist as two Services over one Deployment. Locally the bff Service carries traefik.ingress.kubernetes.io/service.serversscheme: h2c so traefik speaks HTTP/2 cleartext to the gRPC edge on :50051 — and that annotation applies to every port of that Service. The cloud has the mirror-image constraint (nginx backend-protocol: GRPC). An HTTP/1.1 asset fetch dialed as h2c does not work against Go's net/http server, so the shell's bundle would simply fail to load.
• It is why api.<domain> needs two rules, not one: / to svc/bff:50051 for gRPC, and the declared HTTP path set to svc/bff-http:8085 -- plus the one gRPC rule that is not the bff's, in the next section.
• It is why MCP is a host rather than a path: svc/mcp is a different Service on a different port speaking plain HTTP.
• It is why the HTTP path set has to be complete — which is the next section, and the reason it is generated.

Splitting protocols across Services makes each backend's protocol unambiguous, and it is the same shape everywhere. Only which ingress controller points at it differs, and that is a value rather than architecture.

#The one gRPC rule that is not the bff's

WorkerService.Stream -- the stream a cockpit machine opens to the cluster -- is served by the agent node and by nothing else. gRPC puts the fully qualified service name in the request path, so with only the / catch-all a cockpit dialling the documented https://api.<domain> reached the bff and was answered Unimplemented: unknown service znasllc.memql.worker.v1.WorkerService, forever, locally and in the cloud alike (epic memql#5218, D10). The machine never registered and the OS reported "waiting".

So the api host carries one more gRPC rule, ahead of the catch-all:

Three things about it are load-bearing:

• The prefix is spelled once, as component/frontdoor.WorkerServicePath, and deploy/k8s/overlays/frontdoor_worker_test.go holds it equal to the generated service descriptor's name. The trailing slash is what stops the rule matching a service whose name merely begins with this one.
• It is the shape of the system, so it is in the hand-authored local front door, in cmd/frontdoorhosts' generated gRPC Ingress, and in every account's reserved api. host (the bind script and the Go provisioner both emit it). Render gates in each place refuse a front door without it.
• The backend protocol still has to be declared per Service. In the cloud the rule lives in api-front-door-grpc, whose backend-protocol: GRPC is Ingress-scoped and covers both backends. Locally traefik's serversscheme is per Service, so the agent Service carries the same h2c annotation the bff Service does; the rule alone would hand the stream an HTTP/1.1 hop, and that fails with a protocol error naming nothing -- the same failure a missing rule produces, from the other side.

No priority is needed either way: traefik ranks by rule length and nginx by prefix length, so the longer prefix outranks / for exactly the reason every generated HTTP path does.

#WorkerService idle, stickiness, and Azure LB

Prod evidence (Jose): api-front-door-grpc answered HTTP 408 at exactly 60.000s four times while a cockpit held WorkerService.Stream — ingress-nginx defaults proxy-read-timeout / proxy-send-timeout to 60s. The generated Ingress therefore sets both to 14400 (4h), well above the ~30s gRPC keepalive the SDK dial and agent grpc.NewServer share.

Stickiness: agent runs 2 replicas. Without affinity a reconnect after drain can land on the other pod and flap connectedNodeId. The Ingress uses upstream-hash-by: $binary_remote_addr; the agent Service also sets sessionAffinity: ClientIP (gRPC-go has no cookie jar; MCP's cookie affinity remains the HTTP prior art).

Azure Load Balancer idle (DevOps): Standard LB / public IP defaults idle timeout to 4 minutes. Raise the frontend TCP idle timeout to ≥30 minutes on the api ingress Public IP (or set service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout on the ingress controller Service) so the LB does not cut a quiet held stream before nginx/gRPC keepalive. Follow-up: soak hours-idle without 408.

#What is generated

Two things in the front door are derived rather than authored, by two generators that answer different questions and stay separable.

The HOSTS — cmd/frontdoorhosts writes front-door.generated.yaml into each instance overlay (deploy/k8s/overlays/cloud and overlays/cloud-entry) whole: the seven Ingress rules and the cert-manager Certificate with its exact-host SANs. Its whole input is the closed role set, the platform sites and the domain, and it emits ~500 lines from those — which is what earns generation for a listed target.

The DNS-01 ClusterIssuer and the wildcard Certificate beside it (dns01-wildcard-tls.yaml, memql#4347) are not generated, for the same reason: they are neither the role set nor the domain but a certificate regime, chosen per install, and the values in them are Azure coordinates the generator has no input for. The one line they add to the generated file — the wildcard's tls entry on edge-front-door — is a kustomize patch in each overlay, so a hand edit cannot be reverted by the next make frontdoor run.

The PATHS on the api host — cmd/frontdoorpaths fills the block between markers in every api front door, from component/server's own declarations. The hosts generator preserves that block across a regeneration rather than re-deriving it; the rest of this section is about that half.

          # BEGIN generated bff HTTP paths -- make frontdoor-paths
          # END generated bff HTTP paths

The local overlay's five front-door files stay hand-authored: they are traefik rather than nginx, and they carry the measured reasoning for a priority ranking that broke the API once already (above). They are not unchecked — TestFrontDoorServesExactlyTheDerivedHosts computes the host set from the same component/frontdoor package the generator uses, so local's committed defaults cannot drift from what the cloud overlay serves.

The set is exactly server.PublicPaths() ∪ HandlerAuthorizedPaths() ∪ SelfAuthenticatedPaths() minus the gRPC surface. Those functions already exist, and TestContractRoutesMatchesRegistration already verifies them against what the handler really registers — so the front door reads from a list the repo keeps honest instead of from somebody's memory. Hand-edits between the markers are reverted by the generator; the pair mirrors make arch-model / make arch-model-check.

Why it is generated is MemQL's own history. POST /inbound/{source} and GET+POST /unsubscribe are documented public HTTP exceptions that third parties dial — a Shopify webhook, a mail client executing RFC 8058 — and no overlay in this repository routed either one.

And here is the part worth memorising, because it is what makes a hand-maintained list of ingress paths a bad idea rather than merely a chore:

WARNING: a missing rule does not 404. The HTTP/1.1 request falls through to the catch-all / rule, reaches the gRPC backend, and comes back as an HTTP 415 describing a content-type problem.

HTTP/1.1 415 Unsupported Media Type
Content-Type: application/grpc
Grpc-Status: 3
Grpc-Message: invalid gRPC request content-type ""
Content-Length: 0

Measured on a live cluster against api.memql.localhost, on four paths that were not yet in the generated block at the time: /unsubscribe, /inbound/shopify, /memql/query and /spaces/x/attachments. All four answered exactly that, with an empty body.

Read the trap carefully, because it is worse than an error that says nothing. The transport is fine — curl completes, TLS is fine, the door is up. The response names a cause, and it is the wrong one: it says the caller sent no gRPC content-type, when what actually happened is that a routing rule is missing and the request was handed to a server that only speaks gRPC. Nobody debugging a webhook that stopped arriving is going to read "invalid gRPC request content-type" and think "Ingress".

What makes it findable is that Grpc-Status: 3 on a path you expected to be plain HTTP is a specific, greppable signature. If you see it, check whether the path is in the generated block before you look at anything else.

That is why the list became generated output with a CI gate: a new public HTTP path on the bff either reaches the front door or breaks the build. For contrast, on the same host the routed paths answer 200 — /healthz returns the bff's health JSON, os.<domain>/ returns the shell's index.html.

See inbound-delivery.md and campaign-sending.md for the two exceptions themselves.

#Why the count is seven, and must not grow

Because a site is data, not infrastructure.

v1:platform:site carries a hostname, a kind, a bundle reference and a status. Deploying a site is an upload plus a row write; rolling one back is a row write. There is no routing rule per site, no git commit per site and no ArgoCD reconcile per site — so adding the tenth website to a cluster adds a row, not a routing rule and not an overlay patch.

The *.<domain> rule is what makes that possible: one wildcard rule routes every present and future site name to the one node that knows how to look them up — every name, that is, that the six exact hosts have not claimed, which is an assumption about rule ranking rather than a given (above).

The certificate is part of that claim again (memql#4347). It was not: under HTTP-01 the front-door certificate could only name exact hosts, so a new site got routing for free and TLS only with a Certificate and an exact-host Ingress of its own — one object pair per site, the one explicit exception to "a site is data". The DNS-01 wildcard closes that exception: a new site is a row, and *.<domain> already names it. Locally the mkcert pair is a wildcard and always was, so the two targets now agree on this too.

Two things still hold. A hostname outside <domain> — a customer's own apex, a second domain — is not under the wildcard and needs an exact-host Ingress and a Certificate of its own, one object pair each. It is still cluster-owner-only; what is no longer true is that somebody applies that pair by hand (the custom-domain regime, epic memql#4805). And the wildcard covers exactly one label, so shop.<domain> is certified and shop.eu.<domain> is not. The platform sites are the only sites whose rule and SAN the generator writes, because they are the sites whose names are known before any row exists.

This is not a hole in "ArgoCD is the only deploy path". That rule is about the shape of the system: the edge Deployment lives in git and is reconciled like everything else. A customer's landing page is no more a Kubernetes object than a chat message is.

Two consequences worth stating plainly:

• Adding an eighth host rule — a fourth ROLE or a third platform site — is a design change, not a configuration change. If a proposal needs one, the thing being added is probably a site an operator creates, which the wildcard already routes. TestRenderedHostsAreExactlyTheProduct fails on an eighth host rule that is not one of the seven.
• A wildcard DNS record is not a wildcard hosts file. In the cloud *.<domain> resolves at the DNS layer and nothing local is involved. On a developer machine there is no wildcard in /etc/hosts, so each name has to be listed in the managed block scripts/install/hosts-entries.sh owns. That is a property of hosts files, not of the front door.

#Custom domains: a client's own name

A client's own domain — www.acme.com, acme.com — is the one hostname shape this page's seven rules do not cover, and cannot: it is in somebody else's zone. Serving it needs an exact-host Ingress and an exact-host Certificate, which is exactly the object pair the section above calls the one explicit exception to "a site is data".

Epic memql#4805 automates that pair. It does not change the count, the roles, or the wildcard; it adds a second way for the edge to answer, and a state machine that gets the objects created only once the domain has proved it wants them.

#The two records, and why both

A binding is a v1:platform:customDomain row naming a hostname and a site. It does nothing until two DNS records check out:

Neither is redundant. Pointing proves the DNS owner wants traffic to come here; it does not prove the person who typed the hostname into this cluster is that owner — and on a shared install, "points at this cluster" is a property many unrelated tenants share. Ownership proves the reverse and nothing about routing. So issuing is reachable from exactly one place in the engine, and that place requires both.

Nothing is issued before both pass, and the reason is not tidiness: Let's Encrypt's certificates-per-registered-domain limit is shared across everyone under that domain, so an unverified issuance path is a way to spend somebody else's quota.

#Why HTTP-01 works here when it cannot issue *.<domain>

The same fact makes both true. HTTP-01 proves control by serving a token at the name, which a wildcard has no way to be — hence the exact-host certificate this page's whole first half is about. A client's domain, by the time this cluster asks for a certificate, is already pointing at it. The challenge is servable, so it issues.

#The regime in one paragraph

A scheduled sweep walks every non-terminal binding every two minutes, does the two lookups, and writes back what it saw with a typed reason (dns_token_missing, dns_not_pointing, no_acme_issuer, issuance_failed). When both pass it applies the pair; live follows only when the certificate's Ready condition says so. Removal walks the other way, and the row survives — what a cluster served, and when, is the audit. There is deliberately no manual re-check: retries ride the schedule, because a button would invite hammering a recursive resolver and an ACME endpoint.

#It is a VALUE that decides whether issuance is possible at all

MEMQL_CUSTOM_DOMAIN_ACME_ISSUER names the ClusterIssuer. Unset is a supported state, not a misconfiguration: the local k3d target's TLS comes from an mkcert pair and declares no ACME issuer, so a binding there verifies its DNS and then reports no_acme_issuer on every pass, forever. That is the same flow shape as the cloud, honest about what the target can do — as against a Certificate with an empty issuerRef, which the API server accepts and then leaves Pending with a condition nobody reads.

The rest are values too, and all have working defaults: MEMQL_CUSTOM_DOMAIN_EDGE_HOST (derived from MEMQL_DOMAIN as os.<domain> — a host this cluster already serves itself at, so it cannot drift out of agreement with where traffic lands), ..._MAX_PER_SITE (5), ..._NAMESPACE, ..._INGRESS_CLASS, ..._EDGE_SERVICE, ..._EDGE_PORT.

#Where the objects come from

The engine reaches the Kubernetes API directly, with the pod's own ServiceAccount and the narrow Role in deploy/k8s/base/custom-domain-rbac.yaml — two kinds, one namespace, no ClusterRole, no grant on secrets. That is the substrate memql#4257 established for deploy-control and for the same reason: the engine image is distroless, so it has no kubectl and no scripts/ tree, and a path that could only shell out would leave every binding in issuing forever on every deployed cluster. There is no client-go anywhere in the workspace.

scripts/deploy/bind-custom-domain.sh and its unbind mirror apply the identical pair and are the operator's path — contract-conformant capability scripts, allowlisted, runnable by hand or from the cockpit surface.

#What it deliberately does not do

Self-serve binding (v1 is cluster-owner/admin only; the v2 shape is a request-and-approve queue), registrar API integrations, wildcard client domains, certificate export, and local-cluster ACME.

#Per-account front doors: three hosts under a client's reserved name

An account that has proved it owns acme.com can reserve a MemQL name under it — memql.acme.com by default — and this cluster then serves three hosts beneath that name:

Epic memql#5168. It is the custom-domain machinery one level up: same DNS checks, same reconciler discipline, same capability scripts, same two substrates. Read the section above first — everything here assumes it.

#It does not add a source to the derivation

This is the load-bearing sentence on the page. MEMQL_DOMAIN remains the only input to component/frontdoor and to component/envregistry's env derivation. A reserved name never enters a rendered overlay, never changes an env value, never widens the issuer, and cmd/frontdoorhosts does not know accounts exist. The count of generated rules is still seven.

What a reserved name does is put four more Ingresses and one more Certificate into the cluster at runtime, through the reconciler path that already serves a client's own domain.

Teaching the generator instead was the alternative, and it was rejected for a practical reason before an architectural one: every new client would then be a regeneration, a merge and an ArgoCD sync — a release per customer — and a build-time artifact with a --check gate would be deriving itself from runtime rows.

#Why four Ingresses for three hosts

api. needs two. An ingress controller's backend protocol is a per-Service setting, so the bff's gRPC edge (h2c, :50051) and its HTTP edge (:8085) cannot share one object — exactly why api-front-door and api-front-door-grpc are two rules on one host in the generated manifests above. The gRPC object carries the worker-stream rule to svc/agent:50051 ahead of its catch-all, exactly as the cluster's own does (above) -- a cockpit may be told to dial a client's api. host, and a door that routed only the catch-all would answer it Unimplemented: unknown service.

The HTTP half routes the same generated path block the cluster's own api. host routes, from the same generator: cmd/frontdoorpaths writes component/frontdoor/paths.generated.go beside the three Ingress blocks, and the provisioner reads that slice. A hand-kept subset here would be the second source this page keeps warning about, and its failure mode is the one that names nothing — an HTTP/1.1 request handed to an h2c backend.

#One record per host, and one certificate for all three

Ownership is not checked again. The account's domain walk already proved acme.com with a TXT _memql-verify.acme.com token, and the reserved name is stamped only when that succeeded; memql.acme.com is under acme.com. So the only thing left to prove is pointing:

One target for all three, because one ingress controller terminates them all and routes by Host.

The certificate is the activation rule. One Certificate names all three hosts, and an HTTP-01 order cannot go Ready unless every dnsName in it solves — so a front door goes live only when all three names point here, with nothing for the reconciler to police. One order, one Let's Encrypt rate-limit unit, one Ready condition to promote on. A half-served door is the state worth avoiding: sign-in resolving while the API does not reaches a client's employee as a broken app rather than as a setup step an operator can see.

#What the engine does with a live door

• The edge gains a third resolution step, after the custom-domain alias: app.<reservedName> resolves to the OS site — the same os row every cluster serves — with the account attached. A site's own hostname still wins first and a live custom domain second.
• The runtime-config document gains an account block and points identityUrl at the door's own id. host, which is what keeps a client's employee on their own domain through sign-in.
• The identity service resolves the request Host to a live door row — never trusting the header — and uses the reserved name as the WebAuthn RP id (so one passkey works on app. and id. alike) and admits exactly one extra redirect URI for the OS client.
• The token is unchanged. One identity service, one keyset, one issuer: iss is still https://identity.<domain> whatever host minted it, and the per-node verifier is untouched.

#A live door is re-checked, not permanently trusted

Going live is not a permanent fact about DNS. The three names are looked up again every fifteen minutes, and after three consecutive failures the door is demoted to verifying — which stops the edge resolving app. and drops the door's callback out of the identity service's registered redirect URIs in the same row write. It re-verifies on its own if the records come back, and the certificate is still there, so recovery is fast.

This is a security control, not tidiness. A door whose app. host is repointed after it goes live would otherwise keep serving and keep its callback registered — and an authorization code delivered to a host the account no longer points here is a code delivered to whoever it points at instead. Three failures rather than one because the blast radius of a false positive is every one of that client's people, and a resolver hiccup is not drift.

#A withdrawn reservation tears the door down

Changing an account's domain clears the reserved name, the ownership token and the verification. The reconciler notices on its next pass and moves the door to removing; the three hosts stop resolving at that row write, not at the Ingress deletion. Serving three names whose ownership proof has just been discarded is the state this is designed to end.

#What it deliberately does not do

An identity provider per account; per-account branding on the identity service's own pages; a mcp.<reservedName> host (the MCP endpoint is a cluster fact a machine is configured with once); a certificate-renewal watcher of its own — cert-manager owns renewal; and, as for custom domains, self-serve binding and local-cluster ACME. A local cluster gets the same typed no_acme_issuer refusal, every pass, rather than a pretend success.

#Local versus cloud

The front door is the same shape everywhere. What differs is which implementation of each interchangeable part is in play.

WARNING: local proves routing, not certificates — still. The local mkcert wildcard is unconditional; the cloud wildcard depends on install-time Azure values, and when they are not set the Certificate never becomes Ready and every site falls back to ingress-nginx's default. The SAN sets now agree, so the divergence that remains is whether the certificate was actually issued — check kubectl -n memql get certificate memql-wildcard-tls before believing a site is served over TLS in the cloud.

Fixed everywhere: the seven host rules, the paths behind them, and the dial path a client uses (https://api.<domain>, TLS at the proxy, gRPC to the bff). The domain itself is a value too — one MEMQL_DOMAIN, from which every domain-shaped setting is derived at boot — so no file under deploy/ names a domain except the Ingress hosts, which carry a committed default.

The domain reaches every side through ONE derivation: cmd/frontdoorhosts composes the Ingress hosts and certificate SANs from it, component/envregistry/domain.go composes the issuer, CORS origins, redirect URIs and MCP public URL from it, and component/memql's SeedMaterializer composes each platform site row's hostname from it — all through component/frontdoor. Two copies of that rule would be two copies that can disagree, and the disagreement is an issuer nothing is served at, or a certificate naming a host the site row does not carry — which fails as "sign-in is broken" with every manifest looking correct.

What differs between the cloud and local is which of the interchangeable parts above is in play — plus that the local overlay's front door is hand-authored while the cloud one is generated, which is a source question rather than a shape one: the same seven rules either way, gated against the same derivation.

The standard those rows answer to, and the reason the divergences are confined to annotations and secret sources, is environment-parity.md. This page does not restate it.

#Port 5432 is debug-only

The local cluster maps Postgres on 5432 and that is not a connection path. It is there so an operator can attach psql to a local database. No client reaches MemQL that way anywhere, nothing in the connection model depends on it, and a proposal that routes application traffic through it is the port-forward-as-architecture anti-pattern environment-parity.md rejects.

#Verifying a front door

Requires a running cluster, so it belongs on an operator machine or in CI.

kubectl -n memql get ingress -o wide             # the Ingress objects and their hosts
scripts/install/verify-frontdoor.sh              # dns + tls + precedence per host, h2 once
curl -sS --http1.1 https://api.memql.localhost/healthz

verify-frontdoor.sh runs seven checks against the default host set, not three per host: DNS, TLS and precedence for each of api. and identity., plus one gRPC/h2 check, against the first host only. precedence is the one that CAN come back neither passed nor failed — an inconclusive check counts in neither tally, so it leaves allPassed alone (above) — but that is a property of the check, not a permanent state of this cluster: with svc/edge deployed (memql#3714) it is genuinely testable and reports passed, measured live:

allPassed: true, passedCount: 7, failedCount: 0, inconclusiveCount: 0
PASS dns  api.memql.localhost: resolves to 127.0.0.1
PASS tls  api.memql.localhost: TLS handshake ok against a trusted certificate (HTTP 415)
PASS dns  identity.memql.localhost: resolves to 127.0.0.1
PASS tls  identity.memql.localhost: TLS handshake ok against a trusted certificate (HTTP 302)
PASS grpc api.memql.localhost: negotiated HTTP/2 (HTTP 415) -- gRPC can ride this door
PASS precedence api.memql.localhost: /healthz answered by nodeType=bff, not by the
  wildcard's nodeType=edge -- the exact host rule takes precedence
PASS precedence identity.memql.localhost: /healthz answered by nodeType=identity, not
  by the wildcard's nodeType=edge -- the exact host rule takes precedence

Before svc/edge existed, the same command reported allPassed: true, passedCount: 5, failedCount: 0, inconclusiveCount: 2 — both precedence checks INCONCLUSIVE ("the wildcard router is not loaded, so there is nothing for an exact host to take precedence over"), because with no backend Service for the wildcard rule, traefik dropped that router entirely. That is history, not the current state of this front door; it is recorded above as the measurement that motivated declaring the priority explicitly rather than trusting it to still be true once a competing router existed.

Two things in that output are deliberate and neither is obvious.

The status code is informational, not a pass criterion. Note the PASS beside HTTP 415. The script measures transport reachability — does the name resolve to the expected address, does TLS handshake against a trusted certificate, does the door negotiate h2 — not application health. A 4xx passes because a door that answers 415 is a door that is up. (That 415 is the same one described above: the probe hits /, which is the gRPC backend.)

The probe set is smaller than the hosts block, on purpose. DEFAULT_HOSTS is api. plus identity. only, while the managed block scripts/install/hosts-entries.sh writes carries five names — the same five exact hosts the cloud certificate names. Those are different jobs: /etc/hosts has no wildcard semantics so it needs every name spelled out, whereas the probe set is only the doors we assert are up.

WARNING: an h2 probe passes while an HTTP/1.1 path is broken. That is exactly how /unsubscribe stayed unrouted without anyone noticing — and note the h2 check runs once, so it is not even asserting h2 on every host. Extending the script with a plain HTTP/1.1 probe of one bff HTTP path is tracked on epic memql#3700; until it lands, the curl line above is that check done by hand. --http1.1 is not decoration — curl prefers HTTP/2 over TLS by default, and an HTTP/2 request is not what a mail client or a webhook sender will send.

#Reference

• Design: docs/superpowers/specs/2026-08-13-cluster-front-door-design.md (epic memql#3700) — decisions D3 (the five rules as designed; memql#4224 added the platform site's exact rule and retired D2's wildcard SAN in the cloud, and memql#4347 reinstated it on a second, DNS-01-issued certificate), D5 (a site is data) and D9 (the same-origin proxy for hosted sites), each with the alternatives that were rejected.
• The DNS-01 reversal: docs/superpowers/specs/2026-08-22-library-artifacts-and-deployables-design.md section 4.5 (epic memql#4347), and the manifests at deploy/k8s/overlays/{cloud,cloud-entry}/dns01-wildcard-tls.yaml.
• The Service split and its annotations: deploy/k8s/components/engine-bff/bff.yaml.
• The path declarations the block is generated from: component/server/unauthenticated_surface.go.