Inbound delivery -- the webhook receiver, source allowlists, and signatures

Audience: product authors ingesting third-party events from pure DSL, and operators configuring per-deployment ingress. Design: docs/internal/design/inbound-delivery-adr.md (memql#2957).

The counterpart to outbound delivery. Outbound is the outbox a product stages and the engine drains; inbound is the inbox the engine stages and a product drains.

A third party POSTs to /inbound/{source} on the bff. The receiver checks the source against a deploy-configured allowlist, verifies the request's HMAC signature, stages it as a v1:platform:inboundRequest row, and answers 202. A product automation triggers on the new row. No product Go, no sidecar.

#Nothing is admitted until you say so

The route is mounted on every bff, and out of the box it answers 404 to everything. A source is served only when it is both listed and fully configured; a source that is listed but not configured is dropped and also answers 404, with the reason in the node's log at boot.

That is deliberate. A receiver serving one of five sources is a visible partial outage. A receiver serving five sources with one of them unsigned is a silent hole.

#Configuring a source

Global knobs:

Per source. <NAME> is the source name uppercased with - mapped to _, so the source big-corp reads MEMQL_INBOUND_SOURCE_BIG_CORP_*:

There is no vendor list. A source names the encoding its signature uses, so a new sender is a config change rather than a release.

The last three describe a composite header -- one header carrying several k=v elements rather than a single value. That is the same kind of fact as the scheme: how the bytes are laid out, not who sent them. They are all-or-nothing, and a partial configuration is refused at boot rather than at request time, because both halves of a partial one fail as the same flat 401 from a configuration that looks more complete than the one that would work.

A GitHub-shaped sender:

MEMQL_INBOUND_SOURCE_ALLOWLIST=gh
MEMQL_INBOUND_SOURCE_GH_SIGNATURE_SCHEME=hmac-sha256-hex
MEMQL_INBOUND_SOURCE_GH_SIGNATURE_HEADER=X-Hub-Signature-256
MEMQL_INBOUND_SOURCE_GH_SIGNATURE_PREFIX=sha256=
MEMQL_INBOUND_SOURCE_GH_SECRET=<shared secret>

The github source specifically is the one a cluster running GitHub Connect configures, and its secret is not a value you choose here: it is MEMQL_GITHUB_APP_WEBHOOK_SECRET, the GitHub App's own webhook secret, so the app signs with it and this receiver verifies with it. The name must be github rather than gh, because that is the segment MEMQL_PACKAGES_WEBHOOK_SOURCE defaults to and therefore the one the packages feed reads deliveries from.

MEMQL_INBOUND_SOURCE_ALLOWLIST=github
MEMQL_INBOUND_SOURCE_GITHUB_SIGNATURE_SCHEME=hmac-sha256-hex
MEMQL_INBOUND_SOURCE_GITHUB_SIGNATURE_HEADER=X-Hub-Signature-256
MEMQL_INBOUND_SOURCE_GITHUB_SIGNATURE_PREFIX=sha256=
MEMQL_INBOUND_SOURCE_GITHUB_SECRET=<the same value as MEMQL_GITHUB_APP_WEBHOOK_SECRET>

A Stripe-shaped one, whose single header carries both the timestamp and the digest (Stripe-Signature: t=1614556800,v1=<hex>):

MEMQL_INBOUND_SOURCE_STRIPE_SIGNATURE_SCHEME=hmac-sha256-hex
MEMQL_INBOUND_SOURCE_STRIPE_SIGNATURE_HEADER=Stripe-Signature
MEMQL_INBOUND_SOURCE_STRIPE_ELEMENT_SEPARATOR=,
MEMQL_INBOUND_SOURCE_STRIPE_SIGNATURE_ELEMENT=v1
MEMQL_INBOUND_SOURCE_STRIPE_TIMESTAMP_ELEMENT=t
MEMQL_INBOUND_SOURCE_STRIPE_SECRET=<shared secret>

Note there is no ..._SIGNATURE_PREFIX here and it would not help: v1= is not a leading prefix, because t=<unix>, comes first. That is precisely the shape the element vars exist for.

A Shopify-shaped one, for a custom app's webhook secret. The source is named shopify-custom, not shopify: a connector's own name is its app-level source (for Shopify, the three compliance topics, verified with the managed app's secret), and an env pin on a connector's own name is dropped AT BOOT, with an ERROR naming the MEMQL_INBOUND_SOURCE_<NAME>_* vars, exactly like a source with no secret -- the connector would otherwise read a body verified by the env secret as app-signed, and the connector for its part refuses an app-level row it cannot have signed for. The receiver refuses the same pin per request as well, so the answer to every such delivery is 404 either way; the boot line is what says why.

MEMQL_INBOUND_SOURCE_SHOPIFY_CUSTOM_SIGNATURE_SCHEME=hmac-sha256-base64
MEMQL_INBOUND_SOURCE_SHOPIFY_CUSTOM_SIGNATURE_HEADER=X-Shopify-Hmac-Sha256
MEMQL_INBOUND_SOURCE_SHOPIFY_CUSTOM_DEDUPE_HEADER=X-Shopify-Webhook-Id
MEMQL_INBOUND_SOURCE_SHOPIFY_CUSTOM_FORWARD_HEADERS=X-Shopify-Topic,X-Shopify-Shop-Domain,X-Shopify-Webhook-Id
MEMQL_INBOUND_SOURCE_SHOPIFY_CUSTOM_SECRET=<shared secret>

..._SECRET is secret material. It belongs in the deployment's secret store, never in an overlay.

#scheme=none

Accepts without verifying, for a sender already behind another trust boundary (a mesh-internal producer, a gateway that verified upstream). It has to be spelled out per source -- an unset scheme is an error, never a silent downgrade -- it logs a warning at boot, and every row it stages carries signatureVerified=false so an audit query finds them.

#What a connector's Apply returns

A staged row is worked by whatever serves its source. Since epic memql#4378 the engine ships a dispatcher that does the routing, so the usual answer is implement a connector, not write an automation:

dispatchInboundToConnector      an engine automation on inboundRequest.created
  -> the connector whose Name() matches the row's `source`
  -> Connector.Apply(ctx, InboundRequest) -> []MirrorWrite
  -> the runtime writes each one behind the VERSION GUARD
  -> the request row is stamped `processed` or `failed`

Apply receives the row's fields as an InboundRequest — Source, Topic, Body, Headers, ReceivedAt and the staged row's id — and returns the rows it wants written. It does not write them itself: the runtime does, under the connector's own actor, and refuses any write whose version is older than what MemQL already holds (recording it as stale). That is what stops an out-of-order webhook regressing a mirror.

Returning no writes is normal, not a failure. /inbound/{source} is a shared door and most of what comes through it belongs to something else; an unrecognised delivery is stamped processed and forgotten.

See data origins for the contract and integrations/CLAUDE.md for the recipe.

#Writing an automation instead

Still supported, and still the right answer when the delivery is not mirrored data — a notification to act on, a job to enqueue, something with no system of record on the other side:

/// Work a staged order delivery from the store's webhook.
@trigger(event="node.created", concept="v1:platform:inboundRequest")
@filter(row => row.source == "shopify-orders")
automation handleShopifyOrder {
  args {
    id any
    body any
  }
  mutation updateInboundRequestStatus(requestId: args.id, status: "processing")
  order := logic parseShopifyOrder(body: args.body)
  mutation recordOrder(orderId: order.id, total: order.total)
  mutation updateInboundRequestStatus(requestId: args.id, status: "processed", processedAt: now)
}

Filter on source in the trigger's @filter, declare the payload fields the body reads in its args block (body, contentType, ...), and stamp updateInboundRequestStatus with processing / processed / failed as you work the row. The engine only ever writes the initial received; everything after that is yours.

Both can coexist: the dispatcher only ever offers a row to the connector its source names, so an automation on a different source never sees a connector and vice versa.

The row also carries contentType, dedupeKey, signatureVerified and receivedAt. Query staged rows with inboundRequestsByStatus(status: "received"), or look one up with inboundRequestByDedupeKey.

#Redelivery

Every sender in scope retries, so redelivery is normal.

The row id is derived from (source, <digest of the signed payload>) -- the body, or timestamp + "." + body where the source configures a timestamp header. So a redelivery lands on the same row, and the staging mutation preserves your handling state rather than resetting it to received.

The row collapses. The event does not. Staging a redelivery is still a write, and a write still publishes node.created, so an automation triggering on that event fires again. What this gives you is at-least-once delivery with a stable idempotency key -- not exactly-once processing.

That distinction is the contract, so write your automation to be idempotent against it. The row id is the key to be idempotent with: branch on status before doing work, and stamp it as you go, so a second firing is a no-op.

Why identity is the signed digest rather than the sender's dedupe header: the vendor signs the body, not our header. A row id derived from that header could be varied freely by anyone replaying one captured delivery -- minting a row per value while the signature stayed valid. Identity therefore comes from signed material only. The sender's key is still recorded on the row for you to read; it just does not decide the id.

The practical consequence: two deliveries whose signed bytes are identical collapse onto one row. For a redelivery that is exactly what you want. For a sender that legitimately emits two distinct events with byte-identical payloads, configure ..._TIMESTAMP_HEADER -- the timestamp is part of the signed payload, so it separates them and cannot be forged.

#Reading the response codes

401 is deliberately flat: the caller is unauthenticated, so why a check failed is a fact about your configuration and is not theirs to learn. The reason is in the node log.

#Troubleshooting

Everything 404s. The source is not in MEMQL_INBOUND_SOURCE_ALLOWLIST, or it is listed but incomplete. Check the node log at boot: a dropped source logs an error naming the exact env var that is missing.

Everything 401s. Usually the prefix. GitHub sends sha256=<hex>; without ..._SIGNATURE_PREFIX=sha256= the receiver tries to decode the whole thing as hex and refuses. After that, check the secret matches the one configured at the sender, and that the scheme's encoding is right -- hex and base64 of the same digest are both "valid signatures" and neither decodes as the other.

Everything 401s only sometimes. If the source has a ..._TIMESTAMP_HEADER, the replay window is in force in both directions and clock skew between the sender and the node will show up as intermittent refusals. Widen MEMQL_INBOUND_TIMESTAMP_TOLERANCE_SECONDS or fix the clock.

Rows arrive but nothing happens. The receiver's job ends at received. Check the product automation's trigger and filter.

#What this does not do

• No per-source rate limiting. The body cap bounds a single request; it does not bound a flood. Rate limiting is the ingress's job today.

• No engine-side retry of product handling failures. A row stamped failed sits there until something looks at it -- the engine does not know what retrying a business event means.

• No binary payloads. body is text; a non-UTF-8 request is refused with 400 rather than staged corrupted.

• No NUL bytes in the body. U+0000 is valid UTF-8, so it passes the check above, but body is stored in a JSONB column and PostgreSQL's jsonb type cannot represent U+0000. Such a request is refused with 400 (memql#3098). It is refused rather than repaired for the same reason invalid UTF-8 is: the body is signature-verified material, so substituting a byte would stage a row that no longer matches what the sender signed. Note this is the OPPOSITE of what outbound delivery does with the same byte -- there it substitutes, because the affected field is diagnostic text (lastError) where a mangled character beats a permanently stuck row.

  The 400 matters as much as the refusal. Before this, such a body was accepted, the insert failed against the column, and the handler answered 503 -- which every sender in the list above retries, against a request that could never succeed.