Campaign sending

The engine that actually moves campaign mail: what it does, the product decisions baked into it, what an operator has to configure, and what it deliberately does not do.

memql#3323 modelled campaigns as concepts and shipped the authoring UI. It stopped before sending, because sending is a set of product decisions rather than UI. memql#3348 made those decisions and built the engine. Each one is recorded next to the code that implements it; this page is the operator-facing summary.

Related: Environment variables · Outbound delivery (the transactional outbox, which campaigns deliberately do not use — see below)

#Installation defaults

Installation generates a shared unsubscribe signing key once and keeps it on subsequent runs. Local installs store it in memql-secrets; cloud installation uses memql-campaigns-unsubscribe-secret in the instance's Key Vault, delivered by the separate memql-secrets-campaigns ExternalSecret. Every replica uses the same key. Never put it in a package manifest or product repository.

The unsubscribe base URL defaults to https://api.<MEMQL_DOMAIN> on every target. An explicit MEMQL_CAMPAIGNS_UNSUBSCRIBE_BASE_URL overrides that default. The public API origin must be reachable by recipients for external delivery; a local hostname is for local testing and capture only.

For an existing installation, rerun its secret seeder before restarting the nodes with the updated engine. The seeder preserves an existing signing key. For cloud installs, seed the instance vault before applying its ExternalSecret; this separate object prevents a missing campaign key from stopping identity or database credential reconciliation. Signing-key rotation remains an explicit operation, described below, because previously issued links must keep working.

These defaults belong to the engine. Client domains, sender names and Azure resource choices belong to the instance or product repository and its saved cluster configuration; they are never engine defaults.

#Checking a send locally

With MEMQL_EMAIL_TRANSPORT=capture, messages appear in the MemQL OS Email app for owner/developer review; they do not reach external inboxes. Campaigns shows this delivery mode before sending and when confirming a test. A captured test verifies rendering and local persistence, not Azure delivery.

Review the campaign's template and mark it ready before Send now or scheduling. Open the template in the editor and choose Use in Campaigns → Publish template for campaigns after reviewing the preview. Saving keeps it as a draft. The campaign's Review template action opens that editor; send and schedule actions appear when its publication arrives through the live collection. Send a test queues a separate run using a published template and the organization’s testing audience. Ordinary campaigns do not depend on Shopify inventory: connecting an empty store, including a store belonging to another organization, cannot block their sends. Storefront newsletter enrollment keeps its explicit organization, store and audience binding.

#Organization permissions

Select an organization when creating campaign work or a group. Operators who manage several organizations are asked to choose; MemQL never silently assigns new work to the installation owner's organization. A member with access to just one organization sees that organization selected.

A send requires access to the Campaigns app and permission to update data in its organization. Reading a shared campaign or recipient list does not grant permission to send. Campaigns, audiences, templates and sending identities must belong to the same organization. The worker checks the campaign owner's current authority again before each delivery, so revoked membership also stops queued work. Test messages and single-recipient sends use the same organization permission check.

Sending readiness is checked for the selected organization and sender. The operator organization's mailbox status does not enable or block client mail. Use Check again after changing a connection; the backend also repeats the check before scheduling and sending.

Delivery records preserve the organization and authenticated owning user. Editing a shared resource preserves its existing owner; each new version separately records the user who made that change. Rule-triggered deliveries name their email rule; they do not invent a campaign relationship. Historical records are attributed only when an existing parent proves which organization owns them.

#Templates in Productivity Tools

Create a named template in Campaigns → Templates, choose its organization, then open it in the editor. File content lives in MemQL Productivity Tools: Source, Preview, and Split use the same editable document. The OS keeps the template list, organization, status, and archive controls. Files settings select browser VS Code (the default), desktop VS Code, or Cursor.

Create from examples starts a Materializer draft from an image, individual files, or an explicitly selected ZIP of resources plus your brief. MemQL reads the authorized bytes and produces editable HTML, plain text, and a subject in an .email.json file. A resource ZIP is inspected only for that explicit composition; opening a ZIP in Files still downloads it intact. Choose which images are visual inspiration and which should appear in the email. Included PNG, JPEG and GIF bytes stay with the editable template and are sent as inline attachments; the flow does not publish private file URLs. Review the generated layout and links before publishing.

Use Use in Campaigns to save a draft or publish the reviewed template for its organization. Publishing makes it available to send; it does not send mail. Editing a published template returns it to draft. Concurrent edits fail with a comparison prompt, and publishing requires the exact saved revision. An existing template cannot be moved to another organization by editing its file. Create a separate copy instead.

Newly queued and scheduled sends keep a frozen copy of the ready template reviewed at preflight. Later edits do not change those messages. The worker still rechecks the caller's authority and the source template's readability. Jobs queued before this change have no captured copy and retain their previous behavior; pause and recreate those jobs before editing their shared template.

The client write surface is campaignSaveTemplate, with templateId, accountId, name, content, expectedRevision, and action (save, publish, or archive). Its receipt includes the saved revision and status. Raw template writes and the underlying create/update mutations are internal.

#Configure Azure once for the cluster

An owner or developer configures Settings → Connections → Email once for this installation. There is one Azure configuration, shared by every client organization in the cluster. Clients do not need Azure accounts or separate Microsoft sign-ins when the operator runs their email on its own subscription. Campaigns manages client domains and senders; it links to Settings when the cluster needs configuration.

1. Sign in with Microsoft. Enter the displayed code on Microsoft's page. MemQL requests delegated Azure Resource Manager access under your Microsoft account. Your tenant may require administrator consent. No password, client secret, or subscription key is entered into MemQL.
2. Choose the subscription and resource group. Switch Microsoft directories first if necessary. Select an existing resource group, or name a new one and choose its region. Select the email data location.
3. Save configuration. MemQL verifies access to the enabled subscription and an existing selected group, then saves the authorization encrypted in this cluster. Saving creates no Azure services and sends no mail. A new group is created only when you later confirm a domain's resource plan.

All replicas use the same saved configuration and coordinate refresh-token rotation in PostgreSQL. The private browser sign-in session is bound to its initiating MemQL user and expires after one hour once authorized; the saved cluster authorization remains available after it expires. Only the opaque session identifier is kept by the browser. Access tokens, refresh tokens, and sending keys never appear in browser responses. Microsoft can revoke the saved authorization or require another sign-in under its directory policy. Existing verified sends can continue with their sending keys when management sign-in needs renewal; domain changes require renewed authorization.

Disconnect in cluster email settings stops Azure sends and domain setup for every organization. Reconnect before choosing a different subscription or resource configuration. Reconnecting to the same configuration can resume its existing domains. Changing it requires reconfiguring each domain; the cluster never silently reuses credentials from a different configuration. Disconnect does not delete Azure resources, revoke keys in Azure, alter DNS, or stop Azure billing. Historical encrypted versions follow the installation's retention policy. Already submitted messages cannot be recalled.

#Add each client's domain and sender

In Campaigns → Settings, Campaigns → Senders → Connect email domain, or Settings → Connections → Email, select the client organization. An owner or developer also needs write access to that organization. Other campaign authors use its saved sending identities.

1. Add sending domain. Enter the client's domain. Resource names are suggested and can be changed. Subscription, group, and email data location come from the cluster configuration; no client Microsoft sign-in is needed.
2. Review, then create. MemQL registers the Communication resource provider if needed and creates the email service, delivery service, and custom-domain resource. Usage is billed to the operator's selected Azure subscription. MemQL does not convert the subscription, buy support, or send mail during setup. Interrupted steps check the same resources. Resource ownership tags bind services to this cluster configuration and organization; unrelated resources are refused. Names are reserved across replicas before creation and remain assigned after disconnecting. A changed plan must be reviewed again before resources are created. After confirmation, the open wizard continues the setup automatically and advances to domain verification when Azure is ready. Preparing email shows the elapsed wait. Leaving preserves the approved plan; reopening resumes it. An Azure error or an extended wait pauses automatic checks and offers Continue setup for the same resources.
3. Verify domain. Add the exact TXT/CNAME records Microsoft supplies at the domain's DNS provider, then choose Check domain records. Each record uses the same Type, Name, and Value layout as Deployables domain binding; click a name or value to copy it, and read its verification state below. Completed wizard steps remain available to review. Merge an SPF include into an existing SPF record rather than creating a second one. Preserve existing MX and website records. Domain, SPF, and both DKIM checks must succeed. The plan is saved, so you can return after DNS propagates without another Microsoft sign-in.
4. Save sender. Sender news on client.com sends as news@client.com. Set the client's display name and an existing reply mailbox such as help@client.com. ACS sends outbound mail; it does not create receiving mailboxes. no-reply is optional. The verified sender appears in Campaigns.

A client send requires that organization's ready transport and verified sender. It never falls back to the operator's Graph/SMTP mailbox. ACS sets the display name on the sender resource; a campaign override that differs is refused. Disconnect domain stops only that organization's future sends, preserving other organizations and the cluster's Azure authorization.

A successful ACS submission means accepted by Azure, not delivered to an inbox. Ambiguous submission responses are not automatically resubmitted. Delivery feedback still needs the pipeline described below; this setup does not install an Event Grid delivery-feedback route.

#Registration and installation setup

MemQL ships the public application ID of MemQL Azure Connection, its multi-tenant Microsoft Entra application. This identifies the software; it is not a secret, subscription, or authorization. Register the application once for MemQL. Operators do not register another app or supply its ID in environment variables for each client or installation. The same registration supports local and cloud installations in commercial Azure.

Each installation keeps its own authorization and sending credentials. A local connection is never copied into a cloud cluster or another client's installation. The same operator may configure separate clusters against its own directory and subscription. A client's directory consent is relevant only if the operator chooses to use Azure resources controlled by that client. Consent cannot give the signed-in user Azure permissions they do not already hold.

The device code flow uses the existing MemQL stream. It adds no OAuth callback endpoint or per-installation redirect URL. Local capture works without Azure; connecting Microsoft does not switch test emails to external delivery.

Initial installation integration remains planned. The intended onboarding is to configure cluster email during installation and leave Campaigns ready to manage client domains. Azure deployment authorization should be reused where possible, with a durable cluster identity and narrowly scoped access; deployment CLI credentials must not be copied into the cluster. The current implementation uses the one-time Settings flow above. The infrastructure installer does not yet provision this authorization or a verified sender. Existing transactional email bootstrap is separate: do not remove a working bootstrap transport before its replacement can deliver owner verification and login mail. Creating the app registration alone does not solve that bootstrap dependency.

Microsoft directory policy can require administrator approval. Publisher verification is separate from verifying a publisher domain. The registration's publisher verification must be completed before relying on broad customer self-service consent. This does not require a separate registration for each operator-managed client. See Microsoft's application and service principal model and publisher verification requirements.

Protocol references: device authorization flow, email domain setup, email sending API.

#Storefront newsletter signups

In Campaigns → Audiences, open the client's audience and choose Storefront signups → Connect a signup form. Select its deployable, a published welcome template, and the client's sender. Write the sentence the visitor will agree to, then enable the connection. All resources must belong to the same organization. An audience is a marketing list, not a group granting access to MemQL.

Get form → Copy form provides plain HTML to add to the storefront in VS Code. Include /newsletter/thank-you and /newsletter/problem pages in that site's bundle. A form posts to /_memql/forms/campaigns/subscribe on the deployable's own origin. It carries the email, optional name, affirmative consent checkbox, and saved consent revision. Replace the copied form after changing the connection; an old revision is refused rather than recording consent to new wording. This does not subscribe an arbitrary externally hosted Shopify site: the storefront must be a MemQL deployable with the public-form carrier available.

Enabling the connection also enables that deployable's public forms. Turning the newsletter off stops enrollment and blocks pending welcomes; it leaves other declared forms alone. Current organization write authority and access to the deployable are required. A draft or archived welcome and a disabled sender may remain selected while turning an existing connection off.

The server derives the organization, audience, sender and live store binding. Form fields cannot override them. Storefront previews with a different store binding are refused. The current implementation uses single opt-in: checking the box records consent and enrolls the address; it does not prove control of the mailbox with a confirmation link. Previously unsubscribed, bounced, complained-about or suppressed addresses are not silently resubscribed.

Repeated submissions converge on one signup and one intended welcome per normalized address and audience, even across deployables and worker replicas. Consent is stored before a new recipient becomes sendable. The accepted record captures the storefront, original mailbox, consent wording and revision, and published welcome content. Editing the template later does not rewrite that welcome. Changing the recipient's mailbox blocks the queued welcome instead of redirecting it to an address that did not consent. Current authority, sender availability and suppression are checked again before sending.

Recent welcomes shows the latest 50 outcomes, read when the audience opens or Refresh is selected. A rate limit leaves mail pending for a later worker pass. A blocked welcome shows its reason; after resolving it, Check again rechecks that exact saved outcome. An attempt with a lost provider response is uncertain and is never automatically submitted again. sent means the transport accepted the message, not proof of inbox delivery. Local capture sends appear in the Email inbox.

This uses the existing declared shopper-form carrier, its size and rate limits, and borrowed site-owner authority. Campaigns is core and declares the campaigns/subscribe form itself; installing a pack is not required. Enrollment and welcome progress use shared PostgreSQL locks, fresh reads, stable IDs and monotonic revisions. Private mutations cannot be called or forged by clients. Newsletter configuration changes broadcast to other replicas; high-volume signup outcomes update automatically while the audience is open. Configuration uses campaignConfigureNewsletter; rechecking uses campaignRetryNewsletterWelcome, both with saved revisions.

Use the same audience in multiple campaigns, with independent templates and schedules. The automatic welcome is separate from those campaigns.

#Repeating campaigns

Open a campaign and choose Repeat → Set up repeating sends. Select the first send and a weekly interval, such as every two or three weeks. The timezone is shown beside the date. Cadence follows its local calendar time across daylight saving changes. A recurring schedule can be created before its audience has subscribers; an empty occurrence completes without sending mail.

The original campaign is a blueprint. Saving the schedule captures its audience, sender, organization and tracking settings. Save the schedule again after editing those campaign details. Each occurrence reads the then-published template and freezes that copy in a separate campaign/send job. Later edits cannot change an already queued occurrence. The audience is resolved when that occurrence runs, and consent and suppression are checked before delivery.

Each occurrence appears in Campaigns with its scheduled date in the name and its own delivery history. The original draft can still be sent as a one-off; that is an additional campaign, independent of its repeating schedule.

Pause future sends stops new occurrences. Already queued occurrences keep their own pause/cancel controls. Resume future sends skips missed dates. After an unexpected outage, an active schedule queues at most one overdue occurrence, then advances to its next future date, preventing a burst of missed newsletters. A configuration or authority refusal blocks the schedule with a visible reason; review and resume it after resolving the problem.

campaignConfigureSeries accepts a campaign ID, save, pause or resume, and the exact saved series revision for changes. Saving also takes intervalWeeks (1–52), firstSendAt and an IANA timeZone. The cluster captures its definition from the readable campaign, never caller-supplied organization, audience or sender overrides. The person authorizing it must retain current organization write access. Shared PostgreSQL coordination, fresh reads and stable occurrence IDs make retries across replicas converge on the same job, including a crash after queueing but before advancing the schedule. A finished occurrence's job and delivery records are never reset.

#Retrying event emails

Email rules supply a stable identifier automatically for each event and recipient. Their firing reports count suppressed and replayed sends as skipped, and show uncertain outcomes as refusals.

Other automations that call campaignSendToRecipient should supply a stable requestId for each intended message, such as one signup event's welcome. Keep that identifier unchanged when retrying the same event. The recipient, template, rule and sender must also remain unchanged; reusing the identifier with different inputs is refused. The template must be published before sending.

The cluster records an attempt before contacting the provider. A retry on any replica returns the saved result without submitting another message. If the provider response was lost, or the process stopped after the attempt began, the retry returns uncertain: true and does not resend. Inspect the provider and delivery records before deciding whether another message is warranted. This prevents duplicate submissions; it does not guarantee mailbox delivery. Requests that omit requestId retain the existing per-invocation behavior.

Suppression and preflight refusals make no external send attempt and remain retryable. Receipt reads still require current send authority in the recorded organization. The internal receipt cannot be forged through client mutations.

#Local test inbox

The local k3d overlay selects the capture email transport automatically. Open Email → Inbox in MemQL OS as an owner or developer to inspect messages generated by sign-in, invitations, campaigns and email rules. Messages are stored in the installation's shared storage, encrypted with its existing master key; capturing on one node and reading on another works without a separate mail server.

The inbox shows the latest 100 messages from the past seven days. Refresh it after triggering mail, or return focus to the app. Open a message to inspect its sender, recipient and body, and follow its verification link. Previews remove scripts, remote images and styles, so opening test mail does not load tracking pixels. Older encrypted rows are not deleted by the inbox's seven-day display window.

Capture sends no Internet mail, even when external provider credentials exist. It is selected by the installation value MEMQL_EMAIL_TRANSPORT=capture, never by a person's role or an email address. Storage or encryption failures fail the send instead of claiming success. The ordinary external transport remains auto; its Email app states that receiving has not been connected. Neither mode changes bootstrap's email-validation exception or connects Outlook.

This is an operator test inbox and can contain other test users' sign-in links. Ordinary users and admins cannot read it. It is not a connected personal mailbox.

#The shape of a send

  operator                engine                          recipient
  ────────                ──────                          ─────────
  campaignStartSend  ──▶  preflight (sender? unsubscribe?
  campaignScheduleSend    template ready? audience sane?)
                            │ refuses here, or
                            ▼
                          v1:campaigns:sendJob
                          (queued — or `scheduled`, which
                           the worker promotes to queued
                           once the campaign's scheduledAt
                           has passed)
                            │
       drain worker  ──────▶│  every 15s, one batch per job
                            │
                            ├── roster page ⨯ its ledger   = who is left
                            ├── cluster suppression list   = who may not
                            ├── token bucket               = how fast
                            └── send ─────────────────────────────────▶ inbox
                                  │                                     │
                            v1:campaigns:delivery (one row per          │
                            recipient, terminal)                        │
                                                                        │
  suppression list ◀── POST /unsubscribe ◀── one-click ◀────────────────┘

#Configuration

Minimum to send anything:

Tuning (all optional, all documented in env-vars.md): MEMQL_CAMPAIGNS_SEND_RATE_PER_MINUTE, _BATCH_SIZE, _MAX_ATTEMPTS, _MAX_AUDIENCE, _POLL_SECONDS, _STARTUP_DELAY_SECONDS, _CLAIM_TTL_SECONDS, _THROTTLE_SECONDS, _SEND_TIMEOUT_SECONDS, _ENABLED.

Feedback and warming (both opt-in, both off with no value): MEMQL_CAMPAIGNS_FEEDBACK_SOURCES (feeding bounces back in), _WARMUP_ENABLED and the _WARMUP_* ladder and thresholds (warming), _SENDING_IDENTITY.

On a Microsoft Graph deployment there is one more, and without it the suppression list quietly starves on the transport you actually send on: MEMQL_EMAIL_NDR_POLL_SECONDS (default 300; 0 disables) runs the Graph mailbox reader, and the rows it stages mean nothing until graph-mailbox=rfc3464 is listed in MEMQL_CAMPAIGNS_FEEDBACK_SOURCES.

Rotating the unsubscribe signing key is a two-variable operation, and doing it with one variable breaks every link already sent. The procedure is below.

#The decisions

#A campaign is one operator's, and the cluster owner can see it

The nine operator-facing concepts — audience, recipient, template, senderIdentity, campaign, delivery, consentEvent, engagementEvent and emailRule — carry the composite tier, @rowAuthz(owner="ownerUserId", clusterOwner). Every row's ownerUserId is stamped from the caller's own actor rather than from an argument, so a campaign belongs to whoever made it.

What the second term buys, and what it deliberately does not. A cluster owner READS every operator's campaigns, and can drive the builtins on them — campaignStartSend, campaignPauseSend, campaignResumeSend, campaignScheduleSend, campaignStats, campaignTestSend — because each of those is gated on "can the caller read the campaign row" and nothing else. A cluster owner cannot rewrite another operator's campaign directly: the write guard ignores the second argument, so updateCampaign, createTemplate and every other row write stays owner-scoped.

That asymmetry is the point. The person who has to answer "why did this client's send stop" needs to see the row and to be able to stop it; nobody needs to be able to edit somebody else's copy behind their back. Without the second term the oversight would not merely be unimplemented — a plain owner tier has no cluster-owner escape on the READ path at all, so it would not be expressible, and every fleet-wide campaigns view would silently render one person's subset while looking complete.

Full team sharing — several people editing one campaign — is a deliberate non-goal. This is oversight, not sharing.

The four engine concepts (sendJob, suppression, reputationWindow, warmupState) stay clusterOwner-tier and are not readable by an ordinary operator at all.

#Client opt-outs belong to the organization

An unsubscribe blocks marketing mail from the organization that sent it, across that organization's audiences, campaigns, and sending identities. It does not remove the same address from an unrelated client's list. Both queued campaigns and event-triggered sends check this at send time, even if a CSV import has recreated a subscribed membership.

The engine-owned v1:campaigns:suppression rows store an organization ID and an email digest, never a plaintext mailbox. The composite row ID includes both, preventing clients from overwriting each other's opt-outs. Reads bypass per-node caches so a click handled by one replica is visible to another sender.

Existing rows without an organization remain cluster-wide safety blocks. They are not silently narrowed or removed. campaignSuppress and provider hard-bounce/complaint feedback continue to create these administrator-level blocks. A matching block in either scope prevents delivery.

#Suppression is enforced at the point of send

Not at audience-build time. The worker checks both suppression scopes for every recipient immediately before mailing, before it looks at the recipient row's own subscriptionStatus. That ordering is what "outranks every audience" means: a re-imported CSV produces a recipient row saying subscribed, and the applicable suppression still refuses it. An audience assembled last month cannot know about last week's unsubscribe.

A suppressed recipient gets a skipped delivery row with the reason, not a silence — the operator is owed the outcome.

If the suppression lookup fails, the batch stops. An unanswerable list is never read as an empty one.

#A hard bounce suppresses; it does not delete the membership

campaignRecordFeedback(kind: "hard_bounce") adds the address to the cluster list and the next send marks that operator's recipient row bounced. The audience membership stays.

Deleting it looks tidier and is wrong twice. It destroys the audit trail — an operator whose delivery rate fell finds a smaller audience and no explanation. And it makes the address resurrectable: the next import re-adds it as a fresh subscribed row, and mailing a known-dead address is exactly what a reputation system punishes. Keeping the row makes the sendable count drop visibly with the reason attached; keeping the cluster suppression means a re-import cannot undo it, because the list is consulted at send time rather than at import time.

A soft bounce does neither. It is transient — a full mailbox, a greylisting relay — and suppressing on one loses a real subscriber to a bad afternoon. The per-recipient retry budget (MEMQL_CAMPAIGNS_MAX_ATTEMPTS) is what bounds those.

#The audience is walked, not held

The send reads the roster one page at a time, through the engine's keyset cursor, and reads the delivery ledger for that page rather than for the campaign. Both halves matter: the ledger for a large campaign is exactly as large as its roster, so cursoring one and not the other would move the ceiling rather than remove it.

Before this, both reads were whole and bounded at 5000 rows — and a bounded read of an unbounded set is a truncation, so a larger audience would have been mailed as a silent prefix. The send refused instead, and "5000 is the largest mailing list this can send to" was the price.

Why ascending order is safe while the roster is being edited. Reads collapse to the latest version per id, so editing a recipient mid-send — which this worker does, converging a suppressed address — moves it to a newer timestamp, i.e. forward in an ascending walk, never backward past the cursor. A row can therefore be seen twice and can never be missed. A duplicate costs a comparison; a skip costs somebody their mail. Duplicates inside one batch are de-duplicated, because those would be mailed before either send was recorded.

The cursor is an optimization, not the idempotency mechanism. It lives on the send job so a tick resumes instead of re-scanning from the start. Losing it, or having the engine refuse a stale one, costs one re-scan. What decides who gets mailed is the ledger, below.

MEMQL_CAMPAIGNS_MAX_AUDIENCE still exists, and now means something different. It was a correctness guard at 5000. It is now a deliberate refusal at 250,000: no size is unsafe, but a send that large is more often a mis-scoped audience or a bad import than an intent, and it cannot be recalled. Raising it is a decision about how large a send you mean to make — not something you have to do in three places to avoid silent truncation.

#Idempotency is the ledger, not a retry flag

v1:campaigns:delivery has one row per (campaign, recipient), at a deterministically derived id. The worker's batch is "roster members with no delivery row, plus rows still pending whose retry is due". A recipient that reached sent, skipped or terminal failed is simply not in the next batch.

So a resumed send, a restarted process, a paused-and-resumed campaign and a claim lost to a dead replica all converge on the same behaviour, and none of them needs anything to remember what it did. The absence of the row is the work queue.

Cross-replica, a Postgres claim (campaignSend, keyed per job and progress count, leased for MEMQL_CAMPAIGNS_CLAIM_TTL_SECONDS) means one replica drains a given campaign at a time.

#Two rate limits, and both are needed

Ours is a token bucket, MEMQL_CAMPAIGNS_SEND_RATE_PER_MINUTE, per process. It exists so the provider's limit is not discovered by exceeding it on every send. Per-process rather than cluster-wide is correct for what it defends: only one replica drains a given campaign, so a campaign is paced by exactly one bucket.

Theirs is the 429. Microsoft Graph throttles a mailbox and answers 429 with a Retry-After; the sender surfaces that as a typed error, and the worker parks the whole job until the stated deadline and ends the batch. A 429 with no Retry-After is still a throttle and parks for MEMQL_CAMPAIGNS_THROTTLE_SECONDS.

Ignoring the second is the failure the issue names: the loop keeps hammering, the provider keeps refusing, and the retries eventually arrive as duplicates in the recipient's mailbox.

#Unsubscribe: RFC 8058 one-click, plus a visible link

Every campaign message carries both halves:

List-Unsubscribe: <https://api.example.com/unsubscribe?token=...>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

and a link in the body. The header serves the mailbox provider's own "unsubscribe" button; the link serves a person reading in a client that does not surface it.

GET /unsubscribe renders a confirmation page. POST /unsubscribe performs the opt-out. A GET never unsubscribes anyone — mail clients, link scanners and security appliances prefetch URLs found in messages, which is precisely why RFC 8058 specifies a POST.

New tokens bind owner, recipient, campaign, organization, and the original normalized email digest with an HMAC. Moving, renaming, or deleting a recipient cannot redirect the opt-out. Already-mailed u2 links remain valid and derive the organization from the recipient they originally named. An authoritative suppression-write failure returns a retryable 503, never a success page.

The token is not stored: storing one would mean a row per recipient per campaign, minted at send time and never prunable, because a link in somebody's inbox has to keep working for years. It does not expire, for the same reason — a stale unsubscribe link is a compliance failure dressed as hygiene.

The endpoint is HTTP, which is a documented exception to the gRPC-first rule for the same reason POST /inbound/{source} is: the caller is the recipient's mail client, and there is no gRPC version of that conversation.

#Rotating the unsubscribe signing key

The token names the key that signed it, and the node verifies against a ring of two — MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET and the optional MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET_PREVIOUS. Only the first ever signs.

u3.<keyId>.<owner>.<recipient>.<campaign>.<organization>.<emailDigest>.<tag>
   ^^^^^^^ first 4 bytes of HMAC-SHA256(secret, "memql/campaigns/unsubscribe/key-id"), hex

The key id is a digest of the key, not a slot number or a counter. That is deliberate: a link minted today is clicked on a node where that same secret has since become the previous one, so a label like "current" would be wrong exactly when it is needed, and a counter is one more value an operator can set inconsistently across replicas. A digest of the key is true wherever the key is. It leaks nothing — anyone holding a token already holds a 128-bit MAC over known plaintext under the same secret.

The procedure

1. Set MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET_PREVIOUS to the current value of MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET.
2. Set MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET to the new value.
3. Apply both in the same change and roll every node that mints or serves unsubscribe links (the campaign worker and the bff).
4. Leave _PREVIOUS set. Do not remove it later.

Step 4 is where this differs from every other key rotation in the system, and it is the whole reason this page has a section.

How long does an old link keep working? Forever — until a second rotation.

The window is counted in rotations, not in days. There is no time-based expiry anywhere in the token, and adding one would defeat the point.

The usual advice — "drop the previous value once no unsent mail references it" — has no true form here. An unsubscribe link does not expire when the send finishes; it lives in the recipient's mailbox for as long as they keep the message, which for a mailing list is indefinitely. So there is no date after which retiring the old key is safe, and _PREVIOUS is not a migration window that closes. It is a permanent second reader key.

What that buys, stated plainly:

So the standing rule is rotate at most once for any reason short of compromise. A second rotation retires the oldest key and does break every link it signed — which is acceptable in exactly one case: the key leaked, and killing those links is the point. If a routine second rotation is unavoidable, treat the messages signed by the retiring key as messages whose opt-out is now mailto-only, and say so in the next send's footer.

What the engine tells you

• At boot, the campaign worker warns once when this deployment holds only one key and has already sent campaign mail:

  text⧉copy

  WARN campaigns: this cluster has already sent campaign mail and holds only ONE
  unsubscribe signing key. MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET_PREVIOUS is unset,
  so changing MEMQL_CAMPAIGNS_UNSUBSCRIBE_SECRET would invalidate every
  unsubscribe link already sitting in a recipient's mailbox ...

  It is silent before the first send (setting _PREVIOUS then costs nothing) and silent once _PREVIOUS is set.

• When a link arrives signed by a key nobody holds, the endpoint renders the ordinary "This link is not valid" page — never a 500 — and logs a warning carrying the token's keyId (which is not secret). One of those lines means a rotation dropped a key that had already signed live mail.

#SPF / DKIM alignment is structural

MemQL does not sign DKIM — the relay does, with the keys published for the mailbox it authenticated as. What MemQL guarantees is that it never sets a From address the relay did not authenticate as:

• the From address is never a free parameter of a send. It is either the configured sender credential's own mailbox, or the address of a sending identity an operator declared as a row — a closed registry, resolved server-side from the campaign, never a string the caller supplies;
• v1:campaigns:campaign still has no from-address field. It has senderIdentityId, which names a row, and fromName, which is a display name only;
• replyTo is per-campaign and is the correct escape valve: it steers replies without touching the authenticated identity.

Your side of it is DNS, and it is now per identity. Publish SPF and DKIM records covering every mailbox any identity sends as, plus a DMARC policy for each of those domains. Adding a sending identity for a client's domain without their DNS records is the one way to make deliverability worse by using this feature. There is no preflight for it, because any check the engine could run would be a guess about records it cannot see from inside the cluster.

#Campaigns do not use the transactional outbox

v1:platform:outboundRequest and its worker deliver transactional mail. Campaigns send directly instead, for three reasons:

1. the campaign worker has to see the provider's 429 and Retry-After to pace itself; a queue drained by another worker turns that into an opaque retry;
2. a terminal per-recipient outcome is the point of the delivery row, and an async seam leaves every row pending until somebody writes a reconciler;
3. MEMQL_OUTBOUND_EMAIL_ALLOWLIST is a domain allowlist for transactional mail. A marketing audience is by definition arbitrary external domains, so routing campaigns through it would either refuse every send or force that allowlist wide open — weakening the egress control it exists to provide.

delivery.outboundRequestId stays on the schema for a future queued transport.

#Sending identities

One deployment, several clients, several mailboxes. A sending identity (v1:campaigns:senderIdentity) is the operator's declaration that a mailbox exists in this tenant and that campaigns may send as it:

There is no secret material on the row, and that is the design. Azure credentials live in encrypted organization connection storage. The identity row records the address and presentation; the transport still requires that address to be in the organization's verified sender set. Graph/SMTP credentials remain installation configuration for the operator's own mail.

#Resolution order, and why there is no fallback

campaign.senderIdentityId  →  the operator SAID which mailbox
(empty, operator org)      →  the operator organization’s configured default
(empty, client org)         →  refused: choose that client’s sending identity

That is the whole ladder. The engine never infers an identity from a campaign's accountId. The authoring UI prefills the picker from the selected account's identities, and prefill is UX while resolution is explicit — an engine that guessed would mail a client's list from a mailbox nobody chose, and would be right often enough that the wrong case went unnoticed.

Client campaigns require an explicit identity. Legacy campaigns belonging to the operator organization can use its configured default.

A missing or disabled identity is refused, never silently defaulted. Falling back to the cluster default would mail a client's audience from the wrong mailbox, under the wrong From, against the wrong SPF and DKIM records, and nothing would say so — the send would look completely successful. The refusal is classified the same way every other preflight refusal is:

The second is an environment refusal because SMTP AUTH binds to one mailbox: a node with the SMTP sender genuinely cannot send as anything else, and a node that can may be running elsewhere in the cluster. Failing the campaign for a deploy-shaped reason would make an operator re-author a schedule to recover from it. The preflight runs at both authoring time and fire time, so an identity disabled between the two is caught at the second.

#disabled is how you retire a mailbox

Not deletion. Past campaigns name the row, and the reputation history is keyed on its address — derivedSendingIdentity resolves per send to the identity's normalized address, so reputationWindowsSince and warmupStateForIdentity break down per mailbox with no change on their side. They were built for plurality and simply start receiving it.

Organization opt-outs apply across that organization's sending identities. Cluster-wide safety blocks still apply across all identities.

#Organization controls access and sending

New campaigns, audiences, templates, and sender identities select an organization. Read and write access follows the cluster's organization authorization rules; the worker checks current send permission, and the campaign, audience, template, recipient, and sender must agree on organization. A client campaign cannot fall back to the operator's sender identity.

#Adding a sending identity

For ACS, use the guided connection above. It verifies the domain, creates the Azure sender username, and saves the corresponding Campaigns identity. Creating an identity manually does not prove that its address is verified.

The following instructions apply only to an existing Microsoft Graph operator mailbox configuration. They do not configure a client's ACS connection.

1. Create the mailbox in the Microsoft 365 tenant that hosts your sending domain — the tenant MEMQL_EMAIL_AZURE_TENANT_ID names, which is not necessarily the tenant your Azure subscription lives in. A token issued for the wrong tenant is a 404 on a mailbox that plainly exists.

2. Add it to the mail-enabled security group behind your ApplicationAccessPolicy. This is the step that is easy to miss and silent when missed, because everything else works: the row saves, the picker offers the mailbox, the campaign starts, and every message fails at the provider.

   text⧉copy

   Connect-ExchangeOnline -Organization <mailbox-tenant-domain>
   Add-DistributionGroupMember -Identity "memql-mail-senders@<domain>" `
     -Member "news@<domain>"
    
   # Verify BOTH directions. A policy that grants is not evidence it denies,
   # and replication can take ~30 minutes.
   Test-ApplicationAccessPolicy -Identity "news@<domain>"          -AppId <client-id>
   Test-ApplicationAccessPolicy -Identity "<a-real-person>@<domain>" -AppId <client-id>
   # want: Granted, then Denied.

   If you have no such group yet, the app is still scoped tenant-wide — Mail.Send (Application) is literally "Send mail as any user". Create the group and the policy first: Mail.Send is tenant-wide until you scope it.

3. Publish SPF, DKIM and DMARC for the new mailbox's domain, if it is a domain this deployment has not sent from before. A new identity on an already-authenticated domain inherits its records and needs nothing here. A new domain is a new reputation: see SPF / DKIM alignment is structural, and expect to warm it — the ramp keys on the identity's normalized address, so a new mailbox starts its own ramp rather than inheriting an established one's rate.

4. Create the row.

   text⧉copy

   mutation createSenderIdentity(
     address:   "news@<domain>",
     fromName:  "Acme News",
     replyTo:   "hello@<domain>",
     accountId: "<account id>"
   )

   address is normalized lowercase and is the reputation and warmup key, so two spellings of one mailbox would split its ramp in half. fromName is required — a From with no phrase shows the raw mailbox to every recipient.

Mail.Read matters too, and only for the default mailbox. The Graph mailbox reader reads MEMQL_EMAIL_SENDER's inbox for delivery reports, so that mailbox needs Mail.Read under the same policy group. Identity mailboxes do not need it yet: bounces for a send from news@ come back to news@, and the reader does not walk them, so a non-default identity's hard bounces are not yet feeding suppression. Plan around that until it does — the group membership is the only thing to change when it lands.

Retiring one is status: "disabled", never a delete. See disabled is how you retire a mailbox.

#Importing recipients

builtin campaignImportRecipients(audienceId: "<id>", artifactId: "<id>", hasHeader: true)

The file is a CSV already uploaded to the Library. It is read server-side under the caller's own actor, so a file the caller cannot read is a file this cannot import — the artifact id is not a capability.

A header row is required and the column mapping IS the header. email is required (case-insensitive); displayName and name are recognized; every other column lands verbatim in the recipient's fields map, reachable from a template as {{fields.<key>}}. There is no positional import, deliberately: guessing which column holds addresses is how an import mails the wrong list.

Per row the address is normalized and shape-validated, then de-duplicated against the audience's existing recipients and against earlier rows of the same file — first occurrence wins, so a file listing somebody twice with two different names does not produce two recipients.

The import refuses whole rather than truncating. If the resulting roster would exceed MEMQL_CAMPAIGNS_MAX_AUDIENCE, nothing is written. A partially imported list is one nobody knows is partial, and the send that follows is a silent prefix of the audience the operator meant.

It returns {added, duplicates, invalid, total} plus up to twenty sample invalid lines with their line numbers, so the next action is fixing the file rather than guessing at it. Each added recipient gets a consent event (kind: "grant", source: "import") — which is what finally gives source: "import" a writer, and gives the compliance export a record of where an address came from.

#Merge tags

The replacer set is closed, and is a strings.NewReplacer over enumerable keys rather than a template engine — there is no expression evaluation in a campaign body and there is not going to be:

The HTML and text parts escape differently, on purpose. Every substituted value is HTML-escaped on the HTML path and not on the text path, because the text part is not markup and escaping it would show & to a reader. displayName and every fields.* value are recipient-supplied — an imported CSV is untrusted input — so that split is a security property, not a formatting one.

An unknown tag stays literal in the body. It is not an error and it is not blanked, because blanking it would delete the evidence of the typo. The way to catch one before the whole audience does is the test send, which reports every tag it could not resolve.

#Testing a campaign with an audience

Create a separate audience containing your reviewers’ email addresses and merge fields. In Campaigns → Settings → Testing audience, choose the organization, select that audience and save. This is shared cluster configuration, not a browser preference. Each organization has its own selection; an active audience from a different organization is refused. Choose None selected and save to disconnect.

Open any campaign, including a sending, scheduled, paused or completed campaign, and choose Send test. Completed campaigns are available with Show archived items enabled in Settings. A test uses the published template and the same preflight, template snapshot, queue, worker, sender resolution, suppression, merge fields, rate limits, retries, tracking and signed unsubscribe as a live send. For a repeating campaign it copies the series’ saved instruction and does not advance its next occurrence. The testing audience must differ from both the campaign’s and the series’ live audiences. It never falls back to live subscribers.

Each test has its own campaign occurrence, job and delivery ledger. Source counters, status and schedule stay unchanged. Tests are excluded from the regular campaign list and the organization’s campaign rollup. Recent tests shows the latest 50 runs and a Latest test progress bar; expand a run for per-address results. These results, campaign statistics, delivery records and audience rosters update automatically while open, with background reads every five seconds. Returning to the browser tab reads the current state immediately. Campaign results describes the original send only: if it sent one message and a later test skips an opted-out tester, the campaign remains Sent 1 / Skipped 0 and the latest test shows Sent 0 / Skipped 1. Test outcomes are never added to campaign totals. “Accepted” means the transport accepted the message, not proof of inbox delivery. Suppressed or unsubscribed testers are skipped, as they would be in a live run. Unsubscribe links are real: clicking one changes that test recipient’s consent and may suppress their address for subsequent mail from the organization.

The configured transport applies equally to tests and live sends. With capture enabled, mail appears in the Email app. With external delivery enabled, the normal organization provider sends it. A localhost unsubscribe URL can only be exercised where that local installation is reachable; it does not validate public reachability. The test runs a send immediately; it does not simulate waiting for the recurrence clock.

builtin campaignConfigureTestAudience(accountId: "<organization>", audienceId: "<test-audience>")
builtin campaignTestAudienceSend(campaignId: "<campaign>", audienceId: "<test-audience>", requestId: "<stable-request-id>")

Configuration changes include the expectedRevision returned by query campaignTestSettings. The send’s audienceId is the reviewed selection: if another operator changes the setting, a stale send is refused. Retain requestId after a network error and retry with the same inputs. A database lock and fresh reads make retries on separate replicas return the same run without requeuing it. A deliberate new test uses a new request id. The reply identifies testRunId; use query campaignTestRuns and the ordinary delivery queries to inspect outcomes.

#Existing single-address preview API

The existing campaignTestSend DSL builtin below remains a render-only preview for existing callers. It is not used by the Campaigns app’s Send test action and does not verify the normal queue or delivery ledger. Use the audience-based operation above for execution testing.

builtin campaignTestSend(campaignId: "<id>", to: "you@example.com")

Renders the campaign's template against a synthetic recipient — display name Test Recipient, the address you name, and the fields of the audience's first real recipient when one exists, so {{fields.*}} show the shape they will actually have rather than blanks. The subject is prefixed [Test] , the message goes through the campaign's resolved sending identity, and the unsubscribe footer carries an obviously-inert token.

It writes no delivery row and touches no counter, so a test can never make a campaign look partly sent. It does consume the ordinary send-rate token bucket, because a test is a real message to a real mailbox.

It returns the list of merge tags it could not resolve. That is the check that catches a typo'd {{fields.compnay}} while it is still cheap.

to is required and never defaults to the caller's own address. A builtin that mails somewhere you did not name is one whose default you have to remember.

#Reading what a campaign did

builtin campaignStats(campaignId: "<id>")

Every bucket that can be an exact count is one, at any audience size:

They are computed server-side from the delivery ledger (status plus skipReason), the campaign's own consent events, and its engagement events. This replaces counting a page of rows in a browser, which under-reported every campaign past the page bound and did so silently.

skippedCount also now lands on the campaign row itself, where it always should have been: the worker has computed it per job since the sending engine shipped and had nowhere on the campaign to put it, so the number an operator most needs — the gap between the audience and what actually left — was visible only on a row a browser cannot read. recipientCount - sentCount - skippedCount - failedCount is what is still outstanding.

#Two figures are honestly absent rather than rounded

Unique opens and clicks can come back unmeasured. They are folded from a bounded read of the engagement rows, and a read that comes back at its bound is a truncation. Reporting the folded number anyway would print a unique count that is quietly a floor, on the one campaign large enough that the figure mattered. So at the bound the answer is unmeasured, and the totals — which are exact counts — are still there.

There is no soft-bounce figure per campaign at all. Not a zero: the field is absent. Soft bounces are recorded and deliberately do not suppress, but nothing attributes one to a campaign, so a 0 here would be read as "no soft bounces" — a claim nothing in the system can make. An absent figure and a zero are different answers, and a surface rendering this must not turn one into the other.

#Open and click tracking

Two booleans on the campaign, both defaulting true: trackOpens and trackClicks. An operator sending to a privacy-sensitive list turns them off per campaign.

The HTML part only. At render time every http(s) href is rewritten to a signed redirect and a 1x1 pixel is appended. The text part is untouched: there is no honest way to count an open in plain text, and rewriting a URL a reader can see would be visible mangling for a number nobody asked for.

The token is HMAC-signed by the same key ring the unsubscribe link uses, under a different context string, so an unsubscribe token can never verify as a tracking token or the reverse. Its payload carries the delivery, the campaign, the kind and — for a click — the destination URL. The destination is inside the signed payload rather than in a query parameter, which is what makes the redirect open-redirect-proof: there is no unsigned input for an attacker to aim.

Both are served by the bff on the same public origin as /unsubscribe (MEMQL_CAMPAIGNS_UNSUBSCRIBE_BASE_URL), and both are HTTP for the reason /unsubscribe is: the caller is the recipient's mail client, and there is no gRPC version of that conversation. The token has to be a single path segment — the self-authenticated exemption these routes rely on is bounded to exactly one segment under the mount, so a token containing a / is not merely mis-parsed, it is 401'd before the handler runs.

Hits land on v1:campaigns:engagementEvent, one row per hit, written under internal origin. Uniqueness is by delivery, not by address: one address in two audiences is two people as far as a campaign is concerned.

What tracking does not tell you. An open is a mail client fetching an image — clients that block remote images never report one, and clients that prefetch report one nobody read. It is a floor on engagement and a comparison between sends, not a readership figure. There is no delivered event for the same reason there is no delivered column on the reputation window: the transport says accepted and a bounce arrives later.

#Running a send

1. Author the audience, template and campaign in the Campaigns app (Integrations → Campaigns).

2. Mark the template ready. A draft is refused: half-written copy delivered to an audience is unrecallable, and this is the cheapest guard against it.

3. Press Start sending, or call the builtin:

   text⧉copy

   builtin campaignStartSend(campaignId: "<id>")

   The preflight refuses — with the reason — if no sender is registered, if one-click unsubscribe is unconfigured, if the template is not ready, or if the audience is empty or at the ceiling.

   From a client, the same call is a generated SDK method — the builtin is marked @sdk (memql#4239), which is what puts it on that surface: qc.CampaignStartSend(ctx, client.CampaignStartSendArgs{CampaignId: id}) in Go, query.campaignStartSend({ campaignId }) in TypeScript. The pause, resume and schedule calls below have theirs too (campaignPauseSend, campaignResumeSend, campaignScheduleSend), and the app's buttons go through those methods rather than composing the call string by hand.

4. Watch the campaign row's counters, or the delivery ledger (deliveriesForCampaign).

Pause and resume:

builtin campaignPauseSend(campaignId: "<id>")
builtin campaignResumeSend(campaignId: "<id>")

Pausing leaves the ledger alone, so resuming continues exactly where it stopped.

#Scheduling one

builtin campaignScheduleSend(campaignId: "<id>", scheduledAt: "2026-09-01T09:00:00Z")

or the Schedule send button in the editor. The send then starts on its own; nothing else has to be pressed.

scheduledAt is an RFC 3339 instant. A value with no offset is read as UTC — guessing a local zone from a node's TZ would make one string mean different moments on different replicas.

Four things are worth knowing about it:

The preflight runs when you schedule, and again when it fires. Both, and that is the point of scheduling being a builtin rather than a stored date: a campaign whose template is still a draft is refused while you are looking at the screen, not at 3am.

A time already past is refused. Use campaignStartSend to send now. A backdated schedule is far more often a typo in the year or the offset than a request to send immediately.

The campaign row decides when it fires. The send job carries a copy of the time for you to look at; the worker compares the clock against v1:campaigns:campaign.scheduledAt. So moving the date — with the editor, or with updateCampaign — moves the send, and a schedule you postponed does not go out on its original time.

A campaign whose time passed while the cluster was down still sends, when the cluster comes back. There is no window past which a late send is dropped, deliberately: a silently-skipped campaign is the failure this replaced. If you no longer want it, cancel it — a campaign sitting in scheduled is a campaign that is going to send.

If the fire-time preflight refuses, the reason lands on the campaign row's lastError, where the operator who scheduled it is looking, and the two kinds of refusal are treated differently by who can fix them:

The split is by who can fix it, not by severity. Failing a campaign because a node booted without its mail credentials would make an operator re-author a schedule to recover from a bad deploy.

Cross-replica, a due campaign fires once — and the claim is the least of the three reasons. The send job's id is the campaign's id, so two replicas promoting it write one row; the drain claim admits one replica per (job, progress); and the delivery ledger is per (campaign, recipient) underneath both.

Re-sending a campaign is not a thing. The ledger is per (campaign, recipient), so a second run finds every recipient terminal and mails nobody. Author a new campaign.

#Feeding bounces back in

Bounces and complaints arrive out of band — Microsoft Graph's sendMail returns 202 Accepted and the bounce comes back to the sending mailbox later. Two formats parse out of the box (memql#3461):

Anything else needs a parser, and the honest consequence of not writing one is that its bounces never reach the suppression list.

#Wiring a feed

1. Point the provider's feedback webhook at POST /inbound/{source} (inbound delivery) and configure that source's allowlist entry and signing secret there.

2. Name it as a feedback feed:

   shell⧉copy

   MEMQL_CAMPAIGNS_FEEDBACK_SOURCES=postmaster=rfc3464,ses-feedback=ses

That is the whole wiring. A shipped automation (ingestCampaignFeedback) fires on every staged inbound row and offers it to campaignIngestFeedback, which parses and applies it.

Listing the source here is the authorization, and there is deliberately no role check on top of it. The call asserts nothing: every address and every verdict comes out of a body the provider signed with a secret you configured, arriving at a source you allowlisted, in a format you named — three deployment-level decisions, all made by whoever holds the env. A payload from a source configured scheme=none is refused, because unauthenticated input must not write a cluster-wide list. The most anyone can do by calling the builtin directly is re-process a webhook you already trusted, which is idempotent.

#The Graph mailbox reader

On Microsoft Graph nobody dials you. A failed delivery comes back as a multipart/report DSN mailed to the sending mailbox, so the webhook path above — which is the whole ingestion story — had nothing feeding it on the transport this deployment actually sends on. Bounces landed in a mailbox and were read by no one; the suppression list looked healthy because it was empty.

The reader closes that loop. Active only when the resolved sender is Graph and campaigns are enabled, it lists the sending mailbox's inbox for unread multipart/report messages every MEMQL_EMAIL_NDR_POLL_SECONDS (default 300; 0 disables, and a non-integer value falls back to the default rather than to off), fetches each as MIME, and stages it as an ordinary v1:platform:inboundRequest with source: "graph-mailbox". Processed messages are marked read and categorized, so a restart does not re-read the mailbox from the beginning; the Graph message id is the dedupe key underneath, so a redelivery collapses onto the same row.

Two variables and one Entra fact make it work:

MEMQL_EMAIL_NDR_POLL_SECONDS=300
MEMQL_CAMPAIGNS_FEEDBACK_SOURCES=graph-mailbox=rfc3464

Staging a row is not acting on one. Until graph-mailbox=rfc3464 is listed, the reader files DSNs the shipped automation then declines to process — the same "listing the source here is the authorization" rule as every other feed, applied to a feed the engine happens to fill itself. And the mailbox must be readable: the Graph application needs Mail.Read under the same ApplicationAccessPolicy that scopes Mail.Send (azure-entry-install.md), or the poll fails with a 403 the log names and the loop stays open.

signatureVerified is stamped true on these rows, and that is honest rather than a shortcut. Provenance IS the verification here: the payload was read out of our own mailbox over our own authenticated credential, and there is no third-party signature to check. That field is what gates campaignIngestFeedback, so a row that could not honestly carry it would be staged and never acted on.

#Hard vs soft is the provider's word, not ours

Undetermined counts as transient on purpose: SES is saying it could not tell, and a permanent suppression on "could not tell" costs a real subscriber, while the other error costs at most a few more attempts at a dead address. Nothing anywhere classifies by reading the diagnostic text — a parser that looked for "user unknown" would be inventing a verdict.

A bounce is attributed to its send through the X-Campaign-Id header the sender stamps, which a DSN quotes back and SES echoes in mail.headers. Best-effort: attribution improves a review and is never a precondition for suppressing.

#What happens to a payload we cannot read

It is not dropped. The inbound row is stamped failed with the reason and the call errors, so:

query inboundRequestsByStatus(status: "failed")

is the list of feedback this deployment could not understand. A payload that reads fine and carries no bounce — a delivery receipt, an SNS subscription confirmation — is stamped processed with what it was, because recording that as a failure would teach you to ignore failures. An SNS SubscriptionConfirmation says to visit its SubscribeURL, which is the one message that means the wiring is working.

#By hand

An operator can still enter a report or suppress directly:

builtin campaignRecordFeedback(email: "...", kind: "hard_bounce", campaignId: "...")
builtin campaignSuppress(email: "...", reason: "manual", note: "...")

Both are admin-or-cluster-owner, because there the caller is asserting a verdict.

#Reputation telemetry

Every send and every piece of provider feedback is counted per (sending identity, recipient domain, day) on v1:campaigns:reputationWindow (memql#3462). That is the breakdown that matters: mailbox providers judge a sender independently, so "our bounce rate is 1%" is not a number any of them acts on, while "our complaint rate at gmail.com is 0.4% this week" is.

query reputationWindowsSince(since: "2026-08-01")

Cluster-owner gated, and it carries a domain and four integers — no address anywhere, the same property that makes the suppression list safe to keep cluster-wide.

Two things it deliberately does not claim:

• accepted is not delivered. It counts what the transport took; Graph answers 202 and the bounce arrives hours later through the feedback path. Delivery is inferred as accepted minus bounces, and there is no delivered column because nothing on this side could honestly produce one.
• Unsubscribes are not counted. The one-click endpoint is a separate handler with its own store, so a column for them would be a column nothing writes. The ramp thresholds on bounces and complaints, which is what the providers themselves act on.

There is one counter row per replica (nodeId), written as an absolute running total rather than an increment. Two replicas incrementing one shared row would need a read-modify-write on the send path and would lose counts to the interleaving; one row each, summed at read time, has neither problem.

#Warming a new sending domain

Warming is raising volume gradually on a new sending identity, so a provider sees a growing trickle rather than a cold blast.

There is now an automated ramp, and it is off by default.

MEMQL_CAMPAIGNS_WARMUP_ENABLED=true
MEMQL_CAMPAIGNS_WARMUP_STEPS=5,10,25,50,100,200

An established sending domain does not want its rate re-derived by a control loop that has never seen it, which is why you have to ask for it.

It can only ever slow you down. The effective rate is min(the current step, MEMQL_CAMPAIGNS_SEND_RATE_PER_MINUTE). The ramp holds your configured rate down while the evidence is thin; it never raises it past what you allowed.

#It advances on evidence, not on a clock

A fixed-schedule ramp is a guess with a schedule attached, and it produces the worst kind of false confidence — an operator believing they are warming safely while the system has no idea whether it is working. This one advances a step only when all of these hold:

A domain over threshold reduces the ramp a step rather than merely holding it. Holding at a rate that is already producing complaints is not a neutral act.

A domain with less than _WARMUP_MIN_DOMAIN_VOLUME (50) of its own is ignored, or one bounce at a domain we sent three messages to would read as 33% and pin the ramp forever. The cost — a genuinely bad small domain is invisible to the ramp until it grows — is a real gap rather than a hidden one.

#Reading what it decided

query warmupStateForIdentity(sendingIdentity: "sender@example.com")

carries the current step, its ratePerMinute, the last decision (started / held / advanced / reduced) and a reason in words. held is the common and healthy state, so the reason is written on every evaluation: an operator looking at a step that has not moved for two days should read why rather than guess.

A malformed _WARMUP_STEPS leaves the ramp disabled with a boot warning, rather than being sorted into a ladder you did not write.

#Doing it by hand instead

Still perfectly reasonable, and unchanged:

1. Start MEMQL_CAMPAIGNS_SEND_RATE_PER_MINUTE low (a few per minute).
2. Send to your most engaged recipients first — split the audience rather than sampling it, so the send order is deliberate.
3. Raise the rate over days, watching reputationWindowsSince and the domain breakdown on the suppression list.
4. Back off on any rise in complaints.

#Troubleshooting

#Azure send receipts

Before an ACS request leaves the cluster, MemQL persists an encrypted receipt with its own operation UUID and the message's internal delivery identity. A lost response or a restart never authorizes another POST for that identity. The cluster polls the original resource's operation endpoint across replicas using shared database coordination. A confirmed rate-limit refusal can retry after Azure's requested delay; an unknown outcome cannot.

In the organization's email connection detail, Check sends shows the most recent 25 Azure processing receipts. Processing completed means ACS finished processing the request, not that the recipient received it. This setup does not yet provision an Event Grid or Azure Monitor delivery-feedback source; recipient bounces and delivery cannot be inferred from operation success. Existing campaign sent counts continue to mean transport acceptance. An acceptance that initially returned an unknown result can later be visible as accepted in the receipt even when the earlier campaign attempt was recorded as failed; it is never resent automatically.

Polling stops after 24 hours if processing remains unconfirmed, retaining the receipt to prevent accidental resubmission. Reconnecting to a different Azure resource does not move old operations: they belong to the original resource. No message body, mailbox address, or access key appears in the receipt status response. Disconnecting email stops new sends and pauses reconciliation until the original resource is connected again.