Plugin SDK -- the pack extension contract

A pack extends MemQL with product-specific behavior: Go integrations plus a .memql DSL bundle. The Go integrations compile into a node-type binary via build tags; the .memql domains load compiled-in or at runtime from disk (see Scope below). A product's DSL bundle plus its client is the reference consumer.

Three words. A component is engine internals. An integration talks to other systems (integrations/). A pack is a client-agnostic product feature — that is what intake "plugin" means. RegisterPlugin is the registration primitive, not a fourth runtime. Core domains (dsl/todos, dsl/calendar, dsl/campaigns) cannot be shadowed. See Component vs integration vs pack.

This page is the contract reference for the Go surface a pack targets -- the PluginContext it receives, the PluginFactory it implements, the registration primitives it calls, and the contract version the loader checks at startup. For an end-to-end "build your first pack" walkthrough, see Building a pack -- the developer guide, with the in-tree examples/referencepack reference pack as its worked example.

Scope. A pack has two halves that load differently. (a) The Go integration -- the IntegrationProvider, its PluginFactory, and the registration primitives -- is compiled in via build tags: linking its Go package into a binary runs its init() registrations. There is no runtime (non-compiled) loading of the Go half, by design. (b) The .memql DSL domains load either compiled-in (an embedded tree) or at runtime from disk via MEMQL_DSL_PATH, where a product-agnostic engine image mounts a product's DSL bundle at boot with RegisterTree(domain, os.DirFS(...)) and zero compiled-in product code. The contract below is the stable Go surface that half (a) targets.

#Contract version

memql.PluginContractVersion (in component/memql/plugins.go) is the major version of the extension surface: the PluginContext fields, the PluginFactory signature, and the registration primitives.

• Current: 2. It moved from 1 in epic memql#5127, which renamed two PluginContext fields and removed the three provider lookups from IntegrationEngineAccess.
• It bumps only on a breaking change to that surface -- a removed/renamed PluginContext field, a changed PluginFactory signature, a changed registration primitive.
• Additive, backward-compatible changes do NOT bump it -- a new PluginContext field or a new optional primitive leaves existing packs compiling and loading unchanged.

A pack records the version it was built against when it registers (see below). At startup the loader (app.materializePlugins) calls PluginRegistration.ValidateContract for every registered pack and rejects -- fatal, with a descriptive error -- any pack whose declared version is incompatible with the core's PluginContractVersion. Compatibility is exact-major equality: a pack built against a retired older contract (the core may have removed surface it relies on) or one that needs a newer core than the running binary provides (the core lacks surface it relies on) both fail loudly rather than silently mis-binding.

memql.CheckPluginContractCompat(version int) error is the pure, unit-testable form of that check.

#PluginFactory

type PluginFactory func(pctx PluginContext) (IntegrationProvider, error)

A pack's factory builds its IntegrationProvider (the DSL-callable capability set) from the live PluginContext.

• Returning an error aborts startup (fatal log). Use this only for a true misconfiguration a degraded mode cannot paper over.
• Returning (nil, nil) is the documented opt-out signal: the pack is compiled in but its dependencies are not satisfied in this environment (e.g. object storage with no bucket configured). The loader logs plug-in opted out and continues. The factory should log its own warning if the opt-out is worth reporting.

#PluginContext -- the surface a pack receives

PluginContext is the narrow, stable Go surface handed to every factory. A pack either finds what it needs here (or on Engine) or it does not reach into app/ internals. Callbacks are lazily evaluated, so a pack that stashes the context still observes live state.

ResolvePartitionFromContext is the sanctioned way for a pack to scope work to a tenant: partition is the canonical tenant scope. The dedicated reference is Partition scoping.

ConceptDataIsStaged is required reading for any pack that issues a hand-rolled SELECT against "MemoryNodes" (epic memql#3974). A pack that goes through Engine.Execute inherits the engine's enforcement automatically -- the staged predicate is injected into the plan and every emitted row is gated. Raw SQL passes through neither, so nothing is injected into it and this callback is the whole of the enforcement for that read.

Two rules, both learned the expensive way:

• Refuse when it is nil. A factory that treats a missing callback as "nothing is staged" produces a node that serves withheld rows and looks completely healthy doing it. Every core plug-in that consumes it returns an error from its factory instead.
• Ask it in Go, before building the statement, rather than appending a conjunct. Visibility is a pure function of the CONCEPT, and a statement that pins one concept is either wholly admissible or wholly empty -- so an early return is exact, not an approximation. A conjunct additionally has to land INSIDE a latest-version CTE (outside, it drops the id instead of falling through), cannot be appended after a GROUP BY at all, and must not run before a Go-side seen fold -- which would make the fold elect an older version as "latest" rather than skip the id.

And the part that is easy to get backwards: do not gate a read whose rows never leave the process. Counts behind a quota, prior-state reads behind a read-merge or a state machine, idempotency probes, migrations, audit traces and backup exports all consume rows to make a decision. Withholding rows there does not protect anyone -- it corrupts the decision, and in the backup case destroys data. See staged_read_site_classification_test.go in the repository root, which fails the build on a hand-rolled read nobody has ruled on.

#AdmitSourceRow -- the same reads need per-row authorization too

ConceptDataIsStaged answers "is this concept's data visible to anyone yet". AdmitSourceRow answers the other question about the same row: "may this caller see this row" (memql#4029). A hand-rolled SELECT needs both, for the same reason: nothing the engine injects reaches it. Same nil rule -- refuse to construct rather than default to "yes".

Apply it to the rows AS FETCHED, before folding, summarizing or repacking them. Both of the engine's row-authz mechanisms resolve the tier from a concept: filter injection from plan.BoundConcept, the row gate from the row's own concept. So a capability that reads real rows and returns a synthetic node under its own concept defeats both by construction -- the gate is asked about the made-up concept, finds no tier declared, and admits. It fails silently and in the admitting direction.

There is no repairing it afterwards. A summary node has no top-level owner field for an owned tier to read, no per-row identity for granted, a synthesized id, and often carries rows from more than one concept. The fetched rows are the last point on the path where the question can be answered at all.

If the reader dedups versions per id, gate INSIDE the fold, after the id is marked seen. Filtering the slice first lets a denied latest version fall through to an admitted older one, so the caller gets a stale row instead of no row -- the same ordering hazard as the staged conjunct above, arriving from the authorization side. AdmitSourceRows is the convenience form for a reader with no fold; a reader that folds calls AdmitSourceRow per row instead.

Note what this does and does not buy: it makes a direct read correct, not scoped. The gate filters after the fetch and adds no caller predicate to the SQL, so a read narrowed only by a caller-supplied argument still fetches everything that argument names. Routing such reads through the authorized path is tracked separately (memql#3984).

#Registration primitives

A pack wires itself in from init() functions in build-tag-gated .go files. The build tags decide which node-type binaries include the registration.

Minimal shape:

//go:build mypack
 
package mypack
 
import (
	"embed"
 
	memqldsl "github.com/znasllc-io/memql/dsl"
	"github.com/znasllc-io/memql/component/memql"
)
 
//go:embed all:*.memql all:prompts
var packFS embed.FS
 
func init() {
	// Pin the contract version this pack was built against.
	memql.RegisterPluginForContract("mypack", memql.PluginContractVersion, newProvider)
	memqldsl.RegisterTree("mypack", packFS)
}
 
func newProvider(pctx memql.PluginContext) (memql.IntegrationProvider, error) {
	// ... build the IntegrationProvider from pctx ...
}

#Pack model + load-time validation

A pack is exactly these three registration primitives, called together from a pack's build-tag-gated init():

1. RegisterPluginForContract(name, version, factory) -- the Go IntegrationProvider, contract-version-checked by the loader.
2. RegisterTree(domain, fs.FS) -- the embedded .memql subtree, mounted under domain/ and namespace-validated.
3. RegisterRoutingRule(rule) -- cross-node event routing (required for any event that must cross a node boundary).

MemQL validates both halves at load time and fails loudly on a violation -- a broken pack aborts startup rather than silently mis-binding:

• Contract version -- app.materializePlugins calls PluginRegistration.ValidateContract for every registered pack and rejects (fatal) any pack whose declared PluginContractVersion is incompatible with the core's (exact-major equality; see "Contract version" above).
• Namespace ownership -- RegisterTree validates the pack's DSL domain via dsl.ValidatePackDomain(domain, coreDomains, existing) before mounting it. A domain must be non-empty, contain no /, and collide with neither a core embedded domain nor another pack's already-registered domain. A core domain is canonical and owned by MemQL -- a pack cannot shadow or extend one. Two packs claiming the same namespace is ambiguous. Either collision panics at init() time (the only caller), consistent with RegisterTree's other input guards, so the conflict surfaces at startup with an actionable message.

dsl.ValidatePackDomain is the pure, unit-testable form of the namespace-ownership check (the analogue of CheckPluginContractCompat for the DSL tree). The core domain set is read from the embedded tree's top-level directories, so it stays in lockstep with the //go:embed directive.

Go plugins stay build-tag-linked. There is no runtime (non-compiled) loading of the Go half -- an IntegrationProvider registers from init() only when its package is linked in, and contract validation runs at startup against that compiled-in set. The DSL half is different: RegisterTree takes any fs.FS, so a product's .memql domains can be mounted at runtime from disk (os.DirFS under MEMQL_DSL_PATH) as well as embedded -- and the namespace-ownership validation below runs identically for either source.

#Stability promise

Within a major PluginContractVersion, the surface above is append-only: fields and primitives are added, never removed or repurposed. A pack that compiles and loads against version N keeps compiling and loading against any later core that still reports version N. A breaking change bumps the version and is announced; stale packs then fail closed at startup with a clear, actionable error rather than mis-binding.