docsOverview

MemQL Proving Scorecard

What the platform measures about itself, and what it does not. Each number is a median with its spread and its N, stamped with the commit and date it came from. A figure that was not measured says so, in its own words, rather than reporting a zero -- an absent figure and a zero are different answers.

Scorecard 2026-09-27 at commit 9a93b7682
Corpus 7de1803c5d73266a -- two scorecards with different corpus fingerprints measured different things and are not a trend

#The two tiers

Tier	Armed	Last run
`ci`	yes	2026-09-27	replayed on every pull request; it reaches no provider. Responses come from synthetic.
`live`	no	never	dispatched by hand from proving-live.yml; it has not been run, so every live figure reads `tierNotRun`

#What each tier can honestly measure

A CI run replays from recorded model responses, so it is deterministic by construction: pass-rate variance is trivially perfect and wall-clock belongs to the runner rather than to the product. Those figures are unmeasured in CI and stay that way until the live tier fills them.

Family	Metric	Blocks a merge	What it counts
amortizedCost	`amortizedCost.compileCallsOnCatalogHit`	yes	Model calls made while compiling a goal that exactly matches the catalog.
amortizedCost	`amortizedCost.providerCalls`	yes	Provider calls one run made. A run served entirely from the journal makes none.
amortizedCost	`amortizedCost.replaysServedWithoutModel`	yes	Goals a trusted learned procedure answered with no model and no app call.
amortizedCost	`amortizedCost.stepsServedFromJournal`	no	Steps answered from the journal rather than re-executed.
amortizedCost	`amortizedCost.tokensPerGoal`	no	Tokens spent per goal, amortized over N runs with different variables.
amortizedCost	`amortizedCost.usdPerGoal`	no	Dollars per goal, amortized over N runs with different variables.
reliability	`reliability.passRate`	yes	Runs whose verifier passed, over runs attempted.
reliability	`reliability.passVariance`	no	Spread of the pass rate across K runs of one goal.
recovery	`recovery.modelCalls`	no	Model calls spent recovering from an injected failure.
recovery	`recovery.rate`	yes	Runs that reached a passing verifier after an injected failure.
recovery	`recovery.repairStepsVsRestartSteps`	no	Steps a repair ran, over steps a restart from the beginning would have run.
recovery	`recovery.stepsReExecuted`	yes	Already-completed steps re-executed while recovering. Repair from the failed step re-executes none.
recovery	`recovery.wallClockMs`	no	Wall-clock from the injected failure to a passing verifier.
durability	`durability.duplicatedSideEffects`	yes	Side effects delivered twice across a mid-run kill and a resume. Must be zero.
durability	`durability.duplicatedSideEffectsAcrossDivergence`	yes	Side effects delivered twice when a replay diverged and the app took over. Must be zero.
durability	`durability.resumedOnAnotherNode`	yes	Killed runs that resumed on a different node from the journal alone.
durability	`durability.resumedStepsReExecuted`	yes	Completed steps a resumed run executed again. Must be zero.
learningCurve	`learningCurve.catalogServedFraction`	no	Steps served by the catalog, across a sequence of related goals.
learningCurve	`learningCurve.usdPerGoal`	no	Dollars per goal as the catalog fills across a related sequence.
speed	`speed.firstRunVsReplayRatio`	no	A replayed run's wall-clock over the first run's.
speed	`speed.journalPerStepOverheadMs`	no	Milliseconds the journal itself adds per step, measured in one process against the same steps unjournaled.
speed	`speed.wallClockPerGoalMs`	no	Wall-clock for one goal, end to end.
governance	`governance.approvalsBoundToArtifactHash`	yes	Approvals carrying the hash of the artifact approved. Must be 1.
governance	`governance.effectsWithReceipts`	yes	Side-effecting steps that left a receipt. Must be 1.
governance	`governance.modelCallsJournaled`	yes	Model calls the run journaled, over model calls it made. Must be 1.

#The numbers

#Amortized cost

Scenario	Metric	Arm	Figure
`amortizedCost.a-catalog-hit-makes-no-model-call`	`amortizedCost.compileCallsOnCatalogHit`	platform	0 (0-0, N=1)
`amortizedCost.control-a-catalog-miss-reaches-a-model`	`amortizedCost.compileCallsOnCatalogHit`	platform	1 (1-1, N=1)
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.providerCalls`	baseline	1 (1-1, N=1)
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.providerCalls`	platform	0 (0-0, N=1)
`amortizedCost.a-local-fleet-serves-every-call`	`amortizedCost.providerCalls`	baseline	0 (0-0, N=1)
`amortizedCost.a-local-fleet-serves-every-call`	`amortizedCost.providerCalls`	platform	0 (0-0, N=1)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.providerCalls`	baseline	2 (2-2, N=1)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.providerCalls`	platform	1 (1-1, N=1)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.providerCalls`	baseline	1 (1-1, N=1)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.providerCalls`	platform	0 (0-0, N=1)
`amortizedCost.control-a-fresh-goal-reaches-a-model`	`amortizedCost.providerCalls`	baseline	1 (1-1, N=1)
`amortizedCost.control-a-fresh-goal-reaches-a-model`	`amortizedCost.providerCalls`	platform	1 (1-1, N=1)
`amortizedCost.control-a-reasoning-step-does-reach-a-provider`	`amortizedCost.providerCalls`	baseline	1 (1-1, N=1)
`amortizedCost.control-a-reasoning-step-does-reach-a-provider`	`amortizedCost.providerCalls`	platform	1 (1-1, N=1)
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.replaysServedWithoutModel`	baseline	0 (0-0, N=1)
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.replaysServedWithoutModel`	platform	1 (1-1, N=1)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.replaysServedWithoutModel`	baseline	0 (0-0, N=1)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.replaysServedWithoutModel`	platform	1 (1-1, N=1)
`amortizedCost.control-a-disliked-recording-teaches-nothing`	`amortizedCost.replaysServedWithoutModel`	baseline	0 (0-0, N=1)
`amortizedCost.control-a-disliked-recording-teaches-nothing`	`amortizedCost.replaysServedWithoutModel`	platform	0 (0-0, N=1)
`amortizedCost.control-a-shadow-procedure-serves-no-goal`	`amortizedCost.replaysServedWithoutModel`	baseline	0 (0-0, N=1)
`amortizedCost.control-a-shadow-procedure-serves-no-goal`	`amortizedCost.replaysServedWithoutModel`	platform	0 (0-0, N=1)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.stepsServedFromJournal`	baseline	5 (5-5, N=1)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.stepsServedFromJournal`	platform	5 (5-5, N=1)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.tokensPerGoal`	baseline	-- (a replayed run's tokens are the cassette's recorded tokens, not this run's)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.tokensPerGoal`	platform	-- (a replayed run's tokens are the cassette's recorded tokens, not this run's)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.usdPerGoal`	baseline	-- (a replayed run spends nothing; the recorded cost belongs to the capture)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.usdPerGoal`	platform	-- (a replayed run spends nothing; the recorded cost belongs to the capture)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`amortizedCost.a-catalog-hit-makes-no-model-call`	`amortizedCost.compileCallsOnCatalogHit`	not comparable: the baseline arm produced no figure
`amortizedCost.control-a-catalog-miss-reaches-a-model`	`amortizedCost.compileCallsOnCatalogHit`	not comparable: the baseline arm produced no figure
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.providerCalls`	improved (-100.0%)
`amortizedCost.a-local-fleet-serves-every-call`	`amortizedCost.providerCalls`	unchanged
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.providerCalls`	improved (-50.0%)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.providerCalls`	improved (-100.0%)
`amortizedCost.control-a-fresh-goal-reaches-a-model`	`amortizedCost.providerCalls`	unchanged
`amortizedCost.control-a-reasoning-step-does-reach-a-provider`	`amortizedCost.providerCalls`	unchanged
`amortizedCost.a-corrected-step-teaches-the-procedure`	`amortizedCost.replaysServedWithoutModel`	improved (+1 absolute; no relative figure, the earlier value was zero)
`amortizedCost.a-trusted-procedure-replays-without-a-model`	`amortizedCost.replaysServedWithoutModel`	improved (+1 absolute; no relative figure, the earlier value was zero)
`amortizedCost.control-a-disliked-recording-teaches-nothing`	`amortizedCost.replaysServedWithoutModel`	unchanged
`amortizedCost.control-a-shadow-procedure-serves-no-goal`	`amortizedCost.replaysServedWithoutModel`	unchanged
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.stepsServedFromJournal`	unchanged
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.tokensPerGoal`	undecidable (the earlier figure is notMeasurableOnReplay)
`amortizedCost.a-resume-does-not-recall-the-model`	`amortizedCost.usdPerGoal`	undecidable (the earlier figure is notMeasurableOnReplay)

#Reliability

Scenario	Metric	Arm	Figure
`reliability.an-unrecoverable-failure-stays-failed`	`reliability.passRate`	baseline	0.0% (0.0%-0.0%, N=1)
`reliability.an-unrecoverable-failure-stays-failed`	`reliability.passRate`	platform	100.0% (100.0%-100.0%, N=1)
`reliability.the-same-goal-run-repeatedly`	`reliability.passRate`	baseline	100.0% (100.0%-100.0%, N=1)
`reliability.the-same-goal-run-repeatedly`	`reliability.passRate`	platform	100.0% (100.0%-100.0%, N=1)
`reliability.the-same-goal-run-repeatedly`	`reliability.passVariance`	baseline	-- (a replay is deterministic, so its variance is zero by construction rather than by measurement)
`reliability.the-same-goal-run-repeatedly`	`reliability.passVariance`	platform	-- (a replay is deterministic, so its variance is zero by construction rather than by measurement)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`reliability.an-unrecoverable-failure-stays-failed`	`reliability.passRate`	improved (+1 absolute; no relative figure, the earlier value was zero)
`reliability.the-same-goal-run-repeatedly`	`reliability.passRate`	unchanged
`reliability.the-same-goal-run-repeatedly`	`reliability.passVariance`	undecidable (the earlier figure is notMeasurableOnReplay)

#Recovery

Scenario	Metric	Arm	Figure
`recovery.a-transient-failure-is-retried`	`recovery.modelCalls`	baseline	0 (0-0, N=1)
`recovery.a-transient-failure-is-retried`	`recovery.modelCalls`	platform	0 (0-0, N=1)
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.modelCalls`	baseline	0 (0-0, N=1)
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.modelCalls`	platform	0 (0-0, N=1)
`recovery.control-a-novel-failure-reaches-the-classifier`	`recovery.modelCalls`	baseline	0 (0-0, N=1)
`recovery.control-a-novel-failure-reaches-the-classifier`	`recovery.modelCalls`	platform	2 (2-2, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.modelCalls`	baseline	2 (2-2, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.modelCalls`	platform	1 (1-1, N=1)
`recovery.a-stalled-loop-escalates-instead-of-retrying`	`recovery.rate`	baseline	0.0% (0.0%-0.0%, N=1)
`recovery.a-stalled-loop-escalates-instead-of-retrying`	`recovery.rate`	platform	100.0% (100.0%-100.0%, N=1)
`recovery.a-transient-failure-is-retried`	`recovery.rate`	baseline	100.0% (100.0%-100.0%, N=1)
`recovery.a-transient-failure-is-retried`	`recovery.rate`	platform	100.0% (100.0%-100.0%, N=1)
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.rate`	baseline	100.0% (100.0%-100.0%, N=1)
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.rate`	platform	100.0% (100.0%-100.0%, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.rate`	baseline	100.0% (100.0%-100.0%, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.rate`	platform	100.0% (100.0%-100.0%, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.repairStepsVsRestartSteps`	baseline	180.0% (180.0%-180.0%, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.repairStepsVsRestartSteps`	platform	120.0% (120.0%-120.0%, N=1)
`recovery.a-transient-failure-is-retried`	`recovery.stepsReExecuted`	baseline	1 (1-1, N=1)
`recovery.a-transient-failure-is-retried`	`recovery.stepsReExecuted`	platform	0 (0-0, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.stepsReExecuted`	baseline	3 (3-3, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.stepsReExecuted`	platform	0 (0-0, N=1)
`recovery.repair-runs-from-the-failed-step`	`recovery.wallClockMs`	baseline	-- (wall-clock on a shared CI runner belongs to the runner)
`recovery.repair-runs-from-the-failed-step`	`recovery.wallClockMs`	platform	-- (wall-clock on a shared CI runner belongs to the runner)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`recovery.a-transient-failure-is-retried`	`recovery.modelCalls`	unchanged
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.modelCalls`	unchanged
`recovery.control-a-novel-failure-reaches-the-classifier`	`recovery.modelCalls`	regressed (+2 absolute; no relative figure, the earlier value was zero)
`recovery.repair-runs-from-the-failed-step`	`recovery.modelCalls`	improved (-50.0%)
`recovery.a-stalled-loop-escalates-instead-of-retrying`	`recovery.rate`	improved (+1 absolute; no relative figure, the earlier value was zero)
`recovery.a-transient-failure-is-retried`	`recovery.rate`	unchanged
`recovery.an-environment-failure-is-classified-by-rules`	`recovery.rate`	unchanged
`recovery.repair-runs-from-the-failed-step`	`recovery.rate`	unchanged
`recovery.repair-runs-from-the-failed-step`	`recovery.repairStepsVsRestartSteps`	improved (-33.3%)
`recovery.a-transient-failure-is-retried`	`recovery.stepsReExecuted`	improved (-100.0%)
`recovery.repair-runs-from-the-failed-step`	`recovery.stepsReExecuted`	improved (-100.0%)
`recovery.repair-runs-from-the-failed-step`	`recovery.wallClockMs`	undecidable (the earlier figure is notMeasurableOnReplay)

#Durability

Scenario	Metric	Arm	Figure
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.duplicatedSideEffects`	baseline	1 (1-1, N=1)
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.duplicatedSideEffects`	platform	0 (0-0, N=1)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.duplicatedSideEffects`	baseline	1 (1-1, N=1)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.duplicatedSideEffects`	platform	0 (0-0, N=1)
`durability.a-divergence-duplicates-no-side-effect`	`durability.duplicatedSideEffectsAcrossDivergence`	baseline	1 (1-1, N=1)
`durability.a-divergence-duplicates-no-side-effect`	`durability.duplicatedSideEffectsAcrossDivergence`	platform	0 (0-0, N=1)
`durability.control-a-divergence-without-guidance-duplicates`	`durability.duplicatedSideEffectsAcrossDivergence`	baseline	1 (1-1, N=1)
`durability.control-a-divergence-without-guidance-duplicates`	`durability.duplicatedSideEffectsAcrossDivergence`	platform	0 (0-0, N=1)
`durability.a-resume-re-executes-no-completed-step`	`durability.resumedOnAnotherNode`	platform	100.0% (100.0%-100.0%, N=1)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.resumedOnAnotherNode`	platform	100.0% (100.0%-100.0%, N=1)
`durability.a-resume-re-executes-no-completed-step`	`durability.resumedStepsReExecuted`	baseline	2 (2-2, N=1)
`durability.a-resume-re-executes-no-completed-step`	`durability.resumedStepsReExecuted`	platform	0 (0-0, N=1)
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.resumedStepsReExecuted`	baseline	1 (1-1, N=1)
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.resumedStepsReExecuted`	platform	0 (0-0, N=1)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.resumedStepsReExecuted`	baseline	2 (2-2, N=1)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.resumedStepsReExecuted`	platform	0 (0-0, N=1)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.duplicatedSideEffects`	improved (-100.0%)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.duplicatedSideEffects`	improved (-100.0%)
`durability.a-divergence-duplicates-no-side-effect`	`durability.duplicatedSideEffectsAcrossDivergence`	improved (-100.0%)
`durability.control-a-divergence-without-guidance-duplicates`	`durability.duplicatedSideEffectsAcrossDivergence`	improved (-100.0%)
`durability.a-resume-re-executes-no-completed-step`	`durability.resumedOnAnotherNode`	not comparable: the baseline arm produced no figure
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.resumedOnAnotherNode`	not comparable: the baseline arm produced no figure
`durability.a-resume-re-executes-no-completed-step`	`durability.resumedStepsReExecuted`	improved (-100.0%)
`durability.a-script-step-on-a-machine-survives-a-stop`	`durability.resumedStepsReExecuted`	improved (-100.0%)
`durability.a-stopped-run-resumes-with-no-duplicated-effect`	`durability.resumedStepsReExecuted`	improved (-100.0%)

#Learning curve

Scenario	Metric	Arm	Figure
`learningCurve.a-deterministic-plan-needs-no-model-at-all`	`learningCurve.catalogServedFraction`	baseline	100.0% (100.0%-100.0%, N=1)
`learningCurve.a-deterministic-plan-needs-no-model-at-all`	`learningCurve.catalogServedFraction`	platform	100.0% (100.0%-100.0%, N=1)
`learningCurve.a-sequence-of-related-goals`	`learningCurve.catalogServedFraction`	baseline	83.3% (83.3%-83.3%, N=1)
`learningCurve.a-sequence-of-related-goals`	`learningCurve.catalogServedFraction`	platform	83.3% (83.3%-83.3%, N=1)
`learningCurve.a-sequence-of-related-goals`	`learningCurve.usdPerGoal`	baseline	-- (dollars per goal need a live provider; a replay spends nothing)
`learningCurve.a-sequence-of-related-goals`	`learningCurve.usdPerGoal`	platform	-- (dollars per goal need a live provider; a replay spends nothing)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`learningCurve.a-deterministic-plan-needs-no-model-at-all`	`learningCurve.catalogServedFraction`	unchanged
`learningCurve.a-sequence-of-related-goals`	`learningCurve.catalogServedFraction`	unchanged
`learningCurve.a-sequence-of-related-goals`	`learningCurve.usdPerGoal`	undecidable (the earlier figure is notMeasurableOnReplay)

#Speed

Scenario	Metric	Arm	Figure
`speed.a-first-run-against-a-replay`	`speed.firstRunVsReplayRatio`	baseline	-- (the first run and the replay would both be replays here)
`speed.a-first-run-against-a-replay`	`speed.firstRunVsReplayRatio`	platform	-- (the first run and the replay would both be replays here)
`speed.what-the-journal-costs-per-step`	`speed.journalPerStepOverheadMs`	platform	20ms (20ms-20ms, N=1)
`speed.a-first-run-against-a-replay`	`speed.wallClockPerGoalMs`	baseline	-- (wall-clock on a shared CI runner belongs to the runner)
`speed.a-first-run-against-a-replay`	`speed.wallClockPerGoalMs`	platform	-- (wall-clock on a shared CI runner belongs to the runner)
`speed.what-the-journal-costs-per-step`	`speed.wallClockPerGoalMs`	baseline	-- (wall-clock on a shared CI runner belongs to the runner)
`speed.what-the-journal-costs-per-step`	`speed.wallClockPerGoalMs`	platform	-- (wall-clock on a shared CI runner belongs to the runner)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`speed.a-first-run-against-a-replay`	`speed.firstRunVsReplayRatio`	undecidable (the earlier figure is notMeasurableOnReplay)
`speed.what-the-journal-costs-per-step`	`speed.journalPerStepOverheadMs`	not comparable: the baseline arm produced no figure
`speed.a-first-run-against-a-replay`	`speed.wallClockPerGoalMs`	undecidable (the earlier figure is notMeasurableOnReplay)
`speed.what-the-journal-costs-per-step`	`speed.wallClockPerGoalMs`	undecidable (the earlier figure is notMeasurableOnReplay)

#Governance

Scenario	Metric	Arm	Figure
`governance.an-approval-does-not-carry-to-a-changed-artifact`	`governance.approvalsBoundToArtifactHash`	platform	100.0% (100.0%-100.0%, N=1)
`governance.every-side-effect-leaves-a-receipt`	`governance.effectsWithReceipts`	platform	100.0% (100.0%-100.0%, N=1)
`governance.an-approval-does-not-carry-to-a-changed-artifact`	`governance.modelCallsJournaled`	platform	-- (the CI tier's model responses come from a cassette through a fake step registry, so no call reaches the journal seam; the live tier calls a provider down the real path and can count it)

Platform against the bare-loop baseline:

Scenario	Metric	Platform vs baseline
`governance.an-approval-does-not-carry-to-a-changed-artifact`	`governance.approvalsBoundToArtifactHash`	not comparable: the baseline arm produced no figure
`governance.every-side-effect-leaves-a-receipt`	`governance.effectsWithReceipts`	not comparable: the baseline arm produced no figure
`governance.an-approval-does-not-carry-to-a-changed-artifact`	`governance.modelCallsJournaled`	not comparable: the baseline arm produced no figure

#Governance

Pass or fail, with no score. A failure here blocks a merge.

Property	Scenario	Result
approvalsBoundToArtifactHash	`governance.an-approval-does-not-carry-to-a-changed-artifact`	pass
everyEffectHasAReceipt	`governance.every-side-effect-leaves-a-receipt`	pass

#Method

The baseline is the same model, the same tools and the same scenarios in a plain bounded tool loop, with the platform's machinery switched off: no catalog, no journal, no replay, no symptom classifier, no healing. It gets the same retry allowance and the same wall-clock ceiling. What it does not get is a memory of having done this before, which is the whole thesis.

No other framework is named or measured. The comparison is the platform against a model on its own, which is the only comparison this suite is equipped to make fairly.

What a synthetic model id in the CI row means. A CI run has no provider, so both arms replay from a recorded tape. Where that tape holds PLACEHOLDER responses, every figure above is still a fact about the plan and the platform -- how many model exchanges a plan needs, how many steps a resume re-executes, whether an effect duplicated -- because none of those depends on the words a model said. Nothing model-dependent is published from it: tokens, dollars and answer quality all read unmeasured, and they stay that way until the live tier runs.

Design record: the proving suite.