Safe convergence for an experience-reuse product loop: belief provenance, a confirmation state machine, and adversarial design punch-throughs

Problem: why a risk-only guard was bound to break

An experience-reuse "product loop" lets a runtime crystallize answers from prior task attempts so that, over time, it asks fewer blocking clarification questions. The failure under study has its root cause not in the binding logic but in a MISSING DIMENSION in the belief store. The store records (taskClass, slotID) = value together with a source and a turn index, but NOT which task instance acquired the belief. As a result, the binding step cannot distinguish a value the user just confirmed in the current run from a value carried (and possibly stale) from a prior run. Any guard keyed only on a risk class hits both cases at once — which is why a blanket rule such as "never pre-bind irreversible" also broke the legitimate same-session flow.

1. Threat model

Asset: irreversible / high-impact side effects (transfer, ship, sign, consent, authorize). Attack surface: the runtime silently binds a belief to a slot and acts on it. Threats:

• T1 cross-run stale reuse: a historical crystallized value fires though the world changed.
• T2 wrong-source current-instance: a tool- or inference-derived value in the current run is treated as user-confirmed and silently used.
• T3 semantic staleness: the stored value is unchanged (e.g. "consent = yes") but the world it refers to (scope, time) changed.
• T4 risk misclassification: a truly irreversible slot is tagged reversible and falls onto a low-protection path.
• T5 decision split across mixed-risk slots: an irreversible decision leaks through a non-irreversible sub-slot.
• T6 monotonic escalation: for privilege/scope slots, pre-filling the prior value biases toward over-granting.
• T7 confirmation fatigue (human): even a confirmation prompt can be rubber-stamped — an irreducible residual.

2. State model: belief provenance

Each belief carries: a value; a riskClass (cosmetic / reversible / policy / irreversible); a sourceClass (user / tool / inference / default); an acquisition marker (run, turn); an instance scope (an instance identifier — the NEW dimension); and a world reference (a reference to the world-state the value pertains to — load-bearing for consent and T3). The instance identifier is stable across one task attempt's clarify → answer → resume cycle and is fresh for a new attempt.

This split separates current-instance beliefs (acquired in this run, valid now) from carried beliefs (acquired in a prior instance, possibly stale) — the distinction the risk-only fix lacked. Per belief it answers: where from (source), when (acquired), current instance? (scope), silent-bind allowed?, and confirm-needed? (the last two derived in §3).

3. Confirmation model: a minimal state machine

Behaviors: SILENT (bind, no prompt); SURFACED (bind, but listed in a vetoable "assumptions" restatement); CONFIRM (pre-filled blocking confirmation — low-friction, must be passed); ASK (a fresh question, no pre-fill).

* only a USER-sourced current-instance irreversible value is SILENT; a tool- or inference-sourced value (even in the current instance) routes to CONFIRM (T2).

Safety property (high level): an irreversible-and-carried belief always routes to CONFIRM, and an irreversible-and-(tool|inference) belief always routes to CONFIRM. An irreversible value therefore NEVER silently reuses historical state; the CONFIRM pre-fills the prior value to keep friction low. Both goals are met simultaneously.

Convergence (honest restatement): run #1 (current-instance user answer) is SILENT and proceeds, leaving the legitimate primary flow unbroken. Run #2 (all carried) yields SURFACED for reversible/policy (0 blocking questions) and CONFIRM for irreversible (one low-friction confirmation each). So the residual question count equals the number of irreversible decision slots, expressed as confirmations; reversible decisions converge to 0.

4. Adversarial cases (seeking FAILURES, not passes)

• A1 recipient/account change (the original failure): in run #2 the recipient is carried and irreversible, so it routes to CONFIRM and a stale account is caught. HOLDS.
• A2 tool-sourced current instance: a tool fetched a stale or poisoned value this run; the 2-D matrix would give SILENT. PUNCH-THROUGH (T2): current-instance does not imply trusted — only a user source warrants SILENT. Fix: add source class as a third dimension.
• A3 shipping-address change: safety depends on the address being classified irreversible; a misclassification routes to SURFACED and a stale shipment proceeds. PUNCH-THROUGH (T4): safety is a function of classification correctness. Fix: conservative default — ambiguous implies irreversible.
• A4 contract counterparty (composite): a counterparty is name + entity id + jurisdiction; if only the entity id changed and it is classified policy, it routes to SURFACED and leaks. PUNCH-THROUGH (T5): an irreversible decision is split across mixed-risk slots. Fix: propagate risk at the DECISION level — any slot feeding an irreversible action inherits irreversible.
• A5 consent change: a carried "consent = yes" routes to CONFIRM showing "yes, confirm?", but the scope/time it referred to changed. PUNCH-THROUGH (T3): a scalar value hides semantic staleness. Fix: irreversible beliefs need a world reference; the confirmation must surface what and when, not a bare yes/no.
• A6 authorization-scope change: a carried scope of "admin" where only "read" is needed; CONFIRM pre-fills "admin", inviting habitual over-grant. PUNCH-THROUGH (T6): pre-fill biases escalation. Fix: a sub-class of irreversible-and-monotonic routes to ASK (no pre-fill) — escalation must be actively re-added.
• T7 confirmation fatigue (cross-cutting): CONFIRM prevents silent reuse but not rubber-stamping. The runtime can guarantee "never silent", not "the human always reads". This is an irreducible residual and is reported honestly.

Verdict

The CORE design — instance-scoping plus the invariant that a carried irreversible value is never SILENT — STANDS: the original failure is closed and the convergence thesis is NOT overturned. But the naive 2-D (risk × provenance) matrix is PUNCHED THROUGH and must be upgraded with: (1) a source-class third dimension; (2) a conservative risk default (ambiguous implies irreversible); (3) decision-level risk propagation; (4) a world-reference semantic anchor; (5) no pre-fill for monotonic/escalation slots; and (6) an honest residual for confirmation fatigue.

Thesis ruling — NOT overturned, but the "many questions converge to zero" headline must be restated to its honest, safe form: reversible decisions converge to 0 blocking questions with experience; irreversible decisions converge to a low-friction CONFIRM (never a silent historical reuse), whose safety is bounded by classification correctness, a world reference, and anti-fatigue design. The old zero-question claim held only via unsafe silent irreversible reuse; the safe residual is 0 reversible plus k irreversible confirmations, with an irreducible human residual. Convergence is real, but narrower and more conditional than the zero-question framing implied — and that is the honest thesis.

Status: this is a paper design (no code, no implementation, no tests).