Is the count right?

Evals

The counter is judged against labelled clips with a rule written down before any run, every human decision feeds back in, and every agent in the fleet has to change a result to stay.

What the filmed clips showed

1 · The registered gate

INCOMPLETE

Not run as registered: g2, g3, g4, g5, g6, g7, g8 not filmed. g0 scored correctly · g1 scored correctly (merged_mask hold)

2 · Same frames, two answers

Muse Spark alone 4/4 · pipeline 3/4

n=4 filmed clips · wrong refunds: 0 alone, 0 pipeline · false holds: 0 alone, 1 pipeline

The pipeline arm was not run the same way on every clip: on g0 and g1 it ran without the open-weight auditor and Glimmer. g1 shows the same camera the auditor flagged on d2, so the pipeline's score on g1 is flattering.

d2, the complete kit: all three counted → HELD, a false hold.

Loss misread: the prop is a real compact film camera, not a toy (it stands in for the kit's interchangeable-lens body); a false stand-in flag; it held a complete kit (a false hold under the registered scoring)

Loss misread: the label reads BW-CB-55120

Loss missed: the lens prop is a camera-lens-shaped mug; the stand-in check passed it

Per-clip tables, the synthetic clips and the stress sweep follow below.

Scoring the clip set…

Same frames, two answers: Muse Spark alone vs ReturnWitness

RECORDED LIVE RUN · Meta (SAM 3.1, Spark, Wonder) + open-weight auditor (Baseten) + Glimmer (Modal)

Would one strong model, asked “is this kit complete? refund or hold?”, do as well as the pipeline? Each filmed return goes to both: Muse Spark 1.3 alone with the four key frames, the packing slip and the policy, and the pipeline (SAM 3.1 on every frame, the tray counter, the Spark reader, the open-weight auditor on Baseten, the Glimmer label reader on Modal, the Wonder agent) on the same clip. The rule and the prompt were written down before either ran (evals/AB-SAME-INPUT-PREREG.md, prompt sha256 db19d8212987…); every loss stays on this page.

On n=4 filmed clips: Muse Spark alone 4/4 correct (0 wrong refunds, 0 false holds); the pipeline 3/4 (0 wrong refunds, 1 false hold).

The pipeline arm was not run the same way on every clip: on the gate clips (g0, g1) it replayed the gate run's SAM output and ran without the open-weight auditor and Glimmer, which read the crops of a live bench inspection only; on d1 and d2 both were called (an answer can still be lost: Glimmer timed out on d2). g1 shows the same camera the auditor flagged on d2, so the pipeline's score on g1 is flattering.

Arm	False holds	Correct
Muse Spark 1.3 alone	0	4/4
ReturnWitness pipeline	1	3/4
Spark alone, without g1	0	3/3
Pipeline, without g1	1	2/3

Clip	What was in the box	Spark alone	Pipeline
g0	item missing: must hold	correct hold hold · missing: camera_body, lens, battery · “No packing-slip items were observed on the inspection tray in any key frame 3.2s-15.2s. Box remains on left unopened, tray on right remains empty throughout, so presence of camera_body, lens, and battery cannot be confirmed and serials cannot be verified.”	correct hold Held · not_observed
g1 calibration clip: the pipeline's item sizes were measured on it	complete	correct refund refund · “All 3 packing-slip items seen on inspection tray by final frame 36.4s: camera_body with tag BW-CB-55120 matching slip, lens with tag BW-LN-80311 matching slip, and battery+charger unit. Box contains only packing paper. Kit is complete per policy 4.2.”	correct refund AwaitingApproval
d1	item missing: must hold	correct hold hold · missing: lens · “Packing slip lists 3 items. Camera body SN BW-CB-55107 observed in box at 4.3s and on tray 9.6-20.2s with matching serial tag. Battery+charger unit (black dual charger) observed in box at 4.3s and on tray 14.9-20.2s. Zoom lens SN BW-LN-80298 never visible in box or on inspection tray; final box at 20.2s contains only packing paper. Per policy 4.2-4.3 kit is incomplete, refund held for review.”	correct hold Held · count_match, stand_in_suspected, model_disagrees, not_observed the camera is a light-green toy truck · misread: the prop is a real compact film camera, not a toy (it stands in for the kit's interchangeable-lens body); a false stand-in flag
d2	complete	correct refund refund · “All 3 packing-slip items observed on inspection tray: camera_body with serial BW-CB-55120, lens with serial BW-LN-80311, and battery + charger unit. Serials match slip per 4.4, kit complete per 4.2.”	false hold Held · count_match, stand_in_suspected the camera is a toy car · misread: the prop is a real compact film camera, not a toy (it stands in for the kit's interchangeable-lens body); a false stand-in flag; it held a complete kit (a false hold under the registered scoring) label BM-CB-55120 · misread: the label reads BW-CB-55120 clearly legible on the lens, no stand-in · missed: the lens prop is a camera-lens-shaped mug; the stand-in check passed it

Not filmed, so not scored: g2, g3, g4, g5, g6, g7, g8, d3. 4 returns can show the two arms differ on these boxes; they can't give a rate.

“Refund” is each arm's answer before a person: in ReturnWitness no refund moves without a typed-name, single-use signed approval. Spark alone $0.05, pipeline on the gate clips $0.12.

Scoring the clip set…

Safety, and what it costs

DETERMINISTIC · synthetic stress sweep, app code

When the counter isn't sure, the box goes to a person instead of being refunded. That caution is what keeps silent count errors at zero in the sweep below, and its price is paid by honest customers whose complete kit waits for a review. Both numbers sit side by side.

silent count errors in 3,789 runs: no refund drafted on a wrong count

263

false holds: 8.9% of 2,947 complete-kit runs wait for a person

117

of them from merged masks (high)

Synthetic segmentation with mask-level perturbations (scripts/stress-sweep.ts, the same rule as the gate). Silent errors count items; false holds count runs. The filmed clips' numbers are the recorded A/B above, not this sweep.

Label reader learning

The label reader (Muse Glimmer) can be fine-tuned, but a fine-tune is only worth using if it reads serials better on labels it never saw. Its training set is the bench label set · truth from the printed label design: every row's truth is the label that was printed or rendered, not a model's guess. The four serials on the real printed sheet are always held out, so the test runs on the actual labels.

Rows	Train	Held out
sheet · printed sheet crops	0	40
render · re-rendered label design, new seeded serials	150	42
bench · filmed bench clips	0	0
reviewer corrections · people correcting live label reads	0 so far
bench label set total	150	82

Distinct serials: 46 in training, 18 held out (none in both). Held out from the printed sheet: BW-CB-55120, BW-LN-80311, BW-CB-55107, BW-LN-80298.
Filmed bench clips: no filmed clip with a SAM recording yet; rows are added when d1–d3 land.
Floors (at least 50 training rows, 20 held out): met.
Last base-vs-adapter test: kept the base (exact-serial gain is +3.7 points / +3 rows (needs at least +5 points and +2 rows)): base 77/82 exact serials, 1 false clears; adapter 80/82, 0 false clears
The real printed labels: 32 held-out views of the 4 real printed serials. Readable (26): base 26/26, adapter 25/26 (its miss is an abstention, not a wrong read). Covered by a finger or paper (6): the base trusted 2 reads anyway (1 a false clear), the adapter abstained on 6/6.
Switch rule v1 (used for this test): The fine-tuned adapter is used only if, on at least 20 held-out rows (with at least 50 trained on), it beats the base on exact-serial match by at least 5 points and 2 rows, with no more false clears; otherwise the base stays.
Why v1 kept the base although the adapter read more: with the base at 77/82, +5 points needs at least 81.1 of 82, so only a perfect 82/82 could pass; the verdict stands, the rule is replaced by v2 for future runs.
Switch rule v2: on at least 20 held-out rows the adapter has never been scored on, exact-serial errors down by at least 40% (and strictly fewer) than the base, false clears strictly fewer (or zero for both), and no wrong trusted read on any readable real printed serial. (pre-registered before any bench clip existed; pre-registered, not yet run: scored only on rows not in v1.scoredRowIds (the d1–d3 bench crops once the bench clips are filmed, plus freshly seeded rows))
Counts from the summary committed with the generator.

Counting the label reader's training rows…

Ablation: what each agent changes, by effect

DETERMINISTIC · app code, no model

One run of the seeded shift (24 boxes, 34 names, the per-box inspectors one code path, selective dispatch) freezes every model role's output: 61 runs, 2 of them recorded live Meta answers. Each role is then removed and the shift replayed on those frozen outputs, so no other model role is re-run (0 model runs during the removals). Counting, serial matching, the merge, the settlement and routing are app code and do re-run. Each column counts the boxes where that kind of result changed.

Role removed	Hold	Amount	Claim	Route	Artifact	Fees	Δ proposed	Δ claims	Effect	Own runs
label-reader model · live: muse-spark-1.3	1	1	·	1	1	1	+$340	—	decision	1 $0.005 est.
damage-reader model · live: muse-spark-1.3	1	1	·	1	·	1	+$340	—	decision	24 $0.231 est.
voice-note-reader model · live: muse-spark-1.3	1	1	·	1	2	1	+$340	—	decision	3 $0.008 est.
serial-matcher code	1	1	·	1	1	1	+$340	—	decision	24
carrier-claim-agent model · live: muse-wonder-1.0	·	·	1	1	2	·	—	−$199	decision	2 $0.048 est.
policy-agent model · live: muse-wonder-1.0	1	1	·	1	2	1	+$340	—	decision	6 $0.144 est.
restock-grader model · live: muse-spark-1.3	·	·	·	·	19	·	—	—	artifact only	19 $0.139 est.
customer-message-drafter model · live: muse-wonder-1.0	·	·	·	·	6	·	—	—	artifact only	6 $0.132 est.
cost-agent code	·	·	·	·	·	24	—	—	fees only	24
coordinator code	5	5	1	24	24	24	+$1,700	−$199	decision	24
inspector code · 24 agents	24	19	·	18	19	24	−$6,351	—	decision	24

Hold, amount, claim and route are decision effects: they change what a person is asked to approve or where the box goes. Artifact (restock grades, the customer note, work items) and fees changes are reported, but a draft disappearing is not money or loss prevented, and a role's own model cost disappearing is never counted as an effect (it is the last column).
Δ proposed: what the settlement would propose without the role, minus what it proposes with it, summed over the shift. Positive means money a person would have been asked to release that this role held back. Removing the inspectors holds every line, so nothing is proposed.
Frozen outputs measure each role's direct effect. Without the damage reader the frozen carrier claim and restock grades stay as they were; re-running those roles without it would need new model calls, so that knock-on effect is not counted here.
The roster rule: a role whose removal changes nothing is cut. Every shipped role changes something. restock-grader, customer-message-drafter, cost-agent change no decision, but removing each removes work a person uses (a restock grade, the customer note, the fees line), so they stay, labelled “artifact only” or “fees only”.
Each inspector removed on its own: 24 of 24 change only their own box.
Cut: fraud-pattern-agent. Added back on the frozen outputs, it changes 2 boxes, 0 with a decision effect (artifact only). No two boxes in this shift come from one address or show a copied serial. The only cross-box finding is the two demo orders (ORD-58227, ORD-58241) listing the same serials, because the demo films one labelled unit; that is an inventory record conflict, a work item, not a hold. Added back, it changes no hold, amount, claim or route; its only effect is that work item on those two boxes.

[Seeded shift fixture · synthetic] · scenarios are a seeded shift, not real returns · npx tsx scripts/fleet-ablation.ts prints the same table