flaky: the ABLATION leg of `vitest-console-teardown-race.test.ts` reds a full-repo run — its instrument stops reproducing under load (1 red in 6 runs)

[os-steve]

Found while taking the machine-lane baseline for the QA wave (#9296, round #9298). Filed as a finding, not fixed — R0 lands no code.

Summary

examples/app-showcase/test/vitest-console-teardown-race.test.ts is the guard landed by 92a69d813 (#10293, #10374, PR #10605) for the vitest console-forwarding teardown race. It has two legs: a GUARDED leg (the fixture exits 0 under the app's real vitest config) and an ABLATION leg (the same fixture still reddens a green run under vitest defaults — i.e. the instrument still detects the thing it guards).

The ABLATION leg is the one that fails, and it fails by design-intent: it asserts its own instrument still reproduces the defect, and under a loaded full-repo run it does not.

This is the only test failure in the entire repo at this sha.

What failed

FAIL  test/vitest-console-teardown-race.test.ts > [#10293] vitest console-forwarding teardown race
      > ABLATION: the fixture still reddens a green run under vitest defaults

AssertionError: the instrument stopped reproducing: 8 attempts under vitest defaults
produced no "Closing rpc while "onUserConsoleLog" was pending". Either vitest changed its
console forwarding (check sendLog/rpcDone in its dist chunks) or the fixture stopped
leaking. Do NOT relax the guarded leg on the strength of this.: expected undefined to be defined

 ❯ test/vitest-console-teardown-race.test.ts (3 tests | 1 failed) 11284ms
   × ABLATION: the fixture still reddens a green run under vitest defaults  7386ms

 Test Files  1 failed | 25 passed (26)
      Tests  1 failed | 361 passed (362)

Note the assertion message's own instruction — "Do NOT relax the guarded leg on the strength of this" — which is exactly right and is why this is filed rather than touched.

It reproduced once in six runs — the load dependence is the finding

#	condition	result
1	pnpm exec turbo run test --continue --concurrency=2 (full repo, 30m39s)	RED — the failure above
2	pnpm -C examples/app-showcase exec vitest run test/vitest-console-teardown-race.test.ts	green, 3 passed (3), 4 s
3	same, repeat	green, 3 passed (3), 4 s
4	pnpm --filter @objectstack/example-showcase test (the file beside its 25 siblings)	green, 362 passed (362), 24 s
5	same, repeat	green, 362 passed (362), 23 s
6	the file alone under 8 synthetic CPU burners on 4 CPUs (load avg ~7)	green, 3 passed (3), 19 s

So: the file is green in isolation, green at package scope, and green under raw CPU starvation. It went red exactly once, inside a full-repo run. Run 6 is the informative control — plain CPU contention is not sufficient to trigger it, so the trigger is something more specific to the full-repo condition (many concurrent node processes and their memory/scheduling pressure changing the child vitest's rpc teardown timing), not simply "the box is busy".

The ablation spawns child vitest runs and looks for a race window; its reproduction probability is inherently non-deterministic, and under full-repo conditions it apparently drops below what 8 attempts (ABLATION_ATTEMPTS) will catch.

Why this matters more than a normal flake

The failure direction is inverted from an ordinary flaky test. This one goes red when the defect it guards fails to appear. That has two consequences:

1. It reds a run in which nothing is actually wrong, and the message reads like vitest changed under us — an expensive thing to investigate at the wrong moment.
2. If vitest ever genuinely fixes its console forwarding, this leg reds permanently and looks identical to today's load flake. The two are indistinguishable from the assertion alone.

Environment

• framework sha 1f6d04703df8f518c80d0fbaa8b19262068e405d (tip of main)
• .objectui-sha 190fbd01d0615e2e168faf9e08b8ad7844bc039d
• node v22.22.2, pnpm 10.31.0, linux x64, 4 CPU / 15 GB
• fresh worktree, single pnpm install, deps prebuilt, run serialized under the shared verify lock

Suggested direction (not implemented here)

Deliberately not proposing that the ablation be weakened — the guard's own comment forbids exactly that, and the leg is doing real work. Options worth a maintainer's judgement:

• raise ABLATION_ATTEMPTS, or retry the ablation with a bounded budget, so a low-probability window under load does not red the leg;
• give the ablation leg a distinct signal (skip-with-warning under detected load, rather than fail) so that "instrument did not fire this time" and "instrument is permanently dead" stop looking alike — that distinction is the actual gap;
• or pin the file to a serial/isolated lane so it never runs under full-repo pressure.

The first and second change what the guard asserts, so neither is taken here.

---

Generated by Claude Code

[claude]

Triage: lands in examples/app-showcase/test/vitest-console-teardown-race.test.ts; domain:devx (test-infrastructure guard; examples route to the subsystem they exercise). → pm:queue, Bug — a 1-in-6 red on full-repo runs is merge-queue flake tax, and the card's own assertion message forbids the cheap fix (do NOT relax the GUARDED leg). Fix shape is the dev's to measure: make the ABLATION leg robust under load (attempts budget, serialized execution, or an isolated lane) while keeping its instrument value — the guard must still detect the race it exists for. Size/model suggestion: S–M, opus (the judgment sits in not weakening the guard).

---

Generated by Claude Code

[os-steve]

Claim: PM dispatch (devx seat)

• Session: session_015ahemw8RcTgqtxrj15PEZx
• Branch: claude/issue-11939-ablation-leg-load-robustness

⭐ Second finding out of the QA wave's R0 baseline (#9298), after #11938. Both were filed as findings with nothing fixed, because R0 lands no code — and both are worth more than the round that produced them.

The card already identifies the real gap, and it is not "a flaky test"

If vitest ever genuinely fixes its console forwarding, this leg reds permanently and looks identical to today's load flake. The two are indistinguishable from the assertion alone.

That is the defect to close. "The instrument did not fire this time" and "the instrument is permanently dead" are different facts with different responses, and today they produce the same red with the same message. Raising the attempt budget alone would make the symptom rarer without separating them — so it is a partial fix at best, and the card says so.

⛔ The cheap fix is forbidden by the guard's own words

The assertion message ends: "Do NOT relax the guarded leg on the strength of this." Triage restates it. The ABLATION leg exists to prove the GUARDED leg still means something; weakening either to stop the noise destroys the reason the file exists.

Run 6 is the control that makes this dispatchable

The card ran the file alone under 8 synthetic CPU burners on 4 CPUs (load ~7) and it stayed green. So plain CPU starvation is not the trigger — something more specific to the full-repo condition is (many concurrent node processes, memory/scheduling pressure changing the child vitest's rpc teardown timing). Without that control the obvious fix would have been "make it tolerate load", aimed at the wrong mechanism.

⚠️ Note the sibling: #11958 (filed from #11938) measures that vitest's own worker pool scales with host cores, uncapped, in 40/41 packages. That is plausibly the same pressure this leg is sensitive to. Read it before designing — but ⛔ do not fold it in; it has its own card and its own measurement.

Tier: opus — triage suggested S–M/opus and I agree. Derived live, no path-derived mandate, so sonnet is the floor and this is a judgment call: "the judgment sits in not weakening the guard", and every obvious fix here trades instrument value for quiet.

---

Generated by Claude Code

[claude]

{
  "issue": 11939,
  "status": "done",
  "branch": "claude/issue-11939-ablation-leg-load-robustness",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/11990",
  "premise_still_valid": true,
  "summary": "Closed the conflation the card names, rather than the flake it presents as. Added a deterministic PREMISE probe that reads the installed vitest's dist chunks and asserts the five links of the mechanism the ablation's race walks (sendLog discarding the forwarding promise; rpcDone awaiting a snapshot; the cleanup sweep following the drain; the sweep rejecting pending calls; the rejection wording still yielding the grepped string, which TEARDOWN_ERROR is now derived from rather than duplicating). The two axes are graded together: premise intact + race silent is now a loud, countable SKIP; premise broken + race silent FAILS as 'the instrument is dead, and this is not the load flake'; premise broken + race fired FAILS as 'the probe is stale, not vitest'. Neither leg was relaxed and ABLATION_ATTEMPTS is deliberately unchanged at 8 — measurement prices a bigger budget out. Measurement also answered the card's Zone-2 question: the suppressor is CONCURRENT NODE PROCESSES, not CPU starvation (load held 2.8-3.4 of 4 CPUs with 15 GB free while reproduction fell from 100% to ~13%) and not fixture-root contention (suppression survives when every concurrent process gets its own copy of the root). That rules out the card's route 3 as well: the pressure is inter-package, so a lane inside app-showcase cannot isolate the file from it.",
  "tests": "All at head a551091f0 unless stated; heavy runs through scripts/pm/os-verify-lock.sh, verdicts quoted from each gate's own line, never a bare $?. MEASUREMENT (harness in scratchpad, not shipped; every count has its control in the same run): idle ablation config 20/20 reproduced, every child exit 1 — {\"arm\":\"A-idle-ablation\",\"reproduced\":20,\"attempts\":20,\"pattern\":\"RRRRRRRRRRRRRRRRRRRR\"}; idle app config 0/6, every child exit 0 — {\"arm\":\"A-idle-appconfig\",\"reproduced\":0,\"collectedOneFile\":6}; under 4 concurrent vitest processes 0/20 then 3/20 in the measured arms with load1 2.84-3.35 and freeMB ~15250, while those same concurrent processes reproduced 248/1913 (~13%) in the same windows; repeated with each concurrent process on its OWN copy of the fixture root, still suppressed (3/20), so shared-root contention is not the mechanism. TYPECHECK: pnpm --filter @objectstack/example-showcase typecheck EXIT=0 (after building the closure with pnpm --workspace-concurrency=2 --filter '@objectstack/example-showcase^...' build, BUILD_EXIT=0 — the first attempt reported 22 errors, all of them unbuilt-dependency TS2307 in packages/formula and plugin-email reached through tsconfig paths, none in the edited file). PIN FILE IDLE: 'Test Files 1 passed (1) / Tests 5 passed (5)', PIN_IDLE_EXIT=0. BEHAVIOUR CHANGE DEMONSTRATED UNDER THE TRIGGERING CONDITION: pin file run while 4 concurrent vitest processes churn — PIN_UNDER_LOAD_EXIT=0, 'Tests 4 passed | 1 skipped (5)', the ABLATION-DEGRADED [#11939] line printed, and the four load workers reproduced 31/172, 25/173, 19/173, 20/173 during those same 8 attempts (the in-run positive control that the instrument was alive while the pin's own attempts missed). Before this change that run is the red. ABLATION / REVERSE VERIFICATION of the GUARDED leg: removed 'disableConsoleIntercept: true,' from examples/app-showcase/vitest.config.ts. NO BUILD OR DIST IS INVOLVED — vitest reads that config as source when the pin spawns the child — so the on-disk evidence is the anchored grep pair, not an exit code: BEFORE anchor_count=1, AFTER anchor_count=0, git numstat 1/2 on that file. Mutated leg: MUTATED_LEG_EXIT=1, 'x GUARDED: the same fixture exits 0 under this app's real vitest config', 'AssertionError: 4/4 runs hit the teardown race under the app's own config', 'Tests 1 failed | 4 passed (5)'. Restore leg via trap: RESTORED anchor_count=1 and re-measured green — RESTORE_LEG_EXIT=0, 'Tests 5 passed (5)'. The probe additionally carries its own always-run positive control in-tree (PREMISE CONTROL): six minimal mutations of the real chunk text, each asserted to have edited something before grading and asserted to break exactly one named marker. GATES, derived live with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (1 path vs merge base 0cbe79a36), whole union re-run on the final head a551091f0: check:nul-bytes 'OK (scanned 6656 text file(s) ... no raw ASCII control bytes)'; check:cross-package-test-inputs and scripts/check-cross-package-test-inputs.mjs 'OK: 16 package(s) read outside themselves, all declared'; check:examples-live-imports 'examples/** live-import inventory OK'; check:published-files '69 publishable package(s) ... declare a files whitelist'; check:test-source-alias 'OK — 72 packages with tests scanned'; check:type-source-resolution 'OK — 93 tsc program(s) across 77 packages scanned'; check:query-options-erasure 'ratchet holds ... none new'; check:type-check-coverage 'OK — 65/78 workspace packages type-checked'; check:engine-double-contract, check:where-matcher, scripts/check-ci-filter-parity.mjs all EXIT=0 with no new entries; check:type-check-debt refused first ('--re-measure cannot run: 1 workspace dependenc(ies) ... have no built type entry point on disk -- @objectstack/service-knowledge' — NOT MEASURED, not 'not applicable'), so the closure was built (turbo run build --filter='./packages/*' --filter='./packages/*/*', BUILD_ALL_EXIT=0) and it then reported 'OK — 32 ledger entr(ies) re-measured in 220.1s, 1898 raw tsc error(s) total, none above its recorded number'. LINT: full repo, no narrowing — pnpm lint (eslint . --no-inline-config) LINT_FULL_EXIT=0 with empty output. CI is not waited on; gate status at report time is what is written here.",
  "open_questions": [
    {
      "question": "Now that a silent race degrades to a countable SKIP instead of a red, should ABLATION_ATTEMPTS be raised anyway to lower how often the leg degrades?",
      "options": [
        "A — leave it at 8 (what shipped): a miss is no longer a failure, extra attempts only lower the skip rate and are paid on every loaded run, which is the run that can least afford them",
        "B — raise it to ~50: at the measured ~13% per attempt that pushes P(miss) to ~1e-3, at ~45 s per degraded run under load",
        "C — make it adaptive (keep trying until a wall-clock budget expires)"
      ],
      "recommendation": "A, because the rate is environment-dependent, not a constant: the sequentially measured arms came in at 3/40 (7.5%), below the 13% the concurrent processes saw, so any fixed budget sized from today's number is sized from a number that moves. The premise probe already answers 'is the instrument alive' deterministically and for free, which is the question a bigger budget was being asked to answer expensively."
    }
  ],
  "out_of_scope_findings": []
}

---

Generated by Claude Code

---

Generated by Claude Code