Testing

Overview

The AI-DLC test suite is entirely TypeScript — every test is a t*.test.ts file run under bun, with zero shell (.sh) test files. This is the platform-invariance guarantee by construction: the same files run identically on macOS, Linux, and native Windows.

The suite is organized into four levels — smoke, unit, integration, e2e — one directory each under tests/. The four levels map onto the classic three-layer test pyramid that balances speed vs. thoroughness:

            /\
           /  \    ACCEPTANCE — full workflows, artifact + experience verification
          / L3 \   Level: e2e  ·  When: local --release/--all; pre-release confidence
         /------\
        /        \
       /   L2     \  STAGE — individual stages with stub input, verify artifacts
      /------------\ Level: integration  ·  When: local default/--ci; PR gates
     /              \
    /      L1        \  PROTOCOL — contracts, structure, cross-references
   /------------------\ Levels: smoke + unit  ·  When: local changes and PR CI

The --ci profile and the no-flag default both run smoke + unit + integration (so the integration level rides along on every local bun tests/run-tests.ts); --release / --all adds e2e. The pyramid above shows where each level sits conceptually — the profile flags below are how you actually select them.

Test boundaries and ownership

The directory is an execution selector, not a promise about cost. In particular, unit currently includes CLI, installer, native-build and complete workflow tests. --no-llm closes model gates; it does not make a test an isolated unit test. Classify a test by the boundary its assertions exercise:

Keep input variations in the cheapest layer that faithfully exercises the contract. Keep process, concurrency and platform checks when that boundary is the behavior being tested. A live journey does not replace an exact failure injection, and replaying every unit input through every harness is not required to prove adapter wiring.

Before adding a regression, find the suite that owns the same behavior. Extend it when the setup and boundary match. A second covers: claim does not require a second execution of identical inputs and assertions: one case can cover the command, state changes and audit event together. Compare fixtures, hooks, parameter tables and platform implementations before removing a repeated body.

The onboarding contract belongs to t-tui-onboarding; t-e2e-driver-isolation owns the IDE process/profile isolation cases. t47-failure-injection owns the four CLI failures and their ERROR_LOGGED evidence. Their distinct assertions remain in those suites rather than being replayed in a second file.

Release fixtures read the generated copy/native trees and construct independent archives, applying the requested version to each archive's stamp in memory. They do not need another disk copy of every distribution. Each call reads the current source and owns its output; later source changes are visible, and damaging one fixture cannot change another fixture or the generated trees.

The four selectors and current CI destinations remain in place while the broader inventory is reorganized in issue #2774. Moving a file must preserve its discovery, coverage claims, platform selection, shard affinity and required pre-merge execution. Measure setup time, test time and runner use separately; a directory move or a lower case count alone is not a performance result.

The shared policy in tests/harness/test-budget.ts treats timeouts as failure backstops. Cold imports, antivirus, process creation and provider scheduling vary widely between runners; a fast run does not define a suitable timeout. Unspecified deterministic cases get ten minutes on every OS. Choose a shared profile for a whole fixture and remove smaller operational overrides:

These are ceilings: successful operations return immediately. A timeout that a test deliberately exercises remains explicit and short. Ownership, exit status, recursion, human-boundary and other correctness assertions stay enforced. Elapsed wall time is not a substitute for those assertions; use direct state or an injected clock when the test is proving behavior rather than performance.

Production tools and standalone binary gates share operational defaults in core/tools/aidlc-runtime-budget.ts; see runtime and native hook budgets. Explicit caller budgets and intentional responsiveness contracts remain distinct from the default execution backstops.

Budget review includes nested subprocesses, setup, polling deadlines, cleanup and CI limits. Increasing only Bun's case ceiling cannot fix a shorter child cap. Driver operations use the actual remaining parent budget. Sequential operations share that budget; do not reserve every later operation's maximum allowance. The live-case helper supplies a generous Bun ceiling around the work, setup and cleanup phases, while the file deadline remains authoritative. It does not promise every phase its maximum after earlier work has consumed the file time.

Infrastructure uses the same shared policy:

These are maximum operation allowances. File cleanup reserves up to five minutes once, and the parent deadline clips each operation's allowance. remainingCleanupTimeoutMs spends the original hard deadline, including its reserved tail, without subtracting the work reserve again. At hard expiry it allows only a minimal immediate retirement attempt. The terminal client shares one shutdown deadline across its RPC and daemon-retirement wait. Polling returns as soon as the required evidence is available; missing identity or retirement evidence still fails.

Every dispatched file has an independent supervisor deadline, including ordinary integration/SDK files. --file-timeout N caps it in seconds (two hours by default outside isolated E2E). --run-timeout N shares one work deadline across setup and all selected files. Exhausted budgets fail visibly; they do not skip required assertions. The run retains one cleanup cutoff, so finishing a file's cleanup early cannot admit more work into that reserved time. Drivers leave up to five minutes of the file envelope for teardown, while the runner still retires owned processes and records a failure if the work does not cooperate. These flags bound dispatched work; coordinator retirement, fixture retention and report publication follow it. Their native cleanup checks remain bounded, and CI's enclosing step/job limits provide the final wall-clock backstop for collection. Uncertain process cleanup retains its fixtures. Logs record the resolved allowance, absolute deadline and cleanup reserve.

Distribution coverage is split by contract:

• t145-packaging-parity.test.ts proves that copy, native, and plugin projections are deterministic across two independent clean packager runs.
• t238-build-binaries.test.ts compiles and probes the standalone binary closure, including native routes, hooks/adapters, project mutation, and the final installed layout with no bun on PATH.
• t242-plugin-state.test.ts proves host inventory normalization, composition hashes, transactional sync rollback, and ownership-safe prune.
• t243-install-mechanism.test.ts covers archive rejection, the shared transaction engine and recovery paths, project init/refresh ownership, checksums, lifecycle, pins, offline packages, and copy/native projection separation.
• t244-install-management.test.ts covers machine configuration, update discovery, installer harness selection, Windows lifecycle surfaces, completions, and release-workflow candidate continuity.
• t330-release-channel-grammar.test.ts pins the closed stable and preview version-id grammar, the installer and lifecycle literals of it, and a preview install that runs through the launcher shim.
• t331-preview-channel-lifecycle.test.ts covers config --channel, channel-aware update and update --check, API failure as unavailable, switching back to stable, preview retention, and preview pins.
• t332-preview-release-pipeline.test.ts covers the annotated-tag prerelease publication, the preview planner and notes, and the plan record. It checks multiple changed-source previews on one UTC day, including a later manual run after a scheduled publication. Unchanged and overtaken sources skip; drafts and orphan tags permit retry planning with unoccupied ids. Workflow assertions cover a run planning and publishing the commit it started on after main advances, isolation of stable tags from scheduled/manual previews, shared release-preview concurrency, contract gate ancestry, channel-specific provenance signers, build stamping, and a failing Full Suite that still builds the preview without failing the preview workflow.
• t-ci-preview-test-report.test.ts covers the preview's Full Suite report: failing files and cases from each artifact's own run (never the runner's fixture runs), failed jobs, inert markup, the report size budget, and the answers corpus section (read from the live artifacts' last attempt, on a pass and a failure, kept out of the published notes).
• t-live-answers-nightly.test.ts covers the answers corpus's nightly run: the date rotation, the switch that opens it in the Claude SDK live job, a miss reported but not owed while a drive that throws still fails, and the summary the nightly report reads back.
• t-ci-full-suite-evidence.test.ts covers the optional Full Suite evidence inspector and exact commit/run matching. t-release-promotion.test.ts uses real Git histories to cover preview selection, exact source preservation, metadata, control snapshots, retries and main metadata synchronization.

The test runner regenerates all projections under a process lock before test discovery, so a fresh clone has no dependency on pre-existing dist/ bytes. CI also runs an explicit package step before each job that consumes generated trees. Binary and release packagers perform their own regeneration and determinism checks before reading dist-release/.

Release CI adds native smoke on Linux, macOS, and Windows, builds the seven target artifacts on their native runner architectures, and executes both musl probes in disposable Alpine containers after installing the documented libgcc and libstdc++ runtime prerequisites shared by Bun and Node.js. The musl matrix uses fail-fast: false so both architectures report before CI stages one checksum-verified candidate. Unix and Windows lifecycle journeys consume those bytes without signing permissions. The workflow then attests them and uploads one workflow artifact. release rechecks the tag and checksums, creates the GitHub Release in the source repository with GITHUB_TOKEN, and compares the uploaded asset inventory with the candidate. Publishing never rebuilds or repackages the candidate.

tests/harness/release-fixture.ts builds deterministic release directories from the generated projection manifests and can serve them locally with redirect, delay, truncation, captive-portal, oversized-metadata, and missing-asset faults. Run bun tests/harness/release-fixture.ts --output <dir> to author a fixture. The normal suite stays offline; set AIDLC_RELEASE_CONTRACT_LIVE=1 when running t243 to opt into the public release metadata and checksum contract check.

The deterministic e2e slice (bash tests/run-tests.sh --debug -P 8 --e2e --filter "^t[0-9]") runs in CI with --no-llm, while local development loops default to smoke + unit + integration. Branches that touch merge, worktree, or swarm paths should also run this slice locally before review rounds, because mode-boundary regressions between ordinary-Bolt and swarm execution are invisible to the default tier.

Filename convention. A test's filename is t<NN>[-description].test.ts — just the level directory it lives in and an optional human description. There is no mechanism segment in the name: a test's mechanism (whether it spawns a CLI, drives the SDK, or renders a live TUI) is a derived set computed from the drivers its body actually calls, not declared in the filename. The machine-checked index of what each test covers is built fresh from the covers: headers on disk by tests/gen-coverage-registry.ts, never kept in a hand-maintained table here or committed. See Test Registry below.

Markdown adapter coverage

markdownBlocks in core/tools/aidlc-lib.ts is the single block interpreter for visibility, containers, definitions, and claim splitting; visibleMarkdownLines projects it. It is backed by the built-in Bun.markdown renderer. Tests defend consumer outcomes rather than maintaining a second CommonMark oracle. The separate Bun.markdown.render review-authority path keeps its existing security coverage.

• t341-markdown-blocks.test.ts covers authored CommonMark/GFM block-boundary examples, exact inline code and HTML span columns, the four Bun.markdown deviations the adapter corrects (an empty task item, a table running into a heading or fence, a fence leaving its container, a tag indented under a paragraph), probes that must be dropped because they would change the rendering, lines without letters or digits, entity-spelled probe markers, the render budget, the digest's fail-closed question boundary, and control characters that must not forge the rendered tree.
• t343-raw-html-consumer-contracts.test.ts checks summary digests and answers, Change Control sections, and Plan Approval selection/re-baselining inside and outside raw HTML. t344-visible-markdown-goldens.test.ts pins unaffected visibility and digest bytes; t345-receipt-scope-migration.test.ts covers v1/v2 receipt compatibility and parser-upgrade recovery.
• t247-claim-sources-sensor.test.ts defends both visibility safety directions: hidden text cannot ground a claim, and hidden fence-looking text cannot erase a real reference definition or shorten a confirmed assumption. Text after an HTML comment, processing instruction, declaration, CDATA section, or closing raw tag on the same line is a claim.

Layer 1: Protocol (every change, no LLM)

Verifies the orchestrator's structural correctness without invoking the LLM. If these pass, the protocol is internally consistent — stages reference valid files, inputs/outputs chain correctly, routing tables match stage files.

Levels: smoke, unit, integration

What it tests: - File existence, permissions, naming conventions (smoke) - All 17 hook sources through copy/native dispatch, stage frontmatter, knowledge inventory (unit) - Scope-stage mapping, graph consistency, stage I/O contract chains, protocol compliance (integration) - Stage output-to-step validation: all declared outputs referenced in instruction steps (integration, deterministic via the aidlc-validate.ts CLI tool) - Scope runs: every shipped scope driven from the person's first request to done (integration, tests/integration/t-scope-run-*; see below) - Scope-host runs: one classic journey on every harness tree (Claude, Codex, Copilot CLI, Kiro CLI, Kiro IDE, Cursor, opencode), through each tree's own hooks (integration, tests/integration/t-scope-host-*; see below) - Guard matrix: what the person meets when files change under approved work, per Guard Policy, review cap and plan approval (integration, tests/integration/t-guard-matrix-*; see below)

Run: bun tests/run-tests.ts --no-llm. The default profile includes the integration level, so the run needs the LLM and opens the live-model gate whenever the claude CLI is on PATH. --no-llm closes that gate and keeps this layer LLM-free; the deterministic tests still run. bash tests/run-tests.sh is a compatibility wrapper for existing POSIX commands. The scope runs, the scope-host runs and the guard matrix run only when a --filter selects them, or with --release/--all (see the CLI reference).

Scope runs

tests/harness/scope-run.ts drives a shipped scope through the real engine with no model. A scripted stand-in plays the agent: it runs the engine's commands as the Claude skill does and fires the hooks Claude Code would fire. A separate person script is the only part that answers a question or a gate, through the human-turn hook, and every engine call runs under the production guard profile. Each t-scope-run-<scope>.test.ts file checks state, audit records, directives and files, never prose:

• the stages that run and the ones skipped match the stage files' scopes: lists;
• every decision recorded as the person's is backed by a turn the person script sent (tests/harness/person-turns.ts); when one is not, the turn ledger and each intent's audit trail and state are kept under unbacked-decisions-<id>/ in the test's log folder, so the order of events survives the fixture's removal;
• the scope file's switches show in state and directives;
• the run ends done, with every output on disk.

t-scope-run-new-scopes.test.ts drives any shipped scope that has no file of its own, so a new scope is covered the day it ships. t-scope-run-moves-* cover the person's own words, settings typed with new work, a switch mid-run, and a stop for the day then a resume. A stage with a new kind of step fails its run until the stand-in learns that step. A known engine block the runs exempt is listed in KNOWN_STOP_BLOCKS with a test.todo named after it. CI runs the scope runs as their own integration job: Linux on pull requests and in the merge queue, and Linux, macOS and Windows in the nightly Full Suite (its scope_runs job) and in a full_verification Full Suite.

Scope runs on every harness tree

tests/harness/scope-hosts.ts runs the same stand-in and person script on another tool's tree, through that tool's own hooks. Each host installs the tree as it ships, gives the agent's shell what that tool gives it (Codex's CODEX_THREAD_ID, Kiro's KIRO_SESSION_ID), and at every moment (a chat starts, the person types or picks, a shell or file tool runs, a helper is sent, the turn ends) runs each command the tree's wiring registers, fed that tool's payload: .codex/hooks.json, .github/hooks/aidlc.json, Kiro CLI's agent JSON (.kiro/agents/<agent>.json, glob matchers over a tool's name and its alias; a helper runs its own agent's hooks under a session of its own) and Kiro IDE's hook cards (.kiro/hooks/*.json, regex matchers in file-name order; no card runs after a write or a command, so the next card does that work, as in Kiro), Cursor's .cursor/hooks.json (no matchers; every answer is JSON at exit 0, and the turn end can only send a follow-up message, which the journey reads as a blocked hand-off) and opencode's plugin (loaded into one opencode process, a worker thread the host waits on; a refusal is the hook's thrown error). tests/unit/t-scope-hosts.test.ts pins those payloads against the ones recorded from each tool (tests/fixtures/<tool>-hook-payloads). Each tool's answer is read in its own form: Codex and Kiro refuse with exit 2, Copilot with a deny decision, Copilot runs the command its hook rewrote, and Kiro's person-turn hook may run the turn's first next itself, so the agent acts on that reply. Every tool call fires the hooks, so a refusal stops the run, and a hook command that fails is a problem. A host can also run on a project an installed release set up (ScopeHost.launcher): the agent's commands and the wiring's hooks then run through the person's aidlc, and the stand-in acts on the aidlc engine ... commands the installed engine names. The release's update check uses that.

Each t-scope-host-<harness>.test.ts (claude, codex, copilot, kiro, kiro-ide, cursor, opencode) drives one journey: classic with its design stages skipped and two Units built one at a time, a checkpoint each. After the first Unit's checkpoint the person stops for the day, picks the work up from a fresh clone (the folder left with only what a clone holds, the first clone's audit shard sorting last), and carries on in a second chat with /aidlc --resume. The journey checks what the scope runs check, plus two audit shards, two chats, each Unit's checkpoint asked once, and no detour the person should not meet (such as being told to resume work they just picked up). CI runs them as their own integration job (scope-hosts): Linux on pull requests and in the merge queue, and Linux, macOS and Windows in the nightly Full Suite (its scope_runs job) and in a full_verification Full Suite.

Guard matrix

tests/harness/guard-matrix.ts uses the same stand-in to drive classic with two Units (Requirements, Units Generation, Code Generation, Build and Test) under each Guard Policy (off, relaxed, strict), review cap and plan approval setting. Each t-guard-matrix-<change>.test.ts file makes one change at the point a person makes it: a later Unit edits an approved Unit's file, a hand edit, a plan edit while its approval waits, a pull mid-stage, a revert, a second review the person asks for, an engine update between Units, the person's own switch to off, a composed scope and classic as it ships. The document changes walk Functional Design too: while Unit 2 builds, Unit 1's approved functional design or code plan is edited (by Unit 2's agent or by the person), or the design is deleted. Their -waves files run Functional Design stage by stage, then the person asks for one Unit at a time. Every refusal the run meets is followed: the person picks a remedy the engine offers, or the agent runs the command it names. Each case checks:

• under Guard Policy off, nothing refuses the person or asks them again (for a changed document, the same under relaxed; a deleted one may cost the one stop that makes it again, and it must be made again);
• with the policy on, the person is asked at most once about the change;
• no refusal comes back after its step was taken (a deadlock);
• every decision recorded as the person's is backed by a turn they sent, and the run ends done;
• the record keeps at most one CHANGE_ACCEPTED row and one line for the person per changed Unit, naming what changed, and none when nothing changed.

A case the engine blocks today is a test.todo named after the block, with the real check as its body, so the fix turns it on. CI runs the guard matrix where it runs the scope runs, in two duration-balanced jobs. Both select ^t-guard-matrix-; --shard 1/2 and --shard 2/2 partition that inventory without overlaps. The longest change matrices separate agent edits from hand edits, and unit-by-unit deletion from stage-by-stage deletion, into smaller files. Every scenario still starts a fresh project and runs its complete workflow with the same person-experience and record checks. Each split file starts with half its former file's duration as an estimate; the normal weight refresh replaces it with observed timings.

Layer 2: Stage (CI push, LLM, minutes)

Runs individual stages in isolation with known workspace + state fixtures. Verifies each stage produces correct artifacts when given deterministic input.

Levels: integration

What it tests: - Preflight health gate: Claude CLI on PATH, AWS credentials valid, Claude responds (exit 0), response non-empty (preflight) - CLI tool utility handlers: intent-create, --doctor, --status, --stage, --phase (integration) - Individual stages with greenfield/brownfield stubs, artifact verification (integration)

Run: bun tests/run-tests.ts --ci

Layer 3: Acceptance (release, LLM, hours)

Runs full workflows and verifies the experience: beyond state transitions, it checks artifact content, cross-stage coherence, and domain correctness.

Levels: e2e

What it tests: - Full bugfix lifecycle with brownfield stub + artifact assertions - Full POC lifecycle with greenfield stub + artifact assertions - State progression, scope routing, audit completeness, jump mechanics - LLM semantic review of stage instruction quality (clarity, logical flow, ambiguity detection)

Run: bun tests/run-tests.ts --release

Cross-Platform Coverage

The test suite runs on macOS, Linux, and Windows through the native Bun runner:

bun tests/run-tests.ts [--ci | --all --debug -P 8]

bash tests/run-tests.sh ... remains as a POSIX compatibility wrapper and delegates to the same TypeScript runner. Repository tests and copy-channel hooks/tools require bun; native release projections invoke hooks/tools through the installed aidlc binary. Bash is not the primary runner substrate.

Terminal sessions default to native bun on all three platforms; tmux is an explicit alternative on Linux/macOS. On macOS, containment uses start-time-checked process identities and an inherited AIDLC_TUI_CONTAINMENT token to find detached double-fork descendants after they reparent to launchd. A descendant that both escapes ancestry and execs with a scrubbed environment is undetectable on macOS; Linux subreaper containment and Windows Job Objects do not have this limitation.

Portability constraints baked into the suite:

• Paths: createTestProject in tests/harness/fixtures.ts normalizes temporary project paths so they round-trip cleanly through JSON and native bun.
• In-place edits: Prefer TypeScript file writes in tests. If a shell helper is unavoidable, avoid BSD/GNU-specific sed -i forms.
• grep -qiF: Git Bash has a known bug combining -i and -F. Use -i alone if your pattern has no regex metacharacters. Tests hit this in t16 before it was fixed.
• tar archives: macOS tar injects ._* AppleDouble sidecar files by default. When bundling source for cross-platform test runs, use COPYFILE_DISABLE=1 tar … or git archive.
• LLM timing on Windows: Bedrock calls from Windows EC2 can be meaningfully slower than from macOS (first-call cold start, MSYS process fork overhead). SDK/tui tests should assert on driver result surfaces and let the runner's preflight/per-file Claude gate separate absent substrate from real failures.

Running the suite on Windows manually:

1. Install Bun 1.3.14 or newer, Node.js for the suite's other tooling, and the Claude Code CLI.
2. Install Git for Windows if you are running the full suite or the POSIX wrapper compatibility smoke; the native runner path itself does not require Bash.
3. Install the repository's dev dependencies so Bun can resolve @xterm/headless. The default Windows TUI backend uses Bun's native PTY.
4. Set AIDLC_TUI_LIVE=1 for live Claude TUI coverage. Leave AIDLC_TUI_BACKEND unset or set it to auto/bun; use AIDLC_BUN_BIN to select a concrete bun.exe when needed.
5. For the Kiro IDE slice, install and sign in to Kiro IDE, then set AIDLC_KIRO_IDE_LIVE=1. Run the GUI tests in that user's logged-in desktop session. A scheduled task can use “Run only when user is logged on” to launch the runner in this session. The default Windows binary is %LOCALAPPDATA%\Programs\Kiro\Kiro.exe; override it with AIDLC_KIRO_IDE_BIN.
6. Run bun tests/run-tests.ts --all --debug -P 8.

No WSL or Docker is required; the supported validation substrate is native Windows.

For Kiro CLI coverage, put the signed-in native kiro-cli.exe on PATH. Pin AIDLC_BUN_BIN to the validated Bun executable and keep its directory first on PATH; Kiro installations can include an older bundled Bun. Set AIDLC_KIRO_ACP_LIVE=1 for ACP files. For terminal files, set AIDLC_KIRO_TUI_LIVE=1 and AIDLC_TUI_LIVE=1; the native Bun terminal backend handles Windows input and screen capture. These files check the CLI, account, and selected terminal backend before running, with the same journey assertions on each supported platform.

Repeatable MR10 Windows EC2 runbook:

1. Stand up a disposable Windows Server 2022 host with SSM access:

aws cloudformation deploy \
  --stack-name aidlc-windows-test \
  --template-file tests/harness/windows/windows-test.cfn.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides VpcId=vpc-... SubnetId=subnet-...

1. Sync the committed git tree under test:

bun tests/harness/windows/sync.ts --stack-name aidlc-windows-test HEAD

1. Install repo dev dependencies on the box:

bun tests/harness/windows/ssm-run.ts --stack-name aidlc-windows-test -- \
  powershell -ExecutionPolicy Bypass -File C:\aidlc\tests\harness\windows\setup.ps1 -ProjectDir C:\aidlc

1. Run the Windows --all gate with live TUI enabled:

bun tests/harness/windows/ssm-run.ts --stack-name aidlc-windows-test -- \
  powershell -ExecutionPolicy Bypass -File C:\aidlc\tests\harness\windows\run-all.ps1 -ProjectDir C:\aidlc -Parallel 8

1. Tear down the host:

aws cloudformation delete-stack --stack-name aidlc-windows-test

The SSM observer shares one deadline across its API queries and polling. Expiry reports the remote exit as unconfirmed; a late terminal reply cannot turn an expired observation into a successful run.

run-all.ps1 exports AIDLC_BUN_BIN and AIDLC_TUI_LIVE=1 before invoking bun tests/run-tests.ts --all --debug -P <N>. Its preflight calls the shared selectedTuiBackend and tuiUnavailableReason helpers through Bun to check the selected backend's prerequisites. Node remains a full-suite tooling prerequisite. Live opt-in does not establish coverage when a prerequisite check skips a journey: inspect the captured per-file results. The script probes the Claude binary across C:\Users\Administrator\.local\bin and the systemprofile home, since the native installer drops claude.exe under whichever user ran the CloudFormation UserData bootstrap (Administrator under EC2Launch v2).

The stack defaults to c5.4xlarge for the full --all -P 8 live run. Smaller hosts increase contention under parallel load; use a lighter tier selection when reducing InstanceType. The shared timeout backstops above allow for runner variation without treating one host's observed duration as a performance requirement.

Terminal Driver

tests/harness/tui-drive.ts drives an interactive CLI through named sessions. The shared selector in tests/harness/tui-runtime.ts chooses its backend from AIDLC_TUI_BACKEND:

auto chooses by platform; it does not fall back to another backend when prerequisites are missing. Unknown values, including an empty string, are configuration errors. AIDLC_BUN_BIN overrides the Bun executable for the bun and tmux backends. Keep the same backend selection across commands for a session. The native Bun backend does not require tmux.

The native daemon in tests/harness/tui-bun-backend.ts uses inline terminal: { ... } options on Bun.spawn to own the PTY and its output callbacks. tests/harness/tui-screen.ts parses those bytes through @xterm/headless and returns emulator replies to the PTY. Commands reach the daemon over a local socket on Linux or a named pipe on Windows, so the PTY and screen survive separate driver invocations.

Each native session has its own daemon, PTY, screen, and authenticated IPC endpoint. Give parallel workers distinct session names, projects, and CLAUDE_CONFIG_DIR profiles. The test runner must cap Bedrock concurrency independently of the number of terminal sessions.

tests/harness/tui-bun-process.ts supervises descendants with Linux subreaper and pidfd ownership, or a Windows Job Object. kill waits for confirmed process cleanup and daemon retirement; wait-dead checks the daemon's process identity. Start locks are owned by the OS and release when an interrupted starter exits. Fixture cleanup also refuses to remove a project while its native session has unconfirmed cleanup.

Session records and final snapshots remain under AIDLC_TUI_BUN_ROOT (default: $TMPDIR/aidlc-bun-tui, using the OS temporary directory). Both the default and an explicit root must be private: a real directory owned by the current user with mode 0700 on POSIX, or on Windows a current-user owner SID and no allow ACEs for Everyone, BUILTIN\Users or Authenticated Users. Symlinks, junctions and other Windows reparse points are refused. The driver creates a missing root with private permissions; it refuses an unsafe existing root rather than changing its permissions. On POSIX, every explicit-root ancestor must be owned by the current user or root, with no group/other write bits unless sticky; the implicit root's temporary parent must be current-user-owned or mode 1777. On a controlled test host whose filesystem reports those ancestors as owned by a sandbox uid (for example an overlay filesystem where / belongs to nobody), set AIDLC_TUI_ALLOW_UNTRUSTED_ANCESTORS=1 to skip only the ownership requirement; the write-bit rule and every other check here stay enforced. Windows validates root/session owner and DACL, but does not yet validate ancestor ACL trust; the generation handshake below remains enforced. Remove an unsafe pre-created root or point AIDLC_TUI_BUN_ROOT at a private directory under trusted ancestors, and keep that setting consistent across commands.

The test runner gives each file a separate private native root under a trusted OS temporary directory, independent of log-directory permissions. After confirmed transport cleanup it archives session records and snapshots in the file's tui-bun/ artifacts and removes the temporary native root; uncertain cleanup retains the live namespace for diagnosis.

Launch records are private regular files pinned to the root and session directories' filesystem identities. Both clients and the daemon validate ownership, permissions and session identity before trusting them. The daemon also requires a starting record with the fresh generation supplied by the starter through argv, before creating a PTY or supervisor, and rechecks both recorded directory identities immediately before releasing the command. Replayed completed records cannot authorize a command; diagnostics are never written into an untrusted directory.

Target exit status and PTY exit status are separate: Bun reports ordinary Linux slave-close EIO as PTY status 1. The driver accepts that status after confirmed supervisor exit and cleanup; Bun's callback does not expose an errno to distinguish other POSIX read errors. Unreachable sessions and unconfirmed cleanup remain errors.

Captures read the current active viewport after queued output has been parsed. Plain text joins soft-wrapped rows and trims trailing blank lines; ANSI capture is reconstructed from the current cells and their attributes, including palette and RGB colors. Scrollback and previously erased output are excluded. JSON retains the full physical grid, including blank cells and width-zero continuation cells for wide characters.

The existing start, send, plain capture, startup, and wait commands retain their calling conventions. send --keys "<text>" --literal sends literal text; without --literal, named keys are interpreted and arbitrary text such as 1/2 remains literal. Enter is appended unless --no-enter is set. wait --stable-ms N retains its screen-stability condition, and startup --ready-pattern "<regex>" retains its startup navigation behavior.

The Bun backend supports these capture and input commands:

Menu detection always consumes physical rows. Bun derives these from snapshot cells without changing text, wrap flags, or ANSI; tmux omits -J for these reads. This prevents ConPTY wrap metadata from joining visible menu options onto preceding rows. External automation can select capture --physical, separately from --json and --ansi.

wait --pattern and startup --ready-pattern default to --view auto: physical text is tried first, with logical text from the same frame as fallback. This preserves literal patterns spanning a genuine soft wrap while supporting row-anchored UI patterns. Use --view physical or --view logical to select one interpretation. Bun obtains both from one snapshot; tmux captures both in a single synchronous command list. Menu actions always inspect the physical view, regardless of pattern-view selection. Stability is measured on the matched view without restarting the overall deadline.

Live waits end on what the screen shows, not only on the clock. Once a wait has seen the agent working (its status spinner and elapsed timer, a background-agent wait, a running subagent row, or a running command's background hint) and Claude's empty prompt then stays unchanged for 30 seconds, the turn has ended: wait fails with the last pane if its pattern never painted, answer-gate fails if no menu appeared and its terminator is unmet, and revision recovery types free-text feedback only at that idle prompt. A screen the driver does not recognize counts as working, so the hang backstop still applies. Across recorded sessions the longest idle-looking pause inside a turn was under half a second. wait --through-turn-end keeps waiting past the end of a turn. answer-gate answers a menu only once its rows, from its highlighted option through its footer, read the same across half a second, so a menu caught mid-paint is not answered by its first half; rows still changing after five seconds send it back to waiting for a menu. After it answers a menu, it waits up to five seconds while that menu's rows, from its highlighted option through its footer, stay unchanged apart from the caret and checkbox marks its own keys move, so a slow or partial repaint is not answered twice; it stops waiting when those rows repaint or its terminator lands (with --stop-at-approval-gate, only a repaint ends it), and a menu still intact after five seconds is answered again.

resize, paste, and capture --json require the Bun backend. ANSI capture is also available with tmux. --json and --ansi are mutually exclusive.

Invoke these through the same driver, for example:

bun tests/harness/tui-drive.ts resize --session demo --width 120 --height 40
bun tests/harness/tui-drive.ts paste --session demo --text "draft input"
bun tests/harness/tui-drive.ts capture --session demo --json

TuiSnapshot is defined in tests/harness/tui-screen.ts: it contains sequence, cols, rows, zero-based cursor: { x, y }, buffer (normal or alternate), text, ansi, and lines. Each line has a wrapped flag and a cells array. Each cell carries chars, width, fg/bg colors (mode: default|palette|rgb, numeric value), and boolean bold, dim, italic, underline, inverse, invisible, and strikethrough attributes. The sequence advances after parsed output or a resize; an unchanged capture does not advance it. Cursor x can equal cols when a wrap is pending.

The unit suites tests/unit/t-tui-runtime.test.ts and tests/unit/t-tui-screen.test.ts define deterministic checks for runtime selection and terminal transcripts. The screen suite includes UTF-8 regressions for:

• The exact 85-byte ANSI/Unicode preflight payload at all 86 two-chunk split positions, including empty edge chunks.
• Every fixed chunk size from 1 through 85 bytes, with a parser drain between writes; text, physical cells, widths, colors, and ANSI must agree.
• All byte partitions of selected 2-, 3-, and 4-byte characters containing 0x80 continuation bytes.
• Incomplete UTF-8 retained across flush() calls until the remaining bytes arrive.
• Malformed UTF-8 rendered as U+FFFD replacement cells instead of silently disappearing.

The screen uses an incremental StringDecoder before xterm's string API: xterm 5.5's byte decoder can lose a character when a saved continuation byte is 0x80, as in a fragmented em dash (E2 80 94).

tests/unit/t-tui-bun-process-linux.test.ts exercises real Linux supervision; tests/integration/t-tui-bun-lifecycle.test.ts covers Windows Job Object, signal, and descendant cleanup. tests/unit/t-tui-process-identity.test.ts covers identities and lock release, including interrupted starters. tests/integration/t-tui-bun-backend.test.ts runs the public CLI against token-free targets, including eight simultaneous sessions, input, ANSI/cell capture, resizing, paste, restart, IPC errors, and daemon retirement.

The TUI preflight calibrates the selected backend. Native lifecycle has the separate coverage above. Live model journeys retain their existing opt-ins. Keep temporary Claude fixtures outside repository ancestors so they do not inherit development instructions or external-import prompts.

Preflight Validation

Before running unfiltered live-capable levels (integration or e2e), the runner executes tests/integration/t19.test.ts as a gate. It drives a tiny real turn through the Claude Agent SDK (the same live path the integration tier uses) and asserts only on deterministic surfaces. A skipped preflight closes the Claude gate without failing a default run; AIDLC_CLAUDE_SDK_LIVE=1, AIDLC_TUI_LIVE=1, or --require-coverage instead requires complete, non-skipped passing evidence. An actual preflight failure, timeout, or cleanup error fails the run in every mode. Whether the preflight skips or fails, deterministic files still run and Claude-dependent files receive per-file SKIP entries.

The SDK driver gives each driveAidlc() call an ephemeral CLAUDE_CONFIG_DIR and disables session persistence. Live tests therefore leave the user's ~/.claude.json and Claude transcripts untouched, including when the home directory is read-only inside a command sandbox. A per-call env.CLAUDE_CONFIG_DIR remains available for focused calibration.

A drive is one session and, by default, ends at its first turn's result. A journey that needs the person's next message passes nextMessage: when a turn ends, it gets the turn number and the drive's running totals so far (pickers, tool results, Stop hooks; subtract the previous turn's totals for one turn's own) and returns the message to send into the same session (or nothing, to end the drive). chatAboutQuestionWhen answers a picker the way Claude Code's "Chat about this" button does, so the question stays open; an answerScript spec { text } types the person's own words into a picker; captureStopHooks records each Stop hook verdict with its turn. stopWhen ends the drive once the tool results so far satisfy it, for a stop that needs more than one result (an engine line that can arrive before or after the step that creates the work).

A suite launched from inside a Claude Code session (CLAUDECODE=1) does not hand that session's ANTHROPIC_DEFAULT_*_MODEL defaults to its drives: the SDK's bundled Claude Code may be older than the session's model and refuse it. When a drive's final provider is Bedrock, it gets CI's pinned models (CI_BEDROCK_MODELS in scripts/ci-credential-broker.ts) for any model the project settings or the test did not set; otherwise the bundled defaults. A run from any other shell, CI's included, keeps its environment unchanged.

A red here is a real environment finding (missing claude, expired creds), never a flake to soften — exactly the gate's job of bailing the downstream LLM tiers fast.

Test Registry

The suite is discovered, not registered: bun tests/run-tests.ts walks the four level directories (tests/{smoke,unit,integration,e2e}/) and runs every t*.test.ts it finds. There is no hand-maintained per-test table to keep in sync — adding a test file is all it takes for the runner to pick it up.

What each test covers is tracked mechanically by tests/gen-coverage-registry.ts, from the covers: header in a test file's leading comment block (typically line 1; a few files legitimately declare none and simply contribute no coverage claim). The generator enumerates the framework's units across seven classes (function, audit, scope, stage, hook, subcommand, render-surface), maps each covers: claim onto an enumerated unit, and builds one entry per unit with its claims and status.

The registry is never committed. A committed copy changed in nearly every PR, so queued PRs collided on it or had to regenerate it after each rebase. CI builds it fresh instead: the coverage step in the combined Checks job, part of the required Tests (smoke + unit) check, runs --check --base HEAD^1. That builds the registry for the commit under test and for its first parent (the PR's base, or the queue entry ahead), the parent with its own packager and generator in a temp directory, and fails naming each unit the parent covers that the commit still has but no longer covers. Coverage can only grow: a new unit with no claim is listed but never fails, and a unit whose code was deleted or renamed is no loss. To keep a unit covered when its test goes away, add its covers: claim to another test that exercises it.

bun tests/gen-coverage-registry.ts                             # write tests/.coverage-registry.json locally (gitignored)
bun tests/gen-coverage-registry.ts --check                     # the anti-rot guards on a fresh build
bun tests/gen-coverage-registry.ts --check --base origin/main  # and fail if a unit main covers lost its claim

The generator reads the units from the packaged dist/claude tree. If that tree was packaged from other sources (core/, harness/, plugins/ or the packager's own scripts) than the checkout, both commands refuse with one line: run bun scripts/package.ts first. package.ts records a content fingerprint of its inputs per harness in dist/.package-sources.json (scripts/package-sources.ts), so undoing an edit needs no rebuild. A build during which an input changed is not recorded and fails, saying to run it again. Temp trees from the generator's AIDLC_COVERAGE_* seams have no core/ and are not checked.

To find which test exercises a given function, audit event, scope, stage, hook, subcommand, or render surface, write the registry locally and read it, or grep the covers: headers directly. The unit tier runs --check on every tree, so a broken enumerator reds the gate.

Note: t19 appears in both unit (tests/unit/t19.test.ts, the jump CLI tool) and integration (tests/integration/t19.test.ts, the live preflight gate) — a level/file path, not a bare ID, disambiguates such collisions.

Trigger Points

L1 can be enforced via a git pre-commit hook: bun tests/run-tests.ts || exit 1.

The required Tests (smoke + unit) status also requires the combined check job: package determinism, typecheck, lint, installer ShellCheck, Linux smoke and production guards. Contract checks and the test matrix run in parallel; the aggregate waits for both. Its name stays the same so the repository's required status check covers them together.

When the merge queue drops a PR only because a GitHub-hosted runner failed, merge-queue-notice.yml posts one comment on the PR saying so, naming the jobs, and noting that a maintainer can add it back. It runs on workflow_run after each merge-queue CI run, checks out only the default branch, and uses the built-in token with actions: read, checks: read, contents: read and pull-requests: write. .github/scripts/merge-queue-notice.ts owns the signatures, read from GitHub's records:

• for a failed_checks removal, every red job either lost its runner ("The hosted runner lost communication with the server", no failed step) or failed checkout with "Could not resolve host: github.com";
• for a checks_timed_out removal, every job still open at the removal was either waiting for a runner or still in "Prepare unit test substrates".

It posts nothing for test failures, conflicts, manual removals, PRs that change .github/, PRs already back in the queue, or a run it already explained, and it never enqueues, dequeues or merges. t-ci-merge-queue-notice pins this on trimmed records of real drops in tests/fixtures/merge-queue-notice/.

PR CI stays strict: a failing file is red on the PR, with no retry, so a flake is seen where it was introduced. The merge queue is different, because one flaky file drops the PR and rebuilds every group queued behind it. There, ci.yml passes retry-once to the smoke, unit and integration legs, which run the tier with --file-retries 1: a file whose first attempt failed assertions (failed cases, complete JUnit evidence, no file timeout, no cleanup failure, at most ten minutes, and at least five minutes left in the run) runs once more in a fresh process and temporary directory. A case that ran past its own case timeout is a failed case, and a file whose only failures are such case timeouts (a slow Windows runner timing out cases) may have run up to 45 minutes and still gets its one retry; retries.json marks it caseTimeoutsOnly. The rule lives in tests/lib/file-retry.ts, shared with the isolated live retry. A crash or nonzero exit without failed cases, a file that executed no cases, a file that ran past its deadline, and a second failure are never retried. Neither is a person-turn failure: a case that fails with the drives' PersonTurnError (tests/harness/person-turns.ts: a decision recorded as the person's with no reply of theirs behind it, or work started before they answered) keeps its file red with a NO RETRY line, because a pass on a second attempt would hide the break a person would meet. Only a retry that passes every case the first attempt ran replaces the first failure: a second failure, or a retry that skips one of those cases or executes none, stays a failure. The retry starts only after the first attempt's evidence is moved aside whole; if any of it cannot be, the file is not retried. A pass on the second attempt is never silent: the job shows a Flaky test warning (" passed on its second attempt (merge queue)"), the step summary lists it under "Passed on retry", the file's summary.txt row keeps PASS with a "passed on retry" line under it, and retries.json in the run's log directory records every retry for flake triage. The first attempt's log and JUnit stay beside the second as <file>.attempt-1.*. Isolated e2e and Full Suite never take the flag. t-runner-production-guards and t-ci-retry-report cover the rule, the runner and the report.

macOS runners are the slowest and fewest hosted runners, so they set the merge queue's pace: a full macOS unit set costs about 245 job-minutes per merge group. In the queue only, ci.yml passes macos-merge-selection to the macOS unit shards, and scripts/ci-macos-unit-selection.ts picks the unit files whose source names macOS or darwin (computed on each run, never a kept list) plus the unit files the change touches against its first parent (the queue entry ahead), spread over the queue's two macOS unit jobs by weight, about 60 job-minutes in all. A selected file brings the rest of its affinity group (t249 reads the binary t238 builds). Each shard runs its share as one whole shard (--shard 1/1, so the shard rules such as required compiled coverage hold) and leaves the other files out with --exclude, so a file that skips on macOS stays SKIP as in a full shard; a shard with no selected file stops before installing. When the change cannot be diffed, a selected name falls outside the test-file grammar, or the selection fails, the shard runs in full. macOS smoke, integration, isolated E2E and the native-terminal units, Linux and Windows, PR CI and the nightly Full Suite run every file. t345 pins this.

main is not production: PR CI and the merge queue remain the fast gates listed above, while required hosted live tiers run in full-suite.yml. preview-release.yml dispatches it every night as a separate run for the preview source SHA. Its failures remain visible in that run and in the preview notes, without failing the preview workflow. Once contract checks pass, preview native smoke, builds and artifact validation run in parallel with Full Suite. Publication waits for both the validated artifacts and the completed test report, so the release notes retain any Full Suite or upgrade warning. An unchanged source still runs contract checks and Full Suite while skipping the build and publication.

Stable promotion selects a published preview's source and adds only version metadata and pinned publication controls. It does not repeat CI or Full Suite. The preview's prior results remain visible, including failures; choosing to promote is the maintainer's decision. The newly built stable binaries still pass packaging, installer, lifecycle and update checks before publication. See the promotion steps.

A newer dispatch on the same workflow ref with the same verification selection cancels the older run, so redispatching live verification after a push supersedes the run for the previous head. Release-purpose dispatches also key on the ref input, and Full Suite runs called by the preview are never cancelled this way.

ci.yml and full-suite.yml call the same reusable .github/workflows/deterministic-tests.yml. Callers select the immutable ref, runner, tier, unit or integration shard, an optional file filter and exclude (both empty by default, so the whole tier runs) and artifact label. Both split integration in five jobs: the scope runs (filter, t-scope-run-*), the scope-host runs (filter, t-scope-host-*), the guard matrix in two duration-balanced shards (filter: ^t-guard-matrix-, integration-shard: 1/2 or 2/2) and everything else (exclude of all three). PR CI selects Linux smoke, ten weighted unit shards, and the five integration jobs; a full_verification Full Suite selects smoke, sixteen unit shards per OS, the five integration jobs, and isolated E2E on Linux/macOS/Windows. The nightly Full Suite runs no deterministic tier; its scope_runs job calls the shared workflow for the scope runs, the scope-host runs and the guard matrix on all three. Integration and E2E run as independent jobs per OS, each with a fresh Bun runner process. Every call owns a fresh checkout, installs frozen dependencies under Bun 1.4.2, regenerates projections once, and invokes the Bash wrapper with --debug -P 8 --no-llm. E2E runs retain --isolated-e2e; smoke and unit remain serial within each checkout. Sharing the workflow shares the commands and setup, not previous test results.

CI groups code contracts, installer ShellCheck, the coverage ratchet, PR changelog preservation, Linux smoke and production-guard checks in one job. Smoke and production guards use separate runner processes and log directories, sharing only dependency installation and generated projections. macOS and Windows keep their own smoke jobs. Linux smoke retains the merge queue retry and flaky-test reporting policy. Each check retains its own step and failure; independent checks continue after an earlier check fails. Coverage still compares against the first parent, and changelog preservation uses the frozen PR base. The required Tests (smoke + unit) status requires this combined job to pass. PR CI allocates 17 runners; the merge queue allocates 51, with its existing platform selections. Live harness and native build matrices are unchanged.

The repository's default-branch ruleset limits merge queue build concurrency to two groups, down from eight. This setting is managed in GitHub, separately from these workflows; it limits concurrent merge-group builds, not the number of PRs in a merge batch. The required checks still run on each group's commit.

Full Suite limits its manual deterministic matrix to twelve concurrent jobs and its nightly scope/guard matrix to three. All 69 manual deterministic jobs and all twelve nightly scope/guard jobs still run. The suite axis comes first, so job creation interleaves Linux, macOS and Windows. This order is not a per-platform quota: runner availability and job duration determine which jobs execute together. Live shards retain their existing limits of four Linux, two macOS and three Windows jobs. These are per-matrix limits, not an organization quota; concurrent workflows still share hosted capacity. Lower concurrency can lengthen a Full Suite run while reducing contention with PRs and the queue.

Standalone deterministic, native-terminal, and production-guard test jobs use a five-hour job backstop and a 270-minute execution step. The runner shares a four-hour work deadline across the invocation, with a two-hour deadline per file, including smoke and isolated deterministic E2E. This hierarchy leaves time to retire processes, finish reports, sanitize logs and upload evidence after work stops. Unit work remains partitioned without duplication: ten Linux shards in CI, sixteen Windows shards, two selected macOS shards in the merge queue, and sixteen macOS shards in expanded manual CI. Full Suite full verification uses sixteen per OS. Compiled producer/consumer affinity is preserved.

The combined CI check job starts a shared 270-minute work deadline before checkout, leaving thirty minutes of its five-hour job ceiling for process cleanup and evidence collection. Checkout and Bun setup each have a fifteen-minute ceiling; together they cannot consume the work envelope. Every subsequent setup and contract command runs through scripts/ci-check-budget.sh, which enforces the remaining deadline with a process-group timeout and refuses to start after it expires. Independent checks still collect failures, and sanitization stays eligible after a failed command whenever Bun is available. Smoke receives at most half the remaining work time, reserving the other half for production guards; guards receive the actual time left when their step starts. Each invocation also retains the four-hour upper bound. An exhausted budget fails the step before launching tests. Existing case and file ceilings still apply within each invocation's remaining budget.

The same hierarchy applies to merge-queue native-terminal checks, Full Suite native obligations and production-guard checks. These paths must not quietly reintroduce a smaller case, file, run or step ceiling. They all retain captured --debug -P 8 wrapper execution and evidence collection. Credentialed live jobs retain their separate one-hour credential boundary: 60-minute files within 70-minute steps and 80-minute jobs. Their nested driver operations allocate from the remaining file budget.

For a focused deterministic reproduction, dispatch deterministic-tests.yml directly on the candidate branch:

gh workflow run deterministic-tests.yml --ref '<candidate-branch>' \
  -f 'ref=<exact-source-sha>' -f runner=windows-latest -f tier=unit \
  -f unit-shard=1/1 -f 'diagnostic_filter=^t-tui-runtime$'

diagnostic_filter is a manual-only filename regex and is not exposed to reusable CI callers. The unit tier requires unit-shard=N/M; 1/1 selects all unit files before filtering. For smoke, integration or e2e, omit unit-shard; its default is empty. Integration may additionally use integration-shard=N/M to reproduce one partition of its filtered inventory. Each reproduction uses one fresh runner with model gates closed and the same immutable checkout, bounded runner and sanitized evidence paths. The default artifact is ci-deterministic-probe-<OS>. These targeted diagnostics do not qualify a full suite or release.

POSIX unit jobs require tmux: the shared setup first checks command -v, then uses apt on Linux or Homebrew on macOS only when it is missing. Linux unit jobs also install zsh when absent. Missing tools fail setup rather than skipping the compatibility cases.

The shared workflow binds the checked-out commit to the caller's SHA. Release-purpose Full Suite runs authorize that SHA against main in the plan job; manual verification instead binds it to the selected workflow head. PR CI can test its PR merge commit without receiving live credentials. Each run captures stdout/stderr in the checkout root's tmp/ci-deterministic/run.log, prints that path and the actual log stamp, and preserves both this capture and tests/logs/ after successful sanitization. Artifacts use the caller's label plus the actual runner OS and retain evidence for 90 days. ci.yml passes evidence-optional: true because no workflow downloads its ci-deterministic-* artifacts: a failed upload is reported on the job but does not fail it, so an upload timeout cannot drop a PR whose tests passed. The test step's own exit code still fails the job. CI's ci-native-* and ci-check-evidence uploads follow the same rule. Full Suite does not pass the input, so the full-suite-deterministic-* uploads of a manual full verification run stay required: that run exists to hand a person its evidence, and no queued PR waits on it.

For platform verification before a nightly fix lands, dispatch gh workflow run ci.yml --ref <branch> -f platform_regressions=true. This expands the existing deterministic matrix to all three OSes and adds isolated E2E jobs alongside integration; it does not run the Linux pass first or append another broad regression slice. The full unit shards include the tmux, path, workspace-fixture, and macOS regressions, with t238/t249 kept in their weighted affinity group. They use the same POSIX provisioning and capture path as nightly tests. Focused native, production guard, and OS-isolation checks remain required. Manual CI does not make model calls or produce a Full Suite result.

Stubs

Greenfield Stub: tests/fixtures/greenfield-todo/

A project description with no source code. Workspace-detection classifies as greenfield. Gives the LLM deterministic intent context for ideation stages.

Contents: Just README.md describing a React Todo App with TypeScript and Vite.

Brownfield Stub: tests/fixtures/brownfield-todo/

Minimal React+TypeScript+Vite source (~10 files, ~200 LOC). Workspace-detection classifies as brownfield. RE, requirements, and design stages have concrete code to analyze.

Contents: - package.json — react, react-dom, typescript, vite, vitest - tsconfig.json, vite.config.ts, index.html - src/main.tsx, src/App.tsx - src/types/todo.ts — Todo interface (id, title, completed) - src/components/TodoList.tsx — list + add form (~40 lines) - src/components/TodoItem.tsx — checkbox + title + delete button - src/hooks/useTodos.ts — addTodo, toggleTodo, deleteTodo

RE Artifacts Fixture: tests/fixtures/re-artifacts/

Pre-seeded reverse-engineering output for downstream stage tests. Copied into the test project's space-level repository store at $PROJ/aidlc/spaces/default/codekb/<repo>/ during setup.

Contents: 4 minimal .md files (architecture-overview, technology-stack, codebase-analysis, integration-points) describing the brownfield-todo app.

Inception Artifacts Fixture: tests/fixtures/inception-artifacts/

Pre-seeded inception phase output for tests that jump into construction. Copied into $PROJ/aidlc/spaces/default/intents/<record>/inception/{requirements-analysis,domain-design,units-generation}/ during setup.

Contents: minimal .md files (requirements, the consolidated components.md catalogue, unit-of-work, unit-of-work-story-map) describing the Todo app. Unit name: todo-core.

Construction Artifacts Fixture: tests/fixtures/construction-artifacts/

Pre-seeded construction phase output for tests that jump to mid-construction stages (e.g., code-generation). Copied into $PROJ/aidlc/spaces/default/intents/<record>/construction/todo-core/functional-design/ during setup.

Contents: 1 minimal .md file (functional-design) describing the todo-core unit's component specs and state management.

Input Corpus: tests/fixtures/aidlc-input/corpus.json

What people type after /aidlc, and in plain chat once work is under way: one item per typed line at one workspace state (fresh, work-open, plan-question-open, gate-open, two-similar-records), with meaning (what the person means, in plain words) and expected.engine (what the engine's first step must do: the directive kinds allowed, the command or words it must name, the message record's words and settings, whether a question may reach the person). Seeded from every /aidlc example the docs, skills, stage prose and help text ship (source and documented carry the origin), from the input-forms audit, and from variations: typos, word order, quoting per shell, CRLF and unicode, replies at gates and plan questions, workspace words, settings in plain words, retired flags, empty lines, ambiguous lines. Items whose expectation needs a change not merged yet carry after and are skipped until that change is in the source. The shared mechanics live in tests/harness/aidlc-input-corpus.ts; tests/unit/t-aidlc-input-corpus.test.ts runs every item through the Claude Code human-turn hook and next, and a fixed sample of two items per category through the Codex, Kiro CLI and Kiro IDE hooks as well (AIDLC_INPUT_ONLY=<ids> narrows a run, AIDLC_INPUT_PARALLEL sets the fixture pool). tests/e2e/t-live-aidlc-input-corpus.serial.test.ts drives a sample through real agents on demand (AIDLC_INPUT_LIVE=1): what happened on disk is judged by the script against each item's expected.end (the work, the setting set by the person, the record switched, the gate or plan answered, questions asked), what the agent told the person is judged by a model against the item's meaning and a short rubric (plain words, the product's own names, no engine internals, a question only where the item is ambiguous), and a report with a score per category and every miss with its transcript excerpt lands under AIDLC_INPUT_REPORT_DIR.

Answers Corpus: tests/fixtures/aidlc-answers/corpus.json

What people reply to every question AI-DLC asks them: one item per reply at the state where that question is open (the stage gate, the gate with Accept as-is, a blocking-sensor question, the code plan question, the Unit and batch checkpoints, the verification command, a Construction policy change, the summary confirmation, a stage question the agent asked, the interaction-mode question, the routing question, the plan offer and scope confirmation, the record pick, the project-type question, a paused Unit, a guard-recovery ask, re-entry, a reopened stage, the learnings question, a jump target), with meaning (what the person means), reader (tool when the reply is an exact pick as the options were shown, agent otherwise), expected.engine (the choice a tool may record from the reply alone, the words that must be kept, the engine's step) and expected.end (what the person ends with after a real agent acted, for the live check). Replies are typed in plain chat, typed after /aidlc or $aidlc, or picked in the question box. Items whose expectation needs a change not merged yet carry after and are skipped until it lands. The shared mechanics live in tests/harness/aidlc-answers-corpus.ts; tests/unit/t-aidlc-answers-corpus.test.ts runs every item through each host's human-turn hook, then the engine's step and the agent's answer command (AIDLC_ANSWERS_ONLY=<ids> narrows a run, AIDLC_ANSWERS_PARALLEL sets the pool). tests/e2e/t-live-aidlc-answers-corpus.serial.test.ts drives a sample through real agents on demand (AIDLC_ANSWERS_LIVE=1): the real agent shows the question first (a bare /aidlc at the fixture's state), then the person replies in chat, after /aidlc, or in the question box; the script judges what is on disk against expected.end (the choice recorded with the person's words beside it, the message record, a question asked twice), a model judges what the agent told the person against meaning and says, and a report with a score per question and every miss with its transcript excerpt, and a summary for the nightly report, lands under AIDLC_ANSWERS_REPORT_DIR. The nightly runs it too, in the Claude SDK family's Linux live job, where no switch of its own reaches the file: on GitHub Actions with AIDLC_CLAUDE_SDK_LIVE=1 it drives Claude Code only, one item per question chosen by a date rotation (night N of the rotation from 2026-10-15 drives each question's item N modulo the question's item count, in corpus order, so every item comes round within as many nights as the largest question has items), and it reports every miss without failing (NIGHTLY_MISSES_FAIL in the file makes misses fail it, as on demand). A drive that throws, a run that drives no item, or a changed real home still fails the file. The nightly report's Answers corpus section shows the result (see Nightly full-suite matrix and provisioning). The two live checks share tests/harness/live-corpus.ts (gates, the seeded sample and the nightly's rotation, scratch projects and homes, the home digests, the model judge, the report and its summary).

State Fixtures

How to Add a Stage Test

1. Choose the stage to test and identify what state fixture it needs (the state must show that stage as the current/next stage)
2. Create or reuse a state fixture in tests/fixtures/
3. Create tests/integration/tNN-stage-SLUG.test.ts and use the shared TypeScript harness helpers (tests/harness/fixtures.ts, tests/harness/sdk-drive.ts, tests/harness/tui-drive.ts, tests/harness/plugin-kit.ts, or tests/harness/exec-drive.ts) rather than shell TAP helpers.
4. Run with bun tests/run-tests.ts --integration or directly: bun test tests/integration/tNN-stage-SLUG.test.ts

How to Add Acceptance Assertions

To add artifact assertions to an existing e2e workflow test under tests/e2e/:

1. Read the current test and understand what it already checks
2. Add expect(...) assertions inside the existing test(...) block (bun:test counts assertions from the calls themselves — there is no plan line to keep in sync)
3. Use flexible patterns: match /[Tt]odo/ against readFileSync content, not exact strings
4. Use test.skipIf(...) / an early return for assertions that depend on non-deterministic LLM output format
5. Use expect(statSync(path).size).toBeGreaterThan(minBytes) for size-bound checks

Assertion Design Principles

• Keyword classes — Use case-insensitive regex: [Tt]odo, [Rr]eact, [Bb]rownfield
• Flexible discovery — Use find + wc -l to count files rather than checking exact names
• Size bounds — Use statSync(path).size with toBeGreaterThan() for minimum content
• Graceful degradation — Use skip when an assertion depends on non-deterministic LLM output
• Structure over content — Check for markdown headings (^#), file existence, directory creation before checking content

Environment Variables

CLI Reference

# Entrypoints
bun tests/run-tests.ts        # Native cross-platform runner
bash tests/run-tests.sh       # POSIX compatibility wrapper

# Level flags (combinable)
--smoke         # Structural validation
--unit          # Unit directory, including existing process and workflow contracts
--integration   # Cross-component contracts and stage/CLI utilities
--e2e           # Full lifecycle, worktree, and rendered terminal journeys

# Profile flags (shortcuts)
(default)       # smoke + unit + integration
--ci            # smoke + unit + integration
--release       # smoke + unit + integration + e2e
--all           # Same as --release

# Output modifiers
--production-guards # Run selected tests with guard bypasses and direct authority
                    # off; neutralize inherited off-switches. Default: fixture.
--verbose       # Write per-test logs to tests/logs/
--no-llm        # Force all live-model gates closed while deterministic
                # integration/e2e tests still run. Also via AIDLC_NO_LLM=1.
--debug         # Implies --verbose; streams per-test output and writes SDK/TUI
                # driver traces to tests/logs/
--filter PAT    # Only run tests whose filename matches extended regex PAT
--exclude PAT   # Leave out tests whose filename matches PAT; the rest run as an ordinary tier
--parallel N    # Run up to N test files concurrently within a tier (alias: -P N).
                # Default: 1 (serial). Smoke and unit tiers are always serial.
--file-timeout N  # Independent file ceiling in seconds for every tier; caps isolated E2E too.
                  # Default: 7200 outside isolated E2E.
--run-timeout N   # Shared work deadline in seconds across setup and all selected files.
--shard N/M     # Run one duration-balanced unit or integration shard.
                # Requires --unit or --integration with no other tier/profile.
--isolated-e2e  # Run e2e in independent checkouts; -P sets worker count.
--e2e-plan      # Print selected files/resources without builds or test execution.
                # Requires --e2e; implies --isolated-e2e, not --e2e.
--bedrock-parallel N  # Concurrent Bedrock test files (default 2).
--kiro-parallel N     # Concurrent Kiro test files (default 2).
--ide-parallel N      # Concurrent IDE files, also charged to Kiro (default 1).
--e2e-file-timeout N  # Outer isolated-file deadline in seconds (default 10800).
--e2e-timings FILE    # Prior runner summary.txt for duration-based ordering.
--e2e-cancel-file FILE # Create this file to request cancellation on any platform.

--no-llm (or AIDLC_NO_LLM=1) closes the derived Claude gate and forces every live-model opt-in to 0: Claude TUI, Kiro ACP/TUI/IDE, Codex exec, and opencode run. Deterministic tests in those tiers still run, including the token-free TUI substrate preflight. This gives CI a full-tier deterministic profile without live-model cost or flakiness, even when CLIs are installed and live variables were inherited as 1.

Live SDK and TUI harness drivers default to the project's own Claude setting sources: the SDK driver loads the copied test .claude/ project settings and hooks, and the TUI driver also loads that project's .claude/settings.local.json, where the person's choice of AI-DLC's status line lives. Both exclude developer user-level hooks/settings. This mirrors the installed framework surface and prevents local interactive preferences from changing test behavior; explicit driver options or AIDLC_TUI_SETTING_SOURCES remain the escape hatch for calibration.

--all --debug (and --release --debug) defaults AIDLC_TUI_LIVE=1 unless the environment already set it. This makes the "everything with traces" profile run the live, token-spending TUI journeys by default; set AIDLC_TUI_LIVE=0 explicitly to keep those files on their in-test SKIP path.

--exclude matches the same names but selects nothing: the files it matches are left out of the tier, and the rest keep their ordinary, unfiltered rules below. With --shard, the shard is chosen first and then the files it matches are left out of it.

The scope runs, the scope-host runs and the guard matrix (t-scope-run-*, t-scope-host-*, t-guard-matrix-*) take minutes per file, so a run with no --filter leaves them out and says so; the full --release and --all acceptance keeps them. Run them alone with --integration -P 8 --filter '^t-(scope-run|scope-host|guard-matrix)-', as their CI jobs do.

An explicit --filter requires execution in each selected file. A file whose cases are all skipped (or which declares no cases) fails the run even when another selected file passes. A partially skipped file still passes if at least one case executes successfully; expect() counts are not the execution signal. An unmatched filter also fails, retaining Test files: 0 and an explanation in failures.txt rather than inventing a failed test.

Unfiltered suites preserve optional skips, reporting all-skipped files as SKIP. --no-llm (or AIDLC_NO_LLM=1) still deliberately excludes Claude-dependent files from mixed selections. If an explicit filter selects only those excluded files, the run fails because no cases executed. Without --no-llm, an explicitly selected Claude file fails when its substrate is unavailable. Other explicitly filtered live files that report all cases skipped also fail: enable their documented live variable and provide the required CLI/authentication. A live opt-in alone is not execution evidence.

The summary and per-file logs report executed and skipped test-case counts separately from the historical Total assertions field (which counts JUnit test cases, including skips). --verbose / --debug retains Bun's <test-name>.junit.xml next to the per-file log when Bun emits it. Coverage failures name the file and remedy in summary.txt and failures.txt; no assertion failure is invented for a skipped case.

Guard Profiles

The default fixture profile keeps synthetic test setup convenient: it sets AIDLC_SKIP_ARTIFACT_GUARD, AIDLC_SKIP_HUMAN_PRESENCE_GUARD, AIDLC_SKIP_SUMMARY_CONFIRMATION_GUARD, AIDLC_SKIP_REVISION_BACKSTOP, and AIDLC_ALLOW_DIRECT_AUDIT_EVENTS to 1 in test children. Other inherited variables retain their existing behavior.

Use --production-guards with --filter for tests that provide real guard evidence and exercise the owning commands or hooks. It applies to every test selected by the invocation and does not change tier selection or live-model gates. For example, exercise the runner's own profile contract:

bash tests/run-tests.sh --debug -P 8 --unit --production-guards \
  --filter '^t-runner-production-guards'

Production children receive 0 for the known guard skips, direct audit/state authority switches, and recordable bypasses, including ensemble evidence, plan approval, reviewer scope, review freeze, and usage tracking. Any inherited AIDLC_SKIP_*, AIDLC_DISABLE_*, or AIDLC_ALLOW_DIRECT_* variable is also forced to 0. This happens after the runner loads .claude/settings.json environment values, so a shell or project-settings off-switch cannot silently change the child's initial guard profile. Explicit zeroes on recordable bypasses also take precedence over aidlc.settings.json / aidlc.settings.local.json bypass lists; simply deleting the variables would allow that fallback.

The runner prints Guard profile: fixture (...) or Guard profile: production (...) at startup and in the stdout summary, and writes the same label into summary.txt and per-file logs under --verbose or --debug. Tests can assert process.env.AIDLC_TEST_GUARD_PROFILE === "production" before exercising a guard; an inherited marker never selects the profile.

This is a child-launch environment contract. Test code and harness helpers can still override their own environment, write settings, or construct synthetic fixtures. In particular, resetAidlcEnv() only deletes AWS_AIDLC_DEFAULT_SCOPE and AIDLC_SKIP_SOURCE_FRESHNESS; it preserves the profile marker and other guard settings, and does not restore production zeroes after a test changes them. A production-profile label alone is not proof that an individual test used production guard evidence. Review the selected test's environment overrides and setup as part of that claim.

The PR workflow's combined check job (Checks (contracts + smoke + production guards)) runs the deterministic runner and recovery contracts in their own step, with production guards enabled:

bash tests/run-tests.sh --debug -P 8 --production-guards --unit --integration \
  --no-llm --filter '^(t-runner-production-guards|t-guard-recovery-production)' \
  > "$GITHUB_WORKSPACE/tmp/ci-guard-contract/production-run.log" 2>&1

The job creates that log directory and collects smoke and guard logs plus tests/logs/ as ci-check-evidence, after sanitization, even when a test fails; a failed upload does not fail the job. The existing test aggregate (Tests (smoke + unit), retained as the required-check name) requires check to succeed alongside smoke, every unit shard, and deterministic integration on every trigger, plus the native-terminal and live OS-isolation matrices outside PR pushes. The production-guard slice runs on every PR push; both cross-OS matrices join it in the merge queue.

Dropping --production-guards from this filtered command fails because t-guard-recovery-production.test.ts executes no journeys under the fixture profile. Passing runner unit tests cannot mask that missing coverage. The unfiltered deterministic integration jobs in the preview's full-suite.yml keep their deliberate fixture profile and report the production journey file as SKIP; the required production jobs in PR CI and Full Suite exercise those journeys. Full Suite additionally requires complete coverage for that selection.

Parallel Execution

--parallel N (or -P N) runs up to N test files concurrently within a tier. Default is serial (1).

When it helps. Live integration and e2e tests spend much of their wall-clock time on CLI startup and model turns. Integration helpers create a fresh project for each test. E2E families with shared profiles or generated outputs additionally need the isolated checkout mode described below.

Spike results (2026-05-06, Opus 4.7 via Bedrock):

All 8 parallel calls observed cache_read=73789. This historical help-command probe observed prompt-cache reuse without throttling or corruption; it does not establish capacity for concurrent full workflows.

What stays serial. Smoke and unit tiers ignore --parallel and run serially within one checkout. Unit CI reduces wall-clock time with isolated shards instead: each shared-workflow call owns a fresh checkout and runs its assigned files serially, so packaging tests can regenerate dist/ without racing readers. PR CI uses ten weighted unit shards on Linux and eight workers for integration, which start the longest integration files first. Manual Full Suite full_verification uses sixteen shards on each of Linux, macOS, and Windows; smoke runs once per OS, and deterministic integration and isolated E2E run in separate jobs with eight workers each. Adding -P 8 to a combined smoke/unit command alone does not parallelize those tiers. The preflight gate (tests/integration/t19.test.ts) also runs serially because the LLM tiers depend on its exit status.

Output under parallelism. START markers stream live; several can appear before the first DONE. In normal/verbose mode, the TypeScript coordinator buffers each test's TAP body and writes it as one block when that file finishes. In --debug mode, Bun stdout/stderr streams live while still being written to each per-test log; parallel debug output is prefixed by file basename so overlapping workers remain attributable. SDK/TUI/Kiro-ACP driver traces are written beside the logs as $LOG_DIR/sdk-drive-*.ndjson, $LOG_DIR/tui-drive-*.ndjson, and $LOG_DIR/kiro-acp-drive-*.ndjson; isolated E2E places them under the file's artifact directory. The runner prints their paths at startup and at each test start. Kiro-ACP traces include tool calls and updates, output previews, permission answers, process stderr, and result/timeout/end events so a timeout can be investigated from retained evidence.

Worker coordination. The TypeScript runner starts one Bun test subprocess per file and tracks active promises. It waits for a file to finish before admitting another when the parallel limit is reached. Each completed file gets a .meta sidecar in $LOG_DIR/_results/; the runner reads those records to populate the summary tables. The shell wrapper delegates to this runner on every platform.

Guidance. For isolated E2E, use -P 8 for checkout capacity and start with a small explicit provider limit such as --bedrock-parallel 2. Divide that provider allowance between simultaneous jobs and machines. Increase it only after measuring representative workflows for throttling, latency and failures. Use --filter to investigate an individual failure.

Isolated E2E Workers

--e2e --isolated-e2e separates worker capacity from live-provider capacity. -P controls the number of checkout workers. Known TUI, Codex exec, Kiro ACP/IDE, and opencode serial files may run concurrently because each worker owns its checkout, generated trees, application profile, temporary fixtures, terminal namespace, and output directory. Unknown .serial. families retain exclusive execution until their shared resources have been reviewed.

Temporary projects live outside Git checkouts so repository discovery cannot inherit the runner's source tree. Failed fixtures are retained alongside the logs after process cleanup. Fresh Claude profiles receive first-run preparation; an owned startup model-upgrade offer is declined through its visible “No” choice to preserve the shipped model pin.

Each file receives a private AIDLC_TUI_BUN_ROOT under a trusted OS temporary directory; confirmed session records and snapshots are then archived beside its profile and logs. Worker cleanup uses the native driver's authenticated session controls and checks daemon retirement, including interrupted starts. It runs after success, timeout, and cancellation before temporary files can be removed or a worker reused. Unconfirmed cleanup halts dispatch and retains the evidence. tmux cleanup remains scoped to the worker's own transport.

The coordinator snapshots the current authored files, including uncommitted changes, and copies the generated distributions once before creating independent worker checkouts. Default local Git clones hardlink object files where possible, copy them across filesystems, and preserve existing alternate stores without extending their reference chain. Linked worktrees with alternate stores use Git transport for the initial snapshot so relative object references resolve correctly; their selected revision is preserved. Installed dependencies are shared; working files and Git indexes are independent. Tests still execute the same files, fixtures, live opt-ins and assertions. Steps within a test keep their existing order. Do not edit source or regenerate distributions while a snapshot is being prepared.

Relative symlinks retain their original targets, including dangling links, so writing through a fixture alias stays inside that worker. Snapshot preparation rejects absolute links and links that escape the source root; the shared dependency directory is an explicit exception. Tracked descendants are checked component by component before sizing or copying; a symlink or junction in an ancestor position is refused rather than followed, even when its leaf appears to be a regular file. Relative IDE seed paths and path-valued executable overrides resolve against the source checkout before the worker starts. The IDE still copies the seed into its private profile.

Inspect the complete selection without building or starting a CLI. Combine --e2e-plan with --e2e: it implies --isolated-e2e but does not select the e2e tier itself.

bash tests/run-tests.sh --debug -P 8 --e2e --e2e-plan

Example for an explicitly enabled Claude TUI selection:

AIDLC_TUI_LIVE=1 bash tests/run-tests.sh --debug -P 8 \
  --e2e --isolated-e2e --filter 't-tui-(?!kiro)' \
  --bedrock-parallel 2 \
  --e2e-timings tests/logs/<previous-stamp>/summary.txt

--e2e-timings is optional. Historical durations determine scheduling order, never selection: new files still run. Without history, source timeout values provide rough ordering hints. The longest eligible file starts first; work without the saturated resource can use free workers. Bedrock, Kiro and IDE limits are independent, and an IDE file consumes both a Kiro slot and an IDE slot.

These limits count test files, not HTTP requests. A workflow can spawn multiple model-using agents internally. Claude SDK/TUI, Codex and default opencode runs share the Bedrock budget even when configured regions differ. Kiro ACP/TUI/IDE share the Kiro budget. Account authentication remains supplied by the host. Generated IDE seeds prepare onboarding and still require a signed-in host. Budgets apply to one coordinator: divide the account budget between simultaneous jobs or machines instead of giving each job the entire allowance. These provider limits apply to isolated E2E dispatch. Integration still uses its ordinary per-tier scheduler; bound live SDK integration with filtered file lanes when running it alongside another host, and keep deterministic integration in a separate checkout. Unit shards also need separate checkouts because packaging tests regenerate shared outputs within their checkout.

Native Windows Codex files also share a windows-codex serial group. Fresh profiles select the documented elevated Windows sandbox, whose setup uses host accounts; concurrent initialization across profiles has not yet been verified. One Codex file runs at a time while unrelated harnesses can use free workers and provider slots. Run only one Codex coordinator on a Windows host. Use a logged-in interactive session for these native runs; the validated runtime's sandbox commands could not start from the SSH service session. Linux and macOS Codex files continue to use the Bedrock limit.

On Windows, use the same native runner with --e2e --isolated-e2e --filter 't-ide-kiro', set AIDLC_KIRO_IDE_LIVE=1, and choose --ide-parallel. Each IDE launch copies its seed to a private profile and lets Electron allocate the debugging port. The parent waits for that child's endpoint instead of selecting a port using its PID. Begin at one IDE file, then compare two using observed memory and completion times. The native command is bun tests/run-tests.ts --debug -P 8 --e2e --isolated-e2e ...; this path does not depend on the older Windows wrapper's fixed installation directories. Windows profiles use the short per-file temporary directory to keep Electron's nested database and log paths within its path limits. Test artifacts retain their original paths. Teardown requests closure through the owned browser's CDP endpoint and waits for the child to close; a cleanup failure preserves the profile and remains visible alongside any earlier test failure. The IDE adapter also retains an event-provided session ID on prompt submission, so session identity remains available when a workspace startup callback was not observed. Prompts without a session ID preserve the existing fallback state.

Keep the Windows desktop session logged in for GUI runs. Check the account panel in a disposable profile before treating an installed IDE as authenticated. The IDE and CLI have separate login checks; a signed-in IDE does not satisfy the Kiro CLI or Cursor CLI gates. Keep profile seeds on the Windows host and collect test results and traces separately from their user-data directories.

The runner writes these additional artifacts under its timestamped log directory:

• e2e-plan.json: exact selected inventory, estimates and resource requirements.
• e2e-events.ndjson: dispatch/completion events, worker IDs and queue times.
• e2e-results.json: selected/prerequisite files, source revision/dirty state, worker assignment, case pass/fail/skip counts, durations, timeout/cleanup outcomes and diagnostic throttle-pattern counts.
• e2e-artifacts/<test>/: retained JUnit and isolated SDK/TUI/IDE traces.
• e2e-artifacts/<test>/deferred-cleanup.json (under attempt-N/ in isolated-file runs): Windows Codex fixture retention after the coordinator verifies native process retirement. These fixtures move to retained-fixtures/ even on success; the host owns final deletion of the protected sandbox files. A cross-volume copy keeps the original too.
• failed-fixtures/<label>-<pid>-<n>/: a bounded snapshot of a failed live SDK fixture (t183, t193), taken by tests/harness/failed-fixture.ts because the fixture itself lives under the OS temporary directory that a hosted runner discards. The workflow record under aidlc/ is copied first; links are never followed, node_modules/ and .git/ are skipped, and per-file, file-count, total-size and walk caps apply. retained-fixture.json counts every entry left out by reason and lists the first of them by path. The live collectors copy it with the rest of the log tree and the sanitizer redacts it before upload.
• e2e-worker-storage.json: checkout pool location, estimated snapshot size, and whether checkout copies were retained. Windows pools use short private paths under the system temporary directory so deep report paths do not break fixture copies.

The Kiro reviewer journey saves its primary artifact and matched review receipt before assertions. The scope-exclusion journey saves its final state and audit, including on timeout. That journey uses a fully specified miniature security patch while retaining whole-workflow completion and the scope-derived forbidden stage checks; setup and both SDK drives share the original case deadline.

Per-file output is saved incrementally. An outer deadline or interrupted run remains a failure/incomplete result; it is never converted to a passing retry. Verbose/debug runs retain per-file JUnit and an .execution.json record of effective coverage gates for every tier. These records contain coverage controls, not the full process environment or credentials. Every test file runs under an owned process supervisor, including ordinary smoke/unit/integration execution. A debug-log write failure stops further dispatch, retires admitted test processes, and leaves a nonzero runner result. Ordinary files also receive private temporary directories and terminal namespaces, which lets the runner retire native sessions after a test aborts. If the result volume itself is unwritable, missing captures remain incomplete. They cannot establish successful coverage.

Codex live tests use tests/harness/codex-test-lifecycle.ts to retain the original failure and full exec output while reserving cleanup time. On Windows, verified runner-owned fixtures are removed only after the coordinator retires their process tree; failures preserve those fixtures with the diagnostic artifacts. The exec driver disables the interactive request_user_input feature, which Codex exec cannot service, and exercises the skill's prose approval fallback. Approval assertions and sandbox permissions remain required.

Windows provisioning fixtures require their temporary accounts and processes to be retired before returning. A profile hive still held by Windows services is recorded for VM disposal only on GitHub-hosted runners; persistent hosts must delete it within the bounded cleanup operation. Windows deterministic E2E jobs download the checksum-pinned Codex command runner and set AIDLC_CODEX_RUNNER_PROBE so the native bootstrap regression runs alongside the existing filesystem and account tests. It makes no model calls. For a local diagnostic, point that variable at the verified runner executable; optionally set AIDLC_CODEX_RUNNER_PROBE_ONLY=1 to select the bootstrap and launcher checks. That diagnostic selection skips the other provisioning cases and is not a full coverage result.

Worker preparation checks free space for the snapshot and checkout copies, plus a reserve of 512 MiB. Each isolated file checks the reserve before starting. AIDLC_E2E_MIN_FREE_BYTES overrides the reserve in bytes (a nonnegative integer). This admission check cannot predict all runtime fixture growth. Ordinary assertion failures retain their fixtures and diagnostic records, then release the disposable checkout pool. AIDLC_KEEP_TEMP=1 or unconfirmed process cleanup retains checkout copies for investigation. Scheduling does not extend live workflow deadlines. The terminal substrate preflight runs before selected TUI work, including filtered selections. Normal assertion failures do not prevent unrelated files from running.

The revision-loop TUI test runs its clean and reject/revise/approve journeys concurrently in separate projects, Claude profiles, and terminal sessions. Both must reach the same completion milestone within one 60-minute file budget, with time reserved for cleanup. Each journey retains its own terminal state, native fidelity evidence, and cleanup result.

The test also stops waiting after a confirmed, completed root HTTP 5xx failure in its fresh Claude transcript. It records the provider error and native transcript, stops its owned answer-gate client, and performs normal terminal cleanup. This does not retry the model, erase the failure, or change the healthy workflow deadline. Tool output, subagent errors, recovered history, and errors after the test's completion milestone do not trigger this early failure.

Use --e2e-cancel-file <path> when a controller needs portable cancellation, especially on Windows where terminating a process does not deliver POSIX signals. Creating that file stops dispatch and terminates active owned workers. POSIX SIGINT/SIGTERM uses the same cleanup path. Forced host/process termination cannot execute cleanup; the incremental results still show unfinished files.

The legacy summary and failed-file exit convention remain available. Failure exit codes cap at 255 so a run with 256 unfinished files cannot wrap to a successful process status; the summary retains the complete failure count. For a required nightly gate, add --require-coverage: skipped cases, empty files, unexecuted selected files, and an empty selection produce a nonzero exit. Every verbose run writes coverage.json with the selected inventory and case counts, and prints Coverage: COMPLETE or INCOMPLETE separately from test failures. Use platform-specific selections for required gates; intentional platform skips in a broad discovery run do not satisfy them. The effective inventory includes automatically required preflights, even under a narrow filename filter. requestedFiles counts the original selection; selectedFiles includes those prerequisites once. A terminal prerequisite must produce valid, nonempty, passing evidence with no skipped cases before a TUI journey can start. JUnit evidence must be a complete document whose suite totals match its testcase records. A positive header in a truncated report cannot satisfy the gate. Final cleanup/publication errors set the run to ERROR, independently of the testcase outcomes. e2e-results.json additionally distinguishes an all-skipped file from executed coverage, and coverageComplete is false for skipped, empty or incomplete files. A no-LLM or platform-specific run may legitimately have incomplete live coverage; aggregate the expected platform/provider jobs before claiming full nightly coverage. Throttle-pattern counts are diagnostic matches in output, which can include quoted text; they do not establish a provider quota failure.

Required jobs across platforms

tests/native-terminal-profile.json assigns deterministic terminal controls to Linux/Bun, macOS/Bun, Windows/Bun and Linux/tmux jobs. Portable controls run on all three operating systems; native Linux and macOS containment, Windows Job Object and sharing-handle checks, and compatibility backends have explicit owners. The profile targets Linux arm64, macOS arm64 (macos-15) and Windows x64; declare separate matching obligations for another architecture. Platform-specific controls live in separately selectable test files. Their assertions remain required by the profile. A case added to a file the profile names must be added to each job that runs the file: on Linux the unit tier's t-native-terminal-profile.test.ts lists every case of those files without running them and fails on one the profile does not list, so a PR meets it before the Full Suite does.

Prepare one plan before dispatch, using the authored source that every host will receive:

bun tests/reconcile-tests.ts prepare \
  --profile tests/native-terminal-profile.json --cohort "nightly-$(date -u +%Y%m%dT%H%M%SZ)" \
  --repo . --output tmp/native-plan.json

The plan contains the exact job/file/testcase inventories, cohort ID and a digest of authored source, including uncommitted bytes. Copy the same source bytes and plan to the other host. Keep plans and results in ignored or external storage. Root paths, timestamps and Git commit IDs alone are not source identity.

Run each job with matching tier/filter flags and the declared backend. For example, derive a precise filter for the native profile's Linux job:

TEST_MATRIX_PLAN=tmp/native-plan.json
TEST_MATRIX_JOB=linux-bun
TEST_MATRIX_FILTER=$(bun -e '
  const plan = await Bun.file(process.argv[1]).json();
  const job = plan.jobs.find(job => job.id === process.argv[2]);
  if (!job) throw new Error("unknown matrix job");
  const names = job.files.map(({path}) => {
    const parts = path.split("/");
    if (parts[0] !== "tests") throw new Error("expected native-profile test path");
    return `${parts[1]}-${parts.at(-1).replace(/\.test\.ts$/, "")}`
      .replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
  });
  console.log(`^(?:${names.join("|")})$`);
' "$TEST_MATRIX_PLAN" "$TEST_MATRIX_JOB")
AIDLC_TUI_BACKEND=bun bash tests/run-tests.sh --debug -P 8 \
  --unit --integration --e2e --no-llm --isolated-e2e \
  --filter "$TEST_MATRIX_FILTER" \
  --matrix-plan "$TEST_MATRIX_PLAN" --matrix-job "$TEST_MATRIX_JOB" \
  > tmp/native-linux-run.log 2>&1

--matrix-plan and --matrix-job imply strict coverage. The runner verifies the actual source, platform, architecture, selected backend and effective file inventory before dispatch. It seals test-matrix-receipt.json after cleanup and report publication, recording the actual Bun version, whitelisted live gates, JUnit hashes and validated testcase outcomes. Source changes, missing cases, skips and finalization failures prevent a successful receipt. Isolated jobs also hash the worker's authored source before and after each file, before that checkout can be reused or removed; the observations remain in matrix-worker-source.json. Reconciliation refuses an output path that would overwrite its plan, receipts or referenced JUnit evidence. On Windows, Bun 1.3.14 can prematurely time out a synchronous child after an asynchronous pause. Source verification retries only an early ETIMEDOUT from its read-only Git queries, at most once and with the remaining original budget. It logs that recovery; real deadline exhaustion, other errors and test failures are not retried by this mechanism. t-test-source-timeout.test.ts checks both the budget contract and the real asynchronous-gap regression.

Collect each receipt together with its stamp directory, preserving relative JUnit paths, then reconcile all required jobs:

bun tests/reconcile-tests.ts reconcile --plan tmp/native-plan.json \
  --receipt "<linux-bun-stamp>/test-matrix-receipt.json" \
  --receipt "<darwin-bun-stamp>/test-matrix-receipt.json" \
  --receipt "<windows-bun-stamp>/test-matrix-receipt.json" \
  --receipt "<linux-tmux-stamp>/test-matrix-receipt.json" \
  --output tmp/native-matrix-result.json

Only a complete matching matrix exits zero. Missing jobs, wrong source/cohort or backend, changed JUnit, real failures and conflicting receipts remain failures; a later passing receipt cannot erase an earlier failure. Explicitly excluded profile obligations appear as NOT_REQUESTED, never as fulfilled coverage.

This profile verifies the terminal driver and retained compatibility controls. The scheduled preview calls full-suite.yml, which reconciles native receipts and requires every declared job to pass. Skipped live jobs fail the suite; documented excluded families remain untested.

Nightly full-suite matrix and provisioning

Full Suite is callable with an explicit ref and manually dispatchable (default main, both verification flags false). Its ordinary plan accepts the requested branch or commit, including an unmerged PR, and resolves it once. All matrix legs check out that immutable SHA. The preview and stable release workflows enforce their own publication source policy; Full Suite does not require a commit to be on main just to test it. Scheduled and manual preview-release.yml runs call it after packaging determinism, typecheck, lint, and installer shell checks, even when the source already has a published preview. Preview does not call the PR CI test matrix again; Full Suite owns its test coverage. A failing Full Suite does not stop the preview build. The Release tests job renders a report of the failed legs, failed jobs and failing test cases with scripts/ci-preview-test-report.ts. It writes the report to the run summary and a preview-test-report artifact. Passing or failing, the report ends with the answers corpus's nightly reading (### Answers corpus): the replies that passed, the misses by kind (setup, the check's own when the agent never showed the question; record, question, words) and each missed item by id, from the summary the live check leaves in its full-suite-live-* artifact, or a line that none was there. That section is report-only and never goes into the published notes. The published preview notes then open with a warning and end with that report. The planned notes stay whole. The report, and when even that leaves no room the warning, gives way first, so a failing suite never stops a preview whose planned notes fit GitHub's 125,000-character release limit. The update-from-previous job (scripts/ci-update-from-previous.ts) updates an install of the last stable release to the candidate on Linux, macOS and Windows and runs every harness project's hooks, refresh and doctor. Each project holds the person's own committed app, .gitignore and AGENTS.md, and their work waits at a gate: before the update, on the last release's own engine and hooks, the scripted stand-in (tests/harness/work-open.ts, on that tool's scope host) starts a bugfix, answers the Requirements questions and stops for the day at the open Requirements gate, in all seven projects side by side. After the refresh, the person types /aidlc --resume in a new chat and approves. The check fails when the gate does not come back, the stage runs again, an answer is asked for again, the approval is not backed by the person's reply, the next stage does not start, or a file they committed changed outside AI-DLC's marked part. When the stand-in cannot leave work open on the last release at all, the check says so in its own line. A second leg of the job (--channel copy) does the same for the copy channel: each project is runtime/<harness>/ copied from the last release's copy runtime, checked against its .sha256 and set up with its own config --from, and refreshes with that command from the candidate's copy runtime, as the guide's Copy Channel says. When either leg fails, the preview still publishes, its notes open with a warning and the report gains a line for it; release.yml does not publish a stable release that fails it. Release result requires successful preview publication or an intentional skip. Full Suite runs separately: the preview records its run ID, downloads that run's evidence and reports its real result, without propagating its failure. An unchanged source skips publication but still receives the test report. Stable promotion validates the selected preview's source and the new version commit, then builds and checks native binaries, installers, lifecycle flows, upgrades, checksums and provenance without repeating source CI or Full Suite. Deterministic source tiers and production guards remain in PR/merge CI and manual full_verification; the ordinary release-purpose Full Suite omits them. Preview runs contract checks and Full Suite once, with publication deduplication applied only to the subsequent build and publication chain.

For user-approved full validation of an unmerged PR, select its branch and set ref to that branch's exact workflow-head SHA:

gh workflow run full-suite.yml --ref '<candidate-branch>' \
  -f 'ref=<exact-workflow-head-sha>' -f full_verification=true

Full verification runs every job that receives no credentials: native obligations and reconciliation, all deterministic tiers, production guards and Windows release contracts. Because it may select unmerged code, it never runs the three live_prepare_* jobs, live_linux, live_macos or live_windows (the jobs that request OIDC and AWS credentials); cover a candidate's live families with live_verification, the separately authorized mode below. Every checkout in Full Suite sets persist-credentials: false, so candidate code never finds the repository token on disk. It otherwise uses the same file matrices, assertions and timeouts as an ordinary Full Suite run. The separate full-suite-verification-result artifact contains full-suite-result.json with purpose: "full-verification"; passed requires every other job to succeed, the six live/preparation jobs to be skipped, verificationFamily: "all" and omittedLegs naming exactly those six jobs. Neither preview nor stable publication consumes this result, even after the candidate merges.

full_verification exists only on workflow_dispatch. It requires the checked-out SHA to equal the selected workflow head, is mutually exclusive with live_verification, and rejects family/file filters. No push or pull-request event automatically starts privileged full verification.

For user-approved live-only candidate validation, manually dispatch Full Suite on the candidate branch with live_verification=true and set ref to that branch's exact workflow-head SHA:

gh workflow run full-suite.yml --ref '<candidate-branch>' \
  -f 'ref=<exact-workflow-head-sha>' -f live_verification=true

To repeat only one family while other coverage is already running, add -f verification_family=codex. The manual choices are all (default), claude-sdk, claude-tui, codex, copilot, and opencode. A family other than all requires live-verification mode and the same manual event/exact-head checks. It selects only that family's rows on Linux/macOS/Windows, retaining the original per-platform N/M shard numbers. The separate Windows release-contract job is omitted for scoped verification. No family selector is exposed to reusable callers, and ordinary release-purpose runs must use all.

For one test file, also pass its exact repository path:

gh workflow run full-suite.yml --ref '<candidate-branch>' \
  -f 'ref=<exact-workflow-head-sha>' -f live_verification=true \
  -f verification_family=claude-sdk \
  -f verification_test=tests/integration/t238-user-stories-mob.sdk.test.ts

The file must belong to the selected family. Unknown paths and mismatched families fail planning. It runs on its declared platforms: portable tests use all three OSes, while a Windows-only case uses Windows. Preparation covers only those runners. Original shard identities remain intact, and the result records verificationTest, verificationPlatforms and any omitted job explicitly. This selection is not consumed by stable publication.

These verification inputs exist only on workflow_dispatch, never workflow_call. Authorization requires that event and that the checked-out SHA equals github.sha; selecting the workflow on main cannot authorize a different branch's source. No push or pull-request trigger starts privileged verification. The mode runs plan, the three live_prepare_* jobs, live_linux, live_macos, live_windows, and release_contract_windows. It intentionally skips native terminal/reconciliation, deterministic tiers, and production guards, so it does not repeat deterministic CI. Existing provider opt-ins, strict live coverage and credential isolation remain in effect.

The separate full-suite-live-verification-result artifact contains full-suite-result.json with purpose: "live-verification", verificationFamily, the omitted jobs in omittedLegs, and complete: false. Its passed requires every live job to succeed and every intentionally omitted job to be skipped; missing, failed, cancelled or unexpectedly executed jobs fail. Even a successful verification of main is not consumed by stable publication. Ordinary Full Suite runs keep the full-suite-result artifact name and require all jobs except the explicitly omitted deterministic and production-guard jobs; older artifacts without the release purpose do not satisfy the Full Suite result policy.

The three independent live_prepare_* jobs call live-prepare.yml to install dependencies and package projections with contents-read permission only. Each live OS job waits only for its own platform preparation. POSIX preparation selects official Node 22.23.2 and packs its complete install prefix with the pinned CLIs (ci-live-deps.py pack --cli ... --node-runtime ...). The validated archive requires both bin/node and its library directory. The isolated runtime copies that prefix into its root-owned tools tree and puts node/bin on PATH, preserving loader-relative libraries. Node and CLI startup are checked after runner directories are protected. Copying a lone Homebrew executable loses dependencies such as @rpath/libnode and is not supported. Windows CLI installation stays inside its isolated user. This closes #1306: installers never run with OIDC in scope, and credentialed lanes only validate and unpack prepared bytes. Every authorized Full Suite run executes preparation and hosted live jobs using the existing ai-pr-review environment. There is no separate live opt-in switch in this release workflow. Missing prerequisites, skipped jobs, or failed tests fail an ordinary Full Suite run. They do not block preview publication: the preview still builds, its notes end with a Full Suite failure report, and only the separate Full Suite run stays red. Stable promotion preserves those results and warnings; it does not repeat the suite or claim that a previous failure passed. Stable publication still requires the new binaries to pass installation and upgrade checks. The credential-free Windows release-contract job also runs.

The declared coverage is:

• In manual full_verification only: deterministic smoke, sixteen independent unit shards, integration and isolated E2E on Linux, macOS and Windows. Integration and E2E have separate jobs, each with eight workers and its own runner process and evidence. Every unit file is assigned to one shard per OS; each shard retains its own debug logs and results.
• In manual full_verification only: a Linux production-guard slice selects --production-guards and --require-coverage for the runner and recovery contracts. Those cases are therefore exercised even though ordinary fixture-mode tiers skip them.
• Source-bound native terminal obligations on Linux arm64/Bun and tmux, macOS arm64/Bun (macos-15), and Windows/Bun. Hosted Windows supplies Node for other tooling; Bun installs the pinned dependencies.
• Claude SDK, Claude TUI, Codex, Copilot CLI, opencode and release-endpoint contracts on hosted Linux/macOS/Windows, including Claude plugin invocation in the strict SDK leg. Copilot CLI runs in its own-model (BYOK) mode against the credential broker, on the Mantle model Codex uses, so no GitHub sign-in, Copilot seat or token is involved. Its release journey (tests/e2e/t-exec-copilot-release-journey.serial.test.ts) is what a person does on day one: set up through the copy channel, doctor, the documented first step, the plan question, and Reverse Engineering approved, with a message record for every line typed; a second case leaves the folder untrusted and requires doctor to name the trust fix. Codex's golden journey (tests/e2e/t-exec-codex-journey-golden-bugfix.serial.test.ts, Linux and Windows) fixes the same greeter from the first line to Code Generation approved, with the plan question left for the day and approved in a new session; Copilot's (tests/e2e/t-exec-copilot-journey-golden-bugfix.serial.test.ts, Linux only: on Windows it runs past the 25 minutes a retry allows) does the same on the copy-channel install of its release journey. opencode's (tests/e2e/t-run-opencode-journey-golden-change.serial.test.ts, Linux) takes the same request to Reverse Engineering, where the person asks for one change in their own words and approves the revision, on the shipped permission table.
• Kiro ACP, TUI and IDE are declared exclusions pending a dedicated isolated Windows desktop host; local live runs remain their coverage path.
• Cursor is excluded: no credential separation is available because its vendor CLI reads the API key from the agent environment. No exclusion is reported as successful coverage.

scripts/ci-live-filter.ts --list prints the discovered family partition. After authorization and dependency installation, the plan emits --matrix linux, --matrix macos and --matrix windows as the dynamic matrices of the live_linux, live_macos and live_windows jobs. Each row's shard: N/M selects a duration-balanced group for its family/platform. Each OS has two Claude SDK (five on Linux, which alone runs the golden journeys and the answers corpus, the corpus on a shard of its own), three Claude TUI, one Codex (two on Linux and Windows, where the Codex golden journey runs), one Copilot and one opencode shard: 29 live harness jobs in all. Timing weights in tests/live-shard-weights.json are scheduling hints; discovery still assigns every eligible file exactly once per platform. Each OS has its own concurrency cap: four Linux, two macOS and three Windows live jobs. Every shard has at most two file workers; Windows Codex files remain serial. Windows release-contract coverage stays in its separate unsharded job.

--isolated-files restores each worker from the original checkout snapshot before its next file, including generated and Git state. Each file/attempt gets fresh HOME/USERPROFILE, temporary directories, Claude/Codex/opencode and XDG profiles, a private Git config and a private copy of the broker-only AWS config. Installed tools and dependencies are reused. The existing process/transport retirement checks must succeed before a slot is reused. Cleanup uncertainty stops dispatch and marks remaining files incomplete.

Stock permission settings: the two golden journeys drive Claude Code in its default permission mode with the project's shipped settings (setting sources user, project and local; user is the drive's scratch config folder) and answer every prompt yes. A prompt for AI-DLC's own command or record (one that names aidlc/ or .claude/), or a write outside the project, fails the journey; every other prompt is listed in its log (tests/harness/stock-permissions.ts). The other Claude SDK drives keep bypassPermissions. The Codex --json journeys run the shipped workspace-write sandbox, and a command naming .codex/ or aidlc/ that the sandbox refused fails the turn. Copilot CLI runs with --allow-all-tools (AI-DLC ships no Copilot CLI allow rules, so the person's own --allow-tool rules decide), the Claude terminal journeys skip permission prompts, and opencode already runs without --auto.

--file-retries 1 retries only a short failed file after confirmed cleanup, in fresh state, while retaining both attempts under e2e-artifacts/<file>/attempt-N. (Without --isolated-files, the same flag retries smoke, unit and integration files in a fresh process; the merge queue uses that, as described above.) Timeouts, incomplete evidence, cleanup failures and person-turn failures are not retried; a person-turn failure also prints a Person-turn failure error line. A pass on the retry prints a Flaky live test warning. A file that failed because the model service did (the SDK drive's ModelApiError: Claude Code ended the run on a 429, a 5xx or a dropped connection, never a 4xx the request caused) keeps its one retry and prints a Model service failure warning; when every failed case is the service's, its line in failures.txt and the preview report reads "the model service failed, not AI-DLC: infrastructure, not a test result", and a real failure beside one keeps the file's ordinary reason. The Windows live job keeps the isolated child's output out of its log, so it says these lines from the collected e2e-results.json (bun scripts/ci-retry-report.ts --live tests/logs), in fixed words and the test file's name only. Passing files are not rerun. Each shard retains its authenticated isolation and required capability preflights. Manual --family FAMILY --test <repository-path> selects only that file on its declared platforms while preserving its shard identity. Dependency preparation uses fast gzip compression and uploads the resulting archive without a second compression pass to shorten startup.

Every Full Suite job name leads with its runner OS, then its lane and matrix item, for example Linux / claude-tui 3/3, macOS / deterministic unit-3, Windows / native-terminal bun or Windows / release-contract, so the Actions UI groups a run's jobs by OS. Result legs keep the job ids, and the preview report groups failed jobs by OS and lane.

Each credentialed job requests a 3,600-second session from the existing role. Jobs have an 80-minute limit and live test steps have a 70-minute limit. Every live family receives a 3,600-second shared runner budget and independent file deadline, including ordinary integration/SDK files. Driver work reserves up to five minutes within that envelope for cleanup; collection continues after failures and timeouts. An older journey's longer local timeout does not extend the hosted budget. Exhaustion is a visible failure with incomplete coverage.

bun scripts/ci-live-filter.ts claude-tui --platform linux prints an anchored runner filter. Discovery uses each file's own integration/e2e live gate variables, the derived Claude substrate list, and the unit release-contract opt-in; using a generic plugin helper does not claim coverage for every provider it supports. Unit gate-fixture tests are deterministic, not live families. PLATFORM_ONLY names the files that run on some platforms only, and, with no platform, the on-demand checks that open only with a switch of their own that no live job sets (the input corpus check); it never hides a skipped case inside a selected file. All live provider legs use --require-coverage; release-contract files retain platform-conditional cases and do not use that flag.

Outside the native profile, deterministic and release-contract results are job outcomes, not proof that every individual case ran somewhere in the OS matrix. Their debug artifacts retain per-file case counts, skipped cases and JUnit. There is no source-bound inventory assigning all such cases to applicable OSes and reconciling those receipts across jobs. Applying --require-coverage to each whole tier would also reject legitimate platform-inapplicable cases; a green job does not convert those skips into passes. Full case coverage across OSes remains an explicit gap until that inventory and reconciliation exist.

Add --args to print the complete runner arguments, one per line. The script owns tier selection: it emits only tiers with selected files and applies each family's strict-coverage policy. Live integration and E2E files use --isolated-files; unit release contracts retain their existing execution path. Production-guard journeys receive the production profile independently, including when they share a shard with fixture-profile files. The workflow uses --run to spawn the runner directly from the repository root, preserving each argument without shell word splitting or Bash-version-specific builtins:

bun scripts/ci-live-filter.ts claude-sdk --platform linux --run -- --debug -P 8

For a selection containing e2e files, append --e2e-plan to inspect its plan without executing tests. Do not append it to an integration-only or unit-only selection: that mode intentionally rejects an empty e2e selection. Deterministic --no-llm runs omit the closed Claude health preflight and record its file as skipped, without treating the deliberate skip as a prerequisite failure.

Artifacts are full-suite-native-plan, full-suite-native-<job> (complete log stamp directories and JUnit), full-suite-native-result, full-suite-production-guards, full-suite-deterministic-<suite>-<OS> (suite is smoke, unit-1 through unit-16, integration, scope-runs, scope-hosts, guard-matrix-1, guard-matrix-2, or e2e), full-suite-scope-runs-<OS>, full-suite-scope-hosts-<OS>, full-suite-guard-matrix-1-<OS> and full-suite-guard-matrix-2-<OS> (the release scope_runs job), full-suite-live-<family>-<slice-number>-<OS>, full-suite-live-release-contract-Windows, and the purpose-specific result (90-day retention): full-suite-result for purpose: "release", full-suite-live-verification-result for "live-verification", and full-suite-verification-result for "full-verification". The final JSON records sha, runId, runAttempt, purpose, verificationFamily, coveragePolicy, passed, complete, every job's result in legs, disabledLegs: [], omittedLegs, and live families declared with hosting: "excluded" in the sorted excluded list. For purpose: "release" under required-hosted-live-shards-v3, passed requires a 40-hex commit ID, exactly deterministic and production_guards omitted and skipped, and every other declared job, scope_runs included, successful. Missing, failed, cancelled or unexpectedly skipped required jobs fail. disabledLegs is retained so the Full Suite result policy can reject historical disabled-live reports. complete additionally requires no excluded families; it remains false with the documented Kiro and Cursor exclusions and is not the preview-publication predicate. Those exclusions warn without failing the suite; disabled required jobs fail it. The optional inspector, scripts/ci-full-suite-evidence.ts check, accepts a result only when sha is the requested commit, runId is the run it came from, purpose is "release", verificationFamily is "all", coveragePolicy is required-hosted-live-shards-v3, passed is true, disabledLegs is empty, and omittedLegs is exactly the two deterministic/production-guard job IDs. Those jobs must be skipped; all other declared jobs and extra legs must have succeeded. It does not require complete, so the documented exclusions only warn. Promotion does not invoke this inspector or require it to pass. Neither job success nor this policy marker asserts full case coverage across OSes.

Native jobs use the Bash wrapper with --debug -P 8 and their unchanged matrix plan/job selectors. Their artifacts include tests/logs/ plus the sanitized root tmp/full-suite-native/ capture and literal stamp path; native receipt reconciliation still discovers the receipts recursively.

Shared deterministic artifacts contain tests/logs/<stamp>/ and tmp/ci-deterministic/ (full stdout/stderr plus the literal stamp path). CI artifacts use ci-deterministic-<suite>-<OS>, where suite is smoke on expanded macOS/Windows runs, unit-1 through unit-10 on Linux, unit-1 through unit-16 on Windows and manual macOS, unit-1 through unit-2 on merge-queue macOS, integration, scope-runs, scope-hosts, guard-matrix-1, guard-matrix-2, or e2e (expanded matrix only). Linux smoke and production-guard evidence instead share ci-check-evidence, including tmp/ci-smoke/, tmp/ci-guard-contract/ and tests/logs/. An upload requires both log locations to pass sanitization, including after a failed test command.

Kiro ACP/TUI/IDE live families are declared exclusions in the nightly full suite, printed as warnings and leaving complete: false. They need a dedicated isolated Windows desktop host running Kiro under a separate low-privilege identity. Local runs with AIDLC_KIRO_ACP_LIVE=1, AIDLC_KIRO_TUI_LIVE=1 or AIDLC_KIRO_IDE_LIVE=1 remain the coverage path. A follow-up issue tracks the hosted lane.

Before selecting a preview for stable promotion, a maintainer reviews its recorded Kiro CLI release-journey results for the selected source SHA, one for each Kiro tree: kiro, which setup lists as "Kiro CLI (V2)", and kiro-ide, which setup lists as "Kiro" and whose .kiro/settings/cli.json pins Kiro CLI to its v3 engine. Both run in Kiro CLI; neither drives Kiro IDE. If no result exists for that SHA, run the journey (tests/e2e/t-tui-kiro-release-journey.serial.test.ts, the same journey as the Copilot lane's, in the interactive kiro-cli chat terminal, once per tree) on the preview's source commit, on a machine with a signed-in Kiro CLI:

bun scripts/package.ts
AIDLC_KIRO_TUI_LIVE=1 AIDLC_TUI_LIVE=1 bun tests/run-tests.ts --e2e --production-guards \
  --require-coverage --filter t-tui-kiro-release-journey

--require-coverage fails the run when either tree's journey was skipped. When one tree already has a result for that SHA and the other is missing or failed, put AIDLC_KIRO_RELEASE_TREE=kiro or AIDLC_KIRO_RELEASE_TREE=kiro-ide in front of the command: only that tree's journey runs, and --require-coverage then covers that tree alone. On Linux it gives Kiro a scratch home and keeps the machine's sign-in, so ~/.kiro is not touched. Each journey takes about five minutes; the credits follow the model Kiro CLI uses (about 33 each on claude-opus-5.5 with Kiro CLI 2.32.0). It prints one line per tree, for example Kiro CLI 2.32.0 (kiro-ide) release journey on <commit>: PASS (5.1 min of turns, 33.23 credits, 3 permission prompts), and both lines are recorded, including the full source SHA, in the selected preview's GitHub release notes before promotion. Existing results for that source are reused; the promotion workflow does not rerun the journey on its new metadata commit. Nothing in the workflows reads those manual results: the maintainer reviews them in step 2 of Promote a preview to production. A fix ships in another preview before it is selected for promotion.

The live jobs use the existing ai-pr-review environment and role. Verify these prerequisites before running Full Suite:

• Environment ai-pr-review supplies secret AWS_AI_PR_REVIEW_ROLE_ARN. live_linux, live_macos and live_windows select that environment; the secret is resolved inside those jobs. A workflow that calls full-suite.yml must pass secrets: inherit: without it the secret resolved empty in called runs, and every live job failed at "Assume nightly Bedrock role". Each live job now checks the secret first and fails at once with a message naming secrets: inherit. No Kiro/Cursor API-key workflow secret or hosted vendor-key leg is supported. The AWS role's OIDC trust must be scoped to this repository's environment:ai-pr-review subject. Each assumption requests 3,600 seconds, within the existing one-hour role duration. The workflow splits work into bounded file jobs without changing IAM session duration.
• The role needs Bedrock invoke/stream access to the CI-only model table in scripts/ci-credential-broker.ts: Claude Fable 5, Opus 4.8, Sonnet 4.6 and Haiku 4.5 inference profiles in us-east-1, opencode's Sonnet 4.6 default in that region, and Codex openai.gpt-5.5 through Bedrock Mantle in us-east-2, which Copilot CLI's own-model mode uses too. The table pins the broker allowlist and Claude client aliases; shipped user settings remain provider-neutral.

Missing environment access, role trust, model permissions, or required live substrates fails readiness; the workflow does not create a new environment or role.

Claude Code, Copilot CLI and opencode npm versions are pinned in the workflow; update those pins deliberately after verifying the registry and compatibility. CLI/authentication preflights fail loudly rather than letting absent substrates masquerade as passing live tests.

Credential isolation and uploaded evidence

Live models receive repository content and can invoke tools, so treat their shells and tool-result traces as untrusted sinks for credentials. The AWS action returns masked step outputs without exporting AWS credentials to subsequent steps. A trusted startup helper receives those values only in its own step env, sends JSON over stdin to a detached, environment-scrubbed broker process, and exits before live agents run. The broker stores credentials only in memory and performs signed STS GetCallerIdentity at startup; only its nonsecret account, ARN and loopback port leave the broker. There is no credential-returning route, credential process, credentials file or credential-bearing agent environment.

The loopback proxy admits only the configured model IDs and POST /model/<id>/(invoke|invoke-with-response-stream|converse|converse-stream) (optionally under /bedrock), plus Codex Mantle POST /openai/v1/responses with model: openai.gpt-5.5. It removes client authentication headers and signs each upstream request; successful streams are forwarded byte-for-byte. Other routes are forbidden, redirects are not followed, and upstream error bodies are suppressed because AWS errors can echo signatures.

Claude uses documented ANTHROPIC_BEDROCK_BASE_URL and CLAUDE_CODE_SKIP_BEDROCK_AUTH=1; its t19 preflight checks the broker's startup identity, followed by a real SDK turn. Codex 0.151.0 uses a provider base_url ending /openai/v1, verified against a loopback endpoint; the service-specific Bedrock Runtime override alone does not redirect Codex's Mantle traffic. Every scratch Codex home uses the shared Bedrock configuration renderer, including compose, workspace and memory journeys. Outside the isolated CI runtime, leaving AIDLC_CODEX_AWS_PROFILE unset or empty omits the profile from config.toml, allowing the AWS SDK default credential chain to resolve credentials, including an EC2 instance role. To keep using an existing named profile, set AIDLC_CODEX_AWS_PROFILE=codex (or its actual name); a profile named codex is no longer selected implicitly. AIDLC_CODEX_AWS_REGION still defaults to us-east-2. The isolated CI runtime explicitly sets AIDLC_CODEX_AWS_PROFILE=codex to select its dummy broker keys on Linux, macOS and Windows. Opencode profiles also contain only dummy keys. Opencode selects AWS_PROFILE=broker so its prerequisite check recognizes that profile; its documented provider endpoint override routes AI SDK requests through the proxy. Codex shell policy excludes provider/broker/API/GitHub/Actions variables, and the runner strips CI control-plane credentials before launching any live test.

Hosted Linux/macOS live agents run as the separate unprivileged aidlc-live user in a private checkout copy, using an explicit sudo ... env -i environment and root-owned readable/executable tools. They cannot read the launcher process's procfs environment, runner home, original checkout or Actions command files.

Linux Codex preparation installs distro bubblewrap and exposes a root-owned symlink to /usr/bin/bwrap on the live PATH, preserving its AppArmor attachment. When Ubuntu restricts unprivileged user namespaces, preparation loads the existing /etc/apparmor.d/bwrap-userns-restrict profile, or installs the distro's extra profile from apparmor-profiles with apparmor-utils. The global restriction must remain unchanged. This follows the official Codex sandbox prerequisites; Codex retains workspace-write. Before AWS setup, the scrubbed live user must create a bubblewrap user namespace, execute Bun and write in its private temp directory. Even as root inside that namespace, it must not read the runner home, original checkout, Actions environment file, launcher environment, or temporary runner-owned/root-owned sentinel files. The proof repeats after broker startup; failure blocks live execution.

Windows creates a standard Users-only account and ACL-isolated work/home/tools under C:\aidlc-live; Task Scheduler launches each body with a Limited batch logon under that identity, avoiding the runner session's desktop ACL. Preparation grants only SeBatchLogonRight while preserving existing principals, then verifies an actual batch-logon task. Preparation tasks, including Git and smoke probes, default to 30 minutes. Credentialed test tasks retain a 64-minute ceiling inside the workflow's 70-minute live test step. Tasks not started within the five-minute native-startup backstop fail with scheduler status and the last 20 operational events. Explicit safe environments and UTF-8 identity/cwd/output logs remain, and tasks are unregistered after completion. Secondary-logon Start-Process is a reported fallback only if task registration itself fails. Batch sessions may use session 0: the proof requires correct user identity and access denied for launcher modules/PROCESS_VM_READ, runner directories and private credential state. The broker stays under the runner identity; only nonsecret routes and model pins cross into the live user's environment. Failed isolation proofs block execution.

Windows Codex also provisions its two native sandbox identities and verifies both using fresh test homes. A provisioning run that exits non-zero runs once more on the same fresh host, its logs kept beside the first run's; a timeout or an unfinished output drain still fails closed at once. The readiness command runs through PowerShell and Bun under the actual sandbox: the shell must start in the project, Bun must canonicalize its path, and credential reads and writes outside the workspace must remain denied. The sandbox identities receive directory metadata and traversal access on the three private containers above the fixtures, with no inherited permission to list their contents or read their files. Readiness checks the native initializer's exit status independently of its fixed phase diagnostics on stderr; a missing or nonzero status blocks execution. The native launcher allows five minutes for fresh-home initialization to accommodate hosted Windows cold-start variation, then refuses execution if that deadline expires. Each native CLI process is assigned to its own Windows job before being resumed. After the CLI exits, the launcher retires that job's descendants before draining stdout and stderr, so inherited pipe handles cannot hold the invocation open. POSIX collection stops the dedicated account's processes before administrator copying. On macOS it first retires that account's launchd user/GUI domains to stop service restarts. Zombie entries cannot execute; any other remaining process, or an inventory failure, refuses collection. Failures include the remaining PID/state rows. Only this job's newly created account is affected.

Every merge-queue run executes test_live_isolation on all three OSes, using the same preparation and proof scripts plus --smoke --filter '^t01' under the sandbox identity, without provider credentials. Credential-free Windows release-contract units remain separately runnable. Hosted Windows live model coverage is real, not an opt-in stub or a same-user exception.

Agents can still spend through the allowlisted proxy until the job timeout; constrain IAM model permissions, quotas and runner access. This boundary trusts the host kernel and administrators; OS privilege escalation is not in scope. Cursor remains excluded because its CLI exposes the API key to agent tool shells. Kiro uses only the dedicated CI-identity Windows host's existing sign-in, without workflow-injected API keys. Runner-owned collection copies completed logs back for sanitization before upload; it does not execute sandbox-authored code.

After Windows sandbox processes have stopped, test logs and launch logs are collected independently. A failed test-tree copy still preserves validated launch stdout/stderr under tests/logs/windows-launch-<uuid>/. Only complete, validated trees are published; collection remains failed when either copy fails. tests/logs/windows-collection-<uuid>.json records completion per source and, on failure, the operation, safe relative path and exception codes. It omits exception messages, absolute paths and sensitive path components. An uploaded collection report or fallback log does not establish that a test passed.

Windows live legs also keep evidence for a hook that stops making progress. scripts/ci-live-sandbox.ts sets AIDLC_TEST_HOOK_TRACE=1 for every Windows live family, and the runner binds AIDLC_HOOK_TRACE_DIR to each file attempt's e2e-artifacts/<file>/attempt-<n>/hook-trace/, so every hook process in that attempt writes its phases there (see Hook phase trace). While a live run's scheduled task runs, the runner-side wait loop in .github/scripts/prepare-live-runtime.ps1 checks the process table once a minute. When a process owned by the isolated account with engine hook or engine adapter in its command line has run 10 minutes, it writes one hook-stall-<time>.json to tests/logs/windows-launch-<uuid>/hook-stalls-run-<id>/ with the stalled process's tree (its isolated-account parents and all its children) and their thread states, once per process. It reads process metadata only and checks ownership only for the stalled process and its parents. Every query is capped at what is left of a 60-second snapshot budget (15 seconds at most); a query that is skipped, fails or times out marks the snapshot truncated. A stalled process whose owner could not be checked is listed under ownerUnknown with only its id, name and start time (no command line, no children), since it may belong to another account. It never fails the run. The hook trace's .ndjson files follow the trace-retention rule below. Linux and macOS legs turn neither on.

Every full-suite tests/logs/ upload first runs scripts/ci-sanitize-logs.ts and is blocked if sanitization fails. Full-suite and shared deterministic jobs retain eligible driver NDJSON, sdk-drive*, tui-drive* and e2e-artifacts/**/traces by default so tool calls, completion boundaries and timeout behavior remain available for diagnosis. Repository variable AIDLC_NIGHTLY_UPLOAD_TRACES=0 opts out of trace retention. The standalone sanitizer still deletes these traces unless its AIDLC_NIGHTLY_UPLOAD_TRACES environment variable is 1. Invalid UTF-8, UTF-16, NUL-containing and other non-text files are always deleted, with their relative paths and reasons recorded in sanitizer-report.json; there is no binary/screenshot allowlist. Remaining UTF-8 text is redacted for AWS credentials, vendor keys, bearer tokens and Anthropic keys; links are removed without following targets. Redaction remains defense in depth, not proof against encoded secrets; retaining raw text traces increases that residual risk.

Kiro prompt-hook transport controls

Kiro's agent-v1 prompt hook has a default max_output_size of 10 KiB, independent of the shell tool's output budget. The adapter counts the complete UTF-8 pre-dispatch packet, including its explanatory text, before injecting a non-steering directive. Oversized packets and every load-steering directive use the existing exact-argument forwarding latch so the engine's full JSON is returned through the actual tool call. --single skips hook pre-dispatch entirely, leaving its first issuance to that tool call. Only a line the engine reads exactly is pre-dispatched at all (readNextLine): anything it would hand to the agent goes to the agent untouched, so this seam never acts on the person's words before anyone has read them.

t147-kiro-hook-adapter.test.ts checks the 10 KiB boundary with multibyte content, native shell aliases, retained terminal guards, and a real multipart engine round trip. It reconstructs every delivered rule byte and passes each opaque continuation token unchanged. A token surviving alone does not prove complete rule delivery; moving it before truncated rules is not a valid repair.

These deterministic controls test the adapter and engine stdout. They do not calibrate a native Kiro release's tool-channel ceiling. Before changing that transport budget, verify complete near-limit payloads on each supported native platform. Increasing a hook's configured max_output_size also requires that calibration; it does not change this adapter's conservative injection policy.

Unit Sharding

--unit --shard N/M assigns every discovered unit file to exactly one of M duration-balanced shards. Assignment is deterministic and uses tests/unit-shard-weights.json, which records each file's duration per OS. The current platform's measurements take precedence; files not measured there use their slowest-OS fallback. Unlisted files receive a one-second default weight, so new tests join the least-loaded shard without changing the command. The runner exits 2 when M exceeds the number of assignable groups, so no valid shard command can report success after running zero files.

Each shard remains serial. Run separate shards in separate checkouts or CI jobs. Do not run them concurrently against one repository tree because packaging tests regenerate dist/ and can race tests that read generated files.

The affinity list keeps cross-file prerequisites explicit. The native binary builder test currently runs before the Copilot compiled-adapter coverage in the same shard. After all native build/install assertions pass, t238 copies its verified executable and adjacent runtime/ into the runner-owned AIDLC_TEST_COMPILED_DIR. Its private build/install fixtures are still removed after the test; the published copy lives until runner cleanup (or is retained with verbose logs). Each runner assigns a fresh handoff directory, so no shared build/binaries output is overwritten or borrowed from a prior run. Sharded unit execution requires t249 to resolve this handoff; a missing artifact fails instead of silently skipping the compiled cases, even when an explicit executable or an old repository build exists. Direct, non-sharded t249 runs can still opt into AIDLC_TEST_COMPILED_EXECUTABLE or a local native build result. The smoke runner contract verifies that both the ten- and sixteen-shard assignments are non-empty, disjoint, cover the complete unit inventory, preserve producer/consumer ordering, and fail compiled coverage when the producer is filtered out.

With more than one worker, the integration tier starts its parallel files longest-first by tests/integration-weights.json (keyed by summary row name), so a long file never starts last and sets the tier's wall time. Unweighted files and ties keep name order; with no readable weights file, files start in name order. Serial files and the smoke and unit tiers are unaffected.

--integration --shard N/M balances the filtered integration inventory using the same integration weights. Filtering happens before partitioning; exclusions apply afterward, so excluding a file does not move other files between shards. The two guard jobs use this mode with the same filter. The number of hosted jobs and the worker limit remain unchanged.

Weights only balance and order files; they never select, skip, or fail one. scripts/ci-test-weights.ts refresh <dir> rewrites both weight files from deterministic CI evidence downloaded with gh run download <run-id> -p 'ci-deterministic-*' -D <dir> (use several green merge-group runs: each file takes its median per OS; the slowest median also updates its fallback). Platform measurements not present in a refresh are retained. The initial platform weights were collected from successful merge-group run 39029896320; scheduling gains must be checked against subsequent runs, not just those training measurements. After each unit and integration job, the advisory "Report outdated test weights" step runs scripts/ci-test-weights.ts report: a file that ran more than a minute and more than 1.5 times past its weight gets a warning annotation and a row in the job summary naming the refresh command. The step always succeeds, so a stale weight slows CI but never fails a job or drops a PR from the merge queue. t-ci-test-weights.test.ts covers the refresh, the report, and the checked-in weight files.