Testing

Overview

The AI-DLC test suite is entirely TypeScript — every test is a t*.test.ts file run under bun, with zero shell (.sh) test files. This is the platform-invariance guarantee by construction: the same files run identically on macOS, Linux, and native Windows.

The suite is organized into four levels — smoke, unit, integration, e2e — one directory each under tests/. The four levels map onto the classic three-layer test pyramid that balances speed vs. thoroughness:

            /\
           /  \    ACCEPTANCE — full workflows, artifact + experience verification
          / L3 \   Level: e2e  ·  When: before releases (--release / --all)
         /------\
        /        \
       /   L2     \  STAGE — individual stages with stub input, verify artifacts
      /------------\ Level: integration  ·  When: CI push (--ci, every PR)
     /              \
    /      L1        \  PROTOCOL — contracts, structure, cross-references
   /------------------\ Levels: smoke + unit  ·  When: every local change

The --ci profile and the no-flag default both run smoke + unit + integration (so the integration level rides along on every local bun tests/run-tests.ts); --release / --all adds e2e. The pyramid above shows where each level sits conceptually — the profile flags below are how you actually select them.

Filename convention. A test's filename is t<NN>[-description].test.ts — just the level directory it lives in and an optional human description. There is no mechanism segment in the name: a test's mechanism (whether it spawns a CLI, drives the SDK, or renders a live TUI) is a derived set computed from the drivers its body actually calls, not declared in the filename. The machine-checked index of what each test covers lives in tests/.coverage-registry.json (generated by tests/gen-coverage-registry.ts from the covers: headers on disk), not in a hand-maintained table here — see Test Registry below.

Layer 1: Protocol (every change, no LLM, seconds)

Verifies the orchestrator's structural correctness without invoking the LLM. If these pass, the protocol is internally consistent — stages reference valid files, inputs/outputs chain correctly, routing tables match stage files.

Levels: smoke, unit, integration

What it tests: - File existence, permissions, naming conventions (smoke) - Hook scripts (11 TypeScript via bun), stage frontmatter, knowledge inventory (unit) - Scope-stage mapping, graph consistency, stage I/O contract chains, protocol compliance (integration) - Stage output-to-step validation: all declared outputs referenced in instruction steps (integration, deterministic via the aidlc-validate.ts CLI tool)

Run: bun tests/run-tests.ts (default, no flags needed). bash tests/run-tests.sh is a compatibility wrapper for existing POSIX commands.

Layer 2: Stage (CI push, LLM, minutes)

Runs individual stages in isolation with known workspace + state fixtures. Verifies each stage produces correct artifacts when given deterministic input.

Levels: integration

What it tests: - Preflight health gate: Claude CLI on PATH, AWS credentials valid, Claude responds (exit 0), response non-empty (preflight) - CLI tool utility handlers: intent-birth, --doctor, --status, --stage, --phase (integration) - Individual stages with greenfield/brownfield stubs, artifact verification (integration)

Run: bun tests/run-tests.ts --ci

Layer 3: Acceptance (release, LLM, hours)

Runs full workflows and verifies the experience: beyond state transitions, it checks artifact content, cross-stage coherence, and domain correctness.

Levels: e2e

What it tests: - Full bugfix lifecycle with brownfield stub + artifact assertions - Full POC lifecycle with greenfield stub + artifact assertions - State progression, scope routing, audit completeness, jump mechanics - LLM semantic review of stage instruction quality (clarity, logical flow, ambiguity detection)

Run: bun tests/run-tests.ts --release

Cross-Platform Coverage

The test suite runs on macOS, Linux, and Windows through the native Bun runner:

bun tests/run-tests.ts [--ci | --all --debug -P 8]

bash tests/run-tests.sh ... remains as a POSIX compatibility wrapper and delegates to the same TypeScript runner. At runtime this implementation's hooks, CLI tools, and test runner require bun; Bash is no longer the primary runner substrate.

Portability constraints baked into the suite:

• Paths: createTestProject in tests/harness/fixtures.ts normalizes temporary project paths so they round-trip cleanly through JSON and native bun.
• In-place edits: Prefer TypeScript file writes in tests. If a shell helper is unavoidable, avoid BSD/GNU-specific sed -i forms.
• grep -qiF: Git Bash has a known bug combining -i and -F. Use -i alone if your pattern has no regex metacharacters. Tests hit this in t16 before it was fixed.
• tar archives: macOS tar injects ._* AppleDouble sidecar files by default. When bundling source for cross-platform test runs, use COPYFILE_DISABLE=1 tar … or git archive.
• LLM timing on Windows: Bedrock calls from Windows EC2 can be meaningfully slower than from macOS (first-call cold start, MSYS process fork overhead). SDK/tui tests should assert on driver result surfaces and let the runner's preflight/per-file Claude gate separate absent substrate from real failures.

Running the suite on Windows manually:

1. Install bun, Node.js, and the Claude Code CLI.
2. Install Git for Windows if you are running the full suite or the POSIX wrapper compatibility smoke; the native runner path itself does not require Bash.
3. For e2e TUI tests, install the dev dependencies with npm so node can resolve node-pty and @xterm/headless.
4. Set AIDLC_NODE_BIN to the concrete node.exe path and set AIDLC_TUI_LIVE=1 for a full acceptance run.
5. Run bun tests/run-tests.ts --all --debug -P 8.

No WSL or Docker is required; the supported validation substrate is native Windows.

Repeatable MR10 Windows EC2 runbook:

1. Stand up a disposable Windows Server 2022 host with SSM access:

aws cloudformation deploy \
  --stack-name aidlc-windows-test \
  --template-file tests/harness/windows/windows-test.cfn.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides VpcId=vpc-... SubnetId=subnet-...

1. Sync the committed git tree under test:

bun tests/harness/windows/sync.ts --stack-name aidlc-windows-test HEAD

1. Install repo dev dependencies on the box:

bun tests/harness/windows/ssm-run.ts --stack-name aidlc-windows-test -- \
  powershell -ExecutionPolicy Bypass -File C:\aidlc\tests\harness\windows\setup.ps1 -ProjectDir C:\aidlc

1. Run the Windows --all gate with live TUI enabled:

bun tests/harness/windows/ssm-run.ts --stack-name aidlc-windows-test -- \
  powershell -ExecutionPolicy Bypass -File C:\aidlc\tests\harness\windows\run-all.ps1 -ProjectDir C:\aidlc -Parallel 8

1. Tear down the host:

aws cloudformation delete-stack --stack-name aidlc-windows-test

run-all.ps1 exports AIDLC_NODE_BIN and AIDLC_TUI_LIVE=1 before invoking bun tests/run-tests.ts --all --debug -P <N>, so a green result cannot come from silently skipping the live TUI journeys. It probes the claude binary across C:\Users\Administrator\.local\bin and the systemprofile home, since the native installer drops claude.exe under whichever user ran the CloudFormation UserData bootstrap (Administrator under EC2Launch v2).

The stack defaults to c5.4xlarge — the proven size for the full --all -P 8 live run. The e2e tier carries per-test bun:test timeouts (the Bolt-worktree lifecycle test lands at ~5.5s of its 5s budget on c5.4xlarge), so a smaller box (e.g. t3.large) tips deterministic Bolt/runtime tests into spurious timeouts under parallel load. Shrink the InstanceType parameter only when running a lighter tier selection.

Preflight Validation

Before running unfiltered live-capable levels (integration or e2e), the runner executes tests/integration/t19.test.ts as a gate. It drives a tiny real turn through the Claude Agent SDK (the same live path the integration tier uses) and asserts only on deterministic surfaces. If the preflight fails, deterministic files still run and Claude-dependent files are skipped with per-file SKIP entries.

A red here is a real environment finding (missing claude, expired creds), never a flake to soften — exactly the gate's job of bailing the downstream LLM tiers fast.

Test Registry

The suite is discovered, not registered: bun tests/run-tests.ts walks the four level directories (tests/{smoke,unit,integration,e2e}/) and runs every t*.test.ts it finds. There is no hand-maintained per-test table to keep in sync — adding a test file is all it takes for the runner to pick it up.

What each test covers is tracked mechanically in tests/.coverage-registry.json, generated by bun tests/gen-coverage-registry.ts from the covers: header in a test file's leading comment block (typically line 1; a few files legitimately declare none and simply contribute no coverage claim). The generator enumerates the framework's units across seven classes (function, audit, scope, stage, hook, subcommand, render-surface), maps each covers: claim onto an enumerated unit, and emits the coverage counts plus a ratchet floor. Regenerate and verify drift with:

bun tests/gen-coverage-registry.ts          # rewrite the registry from disk
bun tests/gen-coverage-registry.ts --check  # fail if the committed registry is stale

tests/.coverage-registry.json is the authoritative, machine-checked index — consult it (or grep the covers: headers directly) to find which test exercises a given function, audit event, scope, stage, hook, subcommand, or render surface. The --check mode is wired into the suite so a registry that drifts from disk reds the gate.

Note: t19 appears in both unit (tests/unit/t19.test.ts, the jump CLI tool) and integration (tests/integration/t19.test.ts, the live preflight gate) — a level/file path, not a bare ID, disambiguates such collisions.

Trigger Points

L1 can be enforced via a git pre-commit hook: bun tests/run-tests.ts || exit 1.

Stubs

Greenfield Stub: tests/fixtures/greenfield-todo/

A project description with no source code. Workspace-detection classifies as greenfield. Gives the LLM deterministic intent context for ideation stages.

Contents: Just README.md describing a React Todo App with TypeScript and Vite.

Brownfield Stub: tests/fixtures/brownfield-todo/

Minimal React+TypeScript+Vite source (~10 files, ~200 LOC). Workspace-detection classifies as brownfield. RE, requirements, and design stages have concrete code to analyze.

Contents: - package.json — react, react-dom, typescript, vite, vitest - tsconfig.json, vite.config.ts, index.html - src/main.tsx, src/App.tsx - src/types/todo.ts — Todo interface (id, title, completed) - src/components/TodoList.tsx — list + add form (~40 lines) - src/components/TodoItem.tsx — checkbox + title + delete button - src/hooks/useTodos.ts — addTodo, toggleTodo, deleteTodo

RE Artifacts Fixture: tests/fixtures/re-artifacts/

Pre-seeded reverse-engineering output for downstream stage tests. Copied into the test project's space-level repository store at $PROJ/aidlc/spaces/default/codekb/<repo>/ during setup.

Contents: 4 minimal .md files (architecture-overview, technology-stack, codebase-analysis, integration-points) describing the brownfield-todo app.

Inception Artifacts Fixture: tests/fixtures/inception-artifacts/

Pre-seeded inception phase output for tests that jump into construction. Copied into $PROJ/aidlc/spaces/default/intents/<record>/inception/{requirements-analysis,application-design,units-generation}/ during setup.

Contents: 7 minimal .md files (requirements, components, component-methods, services, component-dependency, unit-of-work, unit-of-work-story-map) describing the Todo app. Unit name: todo-core.

Construction Artifacts Fixture: tests/fixtures/construction-artifacts/

Pre-seeded construction phase output for tests that jump to mid-construction stages (e.g., code-generation). Copied into $PROJ/aidlc/spaces/default/intents/<record>/construction/todo-core/functional-design/ during setup.

Contents: 1 minimal .md file (functional-design) describing the todo-core unit's component specs and state management.

State Fixtures

How to Add a Stage Test

1. Choose the stage to test and identify what state fixture it needs (the state must show that stage as the current/next stage)
2. Create or reuse a state fixture in tests/fixtures/
3. Create tests/integration/tNN-stage-SLUG.test.ts and use the shared TypeScript harness helpers (tests/harness/fixtures.ts, tests/harness/sdk-drive.ts, or tests/harness/tui-drive.ts) rather than shell TAP helpers.
4. Run with bun tests/run-tests.ts --integration or directly: bun test tests/integration/tNN-stage-SLUG.test.ts

How to Add Acceptance Assertions

To add artifact assertions to an existing e2e workflow test under tests/e2e/:

1. Read the current test and understand what it already checks
2. Add expect(...) assertions inside the existing test(...) block (bun:test counts assertions from the calls themselves — there is no plan line to keep in sync)
3. Use flexible patterns: match /[Tt]odo/ against readFileSync content, not exact strings
4. Use test.skipIf(...) / an early return for assertions that depend on non-deterministic LLM output format
5. Use expect(statSync(path).size).toBeGreaterThan(minBytes) for size-bound checks

Assertion Design Principles

• Keyword classes — Use case-insensitive regex: [Tt]odo, [Rr]eact, [Bb]rownfield
• Flexible discovery — Use find + wc -l to count files rather than checking exact names
• Size bounds — Use statSync(path).size with toBeGreaterThan() for minimum content
• Graceful degradation — Use skip when an assertion depends on non-deterministic LLM output
• Structure over content — Check for markdown headings (^#), file existence, directory creation before checking content

Environment Variables

CLI Reference

# Entrypoints
bun tests/run-tests.ts        # Native cross-platform runner
bash tests/run-tests.sh       # POSIX compatibility wrapper

# Level flags (combinable)
--smoke         # Structural validation
--unit          # Single-component isolation
--integration   # Cross-component contracts and stage/CLI utilities
--e2e           # Full lifecycle, worktree, and rendered terminal journeys

# Profile flags (shortcuts)
(default)       # smoke + unit + integration
--ci            # smoke + unit + integration
--release       # smoke + unit + integration + e2e
--all           # Same as --release

# Output modifiers
--verbose       # Write per-test logs to tests/logs/
--debug         # Implies --verbose; streams per-test output and writes SDK/TUI
                # driver traces to tests/logs/
--filter PAT    # Only run tests whose filename matches extended regex PAT
--parallel N    # Run up to N test files concurrently within a tier (alias: -P N).
                # Default: 1 (serial). Smoke and unit tiers are always serial.

Live SDK and TUI harness drivers default to project-only Claude setting sources. That means they load the copied test .claude/ project settings and hooks while excluding developer user-level hooks/settings. This mirrors the installed framework surface and prevents local interactive preferences from changing test behavior; explicit driver options or AIDLC_TUI_SETTING_SOURCES remain the escape hatch for calibration.

--all --debug (and --release --debug) defaults AIDLC_TUI_LIVE=1 unless the environment already set it. This makes the "everything with traces" profile run the live, token-spending TUI journeys by default; set AIDLC_TUI_LIVE=0 explicitly to keep those files on their in-test SKIP path.

Parallel Execution

--parallel N (or -P N) runs up to N test files concurrently within a tier. Default is serial (1).

When it helps. Integration and e2e levels each spend most of their wall-clock on claude -p subprocess startup and LLM turns. These tests are already filesystem-isolated — setup_integration_project scaffolds a fresh $PROJ per test — so they can run side-by-side without interfering.

Spike results (2026-05-06, Opus 4.7 via Bedrock):

All 8 parallel calls observed cache_read=73789 — Bedrock prompt caching stays warm across concurrent workers. No throttling or corruption observed at 8-way.

What stays serial. Smoke and unit tiers ignore --parallel and run serially regardless. They already complete in seconds and their interleaved output would hurt debuggability for no wall-clock gain. The preflight gate (tests/integration/t19.test.ts) also runs serially because the LLM tiers depend on its exit status.

Output under parallelism. START markers stream live (several can appear back-to-back before the first DONE — that's the visible signal workers are concurrent). In normal/verbose mode, each worker's TAP body is buffered and flushed to stdout as one contiguous block under a directory-mutex (mkdir $LOG_DIR/.stdout.lock, atomic on POSIX — works on macOS bash 3.2 without flock). So ok/not ok lines from different files never interleave; stdout reads top-to-bottom like a serial run, just with the file completion order determined by how long each test took rather than dispatch order. In --debug mode, Bun stdout/stderr streams live while still being written to each per-test log; parallel debug output is prefixed by file basename so overlapping live workers remain attributable. SDK/TUI/Kiro-ACP driver traces are written beside the logs as $LOG_DIR/sdk-drive-*.ndjson, $LOG_DIR/tui-drive-*.ndjson, and $LOG_DIR/kiro-acp-drive-*.ndjson; their exact filenames depend on process IDs and TUI session names, so the runner prints the glob at startup and at each test start. The Kiro-ACP trace records the live kiro-cli acp turn event-by-event (spawn, prompt start, each tool_call/tool_call_update with its verbatim output preview, permission answers, the spawned process's stderr, and the terminal result/timeout/end), so a session/prompt timeout can be diagnosed after the fact — distinguishing a turn that was progressing (real tool calls firing) from one that stalled.

Worker coordination. The parent backgrounds run_bun_test_file with & and holds a slot gate via jobs -rp | wc -l. Each worker writes an atomic .meta sidecar to $LOG_DIR/_results/; the parent reads them after wait to populate the summary tables. macOS ships bash 3.2.57 (no wait -n), so the gate polls every 200ms — negligible next to minute-long LLM calls.

Guidance. Start with --parallel 4. Raise to 8 if Bedrock capacity and your bill tolerate it. Drop back to serial for debugging a single failing test — or use --filter to isolate it.