> ## Documentation index
> Fetch the complete documentation index at: https://rustynes-git-release-v3-6-0-aaf8d1.docs7.io/llms.txt
> Ask it with GET https://rustynes-git-release-v3-6-0-aaf8d1.docs7.io/ask?q=<keywords or question>. Short keywords rank best.
> Use these to discover all available pages before exploring further.

# Testing strategy

# Testing strategy

**References:** `ref-docs/research-report.md` §Source manifest (test ROMs);
`ref-docs/nesdev-wiki-technical-report.md` §Test Strategy; Nesdev
[Emulator tests](https://www.nesdev.org/wiki/Emulator_tests) and
[Tricky-to-emulate games](https://www.nesdev.org/wiki/Tricky-to-emulate_games).

## Purpose

Define the layered testing approach that takes the project from "compiles" to "passes the cycle-accuracy bar set by Mesen2 and ares". Tests are the primary specification — when written-down rules conflict, the test ROM behavior wins.

## Layer 1 — unit tests (per crate)

Each crate has its own `#[test]` coverage in `src/` and `tests/`:

- `rustynes-cpu`: every opcode × every addressing mode (~600 cases). Flag-update tables for ADC, SBC, CMP, BIT. Property tests via `proptest` for arithmetic flag correctness.
- `rustynes-ppu`: register read/write semantics; OAMADDR rules; loopy `t/v/x/w` updates per the documented rules; sprite evaluation FSM (with the buggy `n+m` increment).
- `rustynes-apu`: register layout; frame counter sequence per mode; length counter halt timing; envelope/sweep arithmetic; mixer table values (compared against the closed-form formula within 0.1%).
- `rustynes-mappers`: per-mapper bank-resolution tables; MMC1 serial protocol; MMC3 IRQ counter (deterministic toggle test); bus conflict cases.

Aim: > 90% line coverage on the chip crates.

## Layer 2 — golden-log compare (CPU only)

`nestest.nes` is the canonical 6502 golden master. The test harness:

1. Forces PC to `$C000` (nestest's automated test entry point).
2. After each instruction, captures `(PC, A, X, Y, P, SP, CYC, PPU dot, scanline)`.
3. Diffs against the bundled `nestest.log` (Nintendulator-generated).
4. First mismatch → test fails, prints the diff.

## Layer 3 — test ROM corpus (subsystem coverage)

The full blargg + kevtris + community test ROM suite, vendored in `tests/roms/` (all are CC0 / public-domain individual ROMs from `christopherpow/nes-test-roms`). Each ROM is run by `rustynes-test-harness::run_until_complete()`, which:

1. Steps until `$6000` reads `$80→$00..$7F` (test complete) or `$81` (needs reset) — automated reset on `$81`.
2. Reads the result code at `$6000` and the message string at `$6004` onward.
3. Compares result code against expected (usually 0); fails the test with the message on mismatch.

| Category | ROMs | Pass target |
|----------|------|-------------|
| CPU instruction | `instr_test_v5/*` (16 sub-ROMs) | All |
| CPU reset/power | `cpu_reset` (2) | **Closed (Phase 7 T-71-002):** power-on register dump asserted strictly; full interactive reset protocol `#[ignore]`'d (not headlessly driveable); semantics covered by `Cpu::power_on`/`Nes::reset` unit tests |
| CPU timing | `cpu_timing_test6`, `instr_timing` (2) | **`instr_timing` closed (Phase 7 T-71-003):** both sub-ROMs strict-pass on the full `Nes` |
| CPU branches | `branch_timing_tests/*` (3) | All |
| CPU interrupts | `cpu_interrupts_v2/*` (5) | All except the 3 C1-axis residuals (`#[ignore]`'d, deferred under the ADR-0002 axis) |
| CPU dummy reads/writes | `cpu_dummy_reads`, `cpu_dummy_writes_*`, `instr_misc` (5) | **`instr_misc` closed (Phase 7 T-71-003):** all 5 strict-pass on the full `Nes` (incl. `04-dummy_reads_apu`) |
| PPU VBL/NMI | `ppu_vbl_nmi/*` (10) | All |
| PPU open bus | `ppu_open_bus` | Pass |
| PPU sprite | `sprite_overflow_tests/*` (5), `sprite_hit_tests_2005.10.05/*` (10), `ppu_sprite_hit/*`, `oam_read`, `oam_stress` | All |
| APU | `apu_test/*` (8), `apu_mixer/*` (4) | All |
| APU PAL (v2.1.5) | `pal_apu_tests/*` (10, forced PAL, on-screen verdict via `run_nes_screen`) | 10/10 strict-pass: `01`/`02`/`03` (region-independent length/table/IRQ-flag) + `04`-`08` (PAL frame-counter-timing, passing since the v2.1.5 region-gated PAL step positions) + `10`/`11` (length halt/reload timing, passing since the v2.1.5 deferred halt/reload write-ordering fix in `length.rs`). Corrects a prior `$6000` false oracle (these NROM ROMs have no PRG-RAM). NTSC byte-identity preserved. Documented in `docs/apu-2a03.md` + `docs/accuracy-ledger.md` |
| APU NTSC (v2.6.2) | `blargg_apu_2005.07.30/*` (11, on-screen **numeric result code** via `run_nes_result_code`) | 11/11 strict-pass. **The verdict was read wrongly twice before this.** (1) The suite asserted `$6000 == 0` through `run_nes_blargg`; these NROM ROMs have no PRG-RAM, so `$6000` reads `0` forever and `0` is blargg's *success* code — eleven vacuous assertions, the same false oracle corrected for the PAL half in v2.1.5 and never migrated here. (2) `run_nes_screen` is also wrong for this corpus: these ROMs never print `PASSED`/`FAILED`, so it returns `Unresolved` for all eleven while the screen reads `$01`. Their `tests.txt` states the convention — *"a result code of 1 always indicates that all tests were passed"*. Demonstrated to fail: half-frame 14913→14914 caught by 2 ROMs, frame IRQ 29828→29827 by 7; the quarter clock is **not** caught and `tests.txt` says why (the corpus does not test envelope, sweep or linear-counter clocking). `vacuity_of_the_6000_protocol_on_this_corpus` pins reason (1) executably |
| DMC DMA | `dmc_dma_during_read4/*` (4) | All |
| MMC3 | `mmc3_test_2/*` (6), `mmc3_irq_tests/*` (6), `mmc3_test` v1 (6) | `mmc3_test_2` 1-5 strict, including `4-scanline_timing` (all 13 sub-tests since v3.0.0, T-MMC3-BG-A12), and `mmc3_test` v1 1-5 strict; v1 `6-MMC6` fails only at its alternate-revision assertion and `mmc3_test_2/6-MMC3_alt` tests the NEC revision, both by design (the project models the Sharp MMC3) |
| TASVideos / extended (C1) | `dpcmletterbox` (DMC-IRQ raster split, visual smoke) | Frame-hash sentinel; committable corpus only — see below |
| Mapper coverage | `holy_mapperel`, `holy_diver_battery_test`; `vrc24test` → in-tree VRC2/4 unit tests + `m22` baseline (T-71-005) | Pass for implemented mappers |
| Input | standard-controller strobe/read tests (T-71-004); DMC-conflict / Four Score / Zapper documented in `compatibility.md` | Standard-pad path strict; expansion devices deferred |
| Accuracy battery | `AccuracyCoin` (single ROM) | Pass-rate target ≥ 90% by v1.0 |
| Battery save (v1.7.0 F1) | `battery_save.rs` (synthetic battery-backed NROM) | `$6000` PRG-RAM round-trips through `snapshot`/`restore`; restore is bit-identical |
| Sub-v2.0 accuracy audit (v1.7.0 F2) | `f2_accuracy_audit.rs` | F2(a) length halt/reload race (`blargg_apu_2005/10`,`11`) strict; F2(b) DMC load even/odd defer (`dmc_tests/latency` audio sig + `sprdma_and_dmc_dma` strict) |
| Extra-scanlines overclock (v1.7.0 F3) | `extra_scanlines.rs` | Byte-identical at `0`; first-frame image-invariant; CPU-cycle growth in band; deterministic |

**v1.7.0 "Forge" Workstream F — accuracy hardening (dot/CPU-cycle-granular).**
F1 added the battery-save round-trip oracle (none existed) and audited the
existing test-ROM bundle (all wired + green; `vbl_nmi_timing/5.nmi_suppression`
kept as a live pin since it passes on this core; holy-mapperel M28/M118/M180 ROMs
are a documented corpus carryover). F2 confirmed the length-counter halt/reload
race and the DMC load-DMA even/odd defer are already implemented and added named
regression pins. F3 added the determinism-gated PPU extra-scanlines overclock
(off by default, byte-identical at zero). None of these alter the default build's
behavior, so AccuracyCoin holds 139/141 — the two newest upstream PPU tests ("ALE + Read", "Hybrid Addresses") added by the v2.0.1 catalog re-sync are known gaps; the 139 previously assigned tests all pass.

## Layer 4 — golden framebuffer / audio comparison

For a curated corpus of freely-distributable demos (NESDev compo entries, homebrew releases under permissive licenses):

- Run for 600 frames with deterministic reset and zero controller input.
- Capture frame 60, 180, 300, 600.
- Compare pixel-exact against a stored reference (generated initially by hand-validation against Mesen2).
- For audio: compute PSNR against a stored reference WAV. Fail if PSNR drops below threshold.

This catches drift caused by accidental changes that the unit and ROM tests miss (e.g., subtle audio mixer drift).

### Expansion-audio decibel oracle (`audio_expansion.rs`, v2.1.6)

The bbbradsmith `nes-audio-tests` `db_*` "hotswap" ROMs each play a sustained full-volume 2A03 reference square, then the expansion-chip square. `crates/rustynes-test-harness/tests/audio_expansion.rs` runs each ROM, measures the **peak amplitude of each segment** in the rendered waveform via `common::capture_frame_peaks` over deterministic frame windows, and **asserts the expansion/reference ratio** against the Mesen2 / hardware target (`level_db_*`): APU triangle ≈0.524, VRC6 ≈1.506, MMC5 ≈1.000, N163 1-ch ≈6.02. This is a real accuracy criterion, not just a hash. `db_5b` (i16-headroom deferral) and `db_vrc7` (patch-dependent FM level) have no `level_db_*` assertion — their real criteria are the `sunsoft5b_volume_dac_follows_logarithmic_step_law` (5B step law) and `vrc7_all_15_melodic_patches_match_nuke_ykt_canonical` (VRC7 patch ROM) unit tests. Every ROM (all 19) is additionally pinned by an `insta` framebuffer+cycle+audio-hash snapshot (byte-exact regression guard); a snapshot is re-blessed only when the new output is provably more accurate (measured vs the ROM target). See `docs/apu-2a03.md` §Expansion-audio levels and `docs/accuracy-ledger.md`.

## Layer 4.5 — commercial-ROM regression-prevention oracle (since 2026-05-17)

The May-2026 SMB / Excitebike / Kid Icarus regression motivated a
dedicated commercial-ROM oracle. Distinct from layer 4 because it
covers the *commercial* library (gitignored under
`tests/roms/external/`), not freely-distributable demos.

- **Harness:** `crates/rustynes-test-harness/tests/external_real_games.rs`
  (60 tests across 15 mappers). Feature-gated on `commercial-roms`
  (default off; CI never depends on non-distributable assets).
- **Per-test contract:** ROM SHA-256 + framebuffer FNV-1a 64-bit hash
  at one or more checkpoints + cumulative CPU cycle count + audio
  FNV-1a hash + audio sample count, all asserted against a committed
  `insta` snapshot (~500 B / file).
- **Visual companion:** `screenshots/` corpus (81 PNGs at 256×240
  RGBA8). Regenerated via `RUSTYNES_DUMP_FRAMES=1
  RUSTYNES_DUMP_DIR=$PWD/screenshots cargo test ...`.
- **Auto-bisect:** `scripts/regression-bisect/` is a permanent
  turn-key wrapper around `git bisect run`. Single-ROM or per-mapper
  filter (`HARNESS_FILTER=external_mmc3_ ./run.sh`). Drove the
  May-2026 recovery in 5 iterations (`0b1d4b66..HEAD` →
  `63d8dea` first-bad).
- **Auto-discovering companion:**
  `crates/rustynes-test-harness/tests/external_coverage.rs` walks
  `tests/roms/external/` at runtime and runs one default boot capture
  per staged ROM against a derived `insta` snapshot — new ROMs need no
  code change. As of T-PS-059 it discovers **every loadable form** the
  frontend accepts, not just bare `.nes`: iNES (`.nes`), UNIF (`.unf` /
  `.unif`), FDS disk images (`.fds`), and `.zip` / `.7z` archives (the
  No-Intro distribution form). The shared loader
  (`common::external::load_nes`) mirrors the frontend's load dispatch —
  it unwraps an archive to its first NES/FDS/UNIF entry (`.zip` via the
  `zip` crate, `.7z` via the `7z` CLI), and routes an FDS disk through
  `Nes::from_disk` with a BIOS resolved from `RUSTYNES_FDS_BIOS` or the
  staged `tests/roms/external/fds/disksys.rom`. A `.fds` on a BIOS-less
  checkout SKIPs cleanly (the BIOS is Nintendo IP and is never
  committed). So a ROM left zipped, or an `.fds` disk, gets a boot
  screenshot just like a loose `.nes`.
- **Trade-off:** snapshots commit emulator output (deterministic
  bytes), not ROM bytes (copyrighted). User-supplied dumps under
  `tests/roms/external/mapper-NNN-NAME/` are gitignored and entirely
  the user's responsibility.

## Layer 4.7 — co-simulation against an independent implementation (since v2.4.1)

The layers above all ask "does RustyNES agree with a reference?" This one inverts
it: RustyNES becomes the **oracle** an independently-written SystemVerilog NES
core is verified against (ADR 0037; `docs/mister.md`). The core lives in a sibling
repository and is written from public hardware documentation only — the
provenance firewall extends to HDL, so third-party `rtl/` is a strict black box.

`crates/rustynes-cosim` is the boundary. It is **excluded from the workspace on
purpose**: it enables trace features on `rustynes-core`, and cargo unifies
features across a workspace build, so as a member it would have made the accuracy
battery validate an instrumented scheduler no user runs. Its `nes_golden_export`
binary emits the golden artifacts a testbench consumes — **ten as of v2.6.3**
(`obs.bin`, `boot.bin`, `apu.bin`, `ckpt.bin`, `fetch.bin`, `index_fb.bin`,
`irq.csv`, `ram.bin`, `ram_init.bin`, `manifest.txt`), enumerated in that
binary's module docs. All ten are gitignored; `*.apu.bin` was the fourth to be
missed and is the reason the check beside those patterns is now written as a
command rather than a list to read.

Verification is a **ladder**, and a rung may not start until the one below is
green. Each rung's compare surface is fixed **before** the rung starts, splitting
every field into one of two roles:

- **Gate** — something a physical device can produce, so a mismatch fails the
  rung. Register values at instruction boundaries, per-cycle bus address / data /
  access type, per-dot PPU fetch addresses, the pre-palette index framebuffer.
- **Diagnostic** — RustyNES's *model*, not a hardware fact. `ppu-state-trace`'s
  internal latches and FSM fields explain a failure and may never cause one,
  because they encode this project's decomposition rather than the chip's.

Two properties of this layer are worth stating because they are easy to lose:

1. **A gate only tests at the resolution it samples.** Five gates that read state
   once per CPU cycle stayed green, in both directions, across a defect that
   moved an access two dots *within* a cycle (v2.5.4). "No existing check sees
   it" is a statement about the checks.
2. **Every new gate is demonstrated to fail by mutation** before it is trusted,
   with three outcomes — CAUGHT / NOT CAUGHT / BUILD-FAILED — and a baseline
   captured once and verified passing first. A NOT CAUGHT can indict the
   **stimulus** rather than the gate: a bit the test ROM never sets is a bit no
   comparison over that ROM can adjudicate.

**These gates do not run in CI**, and saying so is part of the layer's honesty:
they need the oracle's goldens and a build of the excluded crate, neither of
which exists in the sibling repository's workflows. They are run by hand and
recorded in that repository's per-rung docs, each of which carries a
**"what this rung cannot verify"** section. Fetching goldens from a pinned oracle
commit is the missing piece that would automate them.

## Layer 5 — fuzz testing

Nine `cargo-fuzz` harnesses under `fuzz/fuzz_targets/` (a standalone crate,
nightly-only because libFuzzer needs it):

- **`cartridge_parser`**: arbitrary `&[u8]` → `parse()`. Must not panic; errors typed.
- **`cpu_step`**: arbitrary RAM contents + opcode sequence → one instruction.
- **`mapper_writes`**: arbitrary write sequences to mapper registers; bank indices
  stay in range.
- **`ppu_reg_io`** / **`apu_reg_io`**: arbitrary register reads and writes.
- **`netplay_message`**: the netplay message and signalling parsers (untrusted
  network input).
- **`save_state`** / **`movie`**: the `.rns` and `.rnm` deserialisers (untrusted
  file input).
- **`movie_import`** (v3.2.0): the five foreign-movie importers (`.fm2`, `.bk2`,
  `.fcm`, `.fmv`, `.vmv`), selected by the first input byte. Promoted from a
  v2.9.9 probe that pushed 200,000 mutated inputs through them; clean over 2,000,000
  inputs when added.

**In CI since v2.9.4** (the `fuzz` job in `ci.yml`): all nine are built and each
runs 20,000 inputs from seed 1, so a failure is reproducible, and a crash input is
uploaded as an artifact. Before v2.9.4 no workflow built them. Locally:
`cd fuzz && cargo +nightly fuzz run <target> -O -- -runs=20000 -seed=1`. Under a
debugger or a sandbox that blocks ptrace, LeakSanitizer cannot start and reports
every run as a crash; set `ASAN_OPTIONS=detect_leaks=0` there.

## Layer 6 — CI gating

`.github/workflows/ci.yml` (the `CI success` job is the one required check; it
fails if any job it aggregates failed):

- **Lint:** `cargo fmt --check`, clippy `-D warnings` for the workspace and every
  feature set the project enumerates (including `retroachievements`, `full` and
  both wasm32 builds), rustdoc `-D warnings`.
- **Tests:** `cargo test --workspace` on Linux (plus macOS and Windows on full
  runs), on the pinned toolchain (`rust-toolchain.toml`, 1.99). Since v2.9.4 the
  Linux leg also runs the tests behind non-default features: the frontend with
  `full`, `rustynes-core` and `rustynes-ppu` with `debug-hooks,hd-pack`,
  `rustynes-apu` with `debug-hooks`, `rustynes-script` with
  `script-ipc,script-sqlite`.
- **Test ROMs:** `cargo test --workspace --features test-roms` in release mode.
- **Fuzz:** see Layer 5.
- **Coverage (since v2.9.4):** `cargo llvm-cov --workspace --lib
  --fail-under-lines 70`. The scope is the workspace's **lib tests only**; the
  integration suites are too slow instrumented (the `determinism` suite alone ran
  past 16 minutes). Baseline at v2.9.4: **71.66% of lines on the CI runner**
  (76.10% of functions, 74.07% of regions); the same tree measured 72.01% on the
  maintainer's machine, so quote the runner's number. The floor is 70%.
  Raise it when a release raises the number; never lower it to make a change
  pass.
- **Also:** the no_std cross-build, the libretro buildbot targets, and the
  frame-time regression gate.
- **Outside the required check:** `security.yml` (cargo-audit and cargo-deny),
  and `toolchain-canary.yml` (weekly; stable and beta, never blocking).

## Test ROM licensing

All vendored ROMs are individually CC0 or public-domain per their authors' dedications. Redistribution rights are documented in `tests/roms/LICENSES.md`. **No commercial Nintendo ROMs are bundled.** A separate `tests/roms/external/` directory is `.gitignore`'d for users who own ROMs they want to test against locally.

## Nesdev Completeness Audit

When adding or re-baselining tests, use the Nesdev emulator-test index as the
coverage checklist rather than the current repository contents. Gap status
(Phase 7 / v1.5.0):

- `instr_misc` (5) and `instr_timing` (2) — **CLOSED (T-71-003):** vendored
  (blargg PD) and strict-passing on the full `Nes`.
- `cpu_reset` (2) — **CLOSED (T-71-002):** wired; power-on register dump
  asserted strictly. The full interactive reset protocol is `#[ignore]`'d
  because the headless harness cannot supply an externally-timed reset; the
  reset register/RAM semantics are guarded by `Cpu::power_on` / `Nes::reset`
  unit tests.
- `vrc24test` — **REPLACED (T-71-005):** the original forum attachment link is
  permanently rotted (auth-walled; no mirror). Replaced by in-tree VRC2/VRC4
  register/wiring unit tests in `crates/rustynes-mappers` plus the `m22` baseline
  harness.
- Input-device coverage — **standard pad CLOSED (T-71-004):** strobe/serial-read
  bit-order tests on both ports. Four Score, Zapper, Famicom expansion devices,
  microphone, and DMC-DMA controller-bit corruption are a documented decision
  in `docs/compatibility.md` (Sprint 4 T-74-005); they remain deferred unless
  permissive fixtures and user demand surface.
- PAL/Dendy validation needs dedicated timing ROMs and golden snapshots rather
  than NTSC-derived expectations.
- FDS, Vs. System, PlayChoice-10, and non-stock PPU palettes are out of v1.0
  but should be represented as explicit unsupported-platform tests or fixture
  metadata when support begins.

### TASVideos / extended emulator-test pass (v1.5.0 Workstream C1)

An audit against the Nesdev "Emulator tests" + "Tricky-to-emulate games"
indices and the `christopherpow/nes-test-roms` aggregator, looking for
committable tests BEYOND the 139 AccuracyCoin battery. Findings (pinned
2026-06-16):

- **`mmc3_test` v1 (6 sub-ROMs)** — wired (`tests/m004_mmc3.rs`). The older
  kevtris/blargg MMC3 suite (distinct ROMs from the already-wired
  `mmc3_test_2`; same `$6000` protocol). *(Status as of this section's
  writing. Since v3.0.0, `4-scanline_timing` and `5-MMC3` pass strictly in
  both suites, and only `6-MMC6`'s alternate-revision assertion remains, by
  design; `docs/STATUS.md` has the current rows.)* **1/2/3 strict-PASS.** **4/5/6 were
  expected-fail** and converged on the *same* ADR-0002
  fractional-master-clock axis as the existing `mmc3_test_2/4` #3 residual:
  - `4-scanline_timing` #3 — "Scanline 0 IRQ should occur sooner when
    `$2000=$08`" (1-CPU-cycle scanline-IRQ bracket).
  - `5-MMC3` #2 — "Should reload and set IRQ every clock when reload is 0"
    (the structural reload-to-0 assertion is present in `Mmc3::clock_irq`
    path 2; only the sub-scanline IRQ cadence differs).
  - `6-MMC6` #2 — "IRQ should be set when reloading to 0 after clear"
    (MMC6 reload-to-0 cadence, same axis).
  These are pinned with `#[ignore]`'d strict tests + `_currently_fails`
  shape-probes; flip the strict test on and delete the probe if a ROM ever
  passes. **No new bug was found** — the failures are sub-scanline IRQ
  cadence precision deferred to the v2.0 refactor (ADR 0002).
- **`dpcmletterbox`** (Damian Yerrick, royalty-free) — wired
  (`tests/tasvideos_extended.rs`) as a deterministic framebuffer-hash visual
  smoke. It splits the screen with the DMC "sample finished" IRQ as a
  scanline timer (no mapper IRQ), so it is a sensitive DMC-IRQ-cadence +
  sprite-0 + NMI/DMC-phase sentinel.
- **Not committed (licensing):** `scanline-a1`, `tvpassfail`, and the other
  corpus demos lack a clear redistribution license; they stay in the
  gitignored local `tests/roms/nes-test-roms/` checkout only. The
  `cpu_exec_space`, `nmi_sync`, `oam_stress`, `ppu_open_bus`, `read_joy3`,
  `full_palette`, `exram`, and `240p` corpora were already wired in prior
  releases.

## Open questions

- **Snapshot tests for chip state.** `insta` would let us snapshot CPU register state mid-run. Useful for debugging but adds CI noise. Defer until needed.
- **TAS (tool-assisted) movie playback.** Replaying recorded controller inputs over many minutes is the highest-leverage compatibility test. Defer to Phase 5 frontend tooling.
- **Public dashboard of pass rates.** Auto-generate a markdown table of pass/fail for every test ROM and post to the README on each release. Good marketing; defer to v0.5.