aere-docs/AERE-FAULT-INJECTION-HARNESS.md

Spec 3: Deterministic fault-injection harness + on-chain attested recovery

Built 2026-07-19. Purpose: turn "we ran some fault tests" into a reproducible adversarial evidence suite that de-risks the biggest gated item (the post-quantum consensus flip) for zero capital. Extends the existing measured fault-tolerance work into a repeatable chaos matrix plus an on-chain recovery-attestation contract, honest about what runs locally vs what needs a live testnet.

Scope boundary (held throughout): this DE-RISKS the consensus flip. It does NOT activate post-quantum consensus. That flip stays founder-gated and audit-gated.

---

Part 1: AereRecoveryRegistry.sol (REAL, BUILT + TESTED)

• Contract: aerenew/contracts/contracts/AereRecoveryRegistry.sol (solc 0.8.23, repo style).
• Test: aerenew/contracts/test/AereRecoveryRegistry.test.js.

Records recovery events (validator-set drop below quorum -> halt -> subsequent resume), each authenticated by a Falcon-512 signature verified on-chain by the LIVE precompile 0x0AE1.

• Fields per record: haltHeight, resumeHeight, faultKind (1=crash, 2=netsplit, 3=byzantine-proposer), attesting operator key (operatorId), Falcon signature over the record hash, plus validator-set context (nValidators, quorum, aliveDuringFault) that encodes the "drop below quorum."
• Fail-closed: invalid signature, stale nonce, inconsistent heights, or bad validator set records nothing and advances no nonce; an empty/zero/short precompile return (pre-fork / stale chain) is treated as INVALID. Reverts with PQCVerificationFailed.
• Append-only: records live in a push-only array; recordId is an immutable index; a strictly increasing per-operator nonce is consumed each write (no replay). There is NO owner, admin, or overwrite/delete function anywhere (structurally verified in the test's ABI scan).
• Wired to 0x0AE1 for production; Falcon-512 encoding matches the live precompile (pk 897/header 0x09; envelope nonce(40) || esig(0x29..)), mirroring AerePQCAttestation.sol.

REAL TEST RESULT: 10/10 PASSING

Command: npx hardhat test test/AereRecoveryRegistry.test.js from aerenew/contracts. Compiled clean; all 10 tests green (~1s):

• registerOperator: accepts well-formed Falcon-512 key + sequential ids; rejects wrong length/header.
• valid attestation records: kill-2 (f=2 boundary) crash recovery stores every field; netsplit + byzantine-proposer under one operator with nonce advancing.
• fail-closed: tampered signature rejected (nothing recorded); signature bound to different heights/fault-kind rejected; empty precompile return treated invalid; unknown operator / invalid fault kind / bad heights / bad validator set all revert.
• append-only: appends without rewriting an earlier record; replay of a consumed signature rejected; ABI exposes no set/update/delete/overwrite/admin/owner function.

[MEASURE] flag on Part 1: the Hardhat EVM has no 0x0AE1, so the test installs the repo's faithful MockPQCPrecompile (scheme 1) at 0x0AE1 via hardhat_setCode (same pattern the existing AerePQCAttestation.test.js uses). The mock parses input at the LIVE precompile's spec offsets, so a passing record proves the contract builds the 0x0AE1 input correctly. The real-precompile path (that on-chain 0x0AE1 accepts a genuine Falcon-512 signature) is proven separately against live mainnet 2800 via eth_call, not inside the unit test.

---

Part 2: chaos-matrix runner (SCAFFOLD + CONFIG; multi-node runs [MEASURE])

Under aerenew/chaos/:

• matrix.json — single source of truth. N=7 (quorum 5, f=2) and N=9 (quorum 6, crash-tol 3) fault matrix: C1 kill-1, C2 kill-2 (N=7 f=2 boundary), C3 kill-3 (N=7 sub-quorum halt / N=9 crash boundary), C4 kill-4 (N=9 sub-quorum halt), P1 network partition (minority halts), B1 byzantine-proposer. Documented tool = Pumba (toxiproxy / Chaos Mesh as alternatives), plus a process-mode fallback that reuses the existing audit-package-pq-consensus/harness/n7/ scripts.
• run-chaos-matrix.sh — reads the matrix, injects each fault, snapshots per-node heights across the fault + recovery windows, decides halt/resume/fork, and (on a halt-then-recover) invokes the attester. Has a hard guard that REFUSES to run on the live infra box.
• lib/partition.sh — network partition via pumba netem / iptables so the majority keeps quorum.
• lib/attest-recovery.js — ethers script that reads the record hash from the deployed registry, signs it with the operator's offline Falcon-512 key, and submits submitRecovery(...).

Validated locally: matrix.json parses (6 scenarios, N7/q5 + N9/q6); bash -n clean on both shell scripts; node --check clean on the JS.

[MEASURE] flag on Part 2: the ACTUAL multi-node runs require a live isolated QBFT testnet on a SCRATCH box, NEVER the infra box. The runner is inert without a net up (heights read 0). The already-measured numbers are CITED from matrix.json evidence, not re-invented:

• Kill-2 soak (docs/FAULT-TOLERANCE-CHARACTERIZATION-2026-07-14.md): 100 adversarial cycles (40 RT=4 mixed + 40 RT=1 mixed + 20 pure-Besu), 0 forks, 100/100 self-resync, rare temporary self-recovering f=2-boundary stall (3/40 RT=4, 1/40 RT=1, 0/20 Besu) that is inherent QBFT-at-f=2 (FLP), not a client bug.
• Sub-quorum halt-and-recover and partition (spec 5.3/5.4): halts in place, no fork, resumes from the exact halt height on restore, no re-genesis.
• Byzantine negatives (harness/n7/run-b-negatives.sh): 1-2 Falcon-faulty TOLERATED (<= f=2), 3 faulty (> f) HALT post-detection, restore RESUMES with no re-genesis.

---

Part 3: watchtower design (DESIGN)

aerenew/chaos/WATCHTOWER-DESIGN.md. An independent off-chain observer holding a registered Falcon-512 key that: (1) detects a halt by combining a progress stall with quorum loss and no fork (so it does not false-positive on the rare temporary f=2-boundary stall), classifying the fault kind; (2) waits for quorum restoration, doing nothing that could influence consensus; (3) submits the signed recovery attestation to Part 1, verified fail-closed by 0x0AE1. Includes the watchtower's own failure modes and the honest single-operator caveat.

Composition: Part 2 generates adversarial fault evidence reproducibly, Part 3 produces it live and independently, Part 1 makes all of it un-doctorable and quantum-durable on chain. Together they make the flip decision rest on honest, checkable evidence. The flip itself stays a founder + audit decision; the suite does not pull the trigger.

---

All [VERIFY] / [MEASURE] flags

• [MEASURE] Part 1: real-precompile 0x0AE1 acceptance of a genuine Falcon-512 sig is proven vs live mainnet via eth_call, not in the Hardhat unit test (Hardhat has no 0x0AE1; mock installed there).
• [MEASURE] Part 2: multi-node chaos runs need a live isolated QBFT testnet on a scratch box; not run here. Existing soak numbers cited, none invented.
• [VERIFY] Part 2/3: lib/attest-recovery.js needs the operator's offline Falcon-512 signer + key (AERE_FALCON_SIGN / AERE_FALCON_KEY) and a chain where 0x0AE1 is live to complete the submit; on a plain testnet without the precompile the on-chain verify fails closed (correct, safe).
• [MEASURE]/[VERIFY] Part 3: the live watchtower daemon and its detection-threshold tuning against live inter-validator RTT are designed, not built; single-operator vantage is corroboration, not independence, until third-party watchtowers run.

Files

• aerenew/contracts/contracts/AereRecoveryRegistry.sol
• aerenew/contracts/test/AereRecoveryRegistry.test.js
• aerenew/chaos/matrix.json
• aerenew/chaos/run-chaos-matrix.sh
• aerenew/chaos/lib/partition.sh
• aerenew/chaos/lib/attest-recovery.js
• aerenew/chaos/WATCHTOWER-DESIGN.md
• aerenew/chaos/README.md