# Spec 3: Deterministic fault-injection harness + on-chain attested recovery

Built 2026-07-19. Purpose: turn "we ran some fault tests" into a reproducible adversarial evidence
suite that de-risks the biggest gated item (the post-quantum consensus flip) for zero capital.
Extends the existing measured fault-tolerance work into a repeatable chaos matrix plus an on-chain
recovery-attestation contract, honest about what runs locally vs what needs a live testnet.

**Scope boundary (held throughout):** this DE-RISKS the consensus flip. It does NOT activate
post-quantum consensus. That flip stays founder-gated and audit-gated.

---

## Part 1: `AereRecoveryRegistry.sol` (REAL, BUILT + TESTED)

- Contract: `aerenew/contracts/contracts/AereRecoveryRegistry.sol` (solc 0.8.23, repo style).
- Test: `aerenew/contracts/test/AereRecoveryRegistry.test.js`.

Records recovery events (validator-set drop below quorum -> halt -> subsequent resume), each
authenticated by a Falcon-512 signature verified on-chain by the LIVE precompile 0x0AE1.
- Fields per record: haltHeight, resumeHeight, faultKind (1=crash, 2=netsplit,
  3=byzantine-proposer), attesting operator key (operatorId), Falcon signature over the record
  hash, plus validator-set context (nValidators, quorum, aliveDuringFault) that encodes the
  "drop below quorum."
- **Fail-closed:** invalid signature, stale nonce, inconsistent heights, or bad validator set
  records nothing and advances no nonce; an empty/zero/short precompile return (pre-fork / stale
  chain) is treated as INVALID. Reverts with `PQCVerificationFailed`.
- **Append-only:** records live in a push-only array; recordId is an immutable index; a strictly
  increasing per-operator nonce is consumed each write (no replay). There is NO owner, admin, or
  overwrite/delete function anywhere (structurally verified in the test's ABI scan).
- Wired to 0x0AE1 for production; Falcon-512 encoding matches the live precompile
  (pk 897/header 0x09; envelope nonce(40) || esig(0x29..)), mirroring `AerePQCAttestation.sol`.

### REAL TEST RESULT: 10/10 PASSING
Command: `npx hardhat test test/AereRecoveryRegistry.test.js` from `aerenew/contracts`.
Compiled clean; all 10 tests green (~1s):
- registerOperator: accepts well-formed Falcon-512 key + sequential ids; rejects wrong length/header.
- valid attestation records: kill-2 (f=2 boundary) crash recovery stores every field; netsplit +
  byzantine-proposer under one operator with nonce advancing.
- fail-closed: tampered signature rejected (nothing recorded); signature bound to different
  heights/fault-kind rejected; empty precompile return treated invalid; unknown operator / invalid
  fault kind / bad heights / bad validator set all revert.
- append-only: appends without rewriting an earlier record; replay of a consumed signature rejected;
  ABI exposes no set/update/delete/overwrite/admin/owner function.

**[MEASURE] flag on Part 1:** the Hardhat EVM has no 0x0AE1, so the test installs the repo's
faithful `MockPQCPrecompile` (scheme 1) at 0x0AE1 via `hardhat_setCode` (same pattern the existing
`AerePQCAttestation.test.js` uses). The mock parses input at the LIVE precompile's spec offsets, so
a passing record proves the contract builds the 0x0AE1 input correctly. The real-precompile path
(that on-chain 0x0AE1 accepts a genuine Falcon-512 signature) is proven separately against live
mainnet 2800 via eth_call, not inside the unit test.

---

## Part 2: chaos-matrix runner (SCAFFOLD + CONFIG; multi-node runs [MEASURE])

Under `aerenew/chaos/`:
- `matrix.json` — single source of truth. N=7 (quorum 5, f=2) and N=9 (quorum 6, crash-tol 3)
  fault matrix: C1 kill-1, C2 kill-2 (N=7 f=2 boundary), C3 kill-3 (N=7 sub-quorum halt / N=9
  crash boundary), C4 kill-4 (N=9 sub-quorum halt), P1 network partition (minority halts), B1
  byzantine-proposer. Documented tool = Pumba (toxiproxy / Chaos Mesh as alternatives), plus a
  process-mode fallback that reuses the existing `audit-package-pq-consensus/harness/n7/` scripts.
- `run-chaos-matrix.sh` — reads the matrix, injects each fault, snapshots per-node heights across
  the fault + recovery windows, decides halt/resume/fork, and (on a halt-then-recover) invokes the
  attester. Has a hard guard that REFUSES to run on the live infra box.
- `lib/partition.sh` — network partition via pumba netem / iptables so the majority keeps quorum.
- `lib/attest-recovery.js` — ethers script that reads the record hash from the deployed registry,
  signs it with the operator's offline Falcon-512 key, and submits `submitRecovery(...)`.

Validated locally: `matrix.json` parses (6 scenarios, N7/q5 + N9/q6); `bash -n` clean on both
shell scripts; `node --check` clean on the JS.

**[MEASURE] flag on Part 2:** the ACTUAL multi-node runs require a live isolated QBFT testnet on a
SCRATCH box, NEVER the infra box. The runner is inert without a net up (heights read 0). The
already-measured numbers are CITED from `matrix.json` `evidence`, not re-invented:
- Kill-2 soak (docs/FAULT-TOLERANCE-CHARACTERIZATION-2026-07-14.md): 100 adversarial cycles
  (40 RT=4 mixed + 40 RT=1 mixed + 20 pure-Besu), **0 forks, 100/100 self-resync**, rare
  temporary self-recovering f=2-boundary stall (3/40 RT=4, 1/40 RT=1, 0/20 Besu) that is inherent
  QBFT-at-f=2 (FLP), not a client bug.
- Sub-quorum halt-and-recover and partition (spec 5.3/5.4): halts in place, no fork, resumes from
  the exact halt height on restore, no re-genesis.
- Byzantine negatives (harness/n7/run-b-negatives.sh): 1-2 Falcon-faulty TOLERATED (<= f=2),
  3 faulty (> f) HALT post-detection, restore RESUMES with no re-genesis.

---

## Part 3: watchtower design (DESIGN)

`aerenew/chaos/WATCHTOWER-DESIGN.md`. An independent off-chain observer holding a registered
Falcon-512 key that: (1) detects a halt by combining a progress stall with quorum loss and no
fork (so it does not false-positive on the rare temporary f=2-boundary stall), classifying the
fault kind; (2) waits for quorum restoration, doing nothing that could influence consensus; (3)
submits the signed recovery attestation to Part 1, verified fail-closed by 0x0AE1. Includes the
watchtower's own failure modes and the honest single-operator caveat.

Composition: Part 2 generates adversarial fault evidence reproducibly, Part 3 produces it live and
independently, Part 1 makes all of it un-doctorable and quantum-durable on chain. Together they
make the flip decision rest on honest, checkable evidence. The flip itself stays a founder + audit
decision; the suite does not pull the trigger.

---

## All [VERIFY] / [MEASURE] flags

- [MEASURE] Part 1: real-precompile 0x0AE1 acceptance of a genuine Falcon-512 sig is proven vs live
  mainnet via eth_call, not in the Hardhat unit test (Hardhat has no 0x0AE1; mock installed there).
- [MEASURE] Part 2: multi-node chaos runs need a live isolated QBFT testnet on a scratch box; not
  run here. Existing soak numbers cited, none invented.
- [VERIFY] Part 2/3: `lib/attest-recovery.js` needs the operator's offline Falcon-512 signer + key
  (AERE_FALCON_SIGN / AERE_FALCON_KEY) and a chain where 0x0AE1 is live to complete the submit; on
  a plain testnet without the precompile the on-chain verify fails closed (correct, safe).
- [MEASURE]/[VERIFY] Part 3: the live watchtower daemon and its detection-threshold tuning against
  live inter-validator RTT are designed, not built; single-operator vantage is corroboration, not
  independence, until third-party watchtowers run.

## Files
- aerenew/contracts/contracts/AereRecoveryRegistry.sol
- aerenew/contracts/test/AereRecoveryRegistry.test.js
- aerenew/chaos/matrix.json
- aerenew/chaos/run-chaos-matrix.sh
- aerenew/chaos/lib/partition.sh
- aerenew/chaos/lib/attest-recovery.js
- aerenew/chaos/WATCHTOWER-DESIGN.md
- aerenew/chaos/README.md