6cb0140fae
The public history carried kat/__pycache__/mlkem768_reference.cpython-314.pyc, a compiled Python artifact embedding the operator's absolute local path. Text secret scanners do not read compiled binaries, which is exactly how it slipped through, and removing it from the tip would have left it reachable through the old root commits. So this repository is republished from a single clean root. This root also carries, from the previously unpublished line of work: - corrected LICENSE year, LICENSING.md, VERIFY-POLICY.md, and CITATIONS-UNRESOLVED.md remeasured 2026-08-11 (101 paths, README aligned) - O-018: run_consensus_verification.py ran 19 of 29 models and reported PASS; it now runs all 29, and computemarket_smt.py gains resolveByTimeout / reclaimUnsettled cases plus a negative control - O-006: the word 'audited' removed from next to Bouncy Castle, twice, after a concurrent edit resurrected it - O-014: prior art named and dated - Algorand's native falcon_verify shipped about ten months before AERE's precompiles; the primacy claim is withdrawn where it was implied - bench/ scripts parametrized so they actually run for an outsider (the earlier textual sanitization left $STAGING unexpanded inside Python strings) - AIP-2/AIP-3 errata with measured figures, spec remeasurements at 2026-08-01, and the spec-zk-stack retractions (owner is an operational key, not the Foundation; 'maximally sound' withdrawn; aggregator V1 deprecated) The redacted bench-host environment files from the sanitized line are kept exactly as published; the unredacted local variants are not carried.
144 lines
9.7 KiB
Markdown
144 lines
9.7 KiB
Markdown
# MMCS (FieldMerkleTreeMmcs) over BabyBear: construction + conformance KAT (component (c), 0x0AE8)
|
|
|
|
> **Scope caveat (2026-07-19 finding).** "Plonky3/SP1" wording below means Plonky3 `0.4.3-succinct`
|
|
> (Aere's OWN BabyBear + FRI STARK stack). It does NOT mean the pinned SP1 6.1.0, which is a Hypercube
|
|
> release (KoalaBear multilinear: BaseFold + sumcheck-zerocheck + LogUp-GKR) and does not use this
|
|
> BabyBear FRI MMCS config. This component is conformance-confirmed for a BabyBear + FRI STARK
|
|
> verifier, not for an SP1 6.1.0 verifier. See `../../docs/AERE-STARK-SP1-RECURSION-AIR-PORT-SPEC-SUMMARY.md`.
|
|
|
|
Date: 2026-07-19. Status: construction CONFIRMED-FROM-SOURCE; real conformance KAT PASSED.
|
|
|
|
This records exactly which Mixed Matrix Commitment Scheme (MMCS) Plonky3/SP1 use over BabyBear, where
|
|
every part of the construction comes from, and the real known-answer test that confirms conformance.
|
|
The top-level precompile 0x0AE8 stays fail-closed (returns EMPTY for every input) regardless: this
|
|
component being confirmed does not port the sponge challenger, FRI folding, or the recursion AIR.
|
|
|
|
## Pinned target (exact, checksum-matched)
|
|
|
|
The MMCS is `FieldMerkleTreeMmcs` from `p3-merkle-tree`, with the leaf hasher and node compressor from
|
|
`p3-symmetric` and the `Mmcs` trait (verify_batch shape) from `p3-commit`, all crates.io version
|
|
`0.4.3-succinct` (the Succinct Plonky3 fork), the revision pinned in the repo's Cargo.lock files:
|
|
- `p3-merkle-tree` 0.4.3-succinct, sha256 `d5703d9229d52a8c09970e4d722c3a8b4d37e688c306c3a1c03b872efcd204e6`
|
|
- `p3-symmetric` 0.4.3-succinct, sha256 `9047ce85c086a9b3f118e10078f10636f7bfeed5da871a04da0b61400af8793a`
|
|
- `p3-commit` 0.4.3-succinct, sha256 `50acacc7219fce6c01db938f82c1b21b5e7133990b7fff861f91534aeb569419`
|
|
- `p3-matrix` 0.4.3-succinct, sha256 `75c3f150ceb90e09539413bf481e618d05ee19210b4e467d2902eb82d2e15281`
|
|
|
|
Leaf hashing / node compression run on the CONFIRMED Poseidon2-BabyBear permutation (component (b),
|
|
`p3-baby-bear` / `p3-poseidon2` 0.4.3-succinct). Found in `aerenew/zk-circuits/*/Cargo.lock` and
|
|
`aerenew/rollup-evm-validity/*/Cargo.lock`. A fresh cargo build of the extractor (below) resolved the
|
|
SAME checksums, so the crates executed are byte-identical to the ones the AERE prover pins.
|
|
|
|
## The exact SP1 inner config (verbatim from p3-merkle-tree's own tests)
|
|
|
|
`p3-merkle-tree`'s `src/mmcs.rs` test module pins the type aliases the SP1 inner STARK uses:
|
|
|
|
```
|
|
type Perm = Poseidon2<BabyBear, Poseidon2ExternalMatrixGeneral, DiffusionMatrixBabyBear, 16, 7>;
|
|
type MyHash = PaddingFreeSponge<Perm, 16, 8, 8>; // WIDTH 16, RATE 8, OUT 8
|
|
type MyCompress = TruncatedPermutation<Perm, 2, 8, 16>; // N 2, CHUNK 8, WIDTH 16
|
|
type MyMmcs = FieldMerkleTreeMmcs<Packing, Packing, MyHash, MyCompress, 8>; // DIGEST_ELEMS 8
|
|
```
|
|
|
|
So a digest is 8 BabyBear elements (32 bytes). The same aliases appear in the SP1 `BabyBearPoseidon2`
|
|
inner config, so this is the commitment the shrink/wrap ShardProof's trace and FRI matrices use.
|
|
|
|
## Confirmed construction (from source)
|
|
|
|
- Leaf hasher `PaddingFreeSponge<Perm,16,8,8>` (p3-symmetric `sponge.rs`): overwrite-mode, padding-free
|
|
sponge. State starts all-zero; for each chunk of RATE=8 input elements, OVERWRITE the first
|
|
chunk-length lanes (leaving the remaining lanes at their prior value) and permute; the digest is the
|
|
first OUT=8 lanes. An empty input yields the all-zero digest with no permutation.
|
|
`hash_slice`/`hash_item` are the trait defaults: hash the concatenation.
|
|
- Node compressor `TruncatedPermutation<Perm,2,8,16>` (p3-symmetric `compression.rs`):
|
|
`compress([l, r]) = permute(l || r)[0..8]` (l, r each 8 elements padded into a width-16 state).
|
|
- Tree build `FieldMerkleTree::new` (p3-merkle-tree `merkle_tree.rs`): sort matrices tallest-first
|
|
(stable); the first digest layer hashes the concatenated rows of ALL matrices at the max height,
|
|
padded up to a power-of-two length with the zero digest; then repeatedly compress adjacent pairs up a
|
|
layer, and where matrices of the next (padded) height exist, inject them via
|
|
`compress([compress([left, right]), hash(their concatenated row)])` (`compress_and_inject`). Rows
|
|
beyond the injected matrices' height (but within the padded layer) are compressed against the zero
|
|
digest. The commitment is the single top digest (the root).
|
|
- Height property (asserted by Plonky3): matrix heights that round up to the same power of two must be
|
|
equal, so every padded-power-of-two bucket holds matrices of a single exact height. This is what makes
|
|
`height == max_height` (tree build) and `height.next_power_of_two() == curr_padded` (verify) select
|
|
the same set.
|
|
- Opening `open_batch` (p3-merkle-tree `mmcs.rs`): opened rows are each matrix's row at
|
|
`index >> (log_max_height - log2_ceil(matrix.height))`, in ORIGINAL matrix order; the proof is the
|
|
sibling digest `digest_layers[i][(index >> i) ^ 1]` for each of the `log_max_height` layers.
|
|
- Verify `verify_batch` (p3-merkle-tree `mmcs.rs`, the only method the on-chain verifier needs): group
|
|
opened rows by padded height (tallest first, by original index); seed the running root with the
|
|
tallest group's hashed concatenated rows; then walk the proof, at each step ordering `(root, sibling)`
|
|
by `index & 1` (0 -> root is left), compressing, and (when the next group's padded height is reached)
|
|
compressing in that group's hashed rows; accept iff the final root equals the commitment.
|
|
|
|
## Montgomery note
|
|
|
|
Field elements are serialized as canonical u32 (`PrimeField32::as_canonical_u32`), the same canonical
|
|
form the Poseidon2 confirmation used. The permutation carries the Montgomery R^{-1} internal-layer
|
|
factor internally (component (b), resolved there); the MMCS layer above it is pure field data plus
|
|
hashing, with no additional Montgomery subtlety. As a cross-check, the extractor's `permute([0;16])`
|
|
(emitted as `perm_zeros`) equals the confirmed Poseidon2 "zeros" vector
|
|
`[1787823396, 953829438, 89382455, ...]`, proving the extractor's permutation is byte-identical to the
|
|
one the reference uses.
|
|
|
|
## How the KAT vectors were obtained (primary source)
|
|
|
|
A small Rust extractor depending on the pinned crates was compiled and run (cargo 1.97.0). It builds
|
|
the exact `MyMmcs` above with the permutation from `Poseidon2::new_from_rng_128(...,
|
|
Xoroshiro128Plus::seed_from_u64(1))` (the SAME deterministic construction the Poseidon2 confirmation
|
|
used), commits to several known matrix batches, and for each emits the root plus an `open_batch`
|
|
opening (opened rows + sibling path). It also asserts the library's OWN `verify_batch` accepts every
|
|
emitted opening. Each field element is printed as its canonical u32. The extractor and its output
|
|
(`mmcs_ground_truth.json`) are the authoritative source for the numbers below.
|
|
|
|
## Conformance KAT (real known-answer vectors, executed from the pinned crates)
|
|
|
|
Six MMCS cases were emitted and are reproduced EXACTLY by all three references (Python / Node /
|
|
standalone Java) and by `Mmcs` in the precompile:
|
|
|
|
| case | matrices (height x width) | open index | exercises |
|
|
|---|---|---|---|
|
|
| single_8x2 | 8x2 | 3 | single power-of-two height |
|
|
| single_6x2 | 6x2 (padded to 8) | 5 | non-power-of-two height (zero-digest padding) |
|
|
| single_8x1 | 8x1 | 6 | column vector (commit_vec shape) |
|
|
| mixed_8x2_4x3 | 8x2, 4x3 | 5 | two heights, injection at layer 1 |
|
|
| mixed_8x1_4x2_2x2 | 8x1, 4x2, 2x2 | 6 | three heights, injection at two layers |
|
|
| mixed_5x2_3x1 | 5x2, 3x1 | 4 | non-power-of-two mixed; a default (zero) sibling in the proof |
|
|
|
|
Example (single_8x2, matrix rows [10,11],[12,13],...,[24,25], open index 3):
|
|
- root = [35761595, 1133593632, 733114748, 517674920, 1449914397, 74512820, 381712048, 469819303]
|
|
- opened row = [16, 17]
|
|
- proof (3 sibling digests) = [[1513856679, ...], [57846071, ...], [1143223726, ...]]
|
|
|
|
Primitive KATs (also reproduced exactly):
|
|
- hash_item(5) = [881553380, 703286570, 452412164, ...]
|
|
- hash_slice([1,2]) = [1843359319, 912981492, 1448073574, ...]
|
|
- hash_slice([1,2,3]) = [1831345102, 1426305082, 956789587, ...]
|
|
- hash_slice([1..8]) = [8999572, 1033765830, 347083905, ...] (one full RATE block)
|
|
- hash_slice([1..9]) = [1134257664, 40304233, 1823880005, ...] (two blocks, tests the second absorb)
|
|
- compress(hash[1,2], hash[1,2,3]) = [1968576159, 1450511489, 1750728079, ...]
|
|
|
|
Full vectors are embedded in `mmcs_babybear_reference.py` (CONFORMANCE_CASES / PRIM_KATS) and mirrored
|
|
in the `.mjs` and Java references.
|
|
|
|
Result (ran 2026-07-19):
|
|
- `python test_mmcs_babybear.py` -> PASS=300 FAIL=0 (self-consistency: every valid leaf opens to a path
|
|
that recomputes the root, and tampered openings / proof siblings / roots all fail; CONFORMANCE 37/37;
|
|
Python + Node + Java byte-identical; recorded in `../results/kat-results-mmcs-babybear.json`).
|
|
- `java MmcsBabyBearSelfTest` -> PASS=80 FAIL=0 (includes the conformance cases + open-every-leaf).
|
|
|
|
## Status summary
|
|
|
|
- Construction: CONFIRMED-FROM-SOURCE (the exact SP1 inner config from p3-merkle-tree's own tests;
|
|
tree build / open / verify traced against p3-merkle-tree + p3-symmetric + p3-commit 0.4.3-succinct;
|
|
checksums matched Cargo.lock).
|
|
- Conformance KAT: PASSED (6 MMCS cases + 6 primitive KATs + perm sanity reproduced byte-for-byte, and
|
|
verify_batch accepts the emitted openings and rejects tampered ones).
|
|
- `Mmcs.available` = true (the commitment scheme is confirmed conformant).
|
|
- `[MEASURE]` remaining: conformance against a REAL exported SP1 v6.1.0 commitment root (the trace / FRI
|
|
matrices from an actual proof) is a further step needing an exported proof + the challenger.
|
|
- Top-level 0x0AE8: still FAIL-CLOSED (EMPTY for every input). The duplex-sponge challenger
|
|
(`Challenger.spongePorted = false`, component (f)), FRI folding (component (d)), and the SP1
|
|
recursion-AIR (`StarkConstraints` = UNAVAILABLE, component (e)) are un-ported. ACCEPT is unreachable.
|
|
DO NOT ACTIVATE.
|