216 lines
21 KiB
Markdown
216 lines
21 KiB
Markdown
# Conformance suite for the AIP-23 proof kinds
|
|
|
|
Every component beside this one ships its own verifier and its own way of saying no: the AIP-23 reference verifier prints four
|
|
levels, the payment modules return `reasons`, the market judge `motive`, the dataset module `ok`, the audit log and the agents'
|
|
ledger a refusal text. This folder adds the one place that
|
|
says, in the same form for all of them, **this artifact passes or does not, at level N, for reason M**, and a set of vectors that any
|
|
implementation, in any language, can be run against.
|
|
|
|
It adds no verification rule. A verdict here is derived from the verdict of the published verifier of that artifact; the rule ids
|
|
are the ones those verifiers name, read from their code.
|
|
|
|
## The one verdict form
|
|
|
|
```json
|
|
{ "v": 1, "kind": "aere-conformance-verdict", "artifact": "payment-receipt", "verdict": "INVALID",
|
|
"levels": [
|
|
{ "level": "integrity", "state": "FAILED", "rules": ["receipt.chain"], "reason": "the transaction on chain is not this payment ..." },
|
|
{ "level": "authenticity", "state": "NOT_MEASURED", "rules": [], "cause": "absent", "reason": "the envelope carries no signature" },
|
|
{ "level": "anchoring", "state": "NOT_MEASURED", "rules": [], "cause": "absent", "reason": "no notarization claimed and no --rpc given" },
|
|
{ "level": "finality", "state": "NOT_MEASURED", "rules": [], "cause": "absent", "reason": "no notarization claimed and no --rpc given" } ],
|
|
"failed": ["receipt.chain"] }
|
|
```
|
|
|
|
| level | what it says |
|
|
|---|---|
|
|
| `integrity` | the artifact is what it says: its hashes bind its content, and the rules it declares (a policy, deadlines, a challenge window, Merkle paths, a payout) hold when recomputed from the evidence supplied |
|
|
| `authenticity` | the signature the artifact carries verifies, with a key of the scheme declared, and the identity it names is the one derived from that key (an ML-DSA signature on an AIP-23 envelope, the ML-DSA-65 signature of an audit head, the ML-DSA-65 key and signatures of an agent's ledger) |
|
|
| `anchoring` | its `statementHash` (or, for a trust chain, the head of its ledger) is recorded in the notary contract, under a state root that a post-quantum anchor covers |
|
|
| `finality` | that anchor's post-quantum certificate verifies. The finality is the one of the chain the artifact was notarized on |
|
|
|
|
Each level is `PASSED`, `FAILED` or `NOT_MEASURED`, with the reason; a level nobody judged says why (`cause`). The verdict is
|
|
`INVALID` when a level failed; `PARTIAL` when nothing failed but something the artifact **claims** was not measured (a notarization
|
|
claim not checked because the run was offline, a signature scheme the verifier cannot check, a timeline that is not terminal);
|
|
`VALID` otherwise, and only with integrity passed. A claim that was not checked is never `VALID`. `failed` is the sorted list of the
|
|
rule ids that refused; the suite compares it as an exact set.
|
|
|
|
A receipt or a market timeline that carries an AIP-23 envelope gets that envelope judged too, by the reference verifier, so one
|
|
verdict covers the artifact: the receipt recorded on testnet 28001 comes out with integrity, anchoring and finality `PASSED` in one
|
|
verdict (`vectori/payment-receipt/company-p1-28001.valid.json`).
|
|
|
|
## What is enrolled
|
|
|
|
`REGULI.json` is derived from the code (`node deriva-reguli.mjs --verifica`): 97 rules of 9 verifiers, 13 artifact kinds. By level:
|
|
83 rules are reported at `integrity`, 11 at `authenticity`, 3 at `anchoring` (one of them, `envelope.finality`, is reported at
|
|
`anchoring` or at `finality` by the place that refused).
|
|
|
|
| artifact kind | judged by | rules |
|
|
|---|---|---|
|
|
| `envelope` (every AIP-23 kind) | the AIP-23 reference verifier, run as the program it is (`AERE_VERIFY_PROOF=<verify-proof.mjs>`, published at aere.network/tools) | 4: `envelope.form`, `.integrity`, `.signature`, `.finality` |
|
|
| `payment-risk` (a payment presented as allowed under a declared policy) | `payments/risc.mjs` | 6: `risk.form`, `.asset`, `.counterparty`, `.rate`, `.limit`, `.window` |
|
|
| `payment-receipt` | `payments/chitanta.mjs` | 5: `receipt.binding`, `.amount`, `.policy`, `.risk`, `.chain` |
|
|
| `market-timeline` | `market-settlement/piata.mjs` | 10: `market.form`, `.complete`, `.posting`, `.deadline`, `.window`, `.arbiter`, `.payout`, `.result`, `.replay`, `.proof` |
|
|
| `dataset-record`, `-answer`, `-consistency`, `-sample` | `verifiable-data/baza.mjs` | 10: `dataset.form`, `.commitment`, `.field`, `.inclusion`, `.index`, `.range`, `.order`, `.completeness`, `.consistency`, `.sample` |
|
|
| `ai-trust-bundle` (the trust chain of an AI result) | `ai-trust/lant-de-incredere.mjs` | 21: `aitrust.form`, `.identity-mismatch`, `.identity-key`, `.version-attestation`, `.version-artifact-mismatch`, `.version-label-mismatch`, `.runtime-record`, `.runtime-artifact-mismatch`, `.runtime-after-output`, `.audit-log-broken`, `.runtime-not-in-audit-log`, `.audit-head`, `.witness-head`, `.policy-hash`, `.ledger-invalid`, `.output-not-in-ledger`, `.action-refused`, `.ledger-time-mismatch`, `.policy`, `.provenance-envelope`, `.provenance-finality` |
|
|
| `audit-log` (a hash-chained log, alone or against an attested head, signed or not) | `verify-layer/sidecar.mjs` | 13: `auditlog.entry`, `.seq`, `.chain`, `.hash`, `.head-form`, `.head-hash`, `.head-unsigned`, `.head-scheme`, `.head-key`, `.head-signature`, `.head-signer`, `.head-short`, `.head-rewritten` |
|
|
| `audit-inclusion`, `audit-consistency` (RFC 9162 proofs over that log) | `verify-layer/jurnal-arbore.mjs` | 12: `audittree.proof-form`, `.head-form`, `.head-hash`, `.inclusion-index`, `.inclusion-entry`, `.inclusion-envelope`, `.inclusion-leaf`, `.inclusion-path`, `.inclusion-head`, `.consistency-start`, `.consistency-end`, `.consistency` |
|
|
| `agent-ledger` (an agent's signed ledger under its policy) | `agents/agent-ledger.mjs` | 16: `agents.ledger-form`, `.policy`, `.other-policy`, `.session`, `.key`, `.identity`, `.anchor`, `.seq`, `.chain`, `.body`, `.signature`, `.hash`, `.time`, `.revocation`, `.approvals`, `.decision` |
|
|
|
|
A module verifier exports the list of rules it can name (`REGULI`; its helper throws on any other) and marks its rules in one of two
|
|
ways, both read by the derivation: a marker on every check of a rule (seven modules), or the rule named at every call of the
|
|
verifier's one refusal helper (`ai-trust`). The derivation requires declared == marked == mapped here. A rule added to a verifier and
|
|
not mapped is a refusal of the derivation; a mapped rule without a vector is a refusal of the probe. For the reference verifier, a
|
|
published program this suite does not edit, the rules are its level names, and each of the six places where its `finality` level
|
|
can refuse is classified as `anchoring` or `finality` (a seventh place would be a refusal of the derivation).
|
|
|
|
**Not enrolled**, counted in the same file: `identity`. Its presentation verifier reports 41 labelled check rows, not rule ids, and
|
|
its verdict depends on what the verifier brings (audience, nonce, clock, trusted issuers); a rule id per row and a vector input that
|
|
carries those come first. Its SD-JWT, Travel Rule and compliance verifiers are not counted at all. Their own probes and negative
|
|
controls stay what checks them.
|
|
|
|
## The vectors
|
|
|
|
`vectori/` holds 206 vectors (197 offline, 9 that read testnet 28001), listed with their digests in `vectori/INDEX.json`:
|
|
envelope 64, payment-risk 11, payment-receipt 10, market-timeline 26, dataset 22, ai-trust-bundle 24, audit-log 17, audit-inclusion
|
|
10, audit-consistency 4, agent-ledger 18. By expected verdict: 40 `VALID`, 5 `PARTIAL`, 161 `INVALID`. None is typed by hand
|
|
(`node genereaza-vectori.mjs --verifica` regenerates them and compares byte for byte):
|
|
|
|
- the 16 AIP-23 kinds come from their producers on fixed inputs (the same envelopes as the published AIP-23 kind vectors); the audit
|
|
log, its heads and its inclusion and consistency proofs come from the verify-layer producers on fixed inputs;
|
|
- the receipt, the five market timelines, the notarized dataset commitment and one trust chain of an AI result are the evidence
|
|
recorded on testnet 28001 on 2026-10-02 beside those components;
|
|
- `surse-proaspete.json` holds what needed a key, randomness or the chain, made once by `--proaspat` (which only adds what is
|
|
missing). Every key was generated and discarded in that call: a tree head signed with an ML-DSA-65 key and the same head "signed"
|
|
with an Ed25519 key that declares itself ml-dsa-65; an audit head signed by an operator key, by another key, and by a classical
|
|
key; the ledger of an agent (a tool call, a payment, a payment refused above the limit, a tool call) with the forged siblings
|
|
that only the agent's key can make (an empty approvals list, a false decision, a revocation entry without a revocation) and one
|
|
written with a classical key; three trust chains of a fresh agent (allowed, refused by the policy, dated apart); an envelope with
|
|
random content that nobody notarized; a dataset of 16 records with random salts; and the transaction of the recorded receipt read
|
|
from the chain (two read calls).
|
|
|
|
An `INVALID` vector names its `base` and its `mutation`: **one operation** (set, remove or add at one JSON path) applied to the input
|
|
of a vector that is not invalid, and it must be refused for exactly the rules in `expected.failed`, not merely refused: 137 of the
|
|
161 are made this way. Where the rule sits under a hash or a signature, one place is not reachable (changing an amount under the
|
|
instruction hash trips the binding first; a false decision in a ledger must still be signed by the agent): the other 24 are made at
|
|
the producer, the commitments above the change recomputed with the published formulas, and every path that moved is declared
|
|
(`mutation.changed`). Five vectors expect two rules: rules that guard the same field cannot be separated (a looser policy shown
|
|
under a receipt also makes the risk verdict not reproduce; a Proof of AI envelope that is changed is also no longer the one the
|
|
ledger links).
|
|
|
|
## Run it
|
|
|
|
```
|
|
AERE_VERIFY_PROOF=<verify-proof.mjs from aere.network/tools, with verify-anchor.mjs, mpt.mjs and package.json beside it and npm install done>
|
|
node conformance.mjs --impl "node implementare-aere.mjs" # the published verifiers behind the one verdict form: 197 offline vectors
|
|
node conformance.mjs --impl "node implementare-aere.mjs" --online # and the 9 that read testnet 28001 (nothing is sent)
|
|
node conformance.mjs --impl "python3 implementare-integritate.py" # an independent implementation: 62 vectors, 144 reported NOT COVERED
|
|
node conformance.mjs --impl "<your command>" # yours
|
|
node implementare-aere.mjs <artifact.json> [--online] # one artifact { artifact, input }: its verdict
|
|
```
|
|
|
|
For the artifact `envelope` the input is `{ "envelope": <any JSON value> }`, or `{ "raw": "<the text of the file>" }` when the file is
|
|
not JSON at all: a file that is not an envelope is refused at the form level, by name, like any other. The other inputs are the ones
|
|
the vectors show (`{ log, head?, signer? }` for an audit log, `{ ledger, policy }` for an agent's ledger, `{ bundle }` for a trust
|
|
chain).
|
|
|
|
An implementation is a command that answers three calls with JSON on stdout: `--capabilities` (`{ artifacts, rules, online, batch }`:
|
|
what it claims to judge), `<vector.json> [--online]` (the verdict above for the file's `{ artifact, input }`), and, if it says
|
|
`batch`, `--batch <list.json>` (one verdict per line). A vector is scored only if the implementation claims its artifact kind and
|
|
every rule the vector exercises; otherwise it is reported `NOT COVERED`, never passed. A scored vector passes when the verdict, the
|
|
exact set of failed rules and the state of each of the four levels are the expected ones. No output is `BROKEN`: a crash is not a
|
|
refusal. Without `--online` an implementation must not open a connection.
|
|
|
|
`implementare-integritate.py` is the second implementation: Python, standard library only, written from the rules and not from the
|
|
JavaScript. It claims 6 of the 97 rules (the form and the statement hash of an envelope; the form, the commitment, the salted field
|
|
commitments and the RFC 9162 inclusion of a dataset record) and passes all 62 vectors it claims. It cannot check an ML-DSA signature
|
|
or read a chain, says so in its capabilities, and the runner reports the rest as not covered: 144 vectors, among them every vector of
|
|
the trust chain, the audit log and the agents' ledger. For those four verifiers the suite has one implementation, ours.
|
|
|
|
## How the suite itself is checked
|
|
|
|
Measured on 2026-10-05 (Node.js 24.14.1, Python 3.14.4, testnet 28001 read):
|
|
|
|
- `node proba-conformance.mjs --online`: every row green, 10 in this copy (an eleventh, the comparison of the 16 kind envelopes with
|
|
the published AIP-23 kind vectors, runs where their generator is, in the development repository: 11/11 there, and here it is
|
|
printed as information). The rules and the vectors are what the code derives now; every rule has a vector that expects it (97 of
|
|
97); every artifact kind has a valid vector (13); every invalid vector is its base with one operation (137) or one declared
|
|
producer change (24); the published verifiers pass 206/206; the independent implementation passes all 62 it claims; and 9,333
|
|
malformed inputs (every path of every offline vector that is not invalid removed, or replaced by null, a string, a number, a list,
|
|
an object) given to the module verifiers each got a verdict, none threw. That row covers the modules of seven verifiers; the
|
|
trust chain verifier is not in it (each case would run the reference verifier several times), and the reference verifier, a
|
|
program, was tried apart (below). Without `--online` the nine chain vectors are reported unmeasured and the exit code is 2.
|
|
- `node proba-conformance.mjs --control-negativ --online`: 97 of 97 rules. Each rule is planted out of its verifier, one at a time, in
|
|
a copy (every check that carries its marker made true at run time; for `ai-trust`, its refusal helper made to drop the refusals of
|
|
that rule; for the reference verifier, the level made to pass); the suite is run against the planted verifier; the vectors that
|
|
expect the rule, **and only they**, must fail, and none may break. Then five controls of the instruments: an implementation that
|
|
calls everything valid fails every vector that is not plainly valid (165 of the 197 offline vectors); the coverage gate goes red when one rule's vector
|
|
is taken away; a rule added to a verifier and not mapped is refused by the derivation; a judge that throws again turns the
|
|
malformed-input row red; no planted copy is left behind.
|
|
- `node proba-conformance.mjs --pe-loc` is a measurement, not a gate: each of the 122 marked checks of the module verifiers
|
|
planted out alone (the trust chain verifier has no per-check markers and is not in this count). For 87 of them a vector
|
|
then fails. The others are checks whose vector trips a second check of the same rule at the same time, so removing one alone
|
|
changes nothing (the three checks of `market.replay`, the three of `agents.anchor`, the run and neighbour checks of
|
|
`dataset.completeness`), or checks no vector reaches (a dispute timed out before the arbitration timeout; five of the six checks
|
|
of `agents.time`). The control above shows that each **rule** is load-bearing, not each of its checks.
|
|
|
|
The suite found defects in the verifiers it enrols, on 2026-10-03, while they were being enrolled; all are fixed and each is now a
|
|
vector or a probe row:
|
|
|
|
- **three verifiers accepted a classical key under a post-quantum name.** The AIP-23 reference verifier reported a signature that
|
|
declared `ml-dsa-65` and was made with an Ed25519, Ed448, RSA or P-256 key as `signature PASSED: ml-dsa-65: verifies`; the audit
|
|
log verifier accepted a head "signed" the same way as a head signed with ML-DSA-65; the agents' ledger verifier did not require
|
|
the agent key to be an ML-DSA-65 key, so the ledger of a "post-quantum identity" could be written and signed with an Ed25519 key
|
|
and verify. In each, the key was never required to be a key of the scheme declared. Each is now a refusal by name
|
|
(`envelope/signed-head.signature-classical-key`, `audit-log/signed-head.head-scheme-classical-key`,
|
|
`agent-ledger/four-entries.key-classical`);
|
|
- the reference verifier exited with an exception on a file holding the JSON value `null`, let a statement that is a list pass its
|
|
form level, and reported a malformed ML-DSA signature (no key, a key that cannot be parsed) as unmeasured "needs OpenSSL 3.5" on a
|
|
runtime that has it; each is now a refusal by name, and a file that is not JSON is refused at the form level instead of ending
|
|
the run;
|
|
- a malformed payment made the risk module throw instead of refusing by name; malformed dataset answers made the dataset verifier
|
|
throw;
|
|
- the market judge accepted a timeline from which the submission event had been withheld, which skipped the challenge-window rule
|
|
(the judge now requires every event the job state implies: `market.complete`);
|
|
- the market judge took the time an execution envelope was notarized from the envelope's own claim, so a claim written earlier than
|
|
the truth passed. The time now comes from the reference verifier's reading under the anchor, and the claim is only compared with
|
|
it (`market-timeline/s1-28001.proof-claim-early`).
|
|
|
|
## What it is not, and what is not measured
|
|
|
|
- **Not a certification, not an audit, not a mark.** Passing says that an implementation agrees with the published verifiers on these
|
|
206 inputs. It does not say either of them is correct, and no third party has reviewed the verifiers, the vectors or this suite.
|
|
- **The vectors and the verifiers have the same authors.** The expected verdicts were written by the people who wrote the verifiers.
|
|
The second implementation is independent in code and language, not in authorship, and it covers 6 rules of 97; the other
|
|
91 rules, among them every rule of the trust chain, the audit log and the agents' ledger, are judged here by one implementation.
|
|
- **Finality is the chain's.** The level says that the post-quantum certificate of the covering anchor verified on the chain where the
|
|
artifact was notarized. Every notarized vector here is on chain 28001, a testnet; nothing in this suite ran on mainnet.
|
|
- **Five of the six finality refusals have no vector.** The reference verifier can refuse at its finality level in six places; the
|
|
vectors reach one (a notarization claimed that the chain does not hold). The others need an endpoint that lies or an anchor that
|
|
does not verify: they are exercised by that verifier's own probe, through a local proxy that lies to it, not by these vectors.
|
|
- **The chain vectors depend on the testnet's state.** The notary contract is open: anyone can notarize the fresh envelope's hash on
|
|
28001, and `envelope/fresh-28001.*` then change verdict. The runner would show it; `--proaspat` makes a new one.
|
|
- **Integrity is over the evidence supplied.** A market timeline and the transaction facts of a receipt are the caller's reading of the
|
|
chain; this suite does not certify that reading under an anchor. The market judge now refuses a timeline that lacks an event the
|
|
job state implies; it does not prove that the job state shown is the chain's.
|
|
- **Offline, a notarization time is only a claim.** With the chain, `market.proof` takes the time an execution envelope was first
|
|
seen from the reference verifier's reading under the anchor and refuses a claim that differs from it. Offline there is no such
|
|
reading: the claim is all there is, the verdict is `PARTIAL`, and a `certified` field that an artifact brings itself is dropped.
|
|
- **Authenticity is ML-DSA only.** The reference verifier checks ML-DSA signatures on envelopes, the audit log verifier the
|
|
ML-DSA-65 signature of a head, the ledger verifier the agent's ML-DSA-65 key and signatures, each with a key that must be of the
|
|
scheme declared; a hybrid signature (as on identity credentials) is reported unmeasured and the verdict is `PARTIAL`.
|
|
- **Most times in a trust chain and in a ledger are claims.** A trust chain without a witness is `PARTIAL`: every time in it is the
|
|
statement of whoever holds the keys. Three trust-chain vectors read a ledger head notarized on testnet 28001. No vector gives an
|
|
agent's ledger an anchor that verifies: `agents.anchor` is reached only through an anchor that is refused, and five of the six
|
|
checks of `agents.time` have no vector.
|
|
- **What a damaged artifact can still pass.** Of the 9,333 malformed inputs, 1,870 were still accepted, at 94 distinct paths: parts a
|
|
verifier treats as optional (the transaction facts of a receipt, the envelope of a timeline, the head of an audit log), a policy
|
|
field removed that falls back to its default, a log or a ledger without its last entry (still a valid shorter one), and fields no
|
|
rule reads
|
|
(`node proba-conformance.mjs --acceptate` lists them). A policy that cannot be defined makes the risk module throw, by design. The
|
|
reference envelope verifier, a program, was tried apart with 526 malformed envelopes (every field removed or replaced, files that
|
|
hold null, a number, a string, a list, text that is not JSON): each gets a verdict.
|
|
- **Not part of AIP-23.** The rule ids, the four levels and the verdict form are this suite's reading of the published verifiers;
|
|
AIP-23 defines the envelope. The identity verifiers published beside these are not enrolled.
|
|
- **The Python implementation** writes numbers the way JavaScript does only for integers and plain decimals, and sorts keys by code
|
|
point (JavaScript's order for keys in the Basic Multilingual Plane).
|