aere-network/aere-node
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

The heap note, corrected: the first version gave a fraction-of-RAM rule, and the measurement does not support it. Same machine, same database, four values at two heights: 2400m ran at ~82,000 blocks/min up to 8 M and then collapsed to ~640 with the old generation 99.88 percent full and 1,409 full GCs, while 3400m past 9.7 M did zero full GCs at ~28,800. Too large dies of OutOfMemoryError because the storage engine lives off-heap; too small dies of full GC once the post-quantum precompiles make blocks heavier. The advice is now to read jstat, not to pick a fraction

Browse Source
This commit is contained in:
Aere Network 2026-09-06 18:31:54 +03:00
parent 76095a942e
commit 84a6093517
1 changed files with 21 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
RUN-A-NODE.md
View File

@ -160,21 +160,29 @@ the network's reading nodes, and validates every hybrid anchor there, since 2026
## Sizing the JVM heap: bigger is not faster
Measured 2026-09-06 on one machine, one database, one chain, inside a single hour, syncing chain 2800
from genesis in a Linux VM with 8 GB of RAM:
Measured 2026-09-06 on one machine, one database and one chain, syncing chain 2800 from genesis in a
Linux VM with 8 GB of RAM. The same flag was measured at four values and at two different heights,
because the answer changes with the height:
| `-Xmx` | what happened | blocks per minute |
| --- | --- | --- |
| 4500m | 89 `OutOfMemoryError`, process at 7.1 GB RSS, import stopped dead | 0 |
| 3200m | still 24 OOM per 500 log lines, repeated `Chain download failed` | ~1,500 |
| 2400m | no OOM at all | ~82,000 |
| `-Xmx` | height reached | what the JVM did | blocks per minute |
| --- | --- | --- | --- |
| 4500m | ~3.5 M | 89 `OutOfMemoryError`, 7.1 GB RSS, import stopped dead | 0 |
| 3200m | ~6 M | 24 OOM per 500 log lines, repeated `Chain download failed` | ~1,500 |
| 2400m | 6-8 M | no OOM at all | ~82,000 |
| 2400m | past 9.2 M | old generation 99.88% full, 1,409 full GCs, 19 minutes in GC | ~640 |
| 3400m | past 9.7 M | **zero** full GCs, 6 seconds of GC in total | ~28,800 |
The reason is that **heap is not the same as memory**: Besu's storage engine keeps its block cache and
its write buffers OUTSIDE the Java heap, and at six million blocks that off-heap part is larger than the
heap itself. A heap sized to fill the machine leaves nothing for it, and the node spends its time in GC
and failed downloads instead of importing. Give the JVM roughly a third of the RAM you have, not all of
it, and leave the rest to the database and the operating system. If you see `OutOfMemoryError` in the
log, the first thing to try is a SMALLER `-Xmx`, not a larger one.
Two things are going on, and they pull in opposite directions. **Heap is not memory**: the storage
engine keeps its block cache and write buffers OUTSIDE the Java heap, and past six million blocks that
off-heap part is larger than the heap itself, so a heap sized to fill the machine leaves nothing for it
and the node dies of `OutOfMemoryError`. But a heap that is too small is just as bad in the other
direction: past block 9,189,161, where the post-quantum precompiles go live, blocks get heavier, the old
generation fills, and the node spends its time in full GC instead of importing.
So do not pick a fraction of RAM and hope. Watch it: `jstat -gcutil <pid> 5000 3` prints one line every
five seconds. If `FGC` climbs while the height barely moves, the heap is too SMALL - raise it. If you see
`OutOfMemoryError` in the log, it is too LARGE for what the database also needs - lower it. On 8 GB, the
band that worked here was roughly 2.4 GB early and 3.4 GB past nine million blocks.
## The configuration a follower passes (BESU_OPTS)
The Aere fork reads its per-node configuration from JVM system properties passed through the