From 84a60935174cf2ded5b7df3e75f6d0f7c6f99fb3 Mon Sep 17 00:00:00 2001 From: Aere Network Date: Sun, 6 Sep 2026 18:31:54 +0300 Subject: [PATCH] The heap note, corrected: the first version gave a fraction-of-RAM rule, and the measurement does not support it. Same machine, same database, four values at two heights: 2400m ran at ~82,000 blocks/min up to 8 M and then collapsed to ~640 with the old generation 99.88 percent full and 1,409 full GCs, while 3400m past 9.7 M did zero full GCs at ~28,800. Too large dies of OutOfMemoryError because the storage engine lives off-heap; too small dies of full GC once the post-quantum precompiles make blocks heavier. The advice is now to read jstat, not to pick a fraction --- RUN-A-NODE.md | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/RUN-A-NODE.md b/RUN-A-NODE.md index fcf31cb..fcfc54a 100644 --- a/RUN-A-NODE.md +++ b/RUN-A-NODE.md @@ -160,21 +160,29 @@ the network's reading nodes, and validates every hybrid anchor there, since 2026 ## Sizing the JVM heap: bigger is not faster -Measured 2026-09-06 on one machine, one database, one chain, inside a single hour, syncing chain 2800 -from genesis in a Linux VM with 8 GB of RAM: +Measured 2026-09-06 on one machine, one database and one chain, syncing chain 2800 from genesis in a +Linux VM with 8 GB of RAM. The same flag was measured at four values and at two different heights, +because the answer changes with the height: -| `-Xmx` | what happened | blocks per minute | -| --- | --- | --- | -| 4500m | 89 `OutOfMemoryError`, process at 7.1 GB RSS, import stopped dead | 0 | -| 3200m | still 24 OOM per 500 log lines, repeated `Chain download failed` | ~1,500 | -| 2400m | no OOM at all | ~82,000 | +| `-Xmx` | height reached | what the JVM did | blocks per minute | +| --- | --- | --- | --- | +| 4500m | ~3.5 M | 89 `OutOfMemoryError`, 7.1 GB RSS, import stopped dead | 0 | +| 3200m | ~6 M | 24 OOM per 500 log lines, repeated `Chain download failed` | ~1,500 | +| 2400m | 6-8 M | no OOM at all | ~82,000 | +| 2400m | past 9.2 M | old generation 99.88% full, 1,409 full GCs, 19 minutes in GC | ~640 | +| 3400m | past 9.7 M | **zero** full GCs, 6 seconds of GC in total | ~28,800 | -The reason is that **heap is not the same as memory**: Besu's storage engine keeps its block cache and -its write buffers OUTSIDE the Java heap, and at six million blocks that off-heap part is larger than the -heap itself. A heap sized to fill the machine leaves nothing for it, and the node spends its time in GC -and failed downloads instead of importing. Give the JVM roughly a third of the RAM you have, not all of -it, and leave the rest to the database and the operating system. If you see `OutOfMemoryError` in the -log, the first thing to try is a SMALLER `-Xmx`, not a larger one. +Two things are going on, and they pull in opposite directions. **Heap is not memory**: the storage +engine keeps its block cache and write buffers OUTSIDE the Java heap, and past six million blocks that +off-heap part is larger than the heap itself, so a heap sized to fill the machine leaves nothing for it +and the node dies of `OutOfMemoryError`. But a heap that is too small is just as bad in the other +direction: past block 9,189,161, where the post-quantum precompiles go live, blocks get heavier, the old +generation fills, and the node spends its time in full GC instead of importing. + +So do not pick a fraction of RAM and hope. Watch it: `jstat -gcutil 5000 3` prints one line every +five seconds. If `FGC` climbs while the height barely moves, the heap is too SMALL - raise it. If you see +`OutOfMemoryError` in the log, it is too LARGE for what the database also needs - lower it. On 8 GB, the +band that worked here was roughly 2.4 GB early and 3.4 GB past nine million blocks. ## The configuration a follower passes (BESU_OPTS) The Aere fork reads its per-node configuration from JVM system properties passed through the