Pre-warm intra block cache during block execution

[LukaszRozmej]

Changes

• During block processing, run the pending set of transactions in parallel using available processors to populate the read cache; for the sequential run of transactions
• Results in up to 2x performance when processing blocks.

Types of changes

What types of changes does your code introduce?

• New feature (a non-breaking change that adds functionality)
• Optimization

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

Test eth_call and tracing.
Test on blocks that have self-destruct (archive sync?)

Documentation

Requires documentation update

• Yes
• No

Requires explanation in Release Notes

• Yes
• No

[benaadams]

Hive tests https://github.com/NethermindEth/nethermind/actions/runs/9228022573

[smartprogrammer93]

It looks good. One thing, can we have a config that disables the prewarmer in case it results in worse perf in some machines?

[benaadams]

Hive tests https://github.com/NethermindEth/nethermind/actions/runs/9228022573

Bunch of hive tests fail (all Paris)

Failed: 30

✖ Bad Hash on NewPayload (Syncing=true, Sidechain=true) (Paris) (nethermind)
✖ GetPayloadBodiesByHash (Empty Transactions/Withdrawals) (Paris) (nethermind)
✖ Inconsistent Finalized in ForkchoiceState (Paris) (nethermind)
✖ Inconsistent Safe in ForkchoiceState (Paris) (nethermind)
✖ Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P9 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=True, Invalid P9 (Paris) (nethermind)
✖ Invalid NewPayload, PrevRandao, Syncing=False, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, PrevRandao, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, ReceiptsRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, StateRoot, Syncing=False, EmptyTxs=True, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Timestamp, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction ChainID, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction GasTipCapPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Nonce, Syncing=False, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Signature, Syncing=False, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid PayloadAttributes, Zero timestamp, Syncing=False (Paris) (nethermind)
✖ Invalid PayloadAttributes, Zero timestamp, Syncing=True (Paris) (nethermind)
✖ Payload Build after New Invalid Payload: Invalid StateRoot (Paris) (nethermind)
✖ PrevRandao Opcode Transactions Test, DynamicFeeTransactions (Paris) (nethermind)
✖ Re-Org Back into Canonical Chain, Depth=10, Execute Side Payload on Re-Org (Paris) (nethermind)
✖ Sidechain Reorg (Paris) (nethermind)

[benaadams]

Hive tests https://github.com/NethermindEth/nethermind/actions/runs/9233024417

[benaadams]

Failed: 23

✖ GetPayloadBodiesByHash (Empty Transactions/Withdrawals) (Paris) (nethermind)
✖ Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P1 (Paris) (nethermind)
✖ Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P10 (Paris) (nethermind)
✖ Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P9 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=True, Invalid P9 (Paris) (nethermind)
✖ Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Paris) (nethermind)
✖ Invalid NewPayload, Extra VersionedHashes, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Cancun) (nethermind)
✖ Invalid NewPayload, GasLimit, Syncing=False, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, GasLimit, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Incomplete VersionedHashes, Syncing=False, EmptyTxs=False, DynFeeTxs=False (Cancun) (nethermind)
✖ Invalid NewPayload, Number, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, ParentHash, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid NewPayload, Transaction Value, Syncing=True, EmptyTxs=False, DynFeeTxs=True (Paris) (nethermind)
✖ Invalid PayloadAttributes, Parent timestamp, Syncing=False (Paris) (nethermind)
✖ Re-Org Back into Canonical Chain, Depth=5 (Paris) (nethermind)
✖ Re-Org Back to Canonical Chain From Syncing Chain (Paris) (nethermind)
✖ Transaction Re-Org, Re-Org Out (Paris) (nethermind)

[benaadams]

Hive tests pass https://github.com/NethermindEth/nethermind/actions/runs/9234113726

[benaadams]

Consensus hive failures are recent EIP7610 tests ethereum/EIPs#8161

[asdacap]

Not sure about passing in PreBlockCaches in CreateResettableWorldState. But it should work fine.

[LukaszRozmej]

Not sure about passing in PreBlockCaches in CreateResettableWorldState. But it should work fine.

I took big concern passing it there, but it is easy way I found to support default behavior without those caches and then support behavior with caches when needed.