Optimistically queueing unsafe L2 execution payload

[yuanwj210]

version:op-node-v1.7.3
system:ubuntu20.04

After the update to version 1.7.3, the op-geth is stuck, the op-node shows the following log, and the wait for L1 is very long

t=2024-04-16T01:54:07+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801905 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:07+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835
t=2024-04-16T01:54:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:09+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836
t=2024-04-16T01:54:11+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801861 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:11+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837
t=2024-04-16T01:54:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838

[yuanwj210]

@shazow @mslipper @fgimenez

[jwelch-qn]

I am also seeing this at times. Block height will struggle to stay at tip for a while and the logs are full of

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xdc78579b115e5a967ae549af90ab97b1c22a2d8079a85af3e3640fa0205868b9:118944857

and

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=error msg="Payload execution failed" block_hash=0x688b7bebbdacc9a26d5fd2b55152ea49f627a730d27980ae20904a359200685f err="context deadline exceeded"

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"

Running version Geth/v1.101311.0-stable-e9a306ba

[wangjiangw]

we have also found this issue, which makes the block syncing latency become bigger ?

[jwelch-qn]

Yes, nodes start to struggle to stay at blockchain height, they generally resolve on their own, but this appears to be a recent behavior.

[wangjiangw]

jwelch-qn

yes, it will recover at some time in the future, and this happens just in recent monthes