Allow Importing of Historical Blobs

[nisdas]

🚀 Feature Request

Description

With deneb having been activated for more than a month, all prysm full nodes have begun pruning old and expired blobs (beyond 4096 epochs). While this has gone on fine, it has caused issues for power users and those syncing a fresh L2 node. Without these blobs, it becomes more complicated to derive the current L2 state from an older snapshot.

Describe the solution you'd like

There are a few possible/viable solutions for historical blob importing:

1. Using an alternate beacon rpc url. This url is only ever utilized when we are trying to backfill historical blobs for a checkpoint synced node. The beacon node would first check that particular block is within the blob retention period. If the block is beyond the blob retention period, it would request that block's blobs from the provided alternate beacon url.

2. Using the filesystem to import blobs. If the blobs exist in an alternate directory, we can simply import blobs from that directory to our current directory. However if the blobs do actually exist in another path, the question arises whether it makes sense to have a specific command in prysm to achieve this. It might be more straightforward for the node operator to copy + paste manually using the filesystem.

Describe alternatives you've considered

N.A , a relevant issue for context:

OffchainLabs/nitro#2231
sigp/lighthouse#5391

[nisdas]

@kaber2 OffchainLabs/nitro#2231 (comment) Taking the conversation here. Backfilling blocks and blobs are separate concerns. Every full node should have all the historical blocks, but there is no guarantee that a full node will have historical blobs. If we do request blobs beyond the retention period(4096 epochs) then it is pretty likely that an average peer in the network will not hold it because they have already pruned it.

Purpose of this issue is to allow anyone who wants to hold all the historical blobs to simply provide a historical blob rpc provider, and prysm will request it from them and fill it internally on its blob directory.

[ZYS980327]

@nisdas So how should we fix this data, I restarted a prysm and added the flag --blob-retention-epochs=999999999, don't know if it's useful, but still see some logs printed about fetch batches or slot failures, where the peer doesn't have these data

[nisdas]

@ZYS980327 Like i mentioned earlier, you can use the archival rpc provider to sync the missing blobs till your arbitrum node is at its tip and then switch over to using your local prysm node.

[ZYS980327]

@nisdas I don't know how to reboot the nitro node, every reboot is resynced at the very beginning of the snapshot, I redeployed the Ethereum nodes geth and prysm today, hope it helps

[nisdas]

I am pretty sure this is your issue:
OffchainLabs/nitro#2231 (comment)

If you remove that flag it should be fine. You should never be using that flag for any long lived process with persistent state as it will remove all the data.

[ZYS980327]

@nisdas Yes, I removed this flag today, but it still doesn't work, nitro is which directory was retrieved

[nisdas]

When you stop the process, can you confirm the directory exists ?

[ZYS980327]

@nisdas I'm using docker kill [arg], the directory should be in the docker-related directory, it always exists, it's all hash value directory, I just don't know which one

[kaber2]

@kaber2 OffchainLabs/nitro#2231 (comment) Taking the conversation here. Backfilling blocks and blobs are separate concerns. Every full node should have all the historical blocks, but there is no guarantee that a full node will have historical blobs. If we do request blobs beyond the retention period(4096 epochs) then it is pretty likely that an average peer in the network will not hold it because they have already pruned it.

Purpose of this issue is to allow anyone who wants to hold all the historical blobs to simply provide a historical blob rpc provider, and prysm will request it from them and fill it internally on its blob directory.

According to the release announcement (https://github.com/prysmaticlabs/prysm/releases/tag/v5.0.0), this flag does support backfilling blobs. After adding the aforementioned flag, my node did sync a large number of blobs from the network and is now at:

$ ls |wc -l
61361

According to the logs, backfilling is complete. So just to be 100% clear, you are saying that I still don't hold all blobs?

[nisdas]

You will only backfill blobs till the retention period(4096 epochs). There is no guarantee that the p2p network will be able to supply older blobs beyond that, which is why we do not sync older blobs.

[kaber2]

I see. Thank you for the explanation. I'll try to find a suitable RPC provider then.