Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beacon node stops syncing because >64998 subdirectories in blobs folder #13880

Open
Belkaar opened this issue Apr 14, 2024 · 1 comment · May be fixed by #13884 or #14023
Open

Beacon node stops syncing because >64998 subdirectories in blobs folder #13880

Belkaar opened this issue Apr 14, 2024 · 1 comment · May be fixed by #13884 or #14023
Labels
Bug Something isn't working

Comments

@Belkaar
Copy link

Belkaar commented Apr 14, 2024

Describe the bug

The Beacon stops syncing at slot 8854183. It has created 64999 directories in the blobs folder and the fs (ext4 in this case) does not allow any more (without setting dir_nlink feature)

A workaround is setting dir_nlink on the ext4 fs:
tune2fs -O dir_nlink

Maybe the client should try to avoid this by using a deeper directory structure to store hashes?

Has this worked before in a previous version?

Unknown

🔬 Minimal Reproduction

  1. Put data dir on normally formated ext4 fs
  2. Try to sync beacon chain

Error

Apr 14 10:06:44 eth-consensus beacon-chain_active[72397]: time="2024-04-14 10:06:44" level=warning msg="Skip processing batched blocks" error="could not process block in batch: could not validate blob data availability at slot 8854183: failed to save BlobSidecar index 0 for block 0xe546bfeee9cdfb74664dafded853a707e342b40691863eea6929a4f1eef61fa3: mkdir /mnt/store_0/beacon/blobs/0xe546bfeee9cdfb74664dafded853a707e342b40691863eea6929a4f1eef61fa3: too many links" prefix=initial-sync

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

5.0.3

Anything else relevant (validator index / public key)?

No response

@Belkaar Belkaar added the Bug Something isn't working label Apr 14, 2024
@Belkaar
Copy link
Author

Belkaar commented Apr 14, 2024

The workaround and the limitation can be found here:
https://man7.org/linux/man-pages/man5/ext4.5.html

@kasey kasey linked a pull request Apr 16, 2024 that will close this issue
@kasey kasey linked a pull request May 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
1 participant