Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(meta): support rocksdb store mode #3289

Closed
wants to merge 45 commits into from

Conversation

NaturalSelect
Copy link
Collaborator

@NaturalSelect NaturalSelect commented Mar 27, 2024

Main Issue: #3350

Motivation

Implement rocksdb store mode that describe in #3350.

Modifications

  • Each disk own a rocksdb instance, so we can share block cahce and write buffer. Using prefix key to isolate different meta partitions.
  • Support migrate a memory meta partition(replica) to rocksdb meta partition.
  • Refactor behavior of extent deletion, use a new kind of tree to store deleted extent data.
  • Monitor available space of each disk and report it to master, adjust our node and nodeset selector to support disk space based meta partition placement.
  • Support configurate default meta partition store mode of volume.

Types of changes


  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Bugfix (non-breaking change which fixes an issue)
  • Documentation Update (if none of the other choices apply)
  • So on...

Verifying this change


  • Make sure that the change passes the testing checks.

This change added tests and can be verified as follows:

  • Add rocksdb version of existing unit tests, execute docker/run_docker.sh --test to verify.
  • S3 test now using a rocksdb volume, execute docker/run_docker.sh --s3test to verify.

Does this pull request potentially affect one of the following parts:


  • Master
  • MetaNode
  • DataNode
  • ObjectNode
  • AuthNode
  • LcNode
  • Blobstore
  • Client
  • Cli
  • SDK
  • Other Tools
  • Common Packages
  • Dependencies
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Review Expection

  • in-two-days
  • weeks
  • free-time
  • whenever

Matching PR in forked repository

PR in forked repository:

zhangchuanqing5658 and others added 29 commits February 4, 2024 20:07
Signed-off-by: zhangchuanqing1 <zhangchuanqing1@jd.com>
Signed-off-by: NaturalSelect <2145973003@qq.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <2145973003@qq.com>
Signed-off-by: NaturalSelect <2145973003@qq.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Copy link

netlify bot commented Mar 27, 2024

Deploy Preview for cubefs-check ready!

Name Link
🔨 Latest commit 63770f6
🔍 Latest deploy log https://app.netlify.com/sites/cubefs-check/deploys/66502c46c520bf00086afec7
😎 Deploy Preview https://deploy-preview-3289--cubefs-check.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@NaturalSelect NaturalSelect marked this pull request as ready for review March 27, 2024 08:34
@NaturalSelect NaturalSelect marked this pull request as draft March 27, 2024 08:34
NaturalSelect and others added 15 commits April 2, 2024 10:13
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
Signed-off-by: NaturalSelect <2145973003@qq.com>
Signed-off-by: NaturalSelect <2145973003@qq.com>
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
@NaturalSelect NaturalSelect linked an issue May 23, 2024 that may be closed by this pull request
1 task
Signed-off-by: NaturalSelect <huangzhibin1@oppo.com>
@leonrayang leonrayang closed this May 28, 2024
@leonrayang leonrayang deleted the metanode_rocksdb_dev branch May 28, 2024 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: MetaNode support rocksdb store mode
3 participants