stored: add dedup backend #1663
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
from
5bfa1f6
to
cb416e8
Compare
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
2 times, most recently
from
095f8f7
to
2c6f955
Compare
|
One thing i was not sure about was the error reporting. Currently the device itself outputs error messages whenever they are caught and prints an Emsg with it. This makes it easier to give good error reports but there will probably be a lot of double reports ala: An alternative would be to set errno appropriately (printing a dmsg instead) and let the normal bareos routines handle the error reporting. The problem is that some problems do not have an associated error number since they might be internal logic errors so there is no way to give a helpful message in that case. |
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
2 times, most recently
from
4a0948a
to
61553e0
Compare
These are mostly used for parsing device options
Now accepts any number of datafiles. This makes a lot of code much cleaner since you do not need to constantly find out if a certain data file is the "aligned" one or not.
It was previously not possible to abort once a commit was started (since the savestate was moved immediately on the CommitBlock call). This was fixed by instead waiting for the commit to finish before issueing the move.
This allows us to split up records into multiple parts with desirable sizes. For example a 129k record may be split into a 128k part and a 1k part so that at least the first 128k are dedupable. The record header now is not treated special anymore. Its just another (tiny) bit of data.
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
from
5fdb6e7
to
5386512
Compare
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
from
4ddc678
to
e7f1f7f
Compare
Previously we tracked the available capacity instead but this caused problems with mmap wanting offsets page aligned and us not being able to guarantee that cap * element_size is page aligned.
sebsura
force-pushed
the
dev/ssura/fvec/dedup
branch
from
de34fb5
to
8aa5c24
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to the Bareos Project!
The dedup backend makes it possible for deduplicating filesystems to deduplicate backed up data.
Based of pr #1662
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
Make sure you check/merge the PR using
devtools/pr-toolto have some simple automated checks run and a proper changelog record added.General
Source code quality
Tests