Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools: add dedup estimation tool bdedupestimate #1654

Merged
merged 28 commits into from
May 23, 2024

Conversation

sebsura
Copy link
Contributor

@sebsura sebsura commented Jan 9, 2024

Thank you for contributing to the Bareos Project!

This pr adds a tool with which you can estimate how much space you could save when using deduplication.

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
  • Check backport line
  • Required backport PRs have been created
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

@pstorz pstorz self-requested a review January 18, 2024 10:30
@pstorz pstorz self-assigned this Jan 18, 2024
@pstorz pstorz changed the title tools: add dedup estimation tool tools: add dedup estimation tool bdedupestimate Feb 2, 2024
Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a manpage for bdedupestimate please.

@sebsura sebsura force-pushed the dev/ssura/master/dedup-estimate branch from f4a24a2 to db878a4 Compare March 5, 2024 09:29
@sebsura sebsura requested a review from pstorz March 18, 2024 14:38
Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it make sense to have a system test that runs bdedupestimate?

core/src/tools/CMakeLists.txt Show resolved Hide resolved
docs/manuals/source/man/bdedupestimate.rst Outdated Show resolved Hide resolved
docs/manuals/source/man/bdedupestimate.rst Outdated Show resolved Hide resolved
Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • dedup factor format should be 2.34x (like ZFS does the output) and not x2.34. See comment
  • Some commits still contain "Fixup"

core/src/tools/bdedup-estimate.cc Outdated Show resolved Hide resolved
@sebsura sebsura force-pushed the dev/ssura/master/dedup-estimate branch from b178f01 to 1ef2048 Compare May 14, 2024 14:06
@sebsura sebsura requested a review from pstorz May 14, 2024 14:25
@pstorz pstorz force-pushed the dev/ssura/master/dedup-estimate branch from 1ef2048 to c3c347e Compare May 23, 2024 12:32
sebsura and others added 22 commits May 23, 2024 13:07
With this flag enabled we match complete records instead of
subdividing records into datablocks.
dedup_units can now have different sizes if record based dedup is
enabled.  As such we should only compare the sha results if the sizes
are equal.
Now that records are split into multiple blocks to make deduplication
useful even if backing up smaller files, we need to adjust the
estimate logic to account for that.
@BareosBot BareosBot force-pushed the dev/ssura/master/dedup-estimate branch from f995f55 to 88652ef Compare May 23, 2024 13:07
@BareosBot BareosBot merged commit 064cca0 into bareos:master May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants