Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] Prevent memory copy and heap memory allocation when read entry #4259

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

dao-jun
Copy link
Member

@dao-jun dao-jun commented Apr 2, 2024

Motivation

Prevent memory copy when read entry.

Changes

(Describe: what changes you have made)

Master Issue: #


In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks.


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

  • Make sure the PR title is formatted like:
    <BP-#>: Description of bookkeeper proposal
    e.g. BP-1: 64 bits ledger is support
  • Attach the master issue link in the description of this PR.
  • Attach the google doc link if the BP is written in Google Doc.

Otherwise:

  • Make sure the PR title is formatted like:
    <Issue #>: Description of pull request
    e.g. Issue 123: Description ...
  • Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.
  • Replace <Issue #> in the title with the actual Issue number.

@dao-jun
Copy link
Member Author

dao-jun commented Apr 5, 2024

@lhotari @eolivelli @hangc0276 @zymap Could you please take a look?

@shoothzj
Copy link
Member

I am not familiar with netty reference count. I have two questions.

  1. why we need onSendResponseFinished? Can we just change to UnsafeByteOperations.unsafeWrap(entryBody.nioBuffer())? It may leak memory?
  2. I think extract a class field is a little hard to maintenance, can we still pass it by method?

@dao-jun
Copy link
Member Author

dao-jun commented Apr 10, 2024

I am not familiar with netty reference count. I have two questions.

  1. why we need onSendResponseFinished? Can we just change to UnsafeByteOperations.unsafeWrap(entryBody.nioBuffer())? It may leak memory?
  2. I think extract a class field is a little hard to maintenance, can we still pass it by method?

@shoothzj

  1. UnsafeByteOperations is protobuf‘s API, it will not release the memory, so we have to release it manually.
  2. Pass body by method will lead to more changes, this is the most simple way.

@dao-jun dao-jun changed the title [improve] Prevent memory copy when read entry [improve] Prevent memory copy and heap memory allocation when read entry Apr 16, 2024
@@ -350,6 +344,43 @@ protected void registerEvent(boolean failed, OpStatsLogger statsLogger, Stopwatc
}
}

ByteString parseToByteString(ByteBuf entryBody) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parse naming isn't optimal. A better name is unsafeWrapAsByteString etc. It's also worth adding a comment that the lifecycle of the ByteString continues to be managed with the ByteBuf lifecycle (retain/release).

@@ -350,6 +344,43 @@ protected void registerEvent(boolean failed, OpStatsLogger statsLogger, Stopwatc
}
}

ByteString parseToByteString(ByteBuf entryBody) {
ByteString aggregated = null;
ByteString piece;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having this variable introduced at this level isn't needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense to me, fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants