-
Notifications
You must be signed in to change notification settings - Fork 780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations #5213
Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations #5213
Conversation
4bcb81d
to
7b54d0c
Compare
This is nice!!! I wanted to do this before as well!! :D Did u see any improvements on ingesters CPU? |
@alanprot In our test environment, without heavy queries, the CPU usage can be reduced by up to 25%, roughly 10% on average. |
…ory allocations Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
7b54d0c
to
8ba5aa7
Compare
LGTM. Thanks |
Ok.. Seems that something weird here: I was writing unit tests for this and i noticed that: ex:
I think we cannot use decodeLen when using stream. From https://pkg.go.dev/github.com/klauspost/compress/snappy#section-readme
|
…ory allocations (cortexproject#5213) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…mize memory allocations (cortexproject#5213)" This reverts commit 4821ba3. Signed-off-by: Alex Le <leqiyue@amazon.com>
* Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Fix S3 BucketWithRetries upload empty content issue Signed-off-by: Alex Le <leqiyue@amazon.com> * Update CHANGELOG Signed-off-by: Alex Le <leqiyue@amazon.com> * Revert "Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213)" This reverts commit 4821ba3. Signed-off-by: Alex Le <leqiyue@amazon.com> * Only retry if input reader is seekable Signed-off-by: Alex Le <leqiyue@amazon.com> * Rename mock type Signed-off-by: Alex Le <leqiyue@amazon.com> * Add logging Signed-off-by: Alex Le <leqiyue@amazon.com> * nit fixing Signed-off-by: Alex Le <leqiyue@amazon.com> * add comment Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com>
…ory allocations (cortexproject#5213) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
CPU usage can be reduced by up to 25% with this optimization.
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]