Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations #5213

Merged

Conversation

damnever
Copy link
Contributor

@damnever damnever commented Mar 14, 2023

CPU usage can be reduced by up to 25% with this optimization.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@damnever damnever force-pushed the perf/grpc-snappy-decompressed-size branch from 4bcb81d to 7b54d0c Compare March 14, 2023 07:56
@alanprot
Copy link
Member

This is nice!!! I wanted to do this before as well!! :D

Did u see any improvements on ingesters CPU?

@damnever
Copy link
Contributor Author

damnever commented Mar 15, 2023

Did u see any improvements on ingesters CPU?

@alanprot In our test environment, without heavy queries, the CPU usage can be reduced by up to 25%, roughly 10% on average.

…ory allocations

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
@damnever damnever force-pushed the perf/grpc-snappy-decompressed-size branch from 7b54d0c to 8ba5aa7 Compare March 16, 2023 03:08
@alanprot
Copy link
Member

LGTM. Thanks

@yeya24 yeya24 merged commit 9bc9fbf into cortexproject:master Mar 17, 2023
@alanprot
Copy link
Member

alanprot commented Mar 17, 2023

Ok..

Seems that something weird here: DecodedLen does not return the size of the uncompressed data but rather the size of the decoded block.

I was writing unit tests for this and i noticed that:

ex:

				buf = bytes.NewBuffer(compressedBytes)
				r, err := c.Decompress(buf)
				require.NoError(t, err)
				out, err := io.ReadAll(r)
				assert.Equal(t, len(out), sizer.DecompressedSize(compressedBytes))
Expected :18216
Actual   :895

I think we cannot use decodeLen when using stream. From https://pkg.go.dev/github.com/klauspost/compress/snappy#section-readme

There are actually two Snappy formats: block and stream. They are related, but different: trying to decompress block-compressed data as a Snappy stream will fail, and vice versa. The block format is the Decode and Encode functions and the stream format is the Reader and Writer types.

@alanprot alanprot mentioned this pull request Mar 18, 2023
3 tasks
alexqyle pushed a commit to alexqyle/cortex that referenced this pull request Mar 19, 2023
…ory allocations (cortexproject#5213)

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
alexqyle added a commit to alexqyle/cortex that referenced this pull request Mar 20, 2023
…mize memory allocations (cortexproject#5213)"

This reverts commit 4821ba3.

Signed-off-by: Alex Le <leqiyue@amazon.com>
alvinlin123 pushed a commit that referenced this pull request Mar 20, 2023
* Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213)

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>

* Fix S3 BucketWithRetries upload empty content issue

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Update CHANGELOG

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Revert "Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213)"

This reverts commit 4821ba3.

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Only retry if input reader is seekable

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Rename mock type

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Add logging

Signed-off-by: Alex Le <leqiyue@amazon.com>

* nit fixing

Signed-off-by: Alex Le <leqiyue@amazon.com>

* add comment

Signed-off-by: Alex Le <leqiyue@amazon.com>

---------

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
Co-authored-by: Xiaochao Dong <the.xcdong@gmail.com>
alexqyle pushed a commit to alexqyle/cortex that referenced this pull request May 2, 2023
…ory allocations (cortexproject#5213)

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants