Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't raise exception when each message size is smaller than chunk_limit_size #3560

Merged
merged 2 commits into from
Dec 24, 2021
Merged

Don't raise exception when each message size is smaller than chunk_limit_size #3560

merged 2 commits into from
Dec 24, 2021

Commits on Dec 23, 2021

  1. Don't raise exception when each message size is smaller enough 

    Follow up fluent#3553

In the previous versions, even though each record size is smaller
than chunk limit size, but whole message size exceeds
chunk limit size, BufferChunkOverflowError is raised unexpectedly.

For example, if chunk limit size is 1_280_000, when processing 3 event
stream (every 1_000_000 bytes), it throws an exception like this:

  Fluent::Plugin::Buffer::BufferChunkOverflowError(<a 1000025 bytes
  record (nth: 1) is larger than buffer chunk limit size, a 1000025
  bytes record (nth: 2) is larger than buffer chunk limit size>)

Now changed not to raise exception if it's record size is smaller
enough than chunk limit size. Thus each message is stored into
separated chunks.

The idea is based on that adding byte size is smaller than chunk
limit size, chunk should be unstaged and pushed into queue,
If not, it should be skipped like fluent#3553.

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
    @kenhys @ashie
    kenhys authored and ashie committed Dec 23, 2021
    Configuration menu
    Copy the full SHA
    aad2feb View commit details
    Browse the repository at this point in the history

  2. buffer: Make clear the code for checking BufferChunkOverflow 

    Checking original_bytesize isn't needed to detect a big record than
chunk_limit_size.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
    @ashie
    ashie committed Dec 23, 2021
    Configuration menu
    Copy the full SHA
    5d93f36 View commit details
    Browse the repository at this point in the history