Fix line skipping issue in receive_lines method

[yugeeklab]

Which issue(s) this PR fixes:
Fixes #4494

What this PR does / why we need it:
Before this patch, long lines could cause breakdowns in fluentd, potentially posing a vulnerability. With this patch, max_line_size will be integrated into the FIFO, enabling the system to skip lines exceeding the maximum size before executing receive_lines.

Docs Changes:

Release Note:

[daipom]

@yugeeklab Thanks for this fix!
CI is currently unstable because of #4487. We will fix it. Sorry for the trouble.

I see the intent of this fix as follows.

• In the current implementation, large lines that would eventually be discarded in receive_lines are temporarily held in IOHandler's @lines.
• This is a waste of memory.
• This PR resolves the waste.

Surely, such a fix would allow us to limit memory consumption by the max_line_size setting to some extent!

This PR would be effective to some extent, however I believe the problem of memory consumption will remain.
It would be possible that FIFO's @buffer becomes unlimitedly large if the @eol does not appear in the data.

Are these my understandings correct?

[yugeeklab]

Hi, @daipom

I've just published an issue #4491 for more information.

This PR would be effective to some extent, however I believe the problem of memory consumption will remain.
It would be possible that FIFO's @buffer becomes unlimitedly large if the @EOL does not appear in the data.

When max_line_size isn't set, FIFO's @buffer can grow indefinitely. Or if max_line_size has large value, FIFO's buffer will be limited, but there's still a possibility of fluentd experiencing slowdowns.

Summary:

as-is: max_line_size helps you avoid buffer overflow configuring via buffer section.
to-be: max_line_size helps prevent buffer overflow by configuring the buffer section and also ensures FIFO's buffer size remains limited.

If you have any suggestions, such as the fifo_buffer_size parameter or any other ideas, please feel free to discuss them with me.

Thank you for your review!

[daipom]

@yugeeklab

I've just published an issue #4491 for more information.

Thanks so much!

as-is: max_line_size helps you avoid buffer overflow configuring via buffer section. to-be: max_line_size helps prevent buffer overflow by configuring the buffer section and also ensures FIFO's buffer size remains limited.

Now I understand!
The following understanding was not correct.

This PR would be effective to some extent, however I believe the problem of memory consumption will remain.
It would be possible that FIFO's @buffer becomes unlimitedly large if the @eol does not appear in the data.

This fix clears FIFO's @buffer when read_lines.
So, this fix ensures FIFO's buffer size remains limited.

If you have any suggestions, such as the fifo_buffer_size parameter or any other ideas, please feel free to discuss them with me.

Thanks!
Basically, it seems to be a very good idea to limit the FIFO's buffer.

[yugeeklab]

Basically, it seems to be a very good idea to limit the FIFO's buffer.

Thank you for your comment!! @daipom

Please let me know if there's any feedback on my code or idea. I'll review and accept your feedback as soon as possible.

Thank you.

[daipom]

About CI failures, although #4493 has been resolved, we still need to resolve #4487.

[daipom]

@yugeeklab The CI issue has been resolved. Sorry for the trouble.
Could you please rebase this branch on the latest master?

[yugeeklab]

Hi, @daipom

Rebase is done!!

Thank you for your review!!