Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add log throttling per file (revised) #3185

Merged
merged 38 commits into from May 11, 2021
Merged

Add log throttling per file (revised) #3185

merged 38 commits into from May 11, 2021

Commits on May 6, 2021

  1. in_tail: Handle log throttling per file feature 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
Co-authored-by: Anthony Comtois <anthony.comtois@sky.uk>
    @cosmo0920 @rewiko
    cosmo0920 and rewiko committed May 6, 2021
    Configuration menu
    Copy the full SHA
    d6755ed View commit details
    Browse the repository at this point in the history

  2. in_tail: Use size config type for read_bytes_limit_per_second 

    Because file size should be specified as numbers with SI prefix
instead of raw numbers.

Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    a5d3a5a View commit details
    Browse the repository at this point in the history

  3. in_tail: Use actual read bytes to calculate read bytes number 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    d3d7119 View commit details
    Browse the repository at this point in the history

  4. in_tail: Use more meaningful variable name 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    30a63ca View commit details
    Browse the repository at this point in the history

  5. in_tail: Use fixed value instead of variable 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    61554dc View commit details
    Browse the repository at this point in the history

  6. in_tail: test: Add log throttling testcases for w/o stat watcher situ… 

    …ations

Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    a54f25b View commit details
    Browse the repository at this point in the history

  7. in_tail: Guard for read empty lines 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    2b2260e View commit details
    Browse the repository at this point in the history

  8. Use Thread Pool on tailing to prevent Kernel#sleep for entire sleep 

    Because the previous in_tail implementation is single threaded.
Thus, calling Kernel#sleep pauses entire tailing thread and in_tail
plugin.

Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    8ebc227 View commit details
    Browse the repository at this point in the history

  9. in_tail: Use Fluent::EventTime instead of Time 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    6e8a13c View commit details
    Browse the repository at this point in the history

  10. in_tail: Check invalid max_thread_pool_size on #configure 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    5b19e40 View commit details
    Browse the repository at this point in the history

  11. in_tail: Set default value of max_thread_pool_size as 1 

    Because when not using log throttling, is is enough to run on single
thread.

Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    854cca8 View commit details
    Browse the repository at this point in the history

  12. in_tail: Use 2 or more thread pool size on log throttling testcases 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    f76f45a View commit details
    Browse the repository at this point in the history

  13. in_tail: Add warning for specifying less equal than 8192 on read_byte… 

    …s_limit_per_second

Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4b2526b View commit details
    Browse the repository at this point in the history

  14. in_tail: Check thread pool objects existences 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    f88c4ae View commit details
    Browse the repository at this point in the history

  15. in_tail: Use more strict parameter checking for log throttling feature 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    61ae7ca View commit details
    Browse the repository at this point in the history

  16. in_tail: thread_pool: Add tailing thread poll testcases 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    de6bef3 View commit details
    Browse the repository at this point in the history

  17. Use 8192 bytes on number_bytes_read calculation 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    48d9be3 View commit details
    Browse the repository at this point in the history

  18. Fix a typo 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4d9a755 View commit details
    Browse the repository at this point in the history

  19. Use constant instead of interger values 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    a4459ae View commit details
    Browse the repository at this point in the history

  20. in_tail: Fix unstable testcases 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    4da4ae3 View commit details
    Browse the repository at this point in the history

  21. in_tail: Use thread pool only when log throttling feature is enabled 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    f29309b View commit details
    Browse the repository at this point in the history

  22. in_tail: Fix incorrect #join usage 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    490a329 View commit details
    Browse the repository at this point in the history

  23. in_tail: Use Fluent::Clock.new to obtain starting time 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    f1807c9 View commit details
    Browse the repository at this point in the history

  24. in_tail: Use Fluent::Clock.now instead of Fluent::EventTime.now again 

    Signed-off-by: Hiroshi Hatake <hatake@clear-code.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    7023b3f View commit details
    Browse the repository at this point in the history

  25. in_tail: Prevent to call for nil instance 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    ae9deef View commit details
    Browse the repository at this point in the history

  26. in_tail: Prevent to reuse @thread_pool 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    8c63cb1 View commit details
    Browse the repository at this point in the history

  27. in_tail: test: Use safe file removing method 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    43e8f4f View commit details
    Browse the repository at this point in the history

  28. in_tail: Remove needless start_reading variable reset 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    808e27e View commit details
    Browse the repository at this point in the history

  29. Use #bytesize for calculating read bytes 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    1c88b0c View commit details
    Browse the repository at this point in the history

  30. in_tail: thread_pool: Check running status properly 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    e3e9345 View commit details
    Browse the repository at this point in the history

  31. in_tail: thread_pool: Detect running or not with only queue is closed 

    This is because checking `@queue` status is enough to detect whether
thread pool is alived or not.

Signed-off-by: Hiroshi Hatake <cosmo0920.oucc@gmail.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    671c666 View commit details
    Browse the repository at this point in the history

  32. in_tail: test: Reconsider testing caluses and assertion conditions 

    To create test fixture should contain 20 lines or greater.
Then, we should create 30 lines text included file.
num_events should be specify more accurate values.

8192 bytes case, num_events should specify 2 to make more concreate
circumstances.

To prevent hard timout on driver#run,
we should specify `shutdown: false` keyward argument.
This is a culprit to generate plugin hard timeout error messages on
testing.

Signed-off-by: Hiroshi Hatake <cosmo0920.oucc@gmail.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    e4747ea View commit details
    Browse the repository at this point in the history

  33. in_tail: Extract as a method for sleep for log ingestion code 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 6, 2021
    Configuration menu
    Copy the full SHA
    af2ea9b View commit details
    Browse the repository at this point in the history

Commits on May 11, 2021

  1. Avoid to use #sleep for stop reading files 

    Specify `read_more` as `false` is enough to prevent reading beyond bytes
limits.

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 11, 2021
    Configuration menu
    Copy the full SHA
    d3a3023 View commit details
    Browse the repository at this point in the history

  2. Remove needless thread poll implementation 

    If we don't use `#sleep` to prevent reading bytes beyond limits,
it is not needed to implement thread pool mechanism.
We just need to get out reading file loop when bytes limit is reached.

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 11, 2021
    Configuration menu
    Copy the full SHA
    478b487 View commit details
    Browse the repository at this point in the history

  3. Split complicated clause into read lines limits and log throttling 

    limits

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 11, 2021
    Configuration menu
    Copy the full SHA
    147ffbf View commit details
    Browse the repository at this point in the history

  4. Use early return on the log throttling first clause 

    Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>
    @cosmo0920
    cosmo0920 committed May 11, 2021
    Configuration menu
    Copy the full SHA
    6be3533 View commit details
    Browse the repository at this point in the history

  5. in_tail: Replace a magic number with a constant 

    Should use TailWatcher::IOHandler::BYTES_TO_READ as mininum value of
read_bytes_limit_per_second.

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
    @ashie
    ashie committed May 11, 2021
    Configuration menu
    Copy the full SHA
    98616e2 View commit details
    Browse the repository at this point in the history