Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[exporter/awscloudwatchlogsexporter] Improve performance of the awscloudwatchlogs exporter #26692

Merged
merged 29 commits into from
Oct 11, 2023
Merged

[exporter/awscloudwatchlogsexporter] Improve performance of the awscloudwatchlogs exporter #26692

merged 29 commits into from
Oct 11, 2023

Commits on Sep 14, 2023

  1. Chore: Rename PusherKey to StreamKey 

    StreamKey uniquely identify an cloudwatch logs tream and is used to show
where an event should be submited to.

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    406ce61 View commit details
    Browse the repository at this point in the history

  2. Chore: Move StreamKey inside event 

    The idea is that we will identify that an event belongs to a specific stream

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    7ee35f6 View commit details
    Browse the repository at this point in the history

  3. Chore: Refactor to properly initialize the event 

    Properly initialize a cloudwatch.Event in the awsemf exporter, setting the
destination log stream of the event

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    1e55b06 View commit details
    Browse the repository at this point in the history

  4. Feat: Add cwlogs Pusher able to send events to multiple streams 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    7d4c305 View commit details
    Browse the repository at this point in the history

  5. Feat: use MultiStream pusher in awscloudwatchlogsexporter 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    10b4638 View commit details
    Browse the repository at this point in the history

  6. Feat: Add support to consumers in cloudwatchlogs 

    Add support to multiple consumers in the cloudwatchlogs expoters. This will
allow requests to be sent in parallel to cloudwatch logs

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    8b3e0f4 View commit details
    Browse the repository at this point in the history

  7. Feat: Remove locks from cloudwatch logs pusher 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    1d9e0e2 View commit details
    Browse the repository at this point in the history

  8. Feat: Remove throttling limiter 

    Cloudwatch logs no longer limits to 5 request per second per log stream.
Instead the limit is per account.

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    eb38388 View commit details
    Browse the repository at this point in the history

  9. Feat: Stop using sequence token 

    Sequence tokens are no longer necessary to make PutLogEvents calls to
CloudWatch logs. This changes removes all the logic for managing this token and
instead use a "Noop"

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    069f471 View commit details
    Browse the repository at this point in the history

  10. Feat: Send log events directly to the Pusher 

    This optmization will reduce the memroy and CPU usage a bit
because we will not need to keep a big buffer of events that
were translated and insted they can be sent directly to the pusher.

Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    51ddd00 View commit details
    Browse the repository at this point in the history

  11. chore: Update readme 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    034b8ca View commit details
    Browse the repository at this point in the history

  12. Chore: update changelog 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    fb01288 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    d189d48 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    caf1be6 View commit details
    Browse the repository at this point in the history

  15. Chore: fix wrong dependency 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    0b4dbea View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    86e2c38 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    bd7d4a1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    c9f98b1 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    e6383ab View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2023

  1. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    ccd669c View commit details
    Browse the repository at this point in the history

  2. Update internal/aws/cwlogs/pusher.go 

    Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
    @rapphil @Aneurysm9
    rapphil and Aneurysm9 committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    8f183e2 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. Use queue validation from exporterhelper 

    Signed-off-by: Raphael Silva <rapphil@gmail.com>
    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    fb93d1b View commit details
    Browse the repository at this point in the history

  2. Simplify error handling and avoid uncessary code

    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    698e581 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    0cc848c View commit details
    Browse the repository at this point in the history

  4. Chore: Refactor pusher and cwlog_client to not use stream token

    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    eb608f6 View commit details
    Browse the repository at this point in the history

  5. Chore: minor refactoring

    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    0489473 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    f20ffae View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    dec0f2d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into rapphil-parallel-awscloudwatchlogsexporter

    @rapphil
    rapphil committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    c799e53 View commit details
    Browse the repository at this point in the history