Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CW Logs pagination #2

Open
aidansteele opened this issue Oct 25, 2018 · 1 comment
Open

Improve CW Logs pagination #2

aidansteele opened this issue Oct 25, 2018 · 1 comment

Comments

@aidansteele
Copy link
Collaborator

Right now we have the following code:

gossm/pkg/gossm/client.go

Lines 234 to 241 in d236fcd

var lastTimestamp int64 = math.MaxInt64
for _, ts := range lastTimestamps {
if ts < lastTimestamp {
lastTimestamp = ts
}
}
lastTimestamp++
input.StartTime = &lastTimestamp

This will paginate from the most recent timestamp for the instance that least recently posted to CW Logs. We need to do it this way to avoid missing logs from laggards, but it means keeping a cache of "seen" messages to avoid printing duplicates.

This is suboptimal because:

  • Right now the cache of seen messages grows unbounded
  • If one instance in a group finishes command execution long before others, we'll always be paginating from its last message timestamp. This could be minutes, resulting in a lot of duplicated requests. We should remove instances from the lastTimestamps map a few seconds after their command invocation completion. (A few seconds to account for delayed CW Logs appearance)
@aidansteele
Copy link
Collaborator Author

We should also (configurably) support trailing back a few seconds automatically if nothing is being ignored

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant