Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable session leaks prevention by cleaning up long-running tra… #2655

Merged
merged 6 commits into from Nov 15, 2023

Conversation

arpan14
Copy link
Collaborator

@arpan14 arpan14 commented Oct 5, 2023

Adding a couple of option in SessionPoolOptions for auto-detecting long running transactions, logging the stack-trace and removing such transactions from consuming more resources. There can be scenarios where transactions are unexpectedly taking more time and hence can consume the session for longer durations. With this change, customers get two new options which can be set while providing SessionPoolOptions

  1. setWarnIfInactiveTransactions() - Calling this method will generate warning logs and help in identifying faulty code which is leading to incorrect consumption of sessions.
  2. setWarnAndCloseIfInactiveTransactions() - Calling this method will close the faulty transactions that are consuming the resources. We will also be generating the warning logs which can be later referred to know the transactions that were closed through this option.

Reference Feature PR - #2419

@arpan14 arpan14 requested review from a team as code owners October 5, 2023 10:09
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: spanner Issues related to the googleapis/java-spanner API. labels Oct 5, 2023
#### Debugging and Resolving Session Leaks

##### Logging
Enabled by default, the logging option shares warn logs when you have exhausted >95% of your session pool.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for java we won't have this log right so we can remove the line?

session-and-channel-pool-configuration.md Outdated Show resolved Hide resolved
session-and-channel-pool-configuration.md Outdated Show resolved Hide resolved
session-and-channel-pool-configuration.md Outdated Show resolved Hide resolved
session-and-channel-pool-configuration.md Outdated Show resolved Hide resolved
Comment on lines +323 to +326
To help debug which transactions may be causing this session leak, the logs will also contain stack traces of
transactions which have been running longer than expected. The logs are pushed to a destination based on
how the log exporter is configured for the host application.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we logging in the Java client? Based on the comment above, I would expect 'No'.

If we are logging: I'm pretty sure that we are not pushing those logs to any 'destination based on how the log exporter is configured'.

Copy link
Collaborator Author

@arpan14 arpan14 Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the other client libraries such as Node/Python, emit an additional log every-time the application uses > 95% of the sessions. In Java, we use the condition as a pre-condition for the algo but don't emit any log for this condition. We only emit one of the below logs,
a) WARN LOG - when there is an actual long-running transaction. This includes bunch of other pre-conditions conditions (lastUsedTime, type of Transaction, etc.)
OR b) WARN LOG - We have automatically cleaned a long running transaction

Now what I mean as pushing log to the exporter is - the logs will be stored in a destination as per what is configured for the application. For ex - Some apps may configure cloud logging while others could configure logs to be stored on separate files on the app server.

The note about the exporter is not too relevant for Java where there is a default logging behaviour and I think it was more relevant for other languages like Node where a default logging capability was absent. We added a custom logging capability in Node (as part of sessions leaks) where the default logging was console logs or you could define some other exporter.

LMK if this documentation can be better framed or excluded in context of Java client.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using java.util.logging for this, right? Then I would just explicitly call out that. So something like 'The warning is logged using java.util.logging.'. I think that most people know what they need to do to configure that, and can otherwise search for it.

arpan14 and others added 4 commits October 5, 2023 18:53
Co-authored-by: Knut Olav Løite <koloite@gmail.com>
Co-authored-by: Knut Olav Løite <koloite@gmail.com>
Co-authored-by: Knut Olav Løite <koloite@gmail.com>
Co-authored-by: Knut Olav Løite <koloite@gmail.com>
@arpan14 arpan14 added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 14, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 14, 2023
@arpan14 arpan14 added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Nov 14, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 14, 2023
@arpan14 arpan14 added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@arpan14 arpan14 added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@arpan14 arpan14 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 15, 2023
@arpan14 arpan14 added kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 15, 2023
@arpan14 arpan14 added the automerge Merge the pull request once unit tests and other checks pass. label Nov 15, 2023
@arpan14 arpan14 merged commit faa7e5d into googleapis:main Nov 15, 2023
22 of 23 checks passed
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Nov 15, 2023
@arpan14 arpan14 deleted the session-leaks-launch branch November 15, 2023 08:25
Copy link

@Flyboy-60 Flyboy-60 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Copy link

@Flyboy-60 Flyboy-60 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/java-spanner API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants