Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: don't log stacktrace in case of a CancellationException #597

Closed
Stephan202 opened this issue Sep 25, 2021 · 12 comments
Closed

Comments

@Stephan202
Copy link
Collaborator

Caffeine logs a warning including stack trace if a Future completes exceptionally ((1), (2), (3), (4), maybe elsewhere). This is important for flows in which the exception isn't bubbled up, as it may otherwise hide application errors.

However, this code also logs a full stack trace if a Future is simply cancelled. The log line then states than a java.util.concurrent.CancellationException was thrown, while strictly speaking this exception was only created inside CompletableFuture#cancel(boolean).

My change request is this: would it be possible to identify CancellationExceptions and log those differently, e.g. as:

if (error instanceof CancellationException) {
  logger.log(Level.INFO, "Refresh task was cancelled");
} else {
  logger.log(Level.WARNING, "Exception thrown during refresh", error);
}

(This example is for the BoundedLocalCache case; I also changed WARNING to INFO, but have no strong opinion on that, actually.)

Context: we have a custom cache loader, and if its background bulk refresh logic can't keep up, it cancels the oldest refresh requests. When this happens a (very) large number of identical and rather uninformative stack traces is logged. In extreme cases this also causes our log agent to have trouble keeping up.

@ben-manes
Copy link
Owner

ben-manes commented Sep 25, 2021

I think that's a good observation that cancellation should be special cased and is unlike the logging's intent. Do you think it should be logged at all? Either way seems reasonable to me, but I'd lean towards thinking cancellation is an explicit choice and expected if done, whereas most other exceptions are a surprise that the application is likely to silently swallow.

The same logic would also apply with TimeoutException (e.g. via orTimeout).

@Stephan202
Copy link
Collaborator Author

Stephan202 commented Sep 25, 2021

Do you think it should be logged at all?

I'd also be fine with not logging at all. Alternatively Level.TRACE could have the same effect in most setups. From our side either would be just fine. 👍

ben-manes added a commit that referenced this issue Sep 27, 2021
Caffeine logs a warning with the exception when an asynchronous load or
refresh fails. This is to assist in debugging due to a common mistake
of not handling errors, e.g. chaining `thenApply` without an exception
handler in a fire-and-forget fashion. It is expected that if this aid
is an annoyance that application can simply disable this in their
logging configuration.

This logging is not helpful when the future fails due to calling
the future's cancel() method or using orTimeout(duration). In those
cases the cause is expected and desired, so we can assume that the
application is handling the failure appropriately. Therefore the
logging of these causes should be skipped.
@Stephan202
Copy link
Collaborator Author

Tnx @ben-manes!

ben-manes added a commit that referenced this issue Sep 27, 2021
Caffeine logs a warning with the exception when an asynchronous load or
refresh fails. This is to assist in debugging due to a common mistake
of not handling errors, e.g. chaining `thenApply` without an exception
handler in a fire-and-forget fashion. It is expected that if this aid
is an annoyance that application can simply disable this in their
logging configuration.

This logging is not helpful when the future fails due to calling
the future's cancel() method or using orTimeout(duration). In those
cases the cause is expected and desired, so we can assume that the
application is handling the failure appropriately. Therefore the
logging of these causes should be skipped.
@ben-manes
Copy link
Owner

rolling this back from master as the CI is unhappy, but seems unrelated. If it fails only on the v3.dev branch then I'll have something to go off of, but I'm afraid it might be stuck failing on a prior green commit.

Unfortunately soft reference caching is flaky to test because we have to coerce the GC to maybe discard the entries. We do this using -XX:SoftRefLRUPolicyMSPerMB=0 and -XX:+UseParallelGC with Guava's GcFinalization.awaitFullGc(). That works most of the time and locally, but the CI is more finicky. That could be something minor like a new JDK patch release that tweaks the GC to break this, a VM's noisy neighbor causing fewer GCs due to cpu pressure, etc. It mostly. works but sometimes has bad days like now.

@ben-manes ben-manes reopened this Sep 27, 2021
@ben-manes
Copy link
Owner

I'm not sure why, but debugging this and the slf4j-test logger causes the issues. There is nothing special about that logger, nothing is logged which would explain it, and it is cleared between tests. Either the logging tests should be dropped or the logger isolated so that it isn't used by the reference tests.

@ben-manes
Copy link
Owner

disabling the test loggers by default seems to have stabilized it.

@wpanas
Copy link

wpanas commented Oct 26, 2021

@ben-manes when this feature will be released? I'm looking forward to use it 😉

@ben-manes
Copy link
Owner

I'll take a look over the weekend. There haven't been many changes accumulated so I hadn't thought much about a release, but no harm to do so for these QoL touch ups.

@wpanas
Copy link

wpanas commented Oct 27, 2021

Thanks @ben-manes 😄 I would appreciate that!

@taaem
Copy link

taaem commented Oct 29, 2021

@ben-manes would it also be possible to backport this to the 2.x.x releases? I'm looking forward to this in an application, which is sadly bound to Java 8 for now.

@ben-manes
Copy link
Owner

Perhaps, but please note that since this is logging you can easily handle it in your log configuration. A simple exception filter could be applied to this logger to ignore these cases. So while this change is nice, it really isn't necessary for anyone wanting this behavior. (See for example ch.qos.logback.core.filter.Filter if using logback)

@ben-manes
Copy link
Owner

Sorry for dragging my feet here. This is released in v2.9.3 and v3.0.5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants