Argon2 password hashing leads to increased Major GC's in Keycloak's JVM during load tests

[kami619]

Before reporting an issue

• I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

dist/quarkus

Describe the bug

Argon2 password hashing leads to increased Major GC's in Keycloak's JVM during load tests.

With Argon2 occupying larger amounts of heap memory and with the existing GC(ParallelGC) results in continuous Major GC's leading for higher CPU contention and lower performance across the board.

We looked into the tuning options and did several series of tests. And the findings are below.
But we understand there is an intervention needed due to this password hashing algorithm change and there are couple of recommendations suggested in this ticket based on our tests.

Version

nightly

Regression

• The issue is a regression

Expected behavior

We would want the Keycloak's JVM heap to behave consistently in terms of both throughput and endpoint performance.

Actual behavior

With the Argon2 hashing change, we find the application to behave abnormally during medium to high load situations with increased JVM GC overhead and CPU utilization.

With the Keycloak current JVM defaults:

High Major GC counts during a load test run and higher JVM GC Overhead.

How to Reproduce?

Using the keycloak-benchmark and the default Keycloak JVM heap settings, run the below command

./kcb.sh --scenario=keycloak.scenario.authentication.AuthorizationCode \
--server-url=<keycloak-url> --users-per-sec=150 --ramp-up=20 \
--logout-percentage=100 --measurement=900 --users-per-realm=20000

Anything else?

We then disabled the UseAdaptiveSizePolicy and that stabilized the load test runs. Further more we observed the Adaptive policy is aggressively looking to clear the heap resulting in frequent heap resizing events, but with disabling the adaptive policy, we were able to control that better. But disabling a key attribute is not a longer term solution, now we started to look into the G1GC which suits better to our use case and ran few experiments. Below is the result from those experiments.

Based on this data, it seems like, G1GC even with current Keycloak JVM defaults works better than ParallelGC and it offers some more tuning bandwidth by adjusting GCTimeRatio and AdaptiveSizePolicyWeight. We would like to hear how the Cloud Native team interprets this data and take appropriate next steps.

[sschu]

@kami619 As a side note: we have been using G1C in production since we started using Keycloak without problems.

[ahus1]

Steven created an issue in the bouncy castle upstream project to optimize the memory usage of Argon2. We don't know if and when this will be implemented. Therefore, we will need to continue changing the GC settings.

[ahus1]

Re-opened as we need release notes and an updated sizing guide (the sizing guide doesn't reflect Argon2 yet)