Realm export normalization

[sonOfRa]

What this PR does / why we need it:
When trying to move an existing keycloak installation with no configuration management to keycloak-config-cli, it can be hard to figure out which parts of the realm export are actually needed to reproduce the state when re-importing the configuration.

This PR can serve as a (partial) solution to this problem, by taking a realm export from a certain version of Keycloak, comparing it against a reference export of a "barebones" Keycloak realm from the same version, and then creating a yml (or JSON) file that only contains changes that are actually required.

This PR will not be a complete solution that works for all cases, but it will supply an at least somewhat minimized view of clients and some related settings, which can then easily be merged with a full export in order to get a smaller view of the file. Support for additional parts (identity providers, authentication flows, etc) can be added in later steps to improve this export.

This PR includes a "mode switch" that allows users to switch between IMPORT and NORMALIZE modes using the property run.operation=NORMALIZE. IMPORT is the default, and is used when no option is supplied by the user.

This PR includes an upgrade to Spring Boot 2.7.5, which contains a fix for spring-projects/spring-boot#32559. The @DefaultValue annotation is used to move configuration from the default application.properties file directly to the code. This is needed so that only the needed configurations for the chosen mode (IMPORT or NORMALIZE) are activated.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #799

Special notes for your reviewer:
A custom properties application-normalize-dev.properties file for testing the export functionality was added. Activating this profile enables normalization mode, and expects an input realm to be present at the top-level exports directory

exports
├── in
│   └── realm.json
└── out
    └── demo1234.yaml

The file in/realm.json will be imported, and the file out/demo1234.yaml is created by the process, where the realm name of the input realm is demo1234

Run the application with the normalize-dev profile active to normalize a given realm.

The configuration parameter normalization.output-format can be used to switch between YAML (the default, if no option is given) and JSON output.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Merging #818 (93c5fbc) into main (dbb8950) will decrease coverage by 12.39%.
The diff coverage is 20.45%.

❗ Current head 93c5fbc differs from pull request most recent head fe9de52. Consider uploading reports for the commit fe9de52 to get more accurate results

@@              Coverage Diff              @@
##               main     #818       +/-   ##
=============================================
- Coverage     95.46%   83.08%   -12.39%     
- Complexity     1302     1339       +37     
=============================================
  Files            76       96       +20     
  Lines          4213     5037      +824     
  Branches        467      621      +154     
=============================================
+ Hits           4022     4185      +163     
- Misses           94      755      +661     
  Partials         97       97               

Impacted Files	Coverage Δ
...sys/keycloak/config/KeycloakConfigApplication.java	75.00% <ø> (ø)
.../adorsys/keycloak/config/KeycloakConfigRunner.java	93.54% <ø> (ø)
...cloak/config/exception/NormalizationException.java	0.00% <0.00%> (ø)
...ctory/UsedAuthenticationFlowWorkaroundFactory.java	87.55% <ø> (ø)
...eycloak/config/model/AuthenticationFlowImport.java	100.00% <ø> (ø)
.../de/adorsys/keycloak/config/model/RealmImport.java	100.00% <ø> (ø)
...ycloak/config/provider/KeycloakImportProvider.java	94.49% <ø> (ø)
...sys/keycloak/config/provider/KeycloakProvider.java	82.55% <ø> (ø)
...onfig/repository/AuthenticationFlowRepository.java	93.44% <ø> (ø)
...nfig/repository/AuthenticatorConfigRepository.java	100.00% <ø> (ø)
... and 61 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[manuschillerdev]

Would it be possible via some kind of diffing to keep params for fields with env var substitutions?

Example:

{
  "realm": "my-realm",
  "smtpServer": {
    "replyToDisplayName": "$(env:SMTP_REPLY_TO_DISPLAYNAME)",
    "port": "$(env:SMTP_PORT)",
    "host": "$(env:SMTP_HOST)",
    "replyTo": "$(env:SMTP_REPLY_TO)",
    "from": "$(env:SMTP_FROM)",
    "fromDisplayName": "$(env:SMTP_FROM_DISPLAY_NAME)",
    "envelopeFrom": "$(env:SMTP_ENVELOPE_FROM)"
  },

When I import this realm, make changes and export the realm again I'd like to keep the params for SMTP instead of having to replace the actual values from the export with the parameters again.

I have no idea what that means regarding the implementation. Just as an idea to make working with the huge JSON-Files easier :)

[sonOfRa]

@manuschillerdev I think that might be a valuable future extension of this feature!

In theory, one could provide a map of "reverse injections" where you specify something like "When the field realm.smtpServer.host has the same value as the resolved environment variable "SMTP_HOST", then replace the value with $(env:SMTP_HOST). This seems to me like a valuable addition to this feature in order to make the process of going from "I have a keycloak instance that has been configured via admin-ui" to "I have a relatively minimal yaml file that represents my config state"

However, I think I'd like to get the base idea of this PR (minimize a realm.json acquired from a full Keycloak realm export) done first, and then build additional features on top of that

[weand]

i tried this PR on KC20.0.5.

After adding baseline files for 20.0.5 and building the cli, following error occurs when running it against our existing realm. Seems like there are remove(...) calls on an immutable map...

Caused by: java.lang.UnsupportedOperationException: null
        at java.base/java.util.ImmutableCollections.uoe(ImmutableCollections.java:142)
        at java.base/java.util.ImmutableCollections$AbstractImmutableMap.remove(ImmutableCollections.java:1075)
        at de.adorsys.keycloak.config.service.normalize.RoleNormalizationService.normalizeClientRoles(RoleNormalizationService.java:83)
        at de.adorsys.keycloak.config.service.normalize.RoleNormalizationService.normalizeRoles(RoleNormalizationService.java:57)
        at de.adorsys.keycloak.config.service.normalize.RealmNormalizationService.normalizeRealm(RealmNormalizationService.java:150)
        at de.adorsys.keycloak.config.KeycloakConfigNormalizationRunner.run(KeycloakConfigNormalizationRunner.java:92)

[sonOfRa]

Hi @weand!

Can you provide a realm export for the realm you tried this on? Note that realm exports will contain sensitive data like the client secrets and all your realm keys (for signing JWTs). If it's not possible to censor the realm enough for it to be exportable to a public place, could you maybe find a certain configuration of roles that triggers this bug? I'll also take a look in the meantime if I can find what causes this.

[weand]

@sonOfRa Check this reproducer export realm-export-uoe.zip

It's as simple as this: Simply disable one of the default clients, e.g. realm-management and security-admin-console in my example.

BTW: I could not figure out how to enable debug output (which prints the root exception). With that:

java -Dspring.profiles.active=debug \
  -jar $script_dir/keycloak-config-cli-20.0.5.jar \
  --normalization.files.inputLocations=$script_dir/normalize/in/*.json \
  --normalization.files.outputDirectory=$script_dir/normalize/out \
  --run.operation=NORMALIZE
  --debug

Log looks like this:

2023-03-14 15:09:13.382  INFO 2360748 --- [           main] d.a.k.c.s.n.RealmNormalizationService    :
 Exporting realm foo-bar
2023-03-14 15:09:13.486 ERROR 2360748 --- [           main] .a.k.c.KeycloakConfigNormalizationRunner :
 null

To see the root exception I had to remove that condition

Any advice on how to run with debug output enabled would be great.

[sonOfRa]

With the debug profile, there should be debug logging enabled, I'll try to see why that isn't showing up there for those messages.

I see that in the export you provided, there are no roles present whatsoever. Did you, by chance export the realm via the admin console's export functionality? For reasons I've never understood, that export is incomplete and doesn't include everything.

The assumption for this tool is working on a full export, as acquired via one of the methods listed here: https://www.keycloak.org/server/importExport#_exporting_a_specific_realm

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

No Coverage information
0.5% Duplication