Normalized Export

[mlhartme]

Problem Statement

(I'm not sure if this feature request is best placed here or in keycloak itself ... I'll first try here ...)

We use keycloak-config-cli to properly define the configuration of our Keycloak server in my-realm.json. That's great!

But we use manual configuration as well, usually because we need try and error to find the proper configuration.
To get manual changes into my-realm.json (and thus get a well-defined state again), we diff keycloaks export (kcadm.sh create realms/my-realm/partial-export) before and after the manual change.

Problem with that is: the diff gets big, mostly because ordering in some of keycloaks arrays doesn't matter, and keycloak returns entries in different order. And because we regularly delete and create the realm from my-realm.json to play save - which causes Keycloak to choose new id s.

Proposed Solution

Is it possible to provide normalized realm exports that:

• sorts arrays where ordering doesn't matter (e.g. protocolMappers and org.keycloak.keys.KeyProvider)
• normalize id s by replacing the uuids with a numbers: first with 1, then 2, etc

We currently use jq to manually do some normalization (see below). That helps us a lot, but it feels like a poor solution, especially replacing all id s with XXX ...

Best regards, Michael

#!/bin/sh
if [ "$#" -ne 1 ] ; then
echo "Normalize arrays and ids"
echo "usage: normalize.sh <src.json>"
exit 1
fi
set -e

json=$(cat "$1")
json=$(echo "${json}" | jq '.clientScopes = (.clientScopes | sort_by(.name))')
json=$(echo "${json}" | jq '(.. | select(.protocolMappers?) | .protocolMappers) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.keys.KeyProvider"]) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.storage.UserStorageProvider"] | .[] | .subComponents | .["org.keycloak.storage.ldap.mappers.LDAPStorageMapper"]) |= sort_by(.name)')
json=$(echo "${json}" | jq '(.components | .["org.keycloak.services.clientregistration.policy.ClientRegistrationPolicy"]) |= sort_by(.name, .subType)')
json=$(echo "${json}" | jq '(.. | select(.["allowed-protocol-mapper-types"]?) | .["allowed-protocol-mapper-types"] ) |= sort')
json=$(echo "${json}" | jq '(.. | select(.id?) | .id) |= "XXX" ')
echo "${json}" | jq --sort-keys

Environment

• Keycloak Version: 19.0.3
• keycloak-config-cli Version: 5.3.1
• Java Version: 17

[sonOfRa]

I've started a draft implementation of this feature over at #818.

The general idea is to use Javers to diff the objects, comparing them to a "baseline" realm export for any given Keycloak version, and then using that diff to produce a normalized/minimized yaml file that can later be imported using keycloak-config-cli.

Any inputs on particular choices or ideas on how to do it better are much appreciated!

[katoquro]

Hello!
@sonOfRa, how is it going with normalization? I have a similar requirement: show diff before deployment of the new configuration. I've started but I've found this issue and think that it would be better to wait for your implementation and then tune it for my case.

[sonOfRa]

Hi!

I'm making progress, but it's relatively slow going. There's a few edge cases that pop up every now and then when trying the current state out with different realm configurations. Most of the important features of the keycloak config are already present in the normalized config now, but one big block that is still missing is the "components": [] block, which is a quite complex type that isn't very constrained, and as such a bit difficult to "normalize" without accidentally breaking things.

[katoquro]

I see. components are like Map<String, Object> 😄. Currently, I think that we can just make a deep ordering for the keys of that maps and it may cover most cases. At least, it will be reproducible.

[SebastianKunz]

I am also interested in this feature. @sonOfRa can you elaborate what the state of progress is? Are there any blockers for you?

[dominiktopp]

We had a similar problem. Our solution was to replace ids (search by regex, replace with XXX) and use a JSON diff tool like https://www.jsondiff.com/. That reduces the problem to the order of elements in arrays but this is not very hard to check by hand.
After a few roundtrips we had enough experience and examples in our realm.json to do most of the configuration directly in the realm.json.

But for the migration from "configuration by admin ui" to keycloak-config-cli a tool like #818 would have been great 😄