keycloak 20 ispn cache dont work #19693

v-kamerdinerov · 2023-04-13T10:06:46Z

v-kamerdinerov
Apr 13, 2023

Hello! I was trying to use keycloak 20 cluster configuration with 3 pods in kubernetes.

My config:

       - name: JAVA_OPTS
          value: -Xms128m -Xmx384m -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256m
            -Djava.net.preferIPv4Stack=true
        - name: JAVA_OPTS_APPEND
          value: -Djboss.as.management.blocking.timeout=300 -Djgroups.dns.query=keycloak-headless
        - name: JDBC_PARAMS
          value: socketTimeout=0&connectTimeout=10000&autoReconnect=true
        - name: KC_CACHE
          value: ispn
        - name: KC_CACHE_STACK
          value: kubernetes
        - name: KC_DB_ADDR
          value: mysqlendpoint:3306
        - name: KC_DB_DATABASE
          value: keycloak
        - name: KC_DB_POOL_MAX_SIZE
          value: "15"
        - name: KC_DB_POOL_MIN_SIZE
          value: "5"
        - name: KC_DB_USER
          value: keycloak
        - name: KC_DB_VENDOR
          value: mariadb
        - name: KC_HEALTH_ENABLED
          value: "true"
        - name: KC_HOSTNAME_STRICT
          value: "false"
        - name: KC_HOSTNAME_STRICT_BACKCHANNEL
          value: "true"
        - name: KC_HOSTNAME_STRICT_HTTPS
          value: "false"
        - name: KC_HTTP_ENABLED
          value: "true"
        - name: KC_HTTP_RELATIVE_PATH
          value: /auth
        - name: KC_LOG_LEVEL
          value: DEBUG
        - name: KC_METRICS_ENABLED
          value: "true"
        - name: KC_PROXY
          value: edge

kubectl get svc -n keycloak
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
keycloak            ClusterIP   10.16.8.228   <none>        8080/TCP,7800/TCP   25h
keycloak-headless   ClusterIP   None          <none>        8080/TCP,7800/TCP   25h

Also as reverse proxy with TLS i use haproxy, and configured sticky sessions (otherwise UI is not available).

Then i try to check list of users by following these instructions https://quarkus.io/guides/security-keycloak-authorization

The application is using bearer token authorization and the first thing to do is obtain an access token from the Keycloak Server in order to access the application resources:

export access_token=$(\
curl --insecure -X POST https://localhost:8543/realms/quarkus/protocol/openid-connect/token \
-H 'Accept: application/json' \ 
-H 'Content-Type: application/x-www-form-urlencoded' \ 
-H 'cache-control: no-cache' \ 
-d 'grant_type=password&username=admin&password=strongpass&client_id=admin-cli' \
| jq -r '.access_token' \
)

 curl -v -X GET \
   http://localhost:8080/api/admin \
   -H "Authorization: Bearer "$access_token

I get

{
  "error": "HTTP 401 Unauthorized"
}

In logs i find

2023-04-12 21:52:45,834 DEBUG [io.quarkus.vertx.http.runtime.ForwardedParser] (vert.x-eventloop-thread-1) Recalculated absoluteURI to https://keycloak-instance.com/auth/admin/realms/cluster-realm/users
2023-04-12 21:52:45,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-3) new JtaTransactionWrapper
2023-04-12 21:52:45,834 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-3) was existing? false
2023-04-12 21:52:45,836 DEBUG [org.keycloak.services.managers.AuthenticationManager] (executor-thread-3) Failed to verify identity token: Key not found
2023-04-12 21:52:45,836 DEBUG [org.keycloak.services.resources.admin.AdminRoot] (executor-thread-3) Token not valid
2023-04-12 21:52:45,837 DEBUG [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-3) Error response 401: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized

I dived into debugging the problem and realized that the cache is NOT WORKING after all. I tried to create a new realms cluster-realm through UI, with one user - user@two.com and then request it directly in the pods - the realm was created only in the 1st pod! Synchronization failed even after 10 minutes.

[
  {
    "id": "e2269ee3-bb92-4271-9fb4-a8857990ac00",
    "createdTimestamp": 1681374339352,
    "username": "user@two.com",
    "enabled": true,
    "totp": false,
    "emailVerified": false,
    "email": "user@two.com",
    "disableableCredentialTypes": [],
    "requiredActions": [],
    "notBefore": 0,
    "access": {
      "manageGroupMembership": true,
      "view": true,
      "mapRoles": true,
      "impersonate": true,
      "manage": true
    }
  }
]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1926  100  1846  100    80  15603    676 --:--:-- --:--:-- --:--:-- 16322
try on 10.220.1.36
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    28  100    28    0     0    975      0 --:--:-- --:--:-- --:--:--  1000
{
  "error": "Realm not found."
}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1929  100  1849  100    80  15765    682 --:--:-- --:--:-- --:--:-- 16487
try on 10.220.1.101
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    28  100    28    0     0   1179      0 --:--:-- --:--:-- --:--:--  1166
{
  "error": "Realm not found."
}

It's very strange because pods logs with trace on org.jgroups.protocols looks fine

2023-04-13 11:28:01,804 DEBUG [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery requests to hosts [10.220.1.226:0, 10.220.1.36:0, 10.220.1.101:0] on ports [7800 .. 7800]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.226:7800
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.226:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: looping back message [keycloak-0-55692 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL, transient_flags=DONT_LOOPBACK], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.36:7800
2023-04-13 11:28:01,804 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.36:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending discovery request to 10.220.1.101:7800
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=108, headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received [keycloak-0-55692 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL, transient_flags=DONT_LOOPBACK], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_REQ from keycloak-0-55692, sending response keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-0-55692, src=keycloak-0-55692, size=111, headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,805 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: looping back message [keycloak-0-55692 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,806 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-0-55692 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,806 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-0-55692: keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-2-24966, physical_addr=10.220.1.101:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-30,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-2-24966: keycloak-2-24966, name=keycloak-2-24966, addr=10.220.1.101:7800, coord
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-30,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-1-48535, physical_addr=10.220.1.36:7800, TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to keycloak-0-55692, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_RSP from keycloak-1-48535: keycloak-1-48535, name=keycloak-1-48535, addr=10.220.1.36:7800, server
2023-04-13 11:28:01,808 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.36:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:02,676 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 51 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to 10.220.1.101:7800, src=keycloak-0-55692, size=95, headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-0-55692, physical_addr=10.220.1.226:7800, TP: [cluster=ISPN]
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received GET_MBRS_REQ from keycloak-2-24966, sending response keycloak-0-55692, name=keycloak-0-55692, addr=10.220.1.226:7800, server
2023-04-13 11:28:02,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=111, headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-13 11:28:02,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to 10.220.1.226:7800, 0 bytes, flags=INTERNAL], headers are MERGE3: INFO: view_id=[keycloak-2-24966|26], logical_name=keycloak-2-24966, physical_addr=10.220.1.101:7800, TP: [cluster=ISPN]
2023-04-13 11:28:03,388 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:03,388 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:04,015 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:04,015 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:04,646 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending stable msg to keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:04,646 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=143, headers are STABLE: [STABLE_GOSSIP] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:04,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:04,677 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:05,389 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:05,389 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:06,016 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:06,016 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:06,677 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:06,678 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:07,389 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:07,390 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:08,016 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:08,016 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:08,566 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 65 bytes, flags=OOB|NO_RELIABILITY|INTERNAL], headers are STABLE: [STABILITY] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stability msg from keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stable digest keycloak-2-24966: [446 (446)], keycloak-1-48535: [276 (276)], keycloak-0-55692: [184 (184)]
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 446 from keycloak-2-24966
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 276 from keycloak-1-48535
2023-04-13 11:28:08,567 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 184 from keycloak-0-55692
2023-04-13 11:28:09,390 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:09,390 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:10,017 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:10,017 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:10,678 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:10,678 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:11,391 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:11,391 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:11,863 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending stable msg to keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:11,863 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to keycloak-2-24966, src=keycloak-0-55692, size=143, headers are STABLE: [STABLE_GOSSIP] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:12,017 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:12,018 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:12,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:12,679 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:13,391 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:13,392 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:14,018 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:14,018 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 65 bytes, flags=OOB|NO_RELIABILITY|INTERNAL], headers are STABLE: [STABILITY] view-id= [keycloak-2-24966|26], TP: [cluster=ISPN]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stability msg from keycloak-2-24966: keycloak-2-24966: [446], keycloak-1-48535: [276], keycloak-0-55692: [184]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received stable digest keycloak-2-24966: [446 (446)], keycloak-1-48535: [276 (276)], keycloak-0-55692: [184 (184)]
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 446 from keycloak-2-24966
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 276 from keycloak-1-48535
2023-04-13 11:28:14,031 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-34,keycloak-0-55692) keycloak-0-55692: deleting msgs <= 184 from keycloak-0-55692
2023-04-13 11:28:15,392 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:15,392 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:16,018 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:16,019 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:16,679 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:16,679 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:17,393 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:17,393 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535
2023-04-13 11:28:18,019 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sending msg to null, src=keycloak-0-55692, size=36, headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:18,019 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: sent heartbeat
2023-04-13 11:28:18,680 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-2-24966 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:18,680 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-2-24966
2023-04-13 11:28:19,394 TRACE [org.jgroups.protocols.TCP] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received [keycloak-1-48535 to <all>, 0 bytes, flags=INTERNAL], headers are FD_ALL: heartbeat, TP: [cluster=ISPN]
2023-04-13 11:28:19,394 TRACE [org.jgroups.protocols.FD_ALL] (jgroups-34,keycloak-0-55692) keycloak-0-55692: received heartbeat from keycloak-1-48535

v-kamerdinerov · 2023-04-13T12:16:35Z

v-kamerdinerov
Apr 13, 2023
Author

After starting the pod, I found this line

2023-04-13 15:06:26,905 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled

Maybe that's exactly the problem with the cache.

0 replies

DPIDNB · 2023-08-15T10:09:51Z

DPIDNB
Aug 15, 2023

Hi, have you found a solution to this issue

2 replies

v-kamerdinerov Aug 18, 2023
Author

Hello @DPIDNB.

Yes I have solved the problem - and it is primarily my own fault and my inattention.

If you look, I used the following variables to connect to the database:

      KC_DB_ADDR:                         mariadbendpoint:3306
      KC_DB_DATABASE:                     keycloak
      KC_DB_USER:                         keycloak
      KC_DB_VENDOR:                       mariadb

And these parameters, are not valid! Because they were used from old version - keycloak 11.

The replicas were picking up their own database (as @dsteegen mention) and were not linked via the infinispan cache.

The valid parameters I used are shown below:

KC_DB:                              mariadb
KC_DB_SCHEMA:                       keycloak
KC_DB_URL:                          jdbc:mariadb://mariadbendpoint:3306/keycloak
KC_DB_USERNAME:                     keycloak

codespearhead Apr 30, 2024

@v-kamerdinerov Can you close this discussion then?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

keycloak 20 ispn cache dont work #19693

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

keycloak 20 ispn cache dont work #19693

v-kamerdinerov Apr 13, 2023

Replies: 2 comments · 2 replies

v-kamerdinerov Apr 13, 2023 Author

DPIDNB Aug 15, 2023

v-kamerdinerov Aug 18, 2023 Author

codespearhead Apr 30, 2024

v-kamerdinerov
Apr 13, 2023

Replies: 2 comments 2 replies

v-kamerdinerov
Apr 13, 2023
Author

DPIDNB
Aug 15, 2023

v-kamerdinerov Aug 18, 2023
Author