Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/keycloak] Add HPA Behavior when scaling up and down #25681

Merged
merged 1 commit into from May 13, 2024
Merged

[bitnami/keycloak] Add HPA Behavior when scaling up and down #25681

merged 1 commit into from May 13, 2024

Conversation

ashktn
Copy link
Contributor

@ashktn ashktn commented May 10, 2024

Description of the change

This PR adds the ability to configure the behavior of the HorizontalPodAutoscaler of the Keycloak StatefulSet with new parameter block autoscaling.behavior
ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior

This allows us to set policies that control how the autoscaler scales up/down the pods e.g., the keycloak pods can be scaled by X units or percent of current pods during a periodSeconds window.
stabilizationWindowSeconds can be set to set the duration which past recommendations should be considered while scaling
selectPolicy - The priority of policies that the autoscaler will apply when scaling up/down. Can be Max, Min, Disabled

Benefits

Keycloak pods are stateful due to the embedded Infinispan cache that requires cache key rebalancing across all live nodes when a pod joins or leaves the cluster. (This process happens on pod startup and shutdown).

The cache keys have a set number of owners (pods) that can survive one node failure with its default config (https://www.keycloak.org/server/caching#_configuring_caches).

Adding the HPA behavior when scaling down to 1 pod every 300s allows the cache keys to be rebalanced across other pods before terminating, hence avoiding potential data loss if too many pods were to be terminated at the same time by the autoscaler's default behaviour.

Possible drawbacks

The default values set in values.yaml slows down the scaleDown of pods.

Applicable issues

Additional information

The default HPA scaleDown behaviour is now one pod every 300s.

If anybody wishes to go back to the previous behaviour, they can set the values as follows:

  behavior:
    scaleDown:
      policies: []

@javsalgar - I'm happy to set the default value to the above so that this release doesn't change the default behaviour. But i figured the default value I'm proposing are reasonable for an HA deployment of Keycloak.
These are also the default values set in the Codecentric Keycloak Helm Chart

Checklist

  • Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
  • Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
  • Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
  • All commits signed off and in agreement of Developer Certificate of Origin (DCO)

Tests

pods start fine using default values

image

# HPA Enabled but no behavior set

# values.yaml
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70

---
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 300
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 300
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak
Autoscaling enabled - scaleDown policies set to empty to remove the default scaleDown policy

# values.yaml
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70
  behavior:
    scaleDown:
      policies: []

--- 
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak

Autoscaling enabled - scaleUp policy added and some values changes from default

values.yaml

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPU: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 1337
      selectPolicy: Min
      policies:
        - type: Percent
          value: 30
          periodSeconds: 123
    scaleDown:
      stabilizationWindowSeconds: 60

--- 
# applied yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 24.0.4
    helm.sh/chart: keycloak-21.1.3
  name: keycloak
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 300
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 60
    scaleUp:
      policies:
      - periodSeconds: 123
        type: Percent
        value: 30
      selectPolicy: Min
      stabilizationWindowSeconds: 1337
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: keycloak

[bitnami/keycloak] Add HPA Behavior when scaling up and down
a3ed774 
Add ability to control the hpa behaviour when scaling up and down so that we can give Keycloak pods time to rebalance cache entries when scaling up/down


Signed-off-by: Ash Ramasawmy <Ash.Ramasawmy@achievers.com>
@github-actions github-actions bot added keycloak triage Triage is needed labels May 10, 2024
@github-actions github-actions bot requested a review from javsalgar May 10, 2024 20:47
@carrodher carrodher added verify Execute verification workflow for these changes in-progress labels May 13, 2024
@github-actions github-actions bot removed the triage Triage is needed label May 13, 2024
@github-actions github-actions bot removed the request for review from javsalgar May 13, 2024 06:56
@github-actions github-actions bot requested a review from dgomezleon May 13, 2024 06:56
dgomezleon
dgomezleon approved these changes May 13, 2024
View reviewed changes
Copy link
Member

@dgomezleon dgomezleon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for contributing

@dgomezleon dgomezleon merged commit 7664aa5 into bitnami:main May 13, 2024
26 checks passed
@ashktn
Copy link
Contributor Author

ashktn commented May 13, 2024

thanks for the quick reviews!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dgomezleon dgomezleon dgomezleon approved these changes

Assignees

@dgomezleon dgomezleon

Labels
keycloak solved verify Execute verification workflow for these changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[bitnami/keycloak] - Allow configuring HPA Behaviour for scaleUp / scaleDown
4 participants
@ashktn @dgomezleon @javsalgar @carrodher