HelmRelease gets stuck in a state

[orisneh]

Hi,
We running k3s on edge devices, each node runs a single cluster.
These devices are prone to power-cutoffs, this can happen multiple times a day
Whenever a poweroff occurs, if for some reason a helmrelease was in the middle of installing/upgrading/uninstalling, the release gets stuck on something like this:

flux-system        state-manager      19d   False   Could not determine release state: unable to determine state for release with status 'uninstalling'
flux-system        outdoor            19d   False   Could not determine release state: unable to determine state for release with status 'uninstalling'

This will not recover until manually intervening and deleting the helm secret, then waiting for reconciliation or manually reconciling the helmrelease e.g.

kubectl get secret -n flux-system | awk '/state-manager/ {print $1}' | xargs -I {} kubectl delete secret -n flux-system {}
flux reconcile hr state-manager --with-source

Just to shed some more light on our setup, this is an example of one of our helmreleases:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: state-manager
  namespace: flux-system
spec:
  chart:
    spec:
      chart: ./charts/state-manager
      reconcileStrategy: Revision
      sourceRef:
        kind: GitRepository
        name: flux-system
      version: '*'
      valuesFiles:
        - ./charts/state-manager/values.yaml
        - ./presets/office-hw/apps/state-manager/values.yaml
  install:
    timeout: 25m
    remediation:
      retries: -1
  upgrade:
    timeout: 25m
    remediation:
      retries: -1
      strategy: uninstall
      remediateLastFailure: false
  interval: 10m0s
  targetNamespace: green-outdoor

I am assuming this relates to fluxcd/helm-controller#644
Assuming I am ok with losing data, is there a way to have helm-controller recover from this?
maybe ignore the lock or set a stale time on it?

Steps to reproduce:

• Setup a cluster on a node you have direct access to
• Have a helmrelease start an operation (upgrade/install/uninstall)
• Disconnect the power to the device (we unplug it from the power supply)

Unfortunatley setting up a UPS of some sort is not possible for us
If there is any more information you need me to provide I'd be happy to

Any help would be much appreciated,
Thanks!

[stefanprodan]

Can you please upgrade to Flux 2.2 and HelmRelease API v2beta2.

[orisneh]

Thanks for the fast reply!
We already run Flux 2.2.x, I will try HelmRelease API v2beta2 and post here once I have some findings

[orisneh]

Seems like updating HelmRelease API to v2beta2 solved the issue, thank you!