[Feature Request] Update helm chart to allow a secret to be specified

[lknite]

Is your feature request related to a problem? Please describe.

I'd like to pull avi credentials from a vault and then create a secret with those credentials. This can be achieved using sealed-secrets.

The problem is, ako doesn't wait for the secret to appear and use it, instead it just creates its own secret overwriting the secret if it already exists.

Describe the solution you'd like

1. Allow a secretName: to be specified rather than require the secret to be added to the values.yaml in order to be used.

avicredentials:
  # specify an existing secret to use, otherwise a secret will be created automatically
  existingSecretName: "<secretName>"
  # if an 'existingSecretName' was not specified, the following values will be used to create a secret named 'avi-secret'
  username: ""
  password: ""
  authtoken: ""
#  certificateAuthorityData: |
#      PEM-encoded public certificate of avi server with Common Name attribute set as the FQDN of the avi server

If 'existingSecretName' is empty then use the passed in values, otherwise expect a secret with the name 'secretName' to exist with the needed values.

2. (a) Also allow an initContainer to be specified so that I can add an initContainer which will wait for the secret to appear. This can be added by adding the vmware 'common' library as a dependency to this helm chart. Like so:

dependencies:
  - name: common
    repository: oci://registry-1.docker.io/bitnamicharts
    tags:
      - bitnami-common
    version: 2.x.x

2. (b) Or, you could go ahead and add the needed initContainer if 'existingSecretName' was specified:

  initContainers:
  - name: wait-for-secret
    image: radial/busyboxplus
    imagePullPolicy: Always
    command: ['sh', '-c']
    args:
    - |
      # Point to the internal API server hostname
      APISERVER=https://kubernetes.default.svc

      # Path to ServiceAccount token
      SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

      # Read this Pod's namespace
      NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

      # Read the ServiceAccount bearer token
      TOKEN=$(cat ${SERVICEACCOUNT}/token)

      # Reference the internal certificate authority (CA)
      CACERT=${SERVICEACCOUNT}/ca.crt

      # Explore the API with TOKEN
      while [ 200 -ne $(curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" \
        -X GET ${APISERVER}/api/v1/namespaces/avi-system/secrets/<secretName>\
        --head \
        --silent \
        --output /dev/null \
        --write-out '%{http_code}') ]
      do
        echo "waiting for secret: <secretName>"
        sleep 5
      done
      echo "waiting for secret: <secretName>"
      echo "  detected"

Describe alternatives you've considered

I tried creating the 'avi-secret' secret prior to deploying ako but it was simply overwritten.

Additional context

ESXI 8u2, Tanzu, K8s version v1.26.5

[ron1]

See #1145