Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parameters secret_shares,secret_threshold not applicable to seal type alicloudkms #316

Open
wadexu007 opened this issue May 6, 2023 · 0 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@wadexu007
Copy link

wadexu007 commented May 6, 2023

Bugs should be filed for issues encountered whilst operating bank-vaults.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
When we use vault operator deploy vault to Alibaba cloud (Aliyun), we met issue "parameters secret_shares,secret_threshold not applicable to seal type alicloudkms" when vault statefulset initializing at second container bank-vaults

Expected behaviour:
Vault statefulset container started successfully.

Steps to reproduce the bug:
Install Vault operator
Install Vault crd, yaml refer to below.
check logs of Vault satefulset

Additional context:
The problem happens on Vault 1.12.3
but if i downgrade Vault to 1.11.3 which works fine. Vault crd yaml see below.
image: vault:1.12.3 VS image: vault:1.11.3

Environment details:

  • Kubernetes version 1.24.6, ACK 1.24.6-aliyun.1
  • Cloud-provider/provisioner Alibaba Aliyun
  • bank-vaults version vault-operator 1.15.3, also tried latest 1.19.0 same issue.
  • Vault crd 1.12.3
  • Install method, Helm and kubectl apply yaml

Logs:

 kubectl -n vault logs -f vault-0 -c vault
==> Vault server configuration:

              HA Storage: raft
             Api Address: https://vault.vault.svc.cluster.local:8200
                     Cgo: disabled
         Cluster Address: https://vault-0:8201
   Environment Variables: ALICLOUD_ACCESS_KEY, ALICLOUD_SECRET_KEY, GODEBUG, HOME, HOSTNAME, KUBERNETES_PORT, KUBERNETES_PORT_443_TCP, KUBERNETES_PORT_443_TCP_ADDR, KUBERNETES_PORT_443_TCP_PORT, KUBERNETES_PORT_443_TCP_PROTO, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_PORT_HTTPS, PATH, PWD, SHLVL, VAULT_0_PORT, VAULT_0_PORT_8200_TCP, VAULT_0_PORT_8200_TCP_ADDR, VAULT_0_PORT_8200_TCP_PORT, VAULT_0_PORT_8200_TCP_PROTO, VAULT_0_PORT_8201_TCP, VAULT_0_PORT_8201_TCP_ADDR, VAULT_0_PORT_8201_TCP_PORT, VAULT_0_PORT_8201_TCP_PROTO, VAULT_0_PORT_9091_TCP, VAULT_0_PORT_9091_TCP_ADDR, VAULT_0_PORT_9091_TCP_PORT, VAULT_0_PORT_9091_TCP_PROTO, VAULT_0_SERVICE_HOST, VAULT_0_SERVICE_PORT, VAULT_0_SERVICE_PORT_API_PORT, VAULT_0_SERVICE_PORT_CLUSTER_PORT, VAULT_0_SERVICE_PORT_METRICS, VAULT_1_PORT, VAULT_1_PORT_8200_TCP, VAULT_1_PORT_8200_TCP_ADDR, VAULT_1_PORT_8200_TCP_PORT, VAULT_1_PORT_8200_TCP_PROTO, VAULT_1_PORT_8201_TCP, VAULT_1_PORT_8201_TCP_ADDR, VAULT_1_PORT_8201_TCP_PORT, VAULT_1_PORT_8201_TCP_PROTO, VAULT_1_PORT_9091_TCP, VAULT_1_PORT_9091_TCP_ADDR, VAULT_1_PORT_9091_TCP_PORT, VAULT_1_PORT_9091_TCP_PROTO, VAULT_1_SERVICE_HOST, VAULT_1_SERVICE_PORT, VAULT_1_SERVICE_PORT_API_PORT, VAULT_1_SERVICE_PORT_CLUSTER_PORT, VAULT_1_SERVICE_PORT_METRICS, VAULT_K8S_POD_NAME, VAULT_PORT, VAULT_PORT_8200_TCP, VAULT_PORT_8200_TCP_ADDR, VAULT_PORT_8200_TCP_PORT, VAULT_PORT_8200_TCP_PROTO, VAULT_PORT_8201_TCP, VAULT_PORT_8201_TCP_ADDR, VAULT_PORT_8201_TCP_PORT, VAULT_PORT_8201_TCP_PROTO, VAULT_PORT_9091_TCP, VAULT_PORT_9091_TCP_ADDR, VAULT_PORT_9091_TCP_PORT, VAULT_PORT_9091_TCP_PROTO, VAULT_PORT_9102_TCP, VAULT_PORT_9102_TCP_ADDR, VAULT_PORT_9102_TCP_PORT, VAULT_PORT_9102_TCP_PROTO, VAULT_SERVICE_HOST, VAULT_SERVICE_PORT, VAULT_SERVICE_PORT_API_PORT, VAULT_SERVICE_PORT_CLUSTER_PORT, VAULT_SERVICE_PORT_METRICS, VAULT_SERVICE_PORT_STATSD
              Go Version: go1.19.4
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: alicloudoss
                 Version: Vault v1.12.3, built 2023-02-02T09:07:27Z
             Version Sha: 209b3dd99fe8ca320340d08c70cff5f620261f9b

==> Vault server started! Log data will stream in below:

2023-05-05T04:50:09.104Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-05-05T04:50:09.227Z [WARN]  ha.raft.fsm: raft FSM db file has wider permissions than needed: needed=-rw------- existing=-rw-rw----
2023-05-05T04:50:09.263Z [INFO]  core: Initializing version history cache for core
2023-05-05T04:50:09.263Z [INFO]  core: stored unseal keys supported, attempting fetch
2023-05-05T04:50:09.268Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2023-05-05T04:50:09.521Z [INFO]  core: security barrier not initialized
2023-05-05T04:50:10.277Z [INFO]  core: security barrier not initialized
kubectl -n vault logs -f vault-0 -c bank-vaults
{"level":"info","msg":"initializing vault...","time":"2023-05-05T04:53:12Z"}
{"level":"info","msg":"vault metrics exporter enabled: :9091/metrics","time":"2023-05-05T04:53:12Z"}
{"level":"info","msg":"initializing vault","time":"2023-05-05T04:53:12Z"}
{"level":"fatal","msg":"error initializing vault: error initializing vault: Error making API request.\n\nURL: PUT https://127.0.0.1:8200/v1/sys/init\nCode: 400. Errors:\n\n* parameters secret_shares,secret_threshold not applicable to seal type alicloudkms","time":"2023-05-05T04:53:12Z"}

Below is Vault crd yaml

---
# Source: vault/templates/vault.yaml
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: Vault
metadata:
  name: release-name-vault
spec:
  size: 2
  image: vault:1.12.3

  unsealConfig:
    alibaba:
      kmsKeyId: xxxxx
      kmsRegion: cn-shanghai
      ossBucket: xxxx
      ossEndpoint: oss-cn-shanghai.aliyuncs.com
      ossPrefix: vault-operator/
  config:
    storage:
      alicloudoss:
        bucket: xxxxxxx
        endpoint: oss-cn-shanghai.aliyuncs.com
    ha_storage:
      raft:
        path: /data/raft
    listener:
      tcp:
        telemetry:
          unauthenticated_metrics_access: true
        address: "0.0.0.0:8200"
        cluster_address: "0.0.0.0:8201"
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    seal:
      alicloudkms:
        kms_key_id: xxxxxx
        region: cn-shanghai
    api_addr: https://vault.vault.svc.cluster.local:8200
    cluster_addr: "https://${.Env.POD_NAME}:8201"
    ui: true
  vaultEnvsConfig:
    - name: ALICLOUD_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: access_key
          name: vault-aliyun-creds
    - name: ALICLOUD_SECRET_KEY
      valueFrom:
        secretKeyRef:
          key: secret_key
          name: vault-aliyun-creds
  envsConfig:
    - name: BANK_VAULTS_ALIBABA_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: access_key
          name: vault-aliyun-creds
    - name: BANK_VAULTS_ALIBABA_ACCESS_KEY_SECRET
      valueFrom:
        secretKeyRef:
          key: secret_key
          name: vault-aliyun-creds
  volumeClaimTemplates:
    - metadata:
        name: vault-raft
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        volumeMode: Filesystem
  volumeMounts:
    - mountPath: /data/raft
      name: vault-raft

This is vault operator yaml

---
# Source: vault-operator/charts/vault-operator/templates/sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-operator
  labels:
    helm.sh/chart: vault-operator-1.15.8
    app.kubernetes.io/name: vault-operator
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
---
# Source: vault-operator/charts/vault-operator/templates/role.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: release-name-vault-operator
  labels:
    helm.sh/chart: vault-operator-1.15.8
rules:
- apiGroups:
  - vault.banzaicloud.com
  resources:
  - "*"
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - events
  - pods
  - services
  - configmaps
  - secrets
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  - extensions
  resources:
  - replicasets
  verbs:
  - get
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  - deployments/finalizers
  - statefulsets
  verbs:
  - "*"
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - list
  - get
  - create
  - update
  - watch
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs:
  - update
  - list
  - get
  - create
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - "*"
---
# Source: vault-operator/charts/vault-operator/templates/rolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: release-name-vault-operator
  labels:
    helm.sh/chart: vault-operator-1.15.8
subjects:
- kind: ServiceAccount
  name: vault-operator
  namespace: default
roleRef:
  kind: ClusterRole
  name: release-name-vault-operator
  apiGroup: rbac.authorization.k8s.io
---
# Source: vault-operator/charts/vault-operator/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: release-name-vault-operator
  labels:
    helm.sh/chart: vault-operator-1.15.8
    app.kubernetes.io/name: vault-operator
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  - port: 8383
    protocol: TCP
    name: http-metrics
  selector:
    app.kubernetes.io/name: vault-operator
---
# Source: vault-operator/charts/vault-operator/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name-vault-operator
  labels:
    helm.sh/chart: vault-operator-1.15.8
    app.kubernetes.io/name: vault-operator
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
spec:
  strategy:
    type: Recreate
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault-operator
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault-operator
        app.kubernetes.io/instance: release-name
    spec:
      containers:
        - name: vault-operator
          image: "banzaicloud/vault-operator:1.15.3"
          imagePullPolicy: IfNotPresent
          command:
            - vault-operator
            - -sync_period
            - 1m
          env:
            - name: WATCH_NAMESPACE
              value: vault
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: vault-operator
            - name: OPERATOR_LOG_LEVEL
              value: debug
            - name: BANK_VAULTS_IMAGE
              value: "banzaicloud/bank-vaults:1.15.3"
          ports:
          - containerPort: 8080
          - containerPort: 8383
          livenessProbe:
            httpGet:
              path: "/"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: "/ready"
              port: 8080
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 100m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi
      affinity:
        {}
      serviceAccountName: vault-operator
      terminationGracePeriodSeconds: 10      
@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR that has become stale and will be auto-closed. label Nov 29, 2023
@ramizpolic ramizpolic added kind/bug Categorizes issue or PR as related to a bug. and removed lifecycle/stale Denotes an issue or PR that has become stale and will be auto-closed. labels Dec 22, 2023
@ramizpolic ramizpolic transferred this issue from bank-vaults/bank-vaults Dec 22, 2023
@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR that has become stale and will be auto-closed. label Feb 25, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 17, 2024
@csatib02 csatib02 removed the lifecycle/stale Denotes an issue or PR that has become stale and will be auto-closed. label May 5, 2024
@bank-vaults bank-vaults deleted a comment from github-actions bot May 5, 2024
@bank-vaults bank-vaults deleted a comment from github-actions bot May 5, 2024
@bank-vaults bank-vaults deleted a comment from github-actions bot May 5, 2024
@csatib02 csatib02 reopened this May 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants