[Bug]: "caller":"backend/backend.go:189","msg":"failed to open database","path":"/bitnami/etcd/data/member/snap/db"

[erigo]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.3.15
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: -
- GPU: -
- Others: -

Current Behavior

{"level":"panic","ts":"2024-05-13T12:10:12.697Z","caller":"backend/backend.go:189","msg":"failed to open database","path":"/bitnami/etcd/data/member/snap/db","error":"errno 524","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/etcdserver.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55\ngo.etcd.io/etcd/server/v3/etcdserver.openBackend.func1\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76"}
panic: failed to open database
goroutine 52 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0005be300, 0xc0004ca000, 0x2, 0x2)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x58d
go.uber.org/zap.(*Logger).Panic(0xc0000d40a0, 0x12301ac, 0x17, 0xc0004ca000, 0x2, 0x2)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x85
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend(0xc000130a80, 0x21, 0x5f5e100, 0x2710, 0x121f689, 0x7, 0x119999999, 0xc0000d40a0, 0x0, 0x139f320, ...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189 +0x3b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/etcdserver.newBackend(0xc0000420ca, 0x16, 0x0, 0x0, 0x0, 0x0, 0xc0001bfa70, 0x1, 0x1, 0xc00014e5a0, ...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55 +0x1f8
go.etcd.io/etcd/server/v3/etcdserver.openBackend.func1(0xc0004cc060, 0xc0000fa300, 0x139f320, 0xc0003687e0)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76 +0xc5
created by go.etcd.io/etcd/server/v3/etcdserver.openBackend
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:75 +0x145

Expected Behavior

No response

Steps To Reproduce

1. milvus operator v0.9.14
2. cert-manager v1.14.5
3. milvus-cluster.yaml
apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: snrobots-milvus
  labels:
    app: milvus
spec:
  mode: cluster
  dependencies:
    msgStreamType: pulsar
    pulsar:
      inCluster:
        deletionPolicy: Delete
        pvcDeletion: true
    etcd:
      inCluster:
        deletionPolicy: Delete
        pvcDeletion: true
    storage:
      inCluster:
        deletionPolicy: Delete
        pvcDeletion: true
  components:
    enableRollingUpdate: true
    imageUpdateMode: rollingUpgrade
    image: milvusdb/milvus:v2.3.15
    proxy:
      replicas: 1
    dataNode:
      replicas: 1
    indexNode:
      replicas: 1
    queryNode:
      replicas: 1
  config: {}

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

I think you hit some etcd data issue.
here are some fixes might be able to help.
etcd-io/etcd#15834
if you deploy three node etcd, and two of nodes are start, you could delete the broken etcd node.

If you only deployed 1 etcd replica, then I'm not sure you can still recover your data.

[yanliang567]

/assign @erigo
please check as comments above.

/unassign

[erigo]

The issue is likely with etcd; my Kubernetes (k8s) is running on the Docker engine, with version v26.1.2. I have also seen this issue in the official issue tracker, but I didn't quite understand it.
etcd-io/etcd#16596

[erigo]

full logs:
etcd 02:50:33.50
etcd 02:50:33.50 Welcome to the Bitnami etcd container
etcd 02:50:33.50 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 02:50:33.50 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 02:50:33.51
etcd 02:50:33.51 INFO ==> ** Starting etcd setup **
etcd 02:50:33.52 INFO ==> Validating settings in ETCD_* env vars..
etcd 02:50:33.52 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 02:50:33.53 INFO ==> Initializing etcd
etcd 02:50:33.53 INFO ==> Generating etcd config file using env variables
etcd 02:50:48.62 INFO ==>
etcd 02:50:48.62 INFO ==> get_etcd_active_endpoints:
etcd 02:50:48.64 INFO ==> Detected data from previous deployments
etcd 02:50:48.64 INFO ==> The member will try to join the cluster by it's own
etcd 02:50:48.65 INFO ==> ** etcd setup finished! **
etcd 02:50:48.67 INFO ==> ** Starting etcd **
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2379"}
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1000"}
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2024-05-14T02:50:48.692Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"2500"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"500"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"snrobots-milvus-etcd-0=http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380,snrobots-milvus-etcd-1=http://snrobots-milvus-etcd-1.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380,snrobots-milvus-etcd-2=http://snrobots-milvus-etcd-2.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"snrobots-milvus-etcd-0"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"4294967296"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=snrobots-milvus-etcd-headless.default.svc.cluster.local"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-05-14T02:50:48.693Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2024-05-14T02:50:48.693Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"info","ts":"2024-05-14T02:50:48.694Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2024-05-14T02:50:48.694Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2024-05-14T02:50:48.694Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2024-05-14T02:50:48.695Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"19002cfc6","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":false,"name":"snrobots-milvus-etcd-0","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"2.5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":[""],"host-whitelist":[""],"initial-cluster":"snrobots-milvus-etcd-0=http://snrobots-milvus-etcd-0.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380,snrobots-milvus-etcd-1=http://snrobots-milvus-etcd-1.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380,snrobots-milvus-etcd-2=http://snrobots-milvus-etcd-2.snrobots-milvus-etcd-headless.default.svc.cluster.local:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster-k8s","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"panic","ts":"2024-05-14T02:50:48.717Z","caller":"backend/backend.go:189","msg":"failed to open database","path":"/bitnami/etcd/data/member/snap/db","error":"errno 524","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/etcdserver.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55\ngo.etcd.io/etcd/server/v3/etcdserver.openBackend.func1\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76"}
panic: failed to open database
goroutine 68 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001b20c0, 0xc000170000, 0x2, 0x2)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x58d
go.uber.org/zap.(*Logger).Panic(0xc0000d40a0, 0x12301ac, 0x17, 0xc000170000, 0x2, 0x2)
/usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x85
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend(0xc0005325a0, 0x21, 0x5f5e100, 0x2710, 0x121f689, 0x7, 0x119999999, 0xc0000d40a0, 0x0, 0x139f320, ...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189 +0x3b3
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/etcdserver.newBackend(0xc0000420ca, 0x16, 0x0, 0x0, 0x0, 0x0, 0xc00014e750, 0x1, 0x1, 0xc00014efc0, ...)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55 +0x1f8
go.etcd.io/etcd/server/v3/etcdserver.openBackend.func1(0xc0000d61e0, 0xc0000fa300, 0x139f320, 0xc00036a360)
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76 +0xc5
created by go.etcd.io/etcd/server/v3/etcdserver.openBackend
/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:75 +0x145

[erigo]

bitnami/containers#19759

[erigo]

It should be a bug caused by storage, and the specific location should be an issue with the op_lock stored in NFS.

[erigo]

It is recommended that the official provide a best practice for setting up etcd, mq, and minio storage and recommend using local pv for these highly available components.

[xiaofan-luan]

we don't encourage anything, including minio, etcd and kafka to running on nfs or nas.
especially for etcd, high performance SSD is required.

[erigo]

tks for your advise, we will try to deploy the components that Milvus depend on as expected.