4.0.1 kubectl mayastor upgrade failed if 'mayastor.metrics.enabled=true' (this is default value)

[AleksandrSor]

Description

helmchart installed with --set 'mayastor.metrics.enabled=true'
when you run

kubectl mayastor upgrade -n openebs --set 'mayastor.agents.core.rebuild.partial.enabled=false'

you got upgrade job stuck on upgrading data plane
because openebs-io-engine pod did not start after patching

2024-05-08T05:16:35.673250Z ERROR metrics_exporter_io_engine::client::grpc_client: Grpc connection timeout, retrying after 10s
    at metrics-exporter/src/bin/io_engine/client/grpc_client.rs:86

  2024-05-08T05:16:45.674423Z ERROR metrics_exporter_io_engine::client::grpc_client: Grpc connection timeout, retrying after 10s
    at metrics-exporter/src/bin/io_engine/client/grpc_client.rs:86

Expected Behavior

upgrade job completed successfully

Current Behavior

upgrade job stuck

Possible Solution

in general if you disable metrics via installing helm with 'mayastor.metrics.enabled=false'
you can avoid this
2.
if you already on a problem with openebs-io-engine pod not starting you can manually
edit daemon set
and delete metrics exporter containers from spec
apparently patching keeps the old one and it causes crashes

Steps to Reproduce

1. helm chart upgraded from 3.9.0 to 4.0.1 as in instruction
2. values.yaml

mayastor:
  enabled: true
  metrics:
    enabled: true

3. run

kubectl mayastor upgrade -n openebs --set 'mayastor.agents.core.rebuild.partial.enabled=false'

job stuck at one of openebs-io-engine pods not ready after patching

Screenshots

Your Environment

• OS (from /etc/os-release):

NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream

• Kernel (from uname -a): Linux k8host-lon04.l8s.space 5.14.0-437.el9.x86_64 Do we open issues here? Because easy. #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:

kubectl 1.29.4
mayastor plugin 2.6.1

[niladrih]

Hi @AleksandrSor, I tried this on Kubernetes v1.25 (vanilla kubeadm style cluster) with Ubuntu nodes.
Linux master-180253 5.4.0-177-generic #197-Ubuntu SMP Thu Mar 28 22:45:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Seems to work for me. No apparent issues. This was my install command:

helm install openebs openebs/openebs -n openebs --create-namespace --set mayastor.enabled=true --version 3.9.0

Also helm v3.13.3

% helm version
version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.20.11"}

These are the resources I used:

1. Docs: https://openebs.io/docs/user-guides/upgrade
2. Plugin: https://github.com/openebs/mayastor/releases/download/v2.6.1/kubectl-mayastor-x86_64-linux-musl.tar.gz

---

Could you share the logs from the upgrade-job?
kubectl logs -n openebs job/openebs-upgrade-v2-6-1

Also, could you share the logs and kubectl describe pod for any of the failing/stuck io-engine containers or it's init-containers?

cc: @abhilashshetty04

[AleksandrSor]

Sorry, I did not store the logs.
I just fixed the issue and went on

[abhilashshetty04]

@AleksandrSor , Good to know that the issue is resolved. It seems like exporter was stuck in grpc connect phase.

Did it got resolved with metrics.enabled=true ?