Statemachine is not in state ready to do UPGRADE

[kyrychenkoe]

Description:
We have 5 deployed streams on our environment and the issue happens sporadically. We created k8s job to upgrade stream apps to newer versions. We use curl inside our job to perform an upgrade. Here is an example:
link https://{scdf-host}/streams/deployments/update/mystream with body: { "releaseName": "mystream", "packageIdentifier": { "packageName": "mystream" }, "updateProperties": { "version.mystream-source": "1.0.123-SNAPSHOT", "version.mystream-sink": "1.0.103-SNAPSHOT" }, "force": false }
It seems that it works fine but sometimes weird thing happens. Usually, it deploys new versions of these apps on our k8s cluster and after some time (I guess until the readiness probe is green) removes old versions. But sometimes weird things happen and old versions are not removed even after all checks. It means that streams are currently in some upgrading status and I'm not able to perform any action using the UI (like destroy \ upgrade or undeploy). Looks like we are stuck in this state. I didn't find any specific logs that can point me to the issue. Can it happen just because we perform such an upgrade for 5 streams simultaneously?

Release versions:
{ "featureInfo": { "analyticsEnabled": true, "streamsEnabled": true, "tasksEnabled": true, "schedulesEnabled": false, "monitoringDashboardType": "NONE" }, "versionInfo": { "implementation": { "name": "spring-cloud-dataflow-server", "version": "2.11.2" }, "core": { "name": "Spring Cloud Data Flow Core", "version": "2.11.2" }, "dashboard": { "name": "Spring Cloud Dataflow UI", "version": "3.4.2" }, "shell": { "name": "Spring Cloud Data Flow Shell", "version": "2.11.2", "url": "https://repo.maven.apache.org/maven2/org/springframework/cloud/spring-cloud-dataflow-shell/2.11.2/spring-cloud-dataflow-shell-2.11.2.jar" } }, "securityInfo": { "authenticationEnabled": false, "authenticated": false, "username": null, "roles": [] }, "runtimeEnvironment": { "appDeployer": { "deployerImplementationVersion": "2.11.2", "deployerName": "Spring Cloud Skipper Server", "deployerSpiVersion": "2.11.2", "javaVersion": "11.0.19", "platformApiVersion": "", "platformClientVersion": "", "platformHostVersion": "", "platformSpecificInfo": { "default": "kubernetes" }, "platformType": "Skipper Managed", "springBootVersion": "2.7.18", "springVersion": "5.3.31" }, "taskLaunchers": [ { "deployerImplementationVersion": "unknown", "deployerName": "KubernetesTaskLauncher", "deployerSpiVersion": "unknown", "javaVersion": "11.0.19", "platformApiVersion": "v1", "platformClientVersion": "unknown", "platformHostVersion": "unknown", "platformSpecificInfo": { "namespace": "non-prod", "master-url": "https://172.20.0.1:443/" }, "platformType": "Kubernetes", "springBootVersion": "2.7.18", "springVersion": "5.3.31" } ] }, "monitoringDashboardInfo": { "url": "", "refreshInterval": 15, "dashboardType": "NONE", "source": "default-scdf-source" }, "gitAndBuildInfo": { "git": { "branch": "main", "commit": { "id": "86b53e0", "time": "2024-01-10T21:06:58Z" } }, "build": { "artifact": "spring-cloud-dataflow-server", "name": "Spring Cloud Data Flow Server", "time": "2024-01-11T17:00:03.506Z", "version": "2.11.2", "group": "org.springframework.cloud" } }, "_links": { "self": { "href": "https://scdf.eks.elsevierelsa.com/about" } } }

Steps to reproduce:
Unfortunately, I have no specific steps to reproduce. It seems like problem is sporadic and streams sometimes just not able to complete versions upgrade flow