Operator do not upgrade with error "no owned roles found" from olm operator

[teoincontatto]

Type of question

Help troubleshooting a configuration issue

Question

What did you do?

I am testing upgrade of StackGres operator by creating a catalog with current and previous version than a subscription with startingCSV set to the previous version (1.7.0) and installPlanApproval set to manual. Then I set installplan for previous version to approved. After installation of previous version completes I proceed and install the new version (1.8.0-snapshot) by approving the install plan for the new version.

What did you expect to see?

The operator to be fully upgraded and running as expected.

What did you see instead? Under which circumstances?

The operator is not fully upgraded, the install plan Installed condition it True, the CSV InstallSucceeded condition is Secceded but the opeartor Pod is not starting due to the missing secret for webhooks. Also the olm operator has the following messages:

time="2024-01-18T12:11:07Z" level=debug msg="syncing CSV" csv=stackgres.v1.8.0-snapshot id=nqEHX namespace=stackgres-65a90c92 phase=Pending                                                                
time="2024-01-18T12:11:07Z" level=debug msg="annotations correct" annotationTargets= opgroupTargets=                                                                                                       
time="2024-01-18T12:11:07Z" level=debug msg="csv in operatorgroup" csv=stackgres.v1.8.0-snapshot id=bQ0qg namespace=stackgres-65a90c92 opgroup=stackgres phase=Pending                                     
time="2024-01-18T12:11:07Z" level=debug msg="no intersecting operatorgroups provide the same apis" apis="SGBackup.v1.stackgres.io,SGCluster.v1.stackgres.io,SGConfig.v1.stackgres.io,SGDbOps.v1.stackgres.i
o,SGDistributedLogs.v1.stackgres.io,SGInstanceProfile.v1.stackgres.io,SGObjectStorage.v1beta1.stackgres.io,SGPoolingConfig.v1.stackgres.io,SGPostgresConfig.v1.stackgres.io,SGScript.v1.stackgres.io,SGShar
dedBackup.v1.stackgres.io,SGShardedCluster.v1alpha1.stackgres.io,SGShardedDbOps.v1.stackgres.io" csv=stackgres.v1.8.0-snapshot id=bQ0qg namespace=stackgres-65a90c92 phase=Pending                         
time="2024-01-18T12:11:07Z" level=debug msg="checking if csv is replacing an older version"                                                                                                                
time="2024-01-18T12:11:07Z" level=debug msg="unable to get previous csv" error="clusterserviceversions.operators.coreos.com \"stackgres.v1.7.0\" not found" replacing=stackgres.v1.7.0                     
time="2024-01-18T12:11:07Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"                                                                                                                  
time="2024-01-18T12:11:07Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"                                                                                                                   
time="2024-01-18T12:11:07Z" level=debug msg="permissions/requirements not met" minKubeMet=true permMet=false reqMet=true                                                                                   
time="2024-01-18T12:11:07Z" level=debug msg="checking if csv is replacing an older version"                                                                                                                
time="2024-01-18T12:11:07Z" level=debug msg="unable to get previous csv" error="clusterserviceversions.operators.coreos.com \"stackgres.v1.7.0\" not found" replacing=stackgres.v1.7.0                     
time="2024-01-18T12:11:07Z" level=info msg="requirements were not met" csv=stackgres.v1.8.0-snapshot id=bQ0qg namespace=stackgres-65a90c92 phase=Pending                                                   
time="2024-01-18T12:11:07Z" level=debug msg="opgroup is global" csv=stackgres.v1.8.0-snapshot opgroup=stackgres                                                                                            
time="2024-01-18T12:11:07Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"                                                                                                                  
time="2024-01-18T12:11:07Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"                                                                                                                   
time="2024-01-18T12:11:07Z" level=debug msg="lift roles/rolebindings to clusterroles/rolebindings" csv=stackgres.v1.8.0-snapshot opgroup=stackgres                                                         
time="2024-01-18T12:11:07Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=stackgres.v1.8.0-snapshot error="no owned roles found" id=nqEHX namespace=stackgres-65a90c92 phase=Pending      
time="2024-01-18T12:11:07Z" level=debug msg="done syncing CSV" csv=stackgres.v1.8.0-snapshot id=nqEHX namespace=stackgres-65a90c92 phase=Pending                                                           
E0118 12:11:07.885116       1 queueinformer_operator.go:319] sync {"update" "stackgres-65a90c92/stackgres.v1.8.0-snapshot"} failed: no owned roles found                                                   

Environment

• operator-lifecycle-manager version:

$ olm --version
OLM version: v0.26.0
git commit: 95405d81e4c87c8113ccd7a95ba4d088b200a42a

• Kubernetes version information:

$ kubectl version
Client Version: v1.28.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.22.17
WARNING: version difference between client (1.28) and server (1.22) exceeds the supported minor version skew of +/-1

• Kubernetes cluster kind:

$ kind-0.20.0 version
kind v0.20.0 go1.20.4 linux/amd64

Additional context

$ operator-sdk version
operator-sdk version: "v1.30.0", commit: "b794fe909abc1affa1f28cfb75ceaf3bf79187e6", kubernetes version: "1.26.0", go version: "go1.19.10", GOOS: "linux", GOARCH: "amd64"

The CSVs of the stackgres operator previous and current versions:

• 1.7.0: csv-1.7.0.json
• 1.8.0-snapshot: csv-1.8.0-snapshot.json

Also, installing the current version (1.8.0) alone works without issues

[teoincontatto]

After some retries the operator Pod is able to start since the Secret for webhook are created (somehow) by the olm operator. Now I am facing another issue that is an extra service account (stackgres-restapi) that OLM created for the previous version of the operator get removed.

Can someone guide me to find out what is the root cause of that? I think the olm operator error about "no owner roles found" have something to do. From the code there seems to be multiple reasons why that error is shown and debugging just make me think this method contains some hints about the actual check that is failing: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/operators/olm/requirements.go#L260

[teoincontatto]

After rebuilding olm with trace level I was able to found that the extra serviceaccount was indeed the cause of the error "no owner roles found". I am not sure why it get removed but seems that previous version of the operator was creating it since not aware that it was the task of olm operator to create it. Shouldn't olm re-create an extra service account if it does not exists? Is this a bug??

[teoincontatto]

Here is the log with trace level I was talking about above:

time="2024-01-19T11:19:41Z" level=trace msg="popped queue" item="{update stackgres-65aa3592/stackgres.v1.8.0-snapshot}" queue-length=0
time="2024-01-19T11:19:41Z" level=debug msg="syncing CSV" csv=stackgres.v1.8.0-snapshot id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="annotations correct" annotationTargets= opgroupTargets=
time="2024-01-19T11:19:41Z" level=debug msg="csv in operatorgroup" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 opgroup=stackgres phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="no intersecting operatorgroups provide the same apis" apis="SGBackup.v1.stackgres.io,SGCluster.v1.stackgres.io,SGConfig.v1.stackgres.io,SGDbOps.v1.stackgres.io,SGDistributedLogs.v1.stackgres.io,SGInstanceProfile.v1.stackgres.io,SGObjectStorage.v1beta1.stackgres.io,SGPoolingConfig.v1.stackgres.io,SGPostgresConfig.v1.stackgres.io,SGScript.v1.stackgres.io,SGShardedBackup.v1.stackgres.io,SGShardedCluster.v1alpha1.stackgres.io,SGShardedDbOps.v1.stackgres.io" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="checking if csv is replacing an older version"
time="2024-01-19T11:19:42Z" level=trace msg="enqueuing resource event" event="{update stackgres-65aa3592/stackgres}"
time="2024-01-19T11:19:42Z" level=trace msg="popped queue" item="{update stackgres-65aa3592/stackgres}" queue-length=0
2024-01-19T11:19:42Z	DEBUG	controllers.operator	reconciling operator	{"request": {"name":"stackgres.stackgres-65aa3592"}}
time="2024-01-19T11:19:42Z" level=debug msg="subscription has changed, requeuing installed csv" csv=stackgres.v1.8.0-snapshot
2024-01-19T11:19:42Z	DEBUG	controllers.adoption	reconciling subscription	{"request": {"name":"stackgres","namespace":"stackgres-65aa3592"}}
2024-01-19T11:19:42Z	DEBUG	controllers.operator	reconciling operator	{"request": {"name":"stackgres.stackgres-65aa3592"}}
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-operator status="{ v1 ServiceAccount stackgres-operator Present   [{rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\",\"list\",\"watch\",\"update\",\"create\",\"delete\",\"patch\"],\"apiGroups\":[\"\",\"apps\",\"extensions\",\"rbac.authorization.k8s.io\",\"batch\"],\"resources\":[\"pods\",\"pods/exec\",\"pods/log\",\"services\",\"endpoints\",\"endpoints/restricted\",\"persistentvolumeclaims\",\"configmaps\",\"secrets\",\"deployments\",\"statefulsets\",\"serviceaccounts\",\"namespaces\",\"roles\",\"rolebindings\",\"events\",\"cronjobs\",\"jobs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\",\"list\"],\"apiGroups\":[\"storage.k8s.io\"],\"resources\":[\"storageclasses\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"create\",\"watch\",\"list\",\"get\",\"update\",\"patch\",\"delete\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgclusters\",\"sgpgconfigs\",\"sginstanceprofiles\",\"sgpoolconfigs\",\"sgbackupconfigs\",\"sgbackups\",\"sgdistributedlogs\",\"sgdbops\",\"sgobjectstorages\",\"sgscripts\",\"sgshardedclusters\",\"sgshardedbackups\",\"sgshardeddbops\",\"sgconfigs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgconfigs/status\",\"sgclusters/status\",\"sgdistributedlogs/status\",\"sgclusters/finalizers\",\"sgpgconfigs/finalizers\",\"sginstanceprofiles/finalizers\",\"sgpoolconfigs/finalizers\",\"sgbackupconfigs/finalizers\",\"sgbackups/finalizers\",\"sgdistributedlogs/finalizers\",\"sgdbops/finalizers\",\"sgobjectstorages/finalizers\",\"sgscripts/finalizers\",\"sgshardedclusters/finalizers\",\"sgshardedbackups/finalizers\",\"sgshardeddbops/finalizers\",\"sgconfigs/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"\",\"apps\",\"batch\"],\"resources\":[\"statefulsets/finalizers\",\"persistentvolumeclaims/finalizers\",\"deployments/finalizers\",\"services/finalizers\",\"endpoints/finalizers\",\"cronjobs/finalizers\",\"jobs/finalizers\",\"pods/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"sgconfigs.stackgres.io\",\"sgclusters.stackgres.io\",\"sginstanceprofiles.stackgres.io\",\"sgpgconfigs.stackgres.io\",\"sgpoolconfigs.stackgres.io\",\"sgbackups.stackgres.io\",\"sgbackupconfigs.stackgres.io\",\"sgobjectstorages.stackgres.io\",\"sgdbops.stackgres.io\",\"sgdistributedlogs.stackgres.io\",\"sgshardedclusters.stackgres.io\",\"sgshardedbackups.stackgres.io\",\"sgshardeddbops.stackgres.io\",\"sgscripts.stackgres.io\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\",\"watch\",\"create\"],\"apiGroups\":[\"snapshot.storage.k8s.io\"],\"resources\":[\"volumesnapshots\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"prometheuses.monitoring.coreos.com\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\",\"create\",\"delete\",\"update\",\"patch\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"servicemonitors\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"prometheus\",\"prometheuses\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"operators.coreos.com\"],\"resources\":[\"operators\"]}}]}"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-restapi status="{ v1 ServiceAccount stackgres-restapi NotPresent Service account does not exist  []}"
time="2024-01-19T11:19:42Z" level=debug msg="permissions/requirements not met" minKubeMet=true permMet=false reqMet=true
time="2024-01-19T11:19:42Z" level=debug msg="checking if csv is replacing an older version"
time="2024-01-19T11:19:42Z" level=info msg="requirements were not met" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=debug msg="opgroup is global" csv=stackgres.v1.8.0-snapshot opgroup=stackgres
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-operator status="{ v1 ServiceAccount stackgres-operator Present   [{rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\",\"list\",\"watch\",\"update\",\"create\",\"delete\",\"patch\"],\"apiGroups\":[\"\",\"apps\",\"extensions\",\"rbac.authorization.k8s.io\",\"batch\"],\"resources\":[\"pods\",\"pods/exec\",\"pods/log\",\"services\",\"endpoints\",\"endpoints/restricted\",\"persistentvolumeclaims\",\"configmaps\",\"secrets\",\"deployments\",\"statefulsets\",\"serviceaccounts\",\"namespaces\",\"roles\",\"rolebindings\",\"events\",\"cronjobs\",\"jobs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\",\"list\"],\"apiGroups\":[\"storage.k8s.io\"],\"resources\":[\"storageclasses\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"create\",\"watch\",\"list\",\"get\",\"update\",\"patch\",\"delete\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgclusters\",\"sgpgconfigs\",\"sginstanceprofiles\",\"sgpoolconfigs\",\"sgbackupconfigs\",\"sgbackups\",\"sgdistributedlogs\",\"sgdbops\",\"sgobjectstorages\",\"sgscripts\",\"sgshardedclusters\",\"sgshardedbackups\",\"sgshardeddbops\",\"sgconfigs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgconfigs/status\",\"sgclusters/status\",\"sgdistributedlogs/status\",\"sgclusters/finalizers\",\"sgpgconfigs/finalizers\",\"sginstanceprofiles/finalizers\",\"sgpoolconfigs/finalizers\",\"sgbackupconfigs/finalizers\",\"sgbackups/finalizers\",\"sgdistributedlogs/finalizers\",\"sgdbops/finalizers\",\"sgobjectstorages/finalizers\",\"sgscripts/finalizers\",\"sgshardedclusters/finalizers\",\"sgshardedbackups/finalizers\",\"sgshardeddbops/finalizers\",\"sgconfigs/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"\",\"apps\",\"batch\"],\"resources\":[\"statefulsets/finalizers\",\"persistentvolumeclaims/finalizers\",\"deployments/finalizers\",\"services/finalizers\",\"endpoints/finalizers\",\"cronjobs/finalizers\",\"jobs/finalizers\",\"pods/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"sgconfigs.stackgres.io\",\"sgclusters.stackgres.io\",\"sginstanceprofiles.stackgres.io\",\"sgpgconfigs.stackgres.io\",\"sgpoolconfigs.stackgres.io\",\"sgbackups.stackgres.io\",\"sgbackupconfigs.stackgres.io\",\"sgobjectstorages.stackgres.io\",\"sgdbops.stackgres.io\",\"sgdistributedlogs.stackgres.io\",\"sgshardedclusters.stackgres.io\",\"sgshardedbackups.stackgres.io\",\"sgshardeddbops.stackgres.io\",\"sgscripts.stackgres.io\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\",\"watch\",\"create\"],\"apiGroups\":[\"snapshot.storage.k8s.io\"],\"resources\":[\"volumesnapshots\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"prometheuses.monitoring.coreos.com\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\",\"create\",\"delete\",\"update\",\"patch\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"servicemonitors\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"prometheus\",\"prometheuses\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied  cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"operators.coreos.com\"],\"resources\":[\"operators\"]}}]}"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-restapi status="{ v1 ServiceAccount stackgres-restapi NotPresent Service account does not exist  []}"
time="2024-01-19T11:19:42Z" level=debug msg="lift roles/rolebindings to clusterroles/rolebindings" csv=stackgres.v1.8.0-snapshot opgroup=stackgres
time="2024-01-19T11:19:42Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=stackgres.v1.8.0-snapshot error="no owned roles found" id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=debug msg="done syncing CSV" csv=stackgres.v1.8.0-snapshot id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=trace msg="requeuing with rate limiting" cache-key=stackgres-65aa3592/stackgres.v1.8.0-snapshot item="{update stackgres-65aa3592/stackgres.v1.8.0-snapshot}" requeues=2
E0119 11:19:42.543718       1 queueinformer_operator.go:319] sync {"update" "stackgres-65aa3592/stackgres.v1.8.0-snapshot"} failed: no owned roles found

Is there a way to set trace level with some olm command parameter or environment variable?

[ddelnano]

@teoincontatto have you had any updates since your last message? I'm trying to upgrade the pixie operator to bundle OLM v0.26.0 and I'm seeing this same problem consistently with our next release candidate build.

[teoincontatto]

After going back to this I was able to reproduce the issue locally with the same pipeline. I had to set up a Vagrantfile and a script in order to make it work (see attachments Vagrantfile.zip).

It turned out the problem was related to the caBundle field in webhooks that was set to a dummy value. I can not really recall why it was like that but something related to an old Kubernetes version restriction (maybe field was required before?!). The OLM operator was randomly setting the dummy value after changing it (maybe a bug?). Removing the value for caBundle field solved the issue but without reproducing it was impossible to tell what was happening (sometimes the problem didn't appear and the test passed).

[ddelnano]

Thanks for providing that context. I was able to debug my problem and while the error message matched what was reported here, my issue was self inflicted and is resolved.