[BUG] Operator certificate generation / renewals not working

[albgus]

I have recently updated the OpenSearch operator to version 2.6.0. This seems to have actually triggered some sort of certificate genration process, as seen in the log entries. However, it seems that only the admin certificate was updated, the http and transport certificate is still the same old version.

The operator has been logging this for hours with no apparent progress.

{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchactiongroup","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchActionGroup","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchcomponenttemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchComponentTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchindextemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchIndexTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.224Z","msg":"Starting workers","controller":"opensearchrole","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchRole","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.225Z","msg":"Starting workers","controller":"opensearchtenant","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchTenant","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchuserrolebinding","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUserRoleBinding","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:43:40.229Z","msg":"Starting workers","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"http"}
{"level":"info","ts":"2024-05-16T12:43:40.848Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997"}
{"level":"info","ts":"2024-05-16T12:43:40.946Z","logger":"KubeAPIWarningLogger","msg":"would violate PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"dashboards\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"dashboards\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"dashboards\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"dashboards\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}
{"level":"info","ts":"2024-05-16T12:44:10.985Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:11.173Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21"}
{"level":"info","ts":"2024-05-16T12:44:41.274Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:41.293Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:41.294Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:41.536Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095"}
{"level":"info","ts":"2024-05-16T12:45:11.647Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:11.924Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2"}
{"level":"info","ts":"2024-05-16T12:45:42.029Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:42.239Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3"}
{"level":"info","ts":"2024-05-16T12:46:12.334Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:12.488Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7"}
{"level":"info","ts":"2024-05-16T12:46:42.639Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:42.923Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690"}
{"level":"info","ts":"2024-05-16T12:47:13.027Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:13.044Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:13.045Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"http"}
{"level":"info","ts":"2024-05-16T12:47:13.274Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1"}
{"level":"info","ts":"2024-05-16T12:47:43.371Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:43.389Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:43.390Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"http"}

[pasztorl]

+1

[pasztorl]

I've checked with a new cluster install. 2.11.0 works as expected, 2.12.0 the same issue above.

[pasztorl]

update: if I retry multiple times sometimes works sometimes not, race condition?

[Jerrimikkihvatai]

+1 Catch the same error while testing cert renewal after trying this method. Tested on operator versions 2.5.0 and 2.6.0, opensearch 2.13.
The operator didn't recreate certs so I restarted pod and got the error. But certs were regenerated.
The error disappeared only after cluster redeploy.