Quickstart for Helm-based Operators is not working on OpenShift #3767
Assignees
Labels
good first issue
Denotes an issue ready for a new contributor, according to the "help wanted" guidelines.
kind/bug
Categorizes issue or PR as related to a bug.
language/helm
Issue is related to a Helm operator project
Milestone
Comments
|
Hi @nmasse-itix, To follow up the tutorial and quickstart you need to have cluster-admin permissions. The tool scaffold k8s manifests by default which requires this permission. You can change the scope of your project as described in my comment #3733 (comment). The need to create a doc to explain the scope per type is tracked already. See my comment in: #3447 (comment) |
camilamacedo86
added
language/helm
|
Hi @camilamacedo86 ! Thanks for jumping in and helping me. I re-did the whole tutorial (after proper cleanup of course) and I do confirm that:
So, I followed step-by-step the tutorial, while being cluster-admin and the described steps lead to a non-working operator. Can we fix the |
|
Hi @nmasse-itix, I just followed the tutorial and all worked. The finalizer role is not required for the steps performed so far. Also, I checked it with the helm-chart Nginx as well and the Memcached operator example in the samples. You can check also here the steps used to generate the sample. For this one, we need to add customize permissions but it has NOT the finalizer as well. You might not be doing the same steps than I am. Please, check all steps bellow with attention and let us know. Create a Helm Nginx project without a helm-chart (Tutorial)I applied the CR in the same namespace of the Memcached just to make easier get the output for you here
` $ kubectl logs deployment.apps/nginx-operator-controller-manager -n nginx-operator-system -c manager`
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: manager-role
rules:
##
## Base operator rules
##
# We need to get namespaces so the operator can read namespaces to ensure they exist
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
# We need to manage Helm release secrets
- apiGroups:
- ""
resources:
- secrets
verbs:
- "*"
# We need to create events on CRs about things happening during reconciliation
- apiGroups:
- ""
resources:
- events
verbs:
- create
##
## Rules for example.com/v1alpha1, Kind: Nginx
##
- apiGroups:
- example.com
resources:
- nginxes
- nginxes/status
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- verbs:
- "*"
apiGroups:
- ""
resources:
- "serviceaccounts"
- "services"
- verbs:
- "*"
apiGroups:
- "apps"
resources:
- "deployments"
# +kubebuilder:scaffold:rules
See that has no reason at all for we need the finalizer in this scenario. Create a Helm Nginx project with
|
|
Hi @camilamacedo86 and @nmasse-itix I've re-run all the tests from scratch on both Minikube and OpenShift and comes to the conclusion that Nicolas suggestion is mandatory for OpenShift. On MinikubeOn a fresh install, apply the following commands: Operator deploys correctly and is able to deploy the Checking the logs: Clean-up everything On OpenShiftOn a fresh 4.5 cluster, apply the same following commands: Operator deploys correctly but is not able to deploy the Checking the logs: Clean-up everything On OpenShift with modified RBAC on finalizersOn a fresh 4.5 cluster, apply the same following commands: before pursuing, edit [...]
##
## Rules for example.com/v1alpha1, Kind: Nginx
##
- apiGroups:
- example.com
resources:
- nginxes
- nginxes/status
- nginxes/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- verbs:
- "*"
apiGroups:
- ""
resources:
- "serviceaccounts"
- "services"
- verbs:
- "*"
apiGroups:
- "apps"
resources:
- "deployments"
# +kubebuilder:scaffold:rulesFinish with the last commands: Operator deploys correctly and is able to deploy the Checking the logs: There's a remaining issue when deploying the Pod because of security conflict with OpenShift SCC but it's not related to Operator SDK: ConclusionAs a conclusion, I would say that the tests confirm that Operator SDK for Helm does not work out-of-the-box on OpenShift. Finalizer permission on Custom Resource is required for OpenShift because of more restrictive default RBAC permissions on serviceaccount that should not be there on upstream Kubernetes. As this permission is not required by custom code logic and Operator SDK should be the preferred way for customers to develop operators on Red Hat's OpenShift, I suggest adding this permission by default when generating scaffold resources for a new Helm backed API. |
|
Hi @lbroudoux So, it means that we need to:
|
camilamacedo86
added
language/ansible
|
I've made some tests with Ansible but the permission is not required as the SDK does not place a finalizer on the Custom Resource. This seems to be done only with Helm. |
camilamacedo86
added
language/helm
@camilamacedo86 These docs are not related to this issue AFAICT. The operator developer needs cluster admin to be able to create the CRDs and deploy the operator into the cluster. The I think the solution here is just the simple addition of the finalizers permission in the default Helm role scaffold here: @lbroudoux or @nmasse-itix Would either of you be interested in submitting a PR to fix this issue? |
joelanford
added
good first issue
|
Hi @joelanford, You are right. The issue raised in both are |
|
Done PR: #3779 Hi @lbroudoux, could you help to review it? |
|
Re-opening because PR #3779 did not resolve this issue for Helm operators. It applied the change to the Ansible operator scaffolding, but this permission change is not required for Ansible operators because the Ansible operator does not use a finalizer on the main CR. I will submit a PR shortly that makes this change for Helm and reverts the change for Ansible. |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
What did you do?
I followed https://master.sdk.operatorframework.io/docs/building-operators/helm/quickstart/
What did you expect to see?
I expected to see an nginx pod come up. But it never came up.
What did you see instead? Under which circumstances?
The operator logs exhibit an error
failed to install release: serviceaccounts \"nginx-sample\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers onFull log file:
Environment
I'm using OpenShift Version 4.3.5, which is based on Kubernetes v1.16.2.
I'm the cluster-admin of this cluster.
Kubernetes cluster kind: 3 masters, 3 workers, in VM.
Are you writing your operator in ansible, helm, or go?
Helm
Possible Solution
The generated
config/rbac/role.yamlis missing privileges onnginxes/finalizers.The
operator-sdk create apicommand that generates this file needs to be updated to include this privilege since it is required on OpenShift.Since the generated objects include a
ownerReferencefield that references the created CR, on OpenShift the operator needs to be able to set finalizers on the created CR.The text was updated successfully, but these errors were encountered: