fix: Add /metrics non-resource URL to rbac

[thefirstofthe300]

Description: Add the /metrics non-resource URL to the OTEL collector RBAC

By adding this non-resource URL to the RBAC, it becomes possible for an OTEL collector to scrape authenticated endpoints such as control plane components.

[swiatekm-sumo]

Does the k8sprocessor actually scrape these endpoints?

[IshwarKanse]

These RBAC are not documented for the processor. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor

[IshwarKanse]

@thefirstofthe300 @pavolloffay @swiatekm-sumo With the operator built off this PR, I see the following errors in operator logs. The required cluster role is also not created. I do not see this issue from the main branch.

{"level":"error","ts":"2024-04-30T08:15:12.48776853Z","logger":"controllers.OpenTelemetryCollector","msg":"failed to configure desired","opentelemetrycollector":{"name":"simplest","namespace":"chainsaw-k8sattributes"},"object_name":"simplest-chainsaw-k8sattributes-cluster-role","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"clusterroles.rbac.authorization.k8s.io \"simplest-chainsaw-k8sattributes-cluster-role\" is forbidden: user \"system:serviceaccount:opentelemetry-operator:opentelemetry-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:opentelemetry-operator\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{NonResourceURLs:[\"/metrics\"], Verbs:[\"get\"]}","stacktrace":"github.com/open-telemetry/opentelemetry-operator/controllers.reconcileDesiredObjects\n\t/Users/test-user/opentelemetry-operator/controllers/common.go:113\ngithub.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\t/Users/test-user/opentelemetry-operator/controllers/opentelemetrycollector_controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-04-30T08:15:12.495887394Z","logger":"controllers.OpenTelemetryCollector","msg":"failed to configure desired","opentelemetrycollector":{"name":"simplest","namespace":"chainsaw-k8sattributes"},"object_name":"simplest-collector","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"clusterroles.rbac.authorization.k8s.io \"simplest-chainsaw-k8sattributes-cluster-role\" not found","stacktrace":"github.com/open-telemetry/opentelemetry-operator/controllers.reconcileDesiredObjects\n\t/Users/test-user/opentelemetry-operator/controllers/common.go:113\ngithub.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\t/Users/test-user/opentelemetry-operator/controllers/opentelemetrycollector_controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-04-30T08:15:12.496026695Z","msg":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"simplest","namespace":"chainsaw-k8sattributes"},"namespace":"chainsaw-k8sattributes","name":"simplest","reconcileID":"e2739463-7d2f-4fed-9fa8-92f05ed679f6","error":"failed to create objects for simplest: clusterroles.rbac.authorization.k8s.io \"simplest-chainsaw-k8sattributes-cluster-role\" is forbidden: user \"system:serviceaccount:opentelemetry-operator:opentelemetry-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:opentelemetry-operator\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{NonResourceURLs:[\"/metrics\"], Verbs:[\"get\"]}\nclusterroles.rbac.authorization.k8s.io \"simplest-chainsaw-k8sattributes-cluster-role\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/Users/test-user/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

[pavolloffay]

By adding this non-resource URL to the RBAC, it becomes possible for an OTEL collector to scrape authenticated endpoints such as control plane components.

@thefirstofthe300 this file creates RBAC for the k8sattribute processor. This does not seem to be related to metrics

[pavolloffay]

or maybe I am missing something here

[swiatekm-sumo]

I think, rather, this should be added for prometheus receiver, if we had a parser for it.

[thefirstofthe300]

I think, rather, this should be added for prometheus receiver, if we had a parser for it.

My initial thought was to have these permissions be triggered by the k8s attribute processor since 1) it was easy and 2) it's a sure signal that the collector is running in a k8s cluster. After more work trying to scrape authenticated metrics endpoints, I'm inclined to agree with this sentiment.

The Prometheus Operator makes use of secrets to fetch tokens, meaning the target allocator will need permission to fetch those secrets as well. Obviously, everyone is going to have different requirements for what secrets they are willing to grant the OTEL operator access to, meaning it's probably best if there is some sort of config to allow people to granularly configure an RBAC role.