Restricting operator permissions to Thanos components/namespaces only

[sfynx]

The Helm chart for the operator creates a ClusterRole and ClusterRoleBinding that give very broad cluster-wide access, including being able to access all Secrets, manipulate all Deployments, and so on. This is upsetting our security folks who want to decrease attack surfaces whenever possible using the principle of least-privilege, so I wonder if all this access is really needed or that we at least could get away with a regular Role and RoleBinding inside the Helm release namespace (when the operator only manipulates Thanos components in this namespace), or be able to specify namespace(s) that we are allowed to do things in.

I do not have an in-depth understanding of the exact K8s permissions the Thanos operator needs for all its actions, but I think it should be possible to limit it to only manage workloads in Thanos-related namespaces?

[zohairraza]

+1

[gbvikas]

+1

When we tried to change the scope to role and role binding, it fails with below error:

{"level":"error","ts":"2022-09-27T22:44:23.413Z","msg":"pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:thanos-operator:thanosop-thanos-operator" cannot list resource "deployments" in API group "apps" at the cluster scope\n","stacktrace":"k8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220\nk8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73"}
{"level":"error","ts":"2022-09-27T22:44:23.414Z","msg":"pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:thanos-operator:thanosop-thanos-operator" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope\n","stacktrace":"k8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220\nk8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73"}
{"level":"error","ts":"2022-09-27T22:44:23.414Z","msg":"pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:thanos-operator:thanosop-thanos-operator" cannot list resource "statefulsets" in API group "apps" at the cluster scope\n","stacktrace":"k8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220\nk8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73"}
{"level":"error","ts":"2022-09-27T22:44:23.414Z","msg":"pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:thanos-operator:thanosop-thanos-operator" cannot list resource "services" in API group "" at the cluster scope\n","stacktrace":"k8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220\nk8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73"}
{"level":"error","ts":"2022-09-27T22:44:23.415Z","msg":"pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1alpha1.ThanosEndpoint: failed to list *v1alpha1.ThanosEndpoint: thanosendpoints.monitoring.banzaicloud.io is forbidden: User "system:serviceaccount:thanos-operator:thanosop-thanos-operator" cannot list resource "thanosendpoints" in API group "monitoring.banzaicloud.io" at the cluster scope\n","stacktrace":"k8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:222\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:156\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:220\nk8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:56\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.22.2/pkg/util/wait/wait.go:73"}