Support for RemoteWrite CRD

[superbrothers]

Component(s)

Prometheus

What is missing? Please describe.

I would like to allow users to add arbitrary RemoteWrite/Read configs to Prometheus.

Scenario: cluster-admin provides a managed Prometheus server to users using Prometheus CR. This Prometheus CR can only be modified by cluster-admin. The user adds config to the Prometheus server using a RemoteWrite CR to forward metrics to an external managed Prometheus service.

apiVersion: monitoring.coreos.com/v1alpha1
kind: RemoteWrite
metadata:
  name: example
  namespace: default
spec:
  url: xxxx/api/v1/remote_write
  authorization:
    name: cred
    key: token
  writeRelabelConfigs:
  - ...

apiVersion: monitoring.coreos.com/v1alpha1
kind: RemoteRead
metadata:
  name: example
  namespace: default
spec:
  url: xxxx/api/v1/read
  sigv4:
    region: ap-northeast-1

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  RemoteWriteNamespaceSelector: {}
  RemoteWriteSelector: {}
  RemoteReadNamespaceSelector: {}
  RemoteReadSelector: {}

Since RemoteWrite/Read CR are namespaced resources, it may be better to limit the target metrics to the namespace in which the object was created.

Describe alternatives you've considered.

None.

Environment Information.

Environment

Kubernetes Version:
Prometheus-Operator Version:

[simonpasquier]

I've heard about this use case for remote-write before so I'd be inclined to say yes. I'm more curious about the remote-read use caes, can you provide more details about it?

Since RemoteWrite/Read CR are namespaced resources, it may be better to limit the target metrics to the namespace in which the object was created.

I suppose that you mean the operator should automatically add a relabeling config dropping all metrics without a namespace="<remote write's namesapce>" label?

[mviswanathsai]

If the RemoteWrite is namespaced and is only supposed to export metrics from its own namespace, there would also be a relabel config in the final generated config I see. But is there a reason why it should be namespaced?
Could (an alternative) dedicated relabeling rules (edit: without keeping it namespaced) that you could add in the CR help with the separation of concerns among teams a bit more flexibly?
Just bouncing off some ideas.

Edit: Similar to what we already have with the relabeling configs for remote write in the Prometheus CR.

[superbrothers]

I'm more curious about the remote-read use caes, can you provide more details about it?

Actually, I only want RemoteWrite CR; I suggested RemoteRead CR because it seems unnatural that RemoteWrite exists but RemoteRead CR does not.

I suppose that you mean the operator should automatically add a relabeling config dropping all metrics without a namespace="<remote write's namesapce>" label?

Yes, it is. However, it may not be necessary. The behavior may be confusing to users. Also, if it is necessary to do so, cluster-admin can accomplish this by mutating the writeRelabelConfigs field of the RemoteWrite CR with admission webhook.

But is there a reason why it should be namespaced?

This is because users generally cannot create cluster-wide resources in a multi-tenant cluster.

[simonpasquier]

FWIW we already have .spec.enforcedNamespaceLabel which applies label enforcement for monitor and rule objects. I'd apply the same enforcement to RemoteWrite CRD if enabled. And through .spec.excludedFromEnforcement you could have fine-grained control on exactly which resources are enforced.

Regarding RemoteRead, I would leave it out unless we have a concrete use case. It can still be implemented later if needed.

To move it forward, I'd suggest to file a design proposal following https://prometheus-operator.dev/docs/prologue/contributing/#proposal-process

[simonpasquier]

fyi I've removed RemoteRead from the issue title to clarify the scope.

[nicolastakashi]

Amazing idea 👏🏽

[ArthurSens]

I'm a bit concerned about "metrics highjacking", where someone could steal metrics from a Prometheus instance by deploying a RemoteWrite CRD.

I don't mean to mention this as a blocker, but maybe something we need to consider during the proposal

[djluck]

We'd also benefit from having a RemoteWrite CRD.

Let me give you some context on why this is valuable to us. We have a cloud hosted metric store that we ship metrics to via remote write. However, we don't wish to ship all metrics to this cloud store, mostly due to cost-based issues- in these cases, we'll ship an aggregated form of these metrics, generated via recording rules. So we use relabeling rules on the remote_write config to drop these source metrics.
To complicate maters, we also have Prometheus deployed in non-kubenernetes environments with common metrics being collected by both the non-k8s + k8s deployments! This means we need to keep the remote_write relabeling config in sync. Our solution has been to define common rules (e.g alerts, recording rules, relabeling rules) in a central repository which then both deployments can reference.

Unfortunately without this RemoteWrite CRD, keeping the two deployments in sync in response to updates our git repository is difficult.