prom-label-proxy refusing to proxy for Grafana even if the required label is specified

[amine250]

Hello,

I have a Grafana instance with a datasource that points to prom-label-proxy which is in front of a Thanos/Prometheus cluster.

The datasource is configured as below:

The prom-label-proxy is running in EKS as a sidecar, with the following arguments:

- name: prom-label-proxy
        image: quay.io/prometheuscommunity/prom-label-proxy:v0.3.0
        args:
        - --label=perimeter
        - --upstream=http://127.0.0.1:10902
        - --insecure-listen-address=0.0.0.0:8080
        - --unsafe-passthrough-paths=/-/ready

When I issue the following query in Grafana, I get the following response:

When I configure my datasource to always have the query param perimeter set, it works, but it's not good since I would have to create a separate datasource for each value of perimeter:

Any ideas?

[amine250]

After checking out prom-label-proxy source code https://github.com/prometheus-community/prom-label-proxy/blob/master/injectproxy/routes.go#L189 , and saw how it handles query param verification, I did understand the following:

For the same request in my post above, the query params that are parsed are the following:

param	value
query	up{instance="ip-10-170-51-145.eu-west-1.compute.internal", perimeter="eks"}
start	1627464060
end	1627485660
step	60

As a consequence, prom-label-proxy finds no query param with the name perimeter.

I just realized that this is not what prom-label-proxy has been designed for, but could this be a feature to be considered?
Or is there another project that enforces a certain label to be used on Prometheus queries ?

Thanks for your responses.

EDIT: maybe you should edit the description of the project: A proxy that enforces a given label in a given PromQL query.

[zucher]

Hi @amine250 , you have to set the custom query parameter like this:
perimeter=eks
under datasource definition

[simonpasquier]

👋 @amine250 I'm not sure what you're trying to accomplish exactly. The purpose of prom-label-proxy is to enforce dynamically label matchers in PromQL queries, the label key is configured statically and the label value needs to be specified by the caller somehow.

EDIT: maybe you should edit the description of the project: A proxy that enforces a given label in a given PromQL query.

What is confusing in this description?

[simonpasquier]

@amine250 can you check #82? does it match your request about being able to deploy prom-label-proxy with a fixed label key/value to be enforced?

[amine250]

Hi @simonpasquier, I think either of us is confusing PromQL query label and HTTP query parameter. While the latter can be specified in Grafana's datasource, the PromQL query label would be specified inside the PromQL query (e.g. up{enforced_label="something"}.

For my particular use case, I would like to reject HTTP requests that contain PromQL queries, which don't have the enforced_label inside the PromQL query. Technically, in a typical Prometheus HTTP request, the query query parameter will contain the PromQL query.

The purpose of prom-label-proxy is to enforce dynamically label matchers in PromQL queries, the label key is configured statically and the label value needs to be specified by the caller somehow.

From what I saw fiddling with prom-label-proxy, it didn't behave as you described.

While I've seen some code in https://github.com/prometheus-community/prom-label-proxy/blob/master/injectproxy/routes.go#L198-L225 that maybe implements what I want to achieve, I just couldn't make it work. Only enforcing the HTTP query parameter in Grafana's datasource worked for me, using the v0.3.0.

[simonpasquier]

For my particular use case, I would like to reject HTTP requests that contain PromQL queries, which don't have the enforced_label inside the PromQL query.

Ok I get it now: you want all incoming PromQL queries to have a label selector but you don't want to enforce the value. I'm not sure to understand the use case though since if users are free to pass any label value, they have access to all data anyway. Can you give more details about it?

[amine250]

Ok I get it now: you want all incoming PromQL queries to have a label selector but you don't want to enforce the value. I'm not sure to understand the use case though since if users are free to pass any label value, they have access to all data anyway. Can you give more details about it?

Yes, this is exactly what I need. Access to all data is not an issue for me.

In our architecture, we are using a main Thanos querier in front of multiple {Thanos querier+sidecar+prometheus,...} for each application.

Enforcing a PromQL label selector, would help us avoid forwarding a PromQL query to every Thanos instance in the cluster, and would only forward it to the Thanos querier that has advertised the particular label.

[simonpasquier]

Thanks for the reply, really helpful! I think it's quite a departure from the original prom-label-proxy goals but eager to hear the opinion from other @prometheus-community/prom-label-proxy maintainers.