Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[loki.rules.kubernetes] pipelines in the expression #767

Open
ToonTijtgat2 opened this issue May 6, 2024 · 1 comment
Open

[loki.rules.kubernetes] pipelines in the expression #767

ToonTijtgat2 opened this issue May 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@ToonTijtgat2
Copy link

What's wrong?

It's not possible to use the loki.rules.kubernetes component to add rules to the loki ruler using the promehteusrule object.
expressions to filter logs often use | symbols, but it's giving issues for this component.

Steps to reproduce

I created a prometheusrule object to test alerting on loki.

The following spec on the prometheusrule works:

spec:
  groups:
    - name: rabbitmqtest.alerts
      rules:
        - alert: alottofrabbitmqerrorsapplication
          annotations:
            description: High error rate on rabbitmq errors
            summary: High error rate on rabbitmq errors
          expr: sum(rate({app=~"application.*"} [5m]))
          for: 5m

But since I'd like to filter out not all logs, but only logs with the level error the expression is the following:
expr: sum(rate({app=~"application.*"} | json | Level="Error" [5m]))
This give the error saying that | is not allowed:
Failed to save resource: {"error":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid","reason":"Invalid","details":{"name":"prometheusrules","causes":[{"message":"8:11: group "rabbitmqtest.alerts", rule 1, "alottofrabbitmqerrorsapplication": could not parse expression: 1:33: parse error: unexpected character: '|'"}]},"code":422},"messages":["admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid"],"isUsedForNotification":false}

Changing the expr to:

expr: |-
            'sum(rate({app=~"application.*"} | json | Level="Error" [5m]))'

makes it possible to complete the creation of the prometheusrule object but the alloy give the following error:
server returned HTTP status 400 Bad Request: could not parse expression for alert 'alottofrabbitmqerrorsapplication' in group 'rabbitmqtest.alerts': parse error at line 1, col 1: invalid char literal\n\n

Is there a way to work with expressions with pipelines?
Maybe the crd of prometheus rule should be addapted? or can we implement a work around?

Thanks for checkng

System information

kubernetes

Software version

Grafana Alloy v1.0.0

Configuration

No response

Logs

No response

@ToonTijtgat2 ToonTijtgat2 added the bug Something isn't working label May 6, 2024
@agologan
Copy link

Judging by the message there's a prometheus-operator installed which validates the rules.
i.e. https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/validatingWebhookConfiguration.yaml

LokiQL is PromQL-inspired but not fully compatible so the operator will reject that expression.
To disable the validation in kube-prometheus-stack:

prometheusOperator:
  admissionWebhooks:
    enabled: false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants