Prometheus stack not respecting retention

[simonebenati]

What happened?
Prometheus isn't respecting retentionSize nor retention. And my PV keeps growing and growing

Did you expect to see some different?

Yes I expect the pv to stop growing indefinitely

How to reproduce it (as minimally and precisely as possible):

Install prometheus stack version kube-prometheus-stack-56.6.0 v0.71.2 via helm, set a retention value or retentionsize very small and see that your pvc will increase regardless

Environment

on premises, longhorn csi

• Prometheus Operator version:

  quay.io/prometheus-operator/prometheus-operator:v0.71.2

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:38Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.11+rke2r1", GitCommit:"3cd242c51317aed8858119529ccab22079f523b1", GitTreeState:"clean", BuildDate:"2023-11-15T21:08:55Z", GoVersion:"go1.20.11 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  via rancher2

• Manifests:

prometheus.prometheusSpec.retention and prometheus.prometheusSpec.retentionSize

• Prometheus Operator Logs:

I don't have any

• Prometheus Logs:

I think i don't have any, if you know better please point me in the correct direction and i'll be happy to provide for logs

Anything else we need to know?: