New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable deletion of metrics from a vector based on partially specified labels #834
Comments
Thank you for writing up this issue, and suggesting some workaround. We have the same problem in cortexproject/cortex#3784, where we put user-controlled Until that is available, some of your suggested workarounds may actually work fine for us (thinking about "hacky and weird" one). |
We need this feature as well. My preferred approach would be a DeletePartialMatch function. |
I need this feature too! |
Another vote for the Use case for those interestedWe have an exporter which exposes values from some Kubernetes custom resources. An extremely simple conceptual example might be:kind: SomeResource
metadata:
name: my-resource
data:
version: 1.2.3
some_field: 99 The exposed gauge metric for that resource might look like:
Now suppose Our exporter subscribes to changes and deletions of the resource, but (due to Kubernetes client implementation details) we get only the name of the resource when it changes, not the complete original resource. So after our exporter has reconciled the update, we have the following metrics:
But the underlying resource for version 1.2.3 no longer exists. This means
The ability to delete a metric by partial labels would simplify both of those situations. By including the name of the resource, we could simply One way around this would be to have more direct instrumentation and report the state of all resources at scrape time, but my understanding is that is also an anti-pattern. I can imagine this limitation may also arise in other situations where the source of truth for the metric disappears |
I've opened a PR adding this functionality here #1013, I have only so much time to iterate on it but would be open to suggestions |
Since #1013 is merged, we can close this, right? |
Yep this is done. Just waiting for release now |
In most cases, users of a metric vector are well aware of the label combinations for which metric children have been created in the vector. And they should be, because unpredictable label values often go along with unbounded cardinality. However, there are legitimate use cases where tracking the used label names is hard. (This could merely be a code maintenance problem, e.g. a HTTP counter partitioned by endpoints in a situation where other parts of the codebase can add endpoints.)
While those cases exist, it's also rarely necessary to track the used label names. However, once you are in a situation where you want to delete metric children, you have to know them by their full label set. You cannot just say "delete all children with
tenant="42"
plus an unknown set of other labels. (Cortex instrumentation recently ran into this problem when a tenant gets deleted entirely, and all the metrics related to it should disappear.) This is essentially seeing the labeled metric children arranged in a tree in which you want to prune one whole branch.Work-arounds include:
Collect
on the vector and thenWrite
on the collectedMetric
instances. Then look at the resulting protobuf message to find the metric children to delete.tenant
label as aConstLabel
. Those collectors can then be separately registered and de-registered upon creation and removal of a tenant.Note that it is not possible to use vector currying and then call
Reset
an a curried vector becauseReset
always acts on the underlying vector as a whole.I assume the use case, while legitimate, is still fairly niche, so I would like to avoid adding yet another method (like
DeletePartiallyMatching(labels)
) to the already quite fat method set of vectors. In hindsight, applying the currying toReset
would have been a good way to enable this use-case, but now it's too late to change the behavior. Similarly,Delete(labels)
could have been implemented to work on a partial match. Most users probably wouldn't mind to change the behavior now, but given thatgithub.com/prometheus/client_golang/prometheus
is in the top30 of most used Go packages, there are probably a few users around who rely on the current behavior in some weird usage pattern. I was wondering if it would be a good idea to introduce some "wildcard" pattern, similar to the...
we have already used in Alertmanager grouping so that you could write something likemyVec.Delete(prometheus.Labels{"tenant": "42", "...": ""})
. (The label value of the...
pseudo label wouldn't matter.) It's a bit arcane, but it is for a niche use case after all. It should also be noted that the current deletion is optimized to be fast, while the deletion by partial match will require a full scan through all metric children (which could become a problem in those rare cases where a vector has a lot of children).Before doing anything, I'd like to get some feedback from the crowd. What's your preference?
...
in the labels map used forDelete
.DeletePartialMatch
, which will be added to all vector types.@pracucci @pstibrany @bwplotka
The text was updated successfully, but these errors were encountered: