Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

explicitly add +inf bucket in withExemplarsMetric #1094

Merged
merged 4 commits into from Aug 2, 2022
Merged

explicitly add +inf bucket in withExemplarsMetric #1094

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2d3b601
explicitly adding +inf bucket to withExemplarsMetric
arun-shopify Jul 21, 2022
4fe98f8
Update prometheus/metric_test.go
arun-shopify Jul 27, 2022
48be15f
Update prometheus/metric.go
arun-shopify Jul 27, 2022
1b7ee16
updated comment and removed unnecessary test
arun-shopify Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 11 additions & 3 deletions prometheus/metric.go
View file
Open in desktop
Expand Up @@ -15,6 +15,7 @@ package prometheus

import (
"errors"
"math"
"sort"
"strings"
"time"
Expand Down Expand Up @@ -183,9 +184,16 @@ func (m *withExemplarsMetric) Write(pb *dto.Metric) error {
})
if i < len(pb.Histogram.Bucket) {
pb.Histogram.Bucket[i].Exemplar = e
} else {
// This is not possible as last bucket is Inf.
panic("no bucket was found for given exemplar value")
} else { // +inf bucket should be explicitly added if there is an exemplar for it.
arun-shopify marked this conversation as resolved.
Show resolved Hide resolved
b := &dto.Bucket{
CumulativeCount: proto.Uint64(pb.Histogram.Bucket[len(pb.Histogram.GetBucket())-1].GetCumulativeCount()),
UpperBound: proto.Float64(math.Inf(1)),
Exemplar: e,
}
pb.Histogram.Bucket = append(pb.Histogram.Bucket, b)
break
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing this, but I think one thing is still not addressed: #1094 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @arun-shopify - otherwise this PR is rdy to merge (:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix that for you if you don't mind - I am preparing next release

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merging and will fix in separate PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, also response in #1094 (comment)

// end looping after creating +inf bucket and adding one exemplar.
// there could be other exemplars that are in the "inf" range but those will be ignored.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want those comments, let's make them full sentence, but also I am not sure this is safe. Nowhere in the interface/signature we mention that exemplars will be sorted by anything. I think there is no harm to continue the loop, unless we want to optimize this some day. WDYT?

Suggested change
break
// end looping after creating +inf bucket and adding one exemplar.
// there could be other exemplars that are in the "inf" range but those will be ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right that there is no mention (or expectation) of sorting exemplars. If we leave the loop running we pick the last exemplar and if we leave it as is and terminate it here, we pick the first exemplar (in that range) - sort order of the exemplar value being arbitrary. I would consider terminating the loop as it would just avoid running through the remaining exemplars that we are not going to use anyway. But please let me know if you think otherwise or see any safety concerns, I can make the change.

Also, I made the comment into a sentence.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not addressed completely. What do you about lack of sorted order invariant?

Nowhere in the interface/signature we mention that exemplars will be sorted by anything. I think there is no harm to continue the loop, unless we want to optimize this some day. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all the buckets we are comparing the exemplar values to the bucket bound and matching the right exemplar to right bucket (this is the exiting logic):

specifically the check here to get the index of the right bucket for that exemplar:

client_golang/prometheus/metric.go

Line 182 in a528aff

return pb.Histogram.Bucket[i].GetUpperBound() >= e.GetValue()

And assigning the exemplar to the right bucket here:

client_golang/prometheus/metric.go

Line 185 in a528aff

pb.Histogram.Bucket[i].Exemplar = e

At a high level the only change in this PR is that instead of a panic in the else condition, we add the +Inf bucket and add one exemplar that is outside of all previous bucket range and break the loop.

If there are multiple exemplars for the +Inf bucket, we could pick the exemplar that is more representative of the group such as a median - a future improvement, would require further discussion. Currently we are just picking the first in the array in the +inf bucket range.

I hope that addresses your concern, if I understood your question correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bwplotka waiting for your response :) Would be nice if we could include this in the upcoming release.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed this discussion, sorry.

So I like the idea about median or tuning the exemplar we take from those belonging to +Inf. My only problem is that those inputs can be not sorted, that's it. Hope this PR #1100 makes sense to you.

}
}
default:
Expand Down
20 changes: 18 additions & 2 deletions prometheus/metric_test.go
View file
Open in desktop
Expand Up @@ -14,6 +14,7 @@
package prometheus

import (
"math"
"testing"

//nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility.
Expand Down Expand Up @@ -56,16 +57,19 @@ func TestWithExemplarsMetric(t *testing.T) {
{Value: proto.Float64(89.0)},
{Value: proto.Float64(100.0)},
{Value: proto.Float64(157.0)},
{Value: proto.Float64(500.0)},
{Value: proto.Float64(2000.0)},
}}
metric := dto.Metric{}
if err := m.Write(&metric); err != nil {
t.Fatal(err)
}
if want, got := 4, len(metric.GetHistogram().Bucket); want != got {
if want, got := 5, len(metric.GetHistogram().Bucket); want != got {
t.Errorf("want %v, got %v", want, got)
}

expectedExemplarVals := []float64{24.0, 42.0, 100.0, 157.0}
// when there are more exemplars than there are buckets, a +inf bucket will be created and the last exemplar value will be added to the +inf bucket.
arun-shopify marked this conversation as resolved.
Show resolved Hide resolved
expectedExemplarVals := []float64{24.0, 42.0, 100.0, 157.0, 500.0}
for i, b := range metric.GetHistogram().Bucket {
if b.Exemplar == nil {
t.Errorf("Expected exemplar for bucket %v, got nil", i)
Expand All @@ -74,5 +78,17 @@ func TestWithExemplarsMetric(t *testing.T) {
t.Errorf("%v: want %v, got %v", i, want, got)
}
}

infBucket := metric.GetHistogram().Bucket[len(metric.GetHistogram().Bucket)-1].GetUpperBound()
bwplotka marked this conversation as resolved.
Show resolved Hide resolved

if infBucket != math.Inf(1) {
t.Errorf("want %v, got %v", math.Inf(1), infBucket)
}

infBucketValue := metric.GetHistogram().Bucket[len(metric.GetHistogram().Bucket)-1].GetExemplar().GetValue()
arun-shopify marked this conversation as resolved.
Show resolved Hide resolved

if infBucketValue != 500.0 {
t.Errorf("want %v, got %v", 500.0, infBucketValue)
}
})
}