Spring Boot minor upgrade brings major breaking changes to Metrics

[ghost]

While updating our components to the recent 2.5.12 Spring boot version we came across an issue.
A "simple" minor update from version 2.5.3 to the new and secure 2.5.12 contains a hidden breaking change in the prometheus client version 0.10.0 that forces all counters to now have the suffix _total.
Our first approach was to try to force the use of the version 0.9.0 that we used before with no issues, but now, spring boot actuator uses methods that are only available after 0.10.0 and so using an older version of the client is no longer possible.

There were already complaints on the prometheus client side Issue 640 from people having the same issue.
The thing is, right now, some apps are not affected by the Spring4Shell vulnerability but in the near future there might a new vulnerability or even an extension to Spring4Shell that shows its efective against other types of spring boot apps and a lot of people will face this issue and will be unable to update to fix a serious security issue.

[mhalbritter]

I don't quite follow: Spring Boot 2.5.3 uses Prometheus 0.10.0, like Spring Boot 2.5.12. It's the same version. The update has been done in #27349 and #25250.

The only thing which changes from Boot 2.5.3 to 2.5.12 related to prometheus is

io.micrometer:micrometer-registry-prometheus:jar:1.7.2 vs io.micrometer:micrometer-registry-prometheus:jar:1.7.10

Spring Boot 2.4.x uses prometheus 0.9.0, so the breaking change was done from Boot 2.4.x to Boot 2.5.x.

You can use the CVE mitigation which updates only spring-framework to the fixed version and stay on 2.5.3. Details are here. A better approach would be to fix the metric names in the place where you are using them to the "new" prometheus format and then use the lastest Spring Boot 2.5.x (or even 2.6.x)

[ghost]

The issue is that between 2.5.3 and 2.5.12, not exactly sure where that change is made, it starts using org.springframework.boot.actuate.metrics.export.prometheus.TextOutputFormat.CONTENT_TYPE_OPENMETRICS_100
instead of org.springframework.boot.actuate.metrics.export.prometheus.TextOutputFormat.CONTENT_TYPE_004.

There is maybe a way to override that default?
I will def look into the mitigation, thank you

[mhalbritter]

Ah, so the change isn't the version bump, it's the change of the TextOutputFormat somewhere. I will take a look.

[ghost]

The format comes from org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape
on the version 2.5.3 it comes as CONTENT_TYPE_004 but for 2.4.12 it comes as CONTENT_TYPE_OPENMETRICS_100 so it might be something further up.

The prometheus server doing the request is the same in both cases

[mhalbritter]

I can absolutely observe no difference in counter names between 2.4.x, 2.5.3 and 2.5.12. Do you have a sample where this can be reproduced?

2.4.13

# HELP foo_total  
# TYPE foo_total counter
foo_total 1.0

2.5.3

# HELP foo_total  
# TYPE foo_total counter
foo_total 1.0


2.5.12

# HELP foo_total  
# TYPE foo_total counter
foo_total 1.0

[wilkinsona]

CONTENT_TYPE_OPENMETRICS_100 should only be served if it's acceptable to the client (the Prometheus server in this case). In addition to a sample, it would also be useful to know what Accept header your Prometheus server is sending and why, assuming that it indicates that CONTENT_TYPE_OPENMETRICS_100 is acceptable, it's a problem for it to receive content in that format.

[mhalbritter]

In #28130, there has been a change.

When sending Accept: application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1

Spring boot 2.5.3 answers with

HTTP/1.1 200 
Content-Type: text/plain;version=0.0.4;charset=utf-8
Content-Length: 7570
Date: Fri, 08 Apr 2022 12:49:42 GMT

# HELP bar_total  
# TYPE bar_total counter
bar_total 2.0

while 2.5.12 answers with

HTTP/1.1 200 
Content-Type: application/openmetrics-text;version=1.0.0;charset=utf-8
Content-Length: 7469
Date: Fri, 08 Apr 2022 12:50:21 GMT

# TYPE bar counter
# HELP bar  
bar_total 2.0

[wilkinsona]

Thanks, Moritz. That behavior aligns with Prometheus's own TextFormat class so I don't think we should change it. @nandoFromSky I think we really need to know what accept header your server is sending and why the resulting response is a problem to make progress on this.

[ghost]

@mhalbritter its probably it, i will investigate what our prometheus server is doing and if its that it should be an easy work around while we attempt to migrate metrics to use _total in all our systems

[ghost]

Prometheus afaik does not allow us to define the headers.
But, one can always override PrometheusScrapeEndpoint and TextOutputFormat to force the use of the TextFormat.write004.

This way we can still use the 0.9.0 prometheus client to avoid the _total and keep everything working normally.

Still need to run some other tests to see if there is any impact but seems like it should work

[wilkinsona]

Thanks, @nandoFromSky. I’ll close this one for now. Please let us know if it doesn’t work as hoped and we can take another look.

[deepak-auto]

Hi @nandoFromSky and @wilkinsona, would you happen to know of a recommended approach to work around the _total suffix?

[wilkinsona]

IIRC, you should be able to send a different accept header to get the metrics in a different format.