docs: Add URL config behavior spec

[powersj]

Summary

Introduce a new spec to define URL-based config behavior.

Checklist

• No AI generated code was used in this PR

[AndreKR]

Looks good.

A little bit of bike-shedding...

The proposal then is to store the last-modified value when we first obtain the
file and compare the value at each interval. No need to parse the value, just
store the raw string. If there is a difference, trigger a reload.

Given that it's to be used as a opaque token, the ETag header seems more appropriate than Last-Modified? (Neither is currently being provided by the InfluxDB 2 API.)

If anything other than 2xx response code is returned from the HEAD request, Telegraf would print a warning message and retry at the next interval.

Does config-url-watch-interval affect the interval of the initial 3 attempts as well? If yes then config-url-watch-interval can't be disabled by default but needs a default value at least for the initial attempts.

[paulojmdias]

The proposal then is to store the last-modified value when we first obtain the
file and compare the value at each interval. No need to parse the value, just
store the raw string. If there is a difference, trigger a reload.
If anything other than 2xx response code is returned from the HEAD request,
Telegraf would print a warning message and retry at the next interval.
If the value of last-modified is empty, while very unlikely, then Telegraf would
ignore this configuration file.

What happens if the endpoint which is trying to watch is down/timeout?

Is it possible to by default keep the last successful configuration pulled or even add a flag/option to work in this way?

[powersj]

@AndreKR,

Given that it's to be used as a opaque token, the ETag header seems more appropriate than Last-Modified?

When I looked into this I originally was going to use the ETag, but fewer servers seem to provide this. It seemed that Last-Modified was more readily available.

Neither is currently being provided by the InfluxDB 2 API.

Hmm I was unaware of this, nothing in a HEAD request can be used to determine if a change was made. Let me follow-up with some internal folks about this scenario.

$ curl --head https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/telegrafs/0d021699ba828000
HTTP/2 401 
date: Wed, 08 May 2024 13:21:12 GMT
content-type: application/json; charset=utf-8
content-length: 55
trace-id: b7a75074a7235df0
trace-sampled: false
x-platform-error-code: unauthorized
strict-transport-security: max-age=31536000; includeSubDomains
x-influxdb-request-id: 0b69a959f18838135d84f703393bb60d
x-influxdb-build: Cloud

Does config-url-watch-interval affect the interval of the initial 3 attempts as well?

My intention was that config-url-watch-interval did not affect the initial 3 attempts. It would only enable watching the remote URLs when set.

@paulojmdias,

What happens if the endpoint which is trying to watch is down/timeout?

When loading the config the initial time at start, the agent would try config-url-retry-attempts times to collect the config. Telegraf blocks startup until we get a success or run out of attempts.

Once Telegraf is up and running, if the user has set --config-url-watch-interval any response other than a 2xx and telegraf would continue with the currently loaded config.

Is it possible to by default keep the last successful configuration pulled or even add a flag/option to work in this way?

Are you asking to keep the remotely collected config locally and load from that at a restart? That level of config management is out of scope for Telegraf.

[paulojmdias]

Are you asking to keep the remotely collected config locally and load from that at a restart? That level of config management is out of scope for Telegraf.

No @powersj, since any response other than a 2xx will keep Telegraf up and running with the previous successful config pulled, you have answered my request 🙏

[DStrand1]

@powersj

When I looked into this I originally was going to use the ETag, but fewer servers seem to provide this. It seemed that Last-Modified was more readily available.

Is this something that may be worth checking for both headers?

[powersj]

Is this something that may be worth checking for both headers?

Possibly. In general, I found that if someone is actually taking the time to set the etag then last updated was also set.

[srebhan]

@powersj thanks for the spec, just two comments from my side...

[srebhan]

Thanks @powersj!