feat(http_client): Add token_file feature

[unautre]

Add a "token_file" option to http::Auth to read the bearer token from a file.
This is especially useful when using a Kubernetes provided token file (/run/secrets/kubernetes.io/serviceaccount/token).

[tobz]

Overall, the intended functionality makes sense to me... but I think what is less clear is if this actually needs to read the token file every single time.

It would certainly be useful from the perspective of always using the latest token value, but it also feels like it will lead to unintended consequences around errors being throw if the file goes away, or is briefly unavailable, or the actaul call to read the file is slowed down and then in turn slows down the sink/whatever is using this style of authentication.

Can you explain a little more about your use case for this?

[unautre]

Hello @tobz , thanks for the feedback

My use case is to use vector's prometheus_scrape to scrape k8s metrics. This requires authentification, given by a token file that might be refreshed at the kubelet's convenance.

From what I gather from the k8s docs, "The application is responsible for reloading the token when it rotates. Periodic reloading (e.g. once every 5 minutes) is sufficient for most use cases.", so caching the file for a duration is possible ; but that is out of my meager rust expertise for now.

[tobz]

Nice, yeah, that makes sense.

I'll have a chat with the team. I can see the appeal of just reading the file every single time we want to build an authenticated request, but I'm not sure if we're comfortable with that overhead or not.

[jszwedko]

Thanks for opening this @unautre !

I think having Vector read the file each time would be a bit of a departure from how Vector handles the rest of its configuration where it loads it at start-time. I can see that you need the token to be refreshed while Vector is running, though, which makes me think of a few options:

• Having --watch-config watch for changes to this file and reload. This is something we are generally aspiring towards in Vector, that `--watch-config watches files referenced by configuration in addition to the configuration files itself, but implementation so far has been spotty. You can read more about this here: Ensure that --watch-config watches all configuration files for changes #17283
• Another option would be to use Vector's secret loading mechanism instead to load a secret that is just used with auth.bearer. A caveat is that that doesn’t support expiring secrets (yet) but I think it does respect SIGHUP so you could have an external process issues a SIGHUP when the file changes

I think both of those would be more consistent with Vector's current UX than re-reading the file on each request. I know they are also much larger changes though. What do you think?