prometheus.remote_write: mark component unhealthy if sending samples fails

[captncraig]

It accomplishes this by observing the log entries from the remote storage writer. If we see "non-recoverable error" messages, we assume there is some problem (usually with a bad token or some kind of networking or configuration issue). It definitely means no samples are getting through.

Detecting a recovery is a bit harder. There is no clear log message from the prometheus code (even at debug level) to indicate things have resumed. It is possible if we also hooked into the sample append hooks we could find a combination of metrics that would indicate recovery, but for now I am just assuming if we don't see an error log for 2 minutes (fairly arbitrary, may need tuning) that it is recovered. Even flapping health status is better than false positives all the time like we have now.