add tokio metrics collector

[conradludgate]

Problem

It would be interesting to see runtime stats over time

Summary of changes

Adds a prometheus collector for tokio runtime metrics.

TODO:

• Add more of the counters
• Try and get tokio to use counters instead of gauges
• Verify that it doesn't impact performance
• Confirm that 15 seconds prometheus scrape is enough to get interesting stats - otherwise perform eager aggregations

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2340 tests run: 2225 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_crafted_wal_end[last_wal_record_crossing_segment]: debug

Postgres 15

• test_crafted_wal_end[last_wal_record_crossing_segment]: release

Code coverage (full report)

• functions: 53.5% (8633 of 16128 functions)
• lines: 81.5% (50207 of 61595 lines)

---

_{The comment gets automatically updated with the latest test results
ac0b314 at 2023-10-26T22:06:47.381Z :recycle:}

[problame]

There is

https://docs.rs/tokio-metrics

which, I know, doesn't implement the prometheus::core::Collector.

But, it abstracts the idea of looking at the RuntimeMetrics in fixed intervals. Which I think is a very good idea, because many of the tokio::RuntimeMetrics fields are aggregations already.

For example, I think to detect periods of increased executor stalls due to, say, slow blocking IO, you'd want to sample&reset max_busy_duration at a given frequency. (I could be wrong about this though, haven't looked deeply at RuntimeMetrics because they seemed out of reach due to the cfg tokio_unstable)

[conradludgate]

There is https://docs.rs/tokio-metrics

I did look into this, it seemed like managing a timer tasks would add more overhead and complexity than sampling the runtime on demand.

because many of the tokio::RuntimeMetrics fields are aggregations already.

Are they? I see only 2 metrics that fit this:

• https://docs.rs/tokio/latest/tokio/runtime/struct.RuntimeMetrics.html#method.worker_mean_poll_time
• https://docs.rs/tokio/latest/tokio/runtime/struct.RuntimeMetrics.html#method.poll_count_histogram_bucket_count

I'm not keen on enabling poll_count histograms as it's going to add more overhead:
https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.enable_metrics_poll_count_histogram

Task poll times are not instrumented by default as doing so requires calling Instant::now() twice per task poll, which could add measurable overhead

So really it's only 1 aggregate in this case. The rest of the quantities are counters and gauges.

For example, I think to detect periods of increased executor stalls due to, say, slow blocking IO, you'd want to sample&reset max_busy_duration at a given frequency

We can get worker_total_busy_duration as a counter, couldn't you use a sum(rate()) construction to measure this. Unless you think 15s granularity is far too low resolution.

[conradludgate]

Regarding gauges and sampling rates, it's been requested to use counter pairs tokio-rs/tokio#4073 (comment) (which I know @koivunej will be happy about)

[koivunej]

Regarding gauges and sampling rates, it's been requested to use counter pairs tokio-rs/tokio#4073 (comment) (which I know @koivunej will be happy about)

Excellent :) Yes, this brings joy for me :)

[problame]

I'm not keen on enabling poll_count histograms as it's going to add more overhead:
https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.enable_metrics_poll_count_histogram

Well this is the primary use case that I have for these metrics in pageserver.
We want to detect tasks that are stalling the executor.

We're ready to take a constant overhead in exchange for getting that observability.

[conradludgate]

Hmm, fair enough.

[conradludgate]

I was curious so I ran a quick HTTP benchmark and it only had a 1% degradation in performance