Postgres Performance Optimization: Cache baseline metrics and apply updates incrementaly

[amw-zero]

What does this PR do?

#17187 was reverted because we observed that it could lead to inflated metrics. The root cause of that is that computed_derivative_rows currently assumes that all rows in pg_stat_statements are passed as input. It does not compute correct results when a subset of pg_stat_statements is passed in, because that can lead to taking the diff of two different pg_stat_statement rows, which leads to incoherent results.

The approach taken here is to introduce a baseline_metrics cache, which holds one entry for each pg_stat_statements row and is populated once from the full pg_stat_statements dataset. On subsequent check runs, only queries that have been called since the previous run will be returned, and the baseline_metrics will be updated for just those queryids.

This allows the full set of normalized rows to be constructed with one new function: _apply_deltas. This gets run before compute_derivative_rows and uses the baseline_metrics to reconstruct the full set of metrics rows. This relies on the fact that a query that hasn't been called between check runs has no change in metrics, so we can reuse the cached values.

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Changelog entries must be created for modifications to shipped code
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[lu-zhengda]

I'm a bit concerned about increased memory impact of _baseline_metrics because the size of this dict is not bounded by pg_stat_statements.max. In a high cardinality database with lots of evictions in pg_stat_statements, the size of _baseline_metrics could grow quickly.

[amw-zero]

Re the unbounded growth due to query eviction, I added an expiry time (currently set to 10 minutes) past which the baseline will be re-fetched: decf7e8.

[amw-zero]

@lu-zhengda I tweaked the implementation slightly. I realized that aggregating the metric data by query signature in _apply_deltas duplicates the logic in _merge_duplicate_rows. So instead I'm using the baseline_metrics to re-construct the full set of rows (with one row per query signature), and allowing _merge_duplicate_rows to continue doing the aggregation.

[github-actions]

Test Results

296 tests   256 ✅  2m 0s ⏱️
  1 suites   39 💤
  1 files      1 ❌

For more details on these failures, see this check.

Results for commit f7b78bd.

[amw-zero]

Note: This test is failing on PG9. We may have to skip this optimization on that version. It seems like not all expected rows are being returned in the QUERY_CALLS_QUERY which prevents correct functioning.

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[lu-zhengda]

LGTM. minor change on the config default

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.