Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: do fewer heatmap uploads for tiny tenants #7731

Merged
merged 3 commits into from
May 14, 2024

Conversation

jcsp
Copy link
Contributor

@jcsp jcsp commented May 13, 2024

Problem

Currently we do a large number of heatmap uploads for tiny tenants. "tiny" in this context is defined as being less than a single layer in size. These uploads are triggered by atime changes rather than changes in the set of layers.

Uploading heatmaps for atime changes on small tenants isn't useful, because even without bumping these atimes, disk usage eviction still avoids evicting the largest resident layer of a tenant, which in practice keeps tiny/empty tenants mostly resident irrespective of atimes.

Summary of changes

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@jcsp jcsp added c/storage/pageserver Component: storage: pageserver a/tech_debt Area: related to tech debt labels May 13, 2024
Copy link

3060 tests run: 2927 passed, 0 failed, 133 skipped (full report)


Code coverage* (full report)

  • functions: 31.4% (6329 of 20167 functions)
  • lines: 47.3% (47748 of 100999 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
4939e47 at 2024-05-13T13:03:03.545Z :recycle:

@jcsp jcsp marked this pull request as ready for review May 13, 2024 13:08
@jcsp jcsp requested a review from a team as a code owner May 13, 2024 13:08
@jcsp jcsp requested a review from petuhovskiy May 13, 2024 13:08
@jcsp jcsp enabled auto-merge (squash) May 14, 2024 08:31
@jcsp jcsp merged commit cd0e344 into main May 14, 2024
58 checks passed
@jcsp jcsp deleted the jcsp/heatmap-uploads-tiny-tenants branch May 14, 2024 08:31
a-masterov pushed a commit that referenced this pull request May 20, 2024
## Problem

Currently we do a large number of heatmap uploads for tiny tenants.
"tiny" in this context is defined as being less than a single layer in
size. These uploads are triggered by atime changes rather than changes
in the set of layers.

Uploading heatmaps for atime changes on small tenants isn't useful,
because even without bumping these atimes, disk usage eviction still
avoids evicting the largest resident layer of a tenant, which in
practice keeps tiny/empty tenants mostly resident irrespective of
atimes.

## Summary of changes

- For tenants smaller than one checkpoint interval, only upload heatmap
if the set of layers has changed, not if only the atimes have changed.
- Include the heatmap period in the uploaded heatmap, as a precursor to
implementing #6200
(auto-adjusting download intervals to match upload intervals)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/tech_debt Area: related to tech debt c/storage/pageserver Component: storage: pageserver
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants