Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preload adaptative batch #6427

Open
wants to merge 29 commits into
base: develop
Choose a base branch
from

Conversation

MathieuLamiot
Copy link
Contributor

Description

When preparing preload batches, the batch size will be adapted based on the average duration of a request to avoid flooding servers with requests if they take too long.
To get the estimate of the request duration, we do a blocking request from time to time when triggering the preload, and measure the timing.

Fixes #6396

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality).

Is the solution different from the one proposed during the grooming?

Yes, on top of the original idea, we added an average mechanism that keeps track of the average request time through transients.

Checklists

Generic development checklist

  • My code follows the style guidelines of this project, with adapted comments and without new warnings.
  • I have added unit and integration tests that prove my fix is effective or that my feature works.
  • The CI passes locally with my changes (including unit tests, integration tests, linter).
  • Any dependent changes have been merged and published in downstream modules.
  • If applicable, I have made corresponding changes to the documentation. Provide a link to the documentation.

Test summary

  • I triggered all changed lines of code at least once without new errors/warnings/notices.
  • I validated all Acceptance Criteria of the related issues. (If applicable, provide proof).
  • I validated all test plan the QA Review asked me to.

Tested locally and on the gamma website by checking transients and scheduled actions.

@MathieuLamiot MathieuLamiot linked an issue Feb 1, 2024 that may be closed by this pull request
@MathieuLamiot MathieuLamiot requested a review from a team February 27, 2024 11:46
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
inc/Engine/Preload/Controller/PreloadUrl.php Outdated Show resolved Hide resolved
@Khadreal Khadreal self-assigned this Feb 29, 2024
@MathieuLamiot
Copy link
Contributor Author

MathieuLamiot commented Mar 11, 2024

@Khadreal Hey, I just pushed a small change to enforce the max batch size after the min batch size. This is for the case there are already 45 actions in the AS queue. In this case, no matter what, we don't want to add more to the AS queue. So the max has to be enforced after.
Checking the tests now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[R&D] Adaptative batch size on Preload
3 participants