Make max outstanding queries per tenant config in limits #4991
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
And since the query vertical sharding size is also per tenant, the queue size needs to be adjusted per tenant if we enable sharding for some tenants only. |
alanprot
reviewed
Nov 29, 2022
| @@ -2724,6 +2726,11 @@ The `limits_config` configures default and per-tenant limits imposed by Cortex s | |||
| # CLI flag: -frontend.max-queriers-per-tenant | |||
| [max_queriers_per_tenant: <int> | default = 0] | |||
|
|
|||
| # Maximum number of outstanding requests per tenant per request queue (either | |||
| # query frontend or query scheduler); requests beyond this error with HTTP 429. | |||
| # CLI flag: -frontend.max-outstanding-requests-per-tenant | |||
There was a problem hiding this comment.
Yeah.. i think make sense this limit be related to frontend if we use or not the scheduler.
alanprot
reviewed
Nov 29, 2022
pkg/frontend/v1/frontend.go
Outdated
| @@ -37,13 +37,25 @@ type Config struct { | |||
|
|
|||
| // RegisterFlags adds the flags required to config this to the given FlagSet. | |||
| func (cfg *Config) RegisterFlags(f *flag.FlagSet) { | |||
| f.IntVar(&cfg.MaxOutstandingPerTenant, "querier.max-outstanding-requests-per-tenant", 100, "Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.") | |||
| f.IntVar(&cfg.MaxOutstandingPerTenant, "querier.max-outstanding-requests-per-tenant", 0, "Deprecated (user frontend.max-outstanding-requests-per-tenant instead): Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.") | |||
There was a problem hiding this comment.
We need to start documenting when the deprecated flags will be removed (on which version)
alanprot
approved these changes
Nov 29, 2022
yeya24
force-pushed
the
queue-size-per-tenant
branch
from
d0987a9
to
0f538b4
Compare
Signed-off-by: Ben Ye <benye@amazon.com>
yeya24
force-pushed
the
queue-size-per-tenant
branch
2 times, most recently
from
9fa309b
to
6f6cfbb
Compare
Signed-off-by: Ben Ye <benye@amazon.com>
yeya24
force-pushed
the
queue-size-per-tenant
branch
from
6f6cfbb
to
a96f70e
Compare
|
Thanks! LGTM |
alanprot
approved these changes
Dec 1, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ben Ye benye@amazon.com
What this PR does:
Now the QFE queue size is a global configuration https://github.com/cortexproject/cortex/blob/master/pkg/frontend/v1/frontend.go#L40, which means we cannot override the queue size for different tenants.
For different tenants, we may want to use different queue sizes based on their query concurrency. Making it a per-tenant config in limits makes it more flexible and easier to update during runtime.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]