Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingester: Added configuration to set the size of the tsdb in-memory queue used before flushing chunks to the disk #5000

Merged
merged 2 commits into from
Dec 1, 2022

Conversation

t00350320
Copy link
Contributor

@t00350320 t00350320 commented Nov 25, 2022

What this PR does:
Added configuration to set the size of the tsdb in-memory queue used before flushing chunks to the disk
Which issue(s) this PR fixes:
Fixes #
#4957

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@t00350320
Copy link
Contributor Author

configuration is added in cortex.yaml like this:

blocks_storage:
  tsdb:
    xxx
    head_chunks_write_queue_size: 5000

corresponding metric 'cortex_ingester_tsdb_chunk_write_queue_operations_total' is also exported.

FYI, some basic test datas :
4C32G 200G *3 cluster, 3.6M series /30s input, replica factor = 2

For ”head_chunks_write_queue_size“ ,
1、from 0 to 1000, write latency seems better , but it
becomes worse when from 1000 to 5000.
2、from 0 to 1000 , to 5000, the disk busy io seems pretty much the same

For "Distributor Requests/sec"
because our prometheus remote write's queue_config

max_samples_per_send: 500

so, 3.6M series / 500 /30s = 240( Distributor Requests/sec)

For "Write queue"
40000*60s=2.4M=3.6M * 2[replica factor] / 3[nodes number]

case:

head_chunks_write_queue_size = 0,

write latency:
image
cortex_ingester_tsdb_chunk_write_queue_operations_total stat:
image
disk busy:
image

head_chunks_write_queue_size = 1000,

write latency:
image
cortex_ingester_tsdb_chunk_write_queue_operations_total stat:
image
disk busy:
image

head_chunks_write_queue_size = 5000,

write latency:
image
cortex_ingester_tsdb_chunk_write_queue_operations_total stat:
image
disk busy:
image

@yeya24
Copy link
Collaborator

yeya24 commented Nov 28, 2022

Nice! @t00350320 Could you please sign the DCO and fix the lint?

t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 28, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 28, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
@pull-request-size pull-request-size bot added size/M and removed size/S labels Nov 28, 2022
t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 28, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
CHANGELOG.md Outdated Show resolved Hide resolved
pkg/storage/tsdb/config.go Show resolved Hide resolved
t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 29, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
Copy link
Collaborator

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! Thanks for the contribution. Would you mind resolving the conflict again? @t00350320 Thanks!

t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 29, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
@pull-request-size pull-request-size bot added size/L and removed size/M labels Nov 29, 2022
@pull-request-size pull-request-size bot added size/M and removed size/L labels Nov 29, 2022
t00350320 added a commit to t00350320/cortex that referenced this pull request Nov 29, 2022
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
@yeya24
Copy link
Collaborator

yeya24 commented Nov 29, 2022

Maybe @alanprot @friedrichg would like to take another look?

pkg/storage/tsdb/config.go Outdated Show resolved Hide resolved
…llowing to configure the size of the in-memory queue used before flushing chunks to the disk . cortexproject#5000

Signed-off-by: tanghengjian <1040104807@qq.com>
Signed-off-by: tanghengjian <1040104807@qq.com>
@yeya24
Copy link
Collaborator

yeya24 commented Nov 30, 2022

This pr LGTM. It would be appreciated if we could have another 👁️ on it. WDYT? @alanprot @friedrichg

@alanprot
Copy link
Member

alanprot commented Dec 1, 2022

Awesome! thanks!

@alanprot alanprot merged commit a5476bb into cortexproject:master Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants