Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tiflash scale in is slow #8433

Closed
guo-shaoge opened this issue Nov 29, 2023 · 0 comments · Fixed by #8432
Closed

tiflash scale in is slow #8433

guo-shaoge opened this issue Nov 29, 2023 · 0 comments · Fixed by #8432
Labels
type/enhancement Issue or PR for enhancement

Comments

@guo-shaoge
Copy link
Contributor

guo-shaoge commented Nov 29, 2023

Enhancement

For now, when AutoScaler want to scale in one CN, supervisor will send SIGKILL to tiflash process. So tiflash has no chance to do any cleanup work, most importantly, the LAC::stop(), which will send final report rpc to PD to cleanup some slot info.

We can send SIGTERM to let tiflash do its cleanup, but for now tiflash takes almost 10sec from it receive SIGTERM to exit properly, which is too long for AutoScaling. (check the log rec_stop.log.md)

@guo-shaoge guo-shaoge added the type/enhancement Issue or PR for enhancement label Nov 29, 2023
@guo-shaoge guo-shaoge changed the title Add CleanupBeforeScaleIn grpc tiflash scale in is slow Nov 29, 2023
ti-chi-bot bot pushed a commit that referenced this issue Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Issue or PR for enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant