Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Deadlock when querycoord recovers from cluster SyncAll having more than 1024 events #33033

Open
1 task done
congqixia opened this issue May 14, 2024 · 0 comments
Open
1 task done

[Bug]: Deadlock when querycoord recovers from cluster SyncAll having more than 1024 events #33033

congqixia opened this issue May 14, 2024 · 0 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@congqixia
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 4ae7cab
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When querycoord recovers, it triggers a SyncAll operation to fetch all distribution from existing cluster.
If the cluster has more than 1024 channel dist, it shall fire more than 1024 event which will block the update operation forever

Expected Behavior

No deadlock no matter the cluster size are

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@congqixia congqixia added kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on. labels May 14, 2024
@congqixia congqixia self-assigned this May 14, 2024
congqixia added a commit to congqixia/milvus that referenced this issue May 14, 2024
@congqixia
fix: Start LeaderCacheObserver before SyncAll
fe7d2e1 
Related to milvus-io#33033

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue May 14, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue May 14, 2024
@congqixia
fix: Start LeaderCacheObserver before SyncAll (#33035)
861977a 
Related to #33033

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@congqixia