Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add semaphore to limit subchannel connect to prevent race conditions #2422

Merged
merged 6 commits into from Apr 30, 2024

Conversation

JamesNK
Copy link
Member

@JamesNK JamesNK commented Apr 29, 2024

Addresses #2420

I have a theory that concurrent calls to ConnectTransportAsync could put a subchannel in a bad state. Two connection requests happen simultaneously but a race condition occurs as they exit.

Before:

  1. Connect 1 starts.
  2. Connect 2 starts. Cancels connect 1.
  3. Connect 2 succeeds.
  4. Connect 2 exits and updates subchannel status to Ready.
  5. Connect 1 exits and updates subchannel status to TransientFailure.

This PR limits access to connect with a semaphore. A subchannel will have one connect in progress at a time.

After:

  1. Connect 1 acquires semaphore and starts.
  2. Connect 2 tries to start. Cancels connect 1. Waits with semaphore for connect 1 to finish.
  3. Connect 1 exits and updates subchannel status to TransientFailure.
  4. Connect 1 releases semaphore.
  5. Connect 2 acquires semaphore.
  6. Connect 2 succeeds.
  7. Connect 2 exits and updates subchannel status to Ready.
  8. Connect 2 releases semaphore.

Note: New test is unrelated to this exact problem. I wrote it while considering another theory. It's a good test to keep around.

@JamesNK JamesNK merged commit 63914f2 into grpc:master Apr 30, 2024
5 checks passed
@JamesNK JamesNK deleted the jamesnk/connect-threadsafety branch April 30, 2024 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants