Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix to stop forwarding proposals in consumers after scaling down a stream #4556

Merged
merged 4 commits into from Sep 18, 2023
Merged

Fix to stop forwarding proposals in consumers after scaling down a stream #4556

merged 4 commits into from Sep 18, 2023

Conversation

wallyqs
Copy link
Member

@wallyqs wallyqs commented Sep 18, 2023

Sometimes when scaling down a stream, a raft node could continue forwarding proposals after already being closed, in the debug logs this can be confirmed by many entries logging Direct proposal ignored, not leader (state: CLOSED).

wallyqs and others added 2 commits September 18, 2023 12:26
@wallyqs
Add test for scaling replica with pull consumers
2724589 
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
@derekcollison @wallyqs
When scaling a consumer down make sure to pop the loopAndForwardPropo…
850c89e 
…sals go routine

Signed-off-by: Derek Collison <derek@nats.io>
@wallyqs wallyqs requested a review from a team as a code owner September 18, 2023 19:32
derekcollison
derekcollison approved these changes Sep 18, 2023
View reviewed changes
Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - How long does test take?

@wallyqs
Copy link
Member Author

wallyqs commented Sep 18, 2023

Currently it takes about 40s, I think it can be made shorter

neilalexander
neilalexander approved these changes Sep 18, 2023
View reviewed changes
Copy link
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM

server/consumer.go Outdated
defer o.mu.Unlock()
if o.qch != nil {
close(o.qch)
// Note can not close pch here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because of a panic when writing to the channel elsewhere?

server/jetstream_cluster.go Outdated Show resolved Hide resolved
@wallyqs wallyqs changed the title Fix to stop forwarding proposals after scaling a consumer down Fix to stop forwarding proposals in consumers after scaling down a stream Sep 18, 2023
wallyqs and others added 2 commits September 18, 2023 12:46
@wallyqs
Skip TestJetStreamClusterRestartThenScaleStreamReplicas for now
ea775a8 
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
@derekcollison
Update to not pop directly, just bail when we detect leadership change
71b8a33 
Signed-off-by: Derek Collison <derek@nats.io>
neilalexander
neilalexander approved these changes Sep 18, 2023
View reviewed changes
Copy link
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wallyqs wallyqs merged commit 2d23e9b into main Sep 18, 2023
2 checks passed
@wallyqs wallyqs deleted the stream-scaling-issue branch September 18, 2023 20:51
@wallyqs wallyqs mentioned this pull request Oct 11, 2023
Merged
wallyqs added a commit that referenced this pull request Oct 11, 2023
@wallyqs
Backports from v2.10 for v2.9.23 release (#4647)
8b981a2 
Backports the following PRs from v2.10:

- #4510 
- #4556 
- #4578 
- #4605 

Signed-off-by: Your Name <wally@nats.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@derekcollison derekcollison derekcollison approved these changes

@neilalexander neilalexander neilalexander approved these changes

@bruth bruth Awaiting requested review from bruth

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@wallyqs @derekcollison @neilalexander