Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IMPROVED] Performance and Stability under heavy IO loads. #3922

Merged
merged 21 commits into from Feb 28, 2023
Merged

[IMPROVED] Performance and Stability under heavy IO loads. #3922

merged 21 commits into from Feb 28, 2023

Conversation

derekcollison
Copy link
Member

Various improvements based on user test framework that put the system under heavy IO and network loads.

Signed-off-by: Derek Collison derek@nats.io

@derekcollison
Make minimum snapshot time for all assets 10s.
4391629 
Do not lock on clustered test for JetStream, not needed.

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Prevent benign spin between competing leaders with same index but dif…
2711460 
…feren term.

Remove lock from route processing for updating peers progress, altready handled in trackPeer.

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Optimize for high IO workloads. When we know optional metadata will a…
3cebd26 
…lways be correct on restart do not require inline IO all the time.

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Optimize some locking for when under heavy loads.
13167f4 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Optimize locking for when under heavy loads.
2642a8c 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Do not allow meta snapshot processing during recovery to override.
9721309 
Make sure to process all stream updates during recovery through the ru structure.

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Fixup test for new parameters
6078706 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Sometimes do force meta snapshot
576d317 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Only warn on write errors if not closed in case they linger under pre…
fa8afba 
…ssure and blocking on dios

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Do not need lock to grab js here
aad8aa6 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Only need server's rlock here.
bee149b 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Do not block routes/gws on internal stream and consumer info requests
b19fe50 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Fixed dios capacity to 4 due to testing under heavy load.
adbb50f 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Do not block in place on warning, and only warn if consumer not closed
d85bec2 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Always process inbound messages in separate execution context.
3807441 
Do not duplicate work on leader, sealed and clustered state.

Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Fix tests that made assumptions about single server processing.
6bda358 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Do not lock on stream name for consumer write state error
24cb570 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Be more conservative on defaultMaxTotalCatchupOutBytes, default to 64M
68cd312 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Revert closeAndKeepIndex behavior
bab10c1 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison derekcollison requested a review from a team as a code owner February 28, 2023 21:41
neilalexander
neilalexander approved these changes Feb 28, 2023
View reviewed changes
Copy link
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, I particularly like propagating whether we're cleaning up via limits in 3cebd26.

server/jetstream_cluster.go Outdated Show resolved Hide resolved
Jarema
Jarema reviewed Feb 28, 2023
View reviewed changes
Copy link
Member

@Jarema Jarema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some typos and test comments.

server/consumer.go Outdated Show resolved Hide resolved
server/filestore.go Outdated Show resolved Hide resolved
server/jetstream_test.go Show resolved Hide resolved
@derekcollison
Fix flapping tests
724160e 
Signed-off-by: Derek Collison <derek@nats.io>
@derekcollison
Signal a metasnapshot for consumer deletes as well
1956fa3 
Signed-off-by: Derek Collison <derek@nats.io>
wallyqs
wallyqs approved these changes Feb 28, 2023
View reviewed changes
Copy link
Member

@wallyqs wallyqs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎉

neilalexander
neilalexander approved these changes Feb 28, 2023
View reviewed changes
Copy link
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! :shipit:

@derekcollison derekcollison merged commit d920ca6 into main Feb 28, 2023
@derekcollison derekcollison deleted the stress-perf branch February 28, 2023 22:49
@wallyqs wallyqs mentioned this pull request Mar 4, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jarema Jarema Jarema left review comments

@MauriceVanVeen MauriceVanVeen MauriceVanVeen left review comments

@wallyqs wallyqs wallyqs approved these changes

@neilalexander neilalexander neilalexander approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@derekcollison @wallyqs @neilalexander @Jarema @MauriceVanVeen