Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IMPROVED] Performance and Stability under heavy IO loads. #3922

Merged
merged 21 commits into from
Feb 28, 2023
Merged

[IMPROVED] Performance and Stability under heavy IO loads. #3922

merged 21 commits into from
Feb 28, 2023

Commits on Feb 27, 2023

  1. Make minimum snapshot time for all assets 10s. 

    Do not lock on clustered test for JetStream, not needed.

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    4391629 View commit details
    Browse the repository at this point in the history

  2. Prevent benign spin between competing leaders with same index but dif… 

    …feren term.

Remove lock from route processing for updating peers progress, altready handled in trackPeer.

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    2711460 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2023

  1. Optimize for high IO workloads. When we know optional metadata will a… 

    …lways be correct on restart do not require inline IO all the time.

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    3cebd26 View commit details
    Browse the repository at this point in the history

  2. Optimize some locking for when under heavy loads. 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    13167f4 View commit details
    Browse the repository at this point in the history

  3. Optimize locking for when under heavy loads. 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    2642a8c View commit details
    Browse the repository at this point in the history

  4. Do not allow meta snapshot processing during recovery to override. 

    Make sure to process all stream updates during recovery through the ru structure.

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    9721309 View commit details
    Browse the repository at this point in the history

  5. Fixup test for new parameters 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    6078706 View commit details
    Browse the repository at this point in the history

  6. Sometimes do force meta snapshot 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    576d317 View commit details
    Browse the repository at this point in the history

  7. Only warn on write errors if not closed in case they linger under pre… 

    …ssure and blocking on dios

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    fa8afba View commit details
    Browse the repository at this point in the history

  8. Do not need lock to grab js here 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    aad8aa6 View commit details
    Browse the repository at this point in the history

  9. Only need server's rlock here. 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    bee149b View commit details
    Browse the repository at this point in the history

  10. Do not block routes/gws on internal stream and consumer info requests 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    b19fe50 View commit details
    Browse the repository at this point in the history

  11. Fixed dios capacity to 4 due to testing under heavy load. 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    adbb50f View commit details
    Browse the repository at this point in the history

  12. Do not block in place on warning, and only warn if consumer not closed 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    d85bec2 View commit details
    Browse the repository at this point in the history

  13. Always process inbound messages in separate execution context. 

    Do not duplicate work on leader, sealed and clustered state.

Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    3807441 View commit details
    Browse the repository at this point in the history

  14. Fix tests that made assumptions about single server processing. 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    6bda358 View commit details
    Browse the repository at this point in the history

  15. Do not lock on stream name for consumer write state error 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    24cb570 View commit details
    Browse the repository at this point in the history

  16. Be more conservative on defaultMaxTotalCatchupOutBytes, default to 64M 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    68cd312 View commit details
    Browse the repository at this point in the history

  17. Revert closeAndKeepIndex behavior 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    bab10c1 View commit details
    Browse the repository at this point in the history

  18. Fix flapping tests 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    724160e View commit details
    Browse the repository at this point in the history

  19. Signal a metasnapshot for consumer deletes as well 

    Signed-off-by: Derek Collison <derek@nats.io>
    @derekcollison
    derekcollison committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    1956fa3 View commit details
    Browse the repository at this point in the history