Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix reserve over allocating underlying buffer #560

Merged
merged 1 commit into from Jul 30, 2022

Conversation

schultetwin1
Copy link
Contributor

Fixes calls to reserve when the underlying shared buffer was already
big enough to fit the requested capacity. Previously a new even larger
buffer was created anyways. This could eventually lead to an OOM
condition.

Fixes #559

@schultetwin1
Copy link
Contributor Author

This appears to have been introduced by #555

@schultetwin1
Copy link
Contributor Author

Here is an example of how you could cause an OOM error

#[test]
fn reserve_in_arc_unique_does_not_overallocate_after_multiple_splits() {
    let mut bytes = BytesMut::from(LONG);
    let orig_capacity = bytes.capacity();
    for _ in 0..100 {
        drop(bytes.split_off(LONG.len() / 2));

        // now bytes is Arc and refcount == 1

        let new_capacity = bytes.capacity();
        bytes.reserve(orig_capacity - new_capacity);
    }
    assert_eq!(bytes.capacity(), orig_capacity);
}

Fixes calls to `reserve` when the underlying shared buffer was already
big enough to fit the requested capacity. Previously a new even larger
buffer was created anyways. This could eventually lead to an OOM
condition.
Copy link
Contributor

@Darksonn Darksonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. We should be careful to ensure that there aren't any other cases we missed. With this logic we can show that if v_capacity >= 2*new_cap, then it will not reallocate. That seems sufficient to me.

@Darksonn Darksonn merged commit d6e1999 into tokio-rs:master Jul 30, 2022
@schultetwin1 schultetwin1 deleted the fix-oom-condition-on-reserve branch July 30, 2022 16:40
@schultetwin1
Copy link
Contributor Author

Thanks for the quick merge! And I see you are already prepping a new release (1.2.1), thank you!!

blckngm added a commit to godwokenrises/godwoken that referenced this pull request Aug 4, 2022
lelongg pushed a commit to lelongg/bytes that referenced this pull request Jan 9, 2023
Fixes calls to `reserve` when the underlying shared buffer was already
big enough to fit the requested capacity. Previously a new even larger
buffer was created anyways. This could eventually lead to an OOM
condition.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

reserve_inner over allocates which can lead to a OOM conidition
2 participants