Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allocate Vec capacity based on size hints during decoding #293

Merged
merged 5 commits into from Apr 14, 2023

Conversation

jordy25519
Copy link
Contributor

Discovered this issue while profiling another crate

Copy link
Member

@vkgnosis vkgnosis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails CI. I guess there is a case where the code without with_capacity never end ups really allocating that much.

@jordy25519
Copy link
Contributor Author

This fails CI. I guess there is a case where the code without with_capacity never end ups really allocating that much.

Ok I'm glad this happened since some malicious payloads could cause large memory allocation in my previous commit. Iterator should have the same effect to avoid reallocs and not allocate upfront

@vkgnosis
Copy link
Member

Iterator should have the same effect to avoid reallocs and not allocate upfront

I don't think that's true. The current version of the PR works the same as it did before the PR but using collect instead of manually pushing. Either way the Vector reallocate as elements get added because the final size isn't known upfront.

@jordy25519
Copy link
Contributor Author

jordy25519 commented Apr 13, 2023

you're right- after some benchmarking, the iterator size hints didn't work how I thought.
original error caused by test runner on 32bit machine can't allocate enough space in the 'corrupted dynamic array' test when using with_capacity. try_reserve fixes the issue so we can handle the error properly.

before this PR:
Screenshot 2023-04-13 at 13 36 32

with iterator:
still has some reallocs and some over head
Screenshot 2023-04-13 at 12 39 25

with try_reserve_exact:
Screenshot 2023-04-13 at 13 36 56

benchmark code used

use ethabi::{self, decode, encode, ParamType, Token};

fn main() {
    let tokens: Vec<Token> = (0..100)
        .map(|i| Token::Address([i % 3 as u8; 20].into()))
        .collect();
    let param_types = vec![ParamType::Tuple(vec![ParamType::Address; tokens.len()])];
    let expected = vec![Token::Tuple(tokens)];
    let encoded = encode(&expected);

    for _ in 0..=50_000 {
        assert_eq!(decode(&param_types, &encoded,).unwrap(), expected);
    }
}

Comment on lines -85 to -89
impl<T> Default for Topic<T> {
fn default() -> Self {
Topic::Any
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? I think you are changing behavior because the previous version still implemented Default when T wasn't Default while the new one doesn't. Either way, it's unrelated to the rest of the PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no behaviour change, the compiler recommended this change in the CI failure.
this compiles fine:

	#[test]
	fn default_for_topic_T() {
		struct Foo;
		let f = Topic::<Foo>::default();
	}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only changed it due to the CI failure, happy to keep or revert it either way

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does CI fail because of that or is a warning? Apparently derive got smarter and it doesn't change behavior. I'm fine with keeping the change then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI failed, I actually think it was a clippy suggestion but it is configured to fail the job

@vkgnosis vkgnosis merged commit a148a4a into rust-ethereum:master Apr 14, 2023
3 checks passed
@jordy25519 jordy25519 deleted the fix/decode-w-capacity branch April 14, 2023 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants