Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory buffering #401

Open
RReverser opened this issue Feb 4, 2024 · 2 comments
Open

Improve memory buffering #401

RReverser opened this issue Feb 4, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@RReverser
Copy link

I'm using mmap-ed RAW files as the source of data, which should provide best performance as all the data is directly mapped into memory and accessible just &[u8].

Unfortunately, rawler can't benefit from this yet, because 1) RawFile requires only &'static data, which can be worked around by giving it a custom newtype that preserves the mmap handle, but is somewhat cumbersome and 2) rawler does a lot of tiny allocations & copying in methods like subview, which are unnecessarily expensive, as with sources like Mmap or even just Vec it could instead use subslices from existing memory.

In particular, each subview_until_eof copies almost the entire file into new and new allocations over and over. At that point it would be cheaper to have one central Vec<u8> and use &[u8] subslices everywhere.

@RReverser
Copy link
Author

After thinking about this a bit more, I think the best solution here would be to use the common bytes crate: https://docs.rs/bytes/latest/bytes/struct.Bytes.html

It allows to wrap arbitrary buffer types, provides zero-cost shared chunks without the lifetime mess, as well as provides some useful parsing helpers.

@cytrinox
Copy link
Contributor

cytrinox commented Apr 1, 2024

May have a look at #416, feedback welcome.

@cytrinox cytrinox added the enhancement New feature or request label Apr 1, 2024
@cytrinox cytrinox self-assigned this Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants