You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using mmap-ed RAW files as the source of data, which should provide best performance as all the data is directly mapped into memory and accessible just &[u8].
Unfortunately, rawler can't benefit from this yet, because 1) RawFile requires only &'static data, which can be worked around by giving it a custom newtype that preserves the mmap handle, but is somewhat cumbersome and 2) rawler does a lot of tiny allocations & copying in methods like subview, which are unnecessarily expensive, as with sources like Mmap or even just Vec it could instead use subslices from existing memory.
In particular, each subview_until_eof copies almost the entire file into new and new allocations over and over. At that point it would be cheaper to have one central Vec<u8> and use &[u8] subslices everywhere.
The text was updated successfully, but these errors were encountered:
I'm using mmap-ed RAW files as the source of data, which should provide best performance as all the data is directly mapped into memory and accessible just
&[u8]
.Unfortunately, rawler can't benefit from this yet, because 1) RawFile requires only
&'static
data, which can be worked around by giving it a custom newtype that preserves the mmap handle, but is somewhat cumbersome and 2) rawler does a lot of tiny allocations & copying in methods likesubview
, which are unnecessarily expensive, as with sources like Mmap or even just Vec it could instead use subslices from existing memory.In particular, each
subview_until_eof
copies almost the entire file into new and new allocations over and over. At that point it would be cheaper to have one centralVec<u8>
and use&[u8]
subslices everywhere.The text was updated successfully, but these errors were encountered: