How to use rayon to speedup reading files by blocks?

[Evian-Zhang]

We have a huge binary file, whose content can be divided into millions of blocks with same size (order does not matter). What we want is to read each block, map them into instances of same structure, and process each instance.

I think maybe we should use read_exact_at to avoid file pointer conflict, but I cannot come up a whole solution to this problem.

[adamreichold]

Since you appear to be able to compute the block offsets upfront, you should be able to start with Rayon's ParallelIterator implementation for Range, e.g.

let file = File::open("path/to/large/file")?;

let len = file.metadata()?.len();

let block_size = 4096;

let blocks = len / block_size;

(0..blocks)
    .into_par_iter()
    .map(|block| block * block_size)
    .try_for_each_init(
        || vec![0; block_size],
        |buf, offset| {
            file.read_exact_at(buf, offset)?;
            // TODO: process buf...
            buf.clear();
            Ok(())
        },
    )?;

Alternatively, you might want to look into using the memmap2 crate to map the whole file into your process' address space so that you can access it via a byte slice &[u8] to which you can just apply par_chunks_exact.

EDIT: The buffer must be initialized for read_exact_at, not just allocated.

[adamreichold]

Note that the number of faults can be reduced using advise, e.g. using Advice::WillNeed or even Advice::PopulateRead, to trigger aggressive read-ahead.

Sadly, huge pages are still not supported for file-backed mappings AFAIK.

[adamreichold]

I mean, if lives depend on this being as fast as possible, then I guess on recent kernels direct I/O to bypass the page cache and io_uring to amortize the system call overhead would be the way to go. But I guess this would turn into this task into a project instead of being quite straight-forward using positioned I/O or a memory map.

[Evian-Zhang]

Thank you for the in-depth discussion of how to use mmap, though I still have a little question about this.

If mmap is the best way to buffered file reading, why the BufReader provided by std does not map the entire file for disk File specialization?

[adamreichold]

If mmap is the best way to buffered file reading, why the BufReader provided by std does not map the entire file for disk File specialization?

Leaving aside that BufReader can manage any Read implementation as its backing source, not just files which can be mapped, mapping a file into memory also has downsides like requiring the modification of the process' address which is a heavy weight operation that needs synchronization and hence for small files this can actually be slower. Similarly, page faults used to occur the same degree of serialization in multi-threaded program even though this is slowly being resolved in recent kernels. And on 32-bit system, address space is actually a scarce resource, so mapping large files could fail or make other address space operation fail.

Please have a look at e.g. https://blog.burntsushi.net/ripgrep/ which discusses some of this in the context of implementing a grep.

Finally, it is not the job of the standard library to magically make our programs fast. It should give somewhat portable abstractions to access operating system services. Which services to access in which situations, should be the choice of the developer. And so BufReader does exactly what it says on the tin, it uses a memory buffer of configurable size backed by an underlying implementation of Read.

As so often with Rust, it is less about speed than about control. For example, garbage collection is often faster for short-lived programs doing string processing, yet controlling how the overhead implied by garbage collection interacts with your program's performance characteristics can become highly non-trivial.

And so also in your case, I would suggest implementing multiple approaches and measure the results, ideally in different environments, to understand the trade-offs involved and make a choice that is most beneficial for your particular use case.

[Evian-Zhang]

Thank you:)