Support for direct access to memory mapped files and shared memory

[snrostov]

It seems currently OkIo is ByteArray based only.

To achieve native speed for MemoryMapped files (and shared memory), the underlying memory should be accessed via the DirectByteBuffer or other Java API for accessing non-heap memory.

It would be nice to have a way to create such Okio Buffer. Otherwise, it is impossible to use Okio in projects where we need no-copy access to native memory.

[snrostov]

Speaking about multiplatform:

• JVM: DirectByteBuffer
• JS: SharedArrayBuffer
• Native: just pointers

[swankjesse]

DirectByteBuffers can achieve great performance, but they’re significantly more involved to use because they aren’t dynamically sized and because they don’t participate in garbage collection.

If that’s what you want I recommend using a library like Netty that’s built for that. But also note that additional capability might not yield performance gains in practice.

[snrostov]

@swankjesse, is there a way to work with shared memory or memory-mapped files with Okio?

[swankjesse]

@snrostov nope! If you use FileHandle it can do random access reads and writes, and I trust the kernel to give you similar performance to memory mapping.

[snrostov]

If you use FileHandle it can do random access reads and writes, and I trust the kernel to give you similar performance to memory mapping.

I'm sure it is not at least for shared memory and files that are cached in kernel memory (because kernel should copy data from kernel to userspace). Will try to show it in benchmarks.

[snrostov]

But also note that additional capability might not yield performance gains in practice.

Sorry, I read the discussion but found that they said that with fix netty is about 14% faster.. The problem is in Netty internals, not in native memory which is faster. No?

[swankjesse]

Yeah, good point!

Okio will always require memory to be copied into an okio.Buffer, and for certain applications that cost can be significant. Okio has a district disadvantage for implementing say, the sha256sum command line program.

Please don’t write a benchmark if your intention is to convince us to add that functionality though. We’ve made certain tradeoffs in Okio that yield great performance in practice but that also lose to other approaches in benchmarks. For example, although we pay for a copy when reading a file into a Buffer, we save a copy when sending that data in an OkHttp RequestBody, or decoding it as a Wire protocol buffer.

[snrostov]

Ok, fair enough. Thanks for the answer!