Problematic ssize_t to int conversion in uv_fs_write

[zbjornson]

• Version: 1.42.0
• Platform: Linux/Unix (generic)

uv_fs_write returns an int, but the underlying Linux syscalls return a ssize_t. That conversion is problematic for large writes and has caused a few bugs in Node.js, including:

• fs: fix error when writing buffers > INT32_MAX nodejs/node#38546
• Nonsense UNKNOWN error when writing large buffers with fs.writev nodejs/node#40779

There are a handful of issues about related compiler warnings (#708, #2714), but I don't see an issue for the actual bug.

I think the only way to fix this without a breaking change is to add ssize_t uv_fs_write64 or ssize_t uv_fs_write2 or something. uv_fs_s's result property is already a ssize_t, so only the return type needs changing.

[vtjnash]

Somewhat fixed by #1501, but there was enough problems with it at the OS level, I changed my mind and think we are better off documenting the maximum size legal write (~2GB), and then expecting applications to handle that case themselves.

[bnoordhuis]

#1501 stalled on ABI issues, IIRC? I can see two ways to attack this issue:

1. limit reads and writes to ~2GB
2. change uv_fs_write() et al to return ssize_t or int64_t

(1) means libuv users now have to handle partial writes (but the current behavior is broken too)

(2) is technically an API and ABI break, although I expect in practice it'll be fine

[vtjnash]

#1501 stalled for several reasons, the primary one being that most kernels also can't handle reads and writes over 2GB either. This doesn't make much difference to reads (since those are chunked explicitly anyways), but makes it hard to form the return value for the write calls (and is an API break to make these all return off_t / int64_t, though perhaps we want to do that for v2 regardless). My ending suggestion was to make libuv return UV_EINVAL for all writes larger than some documented maximum (empirically, about 2^31 - 4096 bytes for unix and 511 MB for windows, or 0x7ffff000 and 0x1ff00000 respectively), and force the application (e.g. nodejs or julia) to chunk it as needed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[trevnorris]

@vtjnash I see in https://man7.org/linux/man-pages/man2/write.2.html#NOTES that write(2) has a transfer limit of 0x7ffff000. Does that pertain to any individual uv_buf_t (for when writev() is called), or the entire write_queue_size? And what about returning UV_EFBIG instead of UV_EINVAL to be more explicit?

[vtjnash]

Each syscall (summed across all buffers in the vector)