I think this should follow the proposed "project" process: #3443.

I wonder if there are libc compliance test suites that are licensed well enough for us to extract tests from

Hi, I am planning to work on this first since it is related to #3448. I will send the link to the proposal here once I finished localtime_r.

@rustbot claim

Error: The feature assign is not enabled in this repository.
To enable it add its section in the triagebot.toml in the root of the repository.

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

I wonder if there are libc compliance test suites that are licensed well enough for us to extract tests from

Hmm I couldn't find any test for libc for now, if anyone knows where can I find one please let me know, thanks. Currently I imagine minimally the test should look like this (Note: they wrapped libc around their API, so it looks slightly different)

#[test]
pub fn test_socketpair() {
    use nix::sys::socket::{socketpair, AddressFamily, SockFlag, SockType};
    use nix::unistd::{read, write};

    let (fd1, fd2) = socketpair(
        AddressFamily::Unix,
        SockType::Stream,
        None,
        SockFlag::empty(),
    )
    .unwrap();
    write(&fd1, b"hello").unwrap();
    let mut buf = [0; 5];
    read(fd2.as_raw_fd(), &mut buf).unwrap();

    assert_eq!(&buf[..], b"hello");
}
//source: https://github.com/nix-rust/nix/blob/590ab4d5d0b6228f0529b036e4f8a42265ba6d31/test/sys/test_socket.rs#L311. 

source: nix-rust/nix

I might add similar test in the proposal that I am going to send soon.

This is the proposal for socketpair shim: https://hackmd.io/@tiif/Skhc1t0-C. It is still under development, more attention is needed for the problems below:

1. Should we block write when the buffer is full?
2. How can miri differentiate read and write of socketpair file descriptors and other kind of file descriptors?

Feel free to provide any ideas or suggestion especially for the buffer mechanism part.

2. How can miri differentiate read and write of socketpair file descriptors and other kind of file descriptors?

Why does it need to? Shouldn't the abstraction provided by the FileDescriptor trait hide all thatvand only the socket pair type implementing that trait would care?

Left some comments,on the doc

Should we block write when the buffer is full?

Yes, we should.

How can miri differentiate read and write of socketpair file descriptors and other kind of file descriptors?

As Oli said: you "just" have to implement the read and write functions of the SocketPair type; the general file descriptor machinery will make sure to dispatch to that when read/write is called on an FD that is (part of) a socketpair.

This is the proposal for socketpair shim: https://hackmd.io/@tiif/Skhc1t0-C. It is still under development, more attention is needed for the problems below:

Thanks! The API description is very helpful. I have left some comments.

What is missing for me is a plan for how the Miri state looks like. In particular, the proposal should say what the fields of struct SocketPair will be. I currently think it will likely be something like

struct SocketPair {
  write_buf: Rc<SocketBuf>,
  read_buf: Rc<SocketBuf>,
}

but that then raises the question of what the fields of SocketBuf are. :) Maybe data: VecDeque<u8> as you sketched out is sufficient.

FWIW, once this is done it should be fairly easy to also support pipe/pipe2. Those are basically just one-directional sockets, for our purposes.

After reading eventfd, I realised both eventfd and socketpair have non-blocking flag, and I think it is good to implement them (and shouldn't be hard?). But currently blocking file descriptor is not implemented yet, and has higher priority, so maybe non-blocking can be left as a FIXME?

Besides, it is likely that SOCK_DGRAM and SOCK_STREAM have some difference in data size handling, but I think maybe we can leave it as a FIXME too?

Does tokio need non blocking sockets?

Yes, I just realised mio seems to support both SOCK_NONBLOCK and SOCK_CLOEXEC that I initially thought we don't need to support 👀. We also need non-blocking file descriptor for epoll_ctl because there is a flag called EPOLLET for edge-triggered notification¹ file descriptor, and mio used them (tokio used that too). For edge-triggered notification, nonblocking file descriptors is used². I guess the behaviour for non-blocking file descriptors should be similar, so once we get the design down, it should be applicable to other kind of file descriptors too.

FWIW, we might also need to provide support for "block with timeout" at some point too because apparently mio has them, tokio has related functions too :)

But I will go through all the flags and what should we do with them again in detail once we start to discuss epoll.

SOCK_DGRAM needs a pretty different implementation (we need to preserve message boundaries) so I think that is also out for now.

Hmm, now I think about it, should we throw_unsup_format for SOCK_DGRAM? Personally, I prefer to not throw error and let SOCK_DGRAM run without message boundaries. We can let the messages go through first, and message boundaries come later as an enhancement.

Hmm, now I think about it, should we throw_unsup_format for SOCK_DGRAM?

Yes, definitely. We don't currently support it. We should throw_unsup_format on everything we do not support. (The current shim is buggy in that it does not properly do this.)
Not preserving message boundaries gives just the completely wrong semantics, we should never do that.
(I am not even sure what read/write do on such a socket, one would usually use sendmsg/recvmsg I think.)

For edge-triggered notification, nonblocking file descriptors is used2.

I don't know what you mean by that. Does tokio set SOCK_NONBLOCK or not?
If not, then clearly EPOLLET doesn't require SOCK_NONBLOCK.

Not preserving message boundaries gives just the completely wrong semantics, we should never do that.

I see why it is important to throw_unsup_format for that now, thanks for the explanation.

I am not even sure what read/write do on such a socket, one would usually use sendmsg/recvmsg I think.

I didn't know sendmsg/recvmsg exist, thanks for pointing it out. read/recv/recvfrom and send/write/sendto have the same behaviour when it comes to message boundaries ¹. For either of them, if they are datagram socket, read/recv/recvfrom will retrieve exactly one message. If the message size exceed the length byte, the message is silently truncated to length byte.

// Recall that read looks like this:
ssize_t read(int fd, void* buffer, size_t length);
// Return number of bytes read, or -1 on error

recvmsg is different as user can find out if a datagram is truncated through the MSG_TRUNC flag. ²

Does tokio set SOCK_NONBLOCK or not?

Yes, both SOCK_NONBLOCK and SOCK_CLOEXEC. Is it a good idea to delay the support for these two flags until I finish implementing basic functionality of eventfd? We need to support non-blocking and close-on-exec flags for eventfd too because tokio uses it. The design should be similar so I am thinking of doing both together.

I don't know what you mean by that.

Oh that's only for epoll, I will explain it again when we get to epoll.

Yes, both SOCK_NONBLOCK and SOCK_CLOEXEC.

Ah. In that case you probably should implement non-blocking sockets first -- i.e., store during socket creation whether the socket is blocking, and then in read/write, if we would have to block, throw_unsup_format!.

It is up to you whether you want to implement support for blocking sockets at all, if it is not needed for tokio.

SOCK_NONBLOCK and SOCK_CLOEXEC are Linux-specific. ¹

x86_64-apple-darwin doesn't have these two flags. How mio handles them is ²

1. Create a socketpair: socketpair(libc::AF_UNIX, flags, 0, fds.as_mut_ptr())
2. Use fcntl to set the NON_BLOCK flag instead

syscall!(fcntl(fds[0], libc::F_SETFL, libc::O_NONBLOCK))?;
syscall!(fcntl(fds[1], libc::F_SETFL, libc::O_NONBLOCK))?;

I am only mentioning NONBLOCK here becaues SOCK_CLOEXEC will be ignored since miri don't support exec

So to actually support x86_64-apple-darwin, we might need to:

1. Add O_NONBLOCK flag handling in fcntl
2. Find a way to notify socketpair that this socketpair file descriptor has been set to non-blocking.

Don't worry about running mio on macOS for now. That's going to be completely different anyway, it can't be using epoll.

But the NONBLOCK check in the socketpair function should be gated on Linux.