a bug?

[rustMyGod]

hello
I want to test performance on mmap,BIO,AIO,and iouring(by tokio-uring) by write 1KB data to 1G files 1024 times . then i write the code bellow.when program running i find something unexpected.

1. too many threads created and destroyed,maybe 20+ thousands.(my test server is 8c16g,5.15.10-1.el7.elrepo.x86_64)
2. some file succeed ,some not,no exception throwed,program is still alive,but no worker threads work now .many file are only 794MB and not growing, the others are 1024MB. bug here?
3. the writing speed is much more slowly than tranditional io(maybe all AIO model slowly the same).

use std::thread;
use std::time::Duration;
use std::time::Instant;
use std::{path::PathBuf, sync::Arc};
use tokio::sync::mpsc;
use tokio::sync::mpsc::UnboundedSender;
use tokio::sync::Mutex;
use tokio::time::sleep;
use tokio_uring::fs::{File, OpenOptions};
fn main() {
    tokio_uring::start(async {
        let (tx, mut rx) = mpsc::unbounded_channel::<()>();
        let txx = Arc::new(tx);
        let count = Instant::now();
        for i in 0..1024 {
            let t = txx.clone();
            tokio_uring::spawn(create(i, t));
        }

        drop(txx);
        while let Some(_) = rx.recv().await {}
        println!("{:?}", count.elapsed());
    });
}

async fn create(idx: i32, tx: Arc<UnboundedSender<()>>) {
    let path = PathBuf::from(r"/data2/testuring/".to_owned() + idx.to_string().as_str());
    let mut file = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open(&path)
        .await
        .unwrap();
    // file.set_len(1024 * 1024 * 1024).await.unwrap();
    println!("create file={} {:?}", idx, thread::current().id());
    let x=b"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
    let y=&x[..];
    let size_1_g = 1024 * 1024 * 1024;
    let mut idx = 0;
    while idx < size_1_g {
        let (_, _) = file.write_at(y, idx).await;
        idx = idx + 1024;
    }
    println!("end file={:?} {:?}", &path, thread::current().id());
    file.close().await.unwrap();
    drop(tx);
}

[rustMyGod]

I watched 20+threads created and destoryed by htop.

[rustMyGod]

20+thousands threads

[Noah-Kennedy]

I am unable to reproduce. I ran your code and only saw 33 threads.

[Noah-Kennedy]

Also, as far as perf, this is going to be slow. You are making a lot of tiny writes sequentially.

[rustMyGod]

@Noah-Kennedy thanks for reply.

when i started running this code, by htop i got the picture 1. after 1min,i got picture 2. u can see that the pids of threads grown by 1k+. did it mean many threads created and destoryed? is it good way if a threadpool used here?

[rustMyGod]

the parent pid was 102663. then the pids of worker-threads was always growing

[rustMyGod]

I know write_at is a random access call. so it's slow than mmap or traditional io. the issue is not about speed. u can see in the picture when i ran this code a while ,some of the 1024 files length was no longer growing.

[rustMyGod]

no workers any more,but some of the files is still 423M.
@Noah-Kennedy

sorry for my poor english . can u get what i try to say?

[Noah-Kennedy]

I do not get thousands of threads when running, merely dozens. I think that this is probably something to do with the worker pool used internally within io_uring: https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/

[Noah-Kennedy]

What kernel version are you running?

[rustMyGod]

kernel version is 5.15.10-1.el7.elrepo.x86_64.

[rustMyGod]

I try to submit two situation:

1. I watched 33 threads work at a time,but these threads live very shortly.it likes one write_at call use one thread so these threads create then died again and again.

 for i in 0..1024 {
            let t = txx.clone();
            tokio_uring::spawn(create(i, t));
        }

these 1024 tasks not all succeed .some of them may interupt when invoke write_at here.

while idx < size_1_g {
        let (_, _) = file.write_at(y, idx).await;//maybe exception occured here.
        idx = idx + 1024;
    }

sorry again for my poor english.

[Noah-Kennedy]

Don't worry about the language barrier, it's fine, I'm used to this ;).

I think that this is the io_uring work queue. I'll run some experiments tonight though to confirm.

[rustMyGod]

@Noah-Kennedy thanks.
i consider that a threadpool is a better way,now " these threads created and destroyed " is not a bug,but can be optimized.
the real trouble is that some of these 1024 tasks can't finished. only 794MB succeed the first time ,and 423M the second time.
pls run and watch this problem. many many thanks.

[rustMyGod]

hi, anything new about this issue?

[Noah-Kennedy]

I haven't had time to investigate this much further sadly, but I suspect that something within io_uring or the kernel is thrashing, possibly multiple things. There are flags for io_uring and the IO ops that I'd like to provide ways to set for this crate that could help with this. It's also likely that there is some amount of thrashing in the page cache that may also be causing problems, since writing tons of files at once is something that the page cache has a lot of problems with. Unfortunately we don't yet support direct IO, which would likely help with this.