Behavior of nrfiles, openfiles and file_service_type unexpected

[frukto]

Description of the bug:

A combination of nrfiles, openfiles and file_service_type does not work as expected (at least from my perspective). I looks like fio considers only open files when choosing the next file to process. This may also be related to #1676, #1677.
Fio has similar (unexpected) behavior with other file_service_type modes, ie. random:1.

I use a job description like this:

$ cat bug.fio
[job1]
rw=read
nrfiles=10      ; use 10 files
openfiles=1     ; but only keep one open
size=10M        ; so each file is 1M
bs=256K         ; do 4 reads per file
iodepth=1
file_service_type=roundrobin:1
filename_format=__fio/junk.$filenum
directory=.

I would expect fio to do something like

[...]
./__fio/junk.job1.0 open
./__fio/junk.job1.0 read 0 262144
./__fio/junk.job1.0 close
./__fio/junk.job1.1 open
./__fio/junk.job1.1 read 0 262144
./__fio/junk.job1.1 close
./__fio/junk.job1.2 open
[...]

but I see (io.log):

[...]
./__fio/junk.job1.0 open
./__fio/junk.job1.0 read 0 262144
./__fio/junk.job1.0 read 262144 262144
./__fio/junk.job1.0 read 524288 262144
./__fio/junk.job1.0 read 786432 262144
./__fio/junk.job1.0 close
./__fio/junk.job1.1 open
./__fio/junk.job1.1 read 0 262144
./__fio/junk.job1.1 read 262144 262144
./__fio/junk.job1.1 read 524288 262144
./__fio/junk.job1.1 read 786432 262144
./__fio/junk.job1.1 close
./__fio/junk.job1.2 open
[...]

In the debug log (all.log) are a lot of error messages like below. It looks like fio cycles through all files, but fails for all non-open files job1.1 to job1.9 before reaching the open job1.0 again.

[...]
file     1194683 get file ./__fio/junk.0, ref=1
io       1194683 fill: io_u 0x55ce493055c0: off=0x0,len=0x40000,ddir=0,file=./__fio/junk.0
io       1194683 prep: io_u 0x55ce493055c0: off=0x0,len=0x40000,ddir=0,file=./__fio/junk.0
io       1194683 queue: io_u 0x55ce493055c0: off=0x0,len=0x40000,ddir=0,file=./__fio/junk.0
io       1194683 complete: io_u 0x55ce493055c0: off=0x0,len=0x40000,ddir=0,file=./__fio/junk.0
file     1194683 put file ./__fio/junk.0, ref=2
file     1194683 trying file ./__fio/junk.1 200
io       1194683 io_u 0x55ce493055c0, setting file failed
io       1194683 get_io_u failed
file     1194683 trying file ./__fio/junk.2 200
io       1194683 io_u 0x55ce493055c0, setting file failed
io       1194683 get_io_u failed
file     1194683 trying file ./__fio/junk.3 200
io       1194683 io_u 0x55ce493055c0, setting file failed
io       1194683 get_io_u failed
file     1194683 trying file ./__fio/junk.4 200
io       1194683 io_u 0x55ce493055c0, setting file failed
io       1194683 get_io_u failed
[...]

Environment:

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian

$ uname -r
5.15.0-91-generic

fio version: fio-3.28

Reproduction steps

fio bug.fio --write_iolog=io.log --debug=all > all.log

or

fio --name=job1 --rw=read --nrfiles=10 --openfiles=1 --size=10M --bs=256K --iodepth=1 --file_service_type=roundrobin:1 --directory=. --filename_format=__fio/junk.$filenum --write_iolog=io.log --debug=all > all.log