when set nr_files and openfiles simultaneously, performance will decrease significantly

[zqs-Oppenauer]

Please acknowledge the following before creating a ticket

• I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:
I have simultaneously set the openfiles and nr_files parameters. Compared to only setting nr_files, I noticed a significant performance decline. Additionally, I found that this situation only occurs when nr_files is greater than twice the value of openfiles.

Environment: Debian GNU/Linux 10
fio version: fio-3.35

Reproduction steps
I use libaio engine and set nr_files = 2 firstly,

[global]
ioengine=libaio
direct=1

[job]
name=sync_test
iodepth=32
filesize=4G
rw=write
bs=4k
nr_files=2
numjobs=1
directory=/data00/fio/

then, I set nr_files=4 and openfiles=2, the performance decreases significantly

[global]
ioengine=libaio
direct=1

[job]
name=sync_test
iodepth=32
filesize=4G
rw=write
bs=4k
nrfiles=4
openfiles=2
numjobs=1
directory=/data00/fio/

the result as below.

I notice that the main difference is IO depths in fio report. In the second fio test, one half of IO depths is 1 or 2, but in the first fio test, the IO depths is almost 32.

[vincentkfu]

When you limit the number of files open at one time with openfiles fio has to do additional work to close and re-open files. So it's not surprising that performance is worse.

Note that you have an identification problem in your comparison above. We can't tell if the performance difference is due to nrfiles or openfiles because between the two jobs you alter both variables. In order to have better evidence of an effect due to openfiles you should set nrfiles in both jobs to 4.

[zqs-Oppenauer]

thanks for your reply!
I set nrfiles to 4 in both jobs and test again, it seems that the performance is also worse.

• the first job, set nrfiles to 4 and openfiles to 2

[global]
ioengine=libaio
direct=1

[job]
name=sync_test
iodepth=32
filesize=4G
rw=write
bs=4k
nr_files=4
openfiles=2
numjobs=1
directory=/data00/fio/

the result as below

• the second job, only set nrfiles to 4

[global]
ioengine=libaio
direct=1

[job]
name=sync_test
iodepth=32
filesize=4G
rw=write
bs=4k
nr_files=4
numjobs=1
directory=/data00/fio/

the result as below

I notice that when set openfiles to 2, bw is always worse, I think the additional work to close and re-open files may not always influence fio job?

[vincentkfu]

You can also try running with --debug=file and compare the output between the two jobs to gain some insight into what is happening. Try making the job as small as possible as this may produce a lot of output. You may also wish to try --debug=file,io.