FIO reports Illegal byte sequence error

[wangyihanlarry]

Greeting,

I was recently using the FIO to do the data integrity testing, but it always reports Illegal byte sequence error kinds of randomly, I am wondering if my profile was correct set, below are my steps.

1, create 30 volumes, and distribute them randomly to 5 clients
2, on each client, sent IO with following profile, the devices are corrected mapped.

 [test_job]
  rw=write
  numjobs=4
  bs=904k
  ioengine=libaio
  iodepth=4
  verify=pattern
  do_verify=1
  verify_pattern=0xabcd
  size=1G
  filename=/dev/sddc1:/dev/sddb1:/dev/sdcm1:/dev/sdcp1:/dev/sdda1:/dev/sdct1:/dev/sdcs1

3, redistribute the volumes on the 5 clients randomly, the devices mapping ship on each are guaranteed as well, use following profile to validate the content, the only change here is to change the write to read in the rw option.

  [test_job]
  rw=read
  numjobs=4
  bs=904k
  ioengine=libaio
  iodepth=4
  verify=pattern
  do_verify=1
  verify_pattern=0xabcd
  size=1G
  filename=/dev/sddc1:/dev/sddb1:/dev/sdcm1:/dev/sdcp1:/dev/sdda1:/dev/sdct1:/dev/sdcs1

4, here I always met io errors, as in below:

fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: bad pattern block offset 0
pattern: verify failed at file /dev/sddc1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: bad pattern block offset 0
fio: verify type mismatch (64 media, 18 given)
fio: bad pattern block offset 0
pattern: verify failed at file /dev/sddc1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: bad pattern block offset 0
fio: verify type mismatch (64 media, 18 given)
pattern: verify failed at file /dev/sddc1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: verify type mismatch (64 media, 18 given)
pattern: verify failed at file /dev/sddc1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: verify type mismatch (64 media, 18 given)
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: bad pattern block offset 0
pattern: verify failed at file /dev/sddb1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: verify type mismatch (64 media, 18 given)
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: got pattern '00', wanted 'ab'. Bad bits 5
fio: bad pattern block offset 0
fio: bad pattern block offset 0
fio: bad pattern block offset 0
pattern: verify failed at file /dev/sddb1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
pattern: verify failed at file /dev/sddb1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
pattern: verify failed at file /dev/sddb1 offset 134225920, length 4194320 (requested block: offset=134225920, length=925696, flags=8)
fio: verify type mismatch (64 media, 18 given)
fio: verify type mismatch (64 media, 18 given)
fio: verify type mismatch (64 media, 18 given)

I examined the contents of the device with suspicion via dd command, it seems the 1st 128M are written with abcd but I actually want to write the 1G content to each device separated by : as shown in the profile, are there any thing I misused in the profile?

  fio_tmp_1701160406491]$ sudo dd if=/dev/sddc1 bs=1M count=128 | hexdump -C
  00000000  ab cd ab cd ab cd ab cd  ab cd ab cd ab cd ab cd  |................|
  *
  128+0 records in
  128+0 records out
  134217728 bytes (134 MB) copied, 0.529649 s, 253 MB/s
  08000000

[vincentkfu]

See the documentation for size:

https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-size

Fio will divide this size between the available files determined by options such as nrfiles, filename, unless filesize is specified by the job.

Try setting size=7G if you have 7 devices and want 1G each. Also run the debug=io,verify to gain more insight into what fio is doing.

[wangyihanlarry]

one basic question, will this two profiles lead to any data errors which is not relate to the bug of the product, e.g. I if the sequence of the device changes in profile and fio can not evenly divide the size? will this cause a size mismatch in IO and will it lead data mis-compare by mistake？