accelerate test does not work when using 3 GPUs

[Trojaner]

System Info

accelerate 0.30.1

Kernel: linux-5.19.0-1010-nvidia-lowlatency-x86_64-with-glibc2.35
OS: Ubuntu 22.04

Hardware
AMD Ryzen 7 3700x
1x NVIDIA RTX 3060 12GB
1x NVIDIA RTX 3090 24GB
1x NVIDIA RTX 4080 16GB
48GB DDR4 RAM

Software
Python 3.10.1
PyTorch 2.2.0+cu121
Numpy 1.26.4

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

• accelerate config
  • Set GPU count (num_processes) to 3
  • Other configurations such as DeepSpeed or FSDP do not seem to matter
  • Neither does mixed precision matter
• accelerate test
  • Crash @

    accelerate/src/accelerate/test_utils/scripts/test_script.py

    Line 189 in aa21174

    dl = DataLoader(range(length), batch_size=8)

Expected behavior

There is no way of running accelerate test without setting num_processes to 1, 2, 4 etc. since split_batches is enabled and batch_size is hardcoded to 8.

Its expected that accelerate test works regardless of GPU / process count