Fix memory leak with CTC training script on Chinese languages

[lucky-bai]

This PR fixes a memory leak that occurs during the evaluation phase with the CTC training script when attempting to train a CTC model on a language with a very large vocabulary size. I ran into this issue while training for Chinese, but it may affect other languages as well. The problem arises because all the logits are accumulated on the GPU when running the evaluation, even though it is batched, so all the logits for the entire evaluation set are accumulated in the GPU. This is not too much of a problem for most languages, but it quickly runs out of CUDA memory when the vocabulary size is as large as it is in the case for Chinese.

Some community discussion of this problem here: https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941

The bug may be reproduced using this command:

python run_speech_recognition_ctc.py \
	--dataset_name="mozilla-foundation/common_voice_11_0" \
        --model_name_or_path="facebook/wav2vec2-xls-r-300m" \
	--tokenizer_name_or_path="jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn" \
	--dataset_config_name="yue" \
	--output_dir="./wav2vec2-common_voice-tr-demo" \
	--overwrite_output_dir \
	--num_train_epochs="15" \
	--per_device_train_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--save_steps="400" \
	--eval_steps="100" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \
	--fp16 \
	--group_by_length \
	--push_to_hub \
	--do_eval 

cc @sanchit-gandhi

[sanchit-gandhi]

LGTM - thanks @lucky-bai! cc @kamilakesbi

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for digging into this and fixing!