Fix AssertionError in clip conversion script

[ydshieh]

What does this PR do?

Fix #30037(AssertionError) part. The failure comes from the fact that the prepared inputs in not really meaningful (not processed by clip tokenizer).

The conversion is still failing with saftensors error

ValueError: You are trying to save a non contiguous tensor

I will ping people once this PR merged.

Full log

Traceback (most recent call last):
  File "convert_clip_original_pytorch_to_hf.py", line 146, in <module>
    convert_clip_checkpoint(args.checkpoint_path, args.pytorch_dump_folder_path, args.config_path)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "convert_clip_original_pytorch_to_hf.py", line 136, in convert_clip_checkpoint
    hf_model.save_pretrained(pytorch_dump_folder_path)
  File "/transformers/src/transformers/modeling_utils.py", line 2584, in save_pretrained
    safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 284, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 488, in _flatten
    return {
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 492, in <dictcomp>
    "data": _tobytes(v, k),
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 406, in _tobytes
    raise ValueError(
ValueError: You are trying to save a non contiguous tensor: `visual_projection.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Awesome - thanks for digging into this and fixing!