[Bug]: Llama 3 - Out of memory - RTX 4060 TI

[savi8sant8s]

Your current environment

vllm 0.4.2

🐛 Describe the bug

Hello. Would anyone have an example of how I could run Llama 3 on an NVIDIA RTX 4060 TI 16GB? I tried to do inference with this model https://huggingface.co/rhaymison/Llama-3-portuguese-Tom-cat-8b-instruct and a lora adapter but it always gives out of memory. I enabled enforce eager, increased the max GPU memory to 1, even reduced the size of the SWAP but it still overflowed the memory.

[robertgshaw2-neuralmagic]

Llama3 at fp16 > 16GB of RAM.

Try using a quantized model.