How to specify which gpu to use? #691
-
If I have multiple GPUs, how can I specify which GPU to use individually? Previously, I used exp. if i want to specify the model_kwargs = {"torch_dtype": torch.float16, 'device_map': 'sequential'} Then, I manually set the available memory of
|
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 3 replies
-
Use |
Beta Was this translation helpful? Give feedback.
-
Thank you for the suggestion. |
Beta Was this translation helpful? Give feedback.
-
In addition to @gesanqiu's method, you can also use Ray's placement group feature for this. Feel free to follow up on discussion if you meet any issues or have any other suggestions! |
Beta Was this translation helpful? Give feedback.
-
A simple solution is adding this to your code: Then, only GPU-0,1 are visible. |
Beta Was this translation helpful? Give feedback.
-
export CUDA_VISIBLE_DEVICES=1 |
Beta Was this translation helpful? Give feedback.
-
Hi |
Beta Was this translation helpful? Give feedback.
-
Any solution for this |
Beta Was this translation helpful? Give feedback.
In addition to @gesanqiu's method, you can also use Ray's placement group feature for this. Feel free to follow up on discussion if you meet any issues or have any other suggestions!