fix duplicate elements in split_between_processes #2781
Conversation
There was a problem hiding this comment.
Thanks! Can you write a test for it over in https://github.com/huggingface/accelerate/blob/main/src/accelerate/test_utils/scripts/test_script.py#L594-L693 with your failing case?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
src/accelerate/state.py
Outdated
| num_samples_per_process = length // self.num_processes | ||
| num_extras = length % self.num_processes |
There was a problem hiding this comment.
How about using the underappreciated divmod to calculate both in a single call:
| num_samples_per_process = length // self.num_processes | |
| num_extras = length % self.num_processes | |
| num_samples_per_process, num_extras = divmod(length, self.num_processes) |
|
@muellerzr @BenjaminBossan Is there anything else that needs to be resolved or checked in this PR? |
|
Nope looks good to me! I'll be running this and a few other PR's locally on my GPUs just to make sure it works fine there, then I'll give a ✅ and merge! |
|
All green! Thanks! |
Fixes #2750
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@muellerzr