Dataloader stuck on multiple GPUs
COLLABORATOR
What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo.
2023-09-14T12:37:12Z
https://github.com/huggingface/datasets/issues/6240#issuecomment-1719371414
https://api.github.com/repos/huggingface/datasets/issues/6240
IC_kwDODunzps5me4aW
https://api.github.com/repos/huggingface/datasets/issues/comments/1719371414/reactions
2023-09-14T12:37:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1719371414
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.
2023-09-14T23:54:42Z
https://github.com/huggingface/datasets/issues/6240#issuecomment-1720299017
https://api.github.com/repos/huggingface/datasets/issues/6240
IC_kwDODunzps5mia4J
https://api.github.com/repos/huggingface/datasets/issues/comments/1720299017/reactions
2023-09-14T23:54:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1720299017
https://avatars.githubusercontent.com/u/40049003?v=4
https://api.github.com/users/kuri54/events{/privacy}
https://api.github.com/users/kuri54/followers
https://api.github.com/users/kuri54/following{/other_user}
https://api.github.com/users/kuri54/gists{/gist_id}
https://github.com/kuri54
kuri54
MDQ6VXNlcjQwMDQ5MDAz
https://api.github.com/users/kuri54/orgs
https://api.github.com/users/kuri54/received_events
https://api.github.com/users/kuri54/repos
https://api.github.com/users/kuri54/starred{/owner}{/repo}
https://api.github.com/users/kuri54/subscriptions
User
https://api.github.com/users/kuri54
public
### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1