Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
COLLABORATOR
[This doc section](https://huggingface.co/docs/datasets/main/en/process#multiprocessing) explains how to modify the script to avoid this error.
2023-11-23T15:29:03Z
https://github.com/huggingface/datasets/issues/6435#issuecomment-1824617283
https://api.github.com/repos/huggingface/datasets/issues/6435
IC_kwDODunzps5swXND
https://api.github.com/repos/huggingface/datasets/issues/comments/1824617283/reactions
2023-11-23T15:29:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1824617283
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko thank you very much, i'll check it
2023-11-23T16:55:39Z
https://github.com/huggingface/datasets/issues/6435#issuecomment-1824726414
https://api.github.com/repos/huggingface/datasets/issues/6435
IC_kwDODunzps5swx2O
https://api.github.com/repos/huggingface/datasets/issues/comments/1824726414/reactions
2023-11-23T16:55:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1824726414
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
NONE
@mariosasko no it does not

`Dataset.filter() got an unexpected keyword argument 'with_rank'`
2024-01-06T23:44:42Z
https://github.com/huggingface/datasets/issues/6435#issuecomment-1879879391
https://api.github.com/repos/huggingface/datasets/issues/6435
IC_kwDODunzps5wDK7f
https://api.github.com/repos/huggingface/datasets/issues/comments/1879879391/reactions
2024-01-06T23:44:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1879879391
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
### Describe the bug

1. I ran dataset mapping with `num_proc=6` in it and got this error:
`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method`

I can't actually find a way to run multi-GPU dataset mapping. Can you help?

### Steps to reproduce the bug

1. Rund SDXL training with `num_proc=6`: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py

### Expected behavior

Should work well

### Environment info

6x A100 SXM, Linux