nlp.load_dataset is not safe for multi processes when loading from local files
MEMBER
I'll take a look!
2020-08-31T08:05:39Z
https://github.com/huggingface/datasets/issues/543#issuecomment-683631541
https://api.github.com/repos/huggingface/datasets/issues/543
MDEyOklzc3VlQ29tbWVudDY4MzYzMTU0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/683631541/reactions
2020-08-31T08:05:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683631541
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
Loading from local files, e.g., `dataset = nlp.load_dataset('csv', data_files=['file_1.csv', 'file_2.csv'])`
concurrently from multiple processes, will raise `FileExistsError` from builder's line 430, https://github.com/huggingface/nlp/blob/6655008c738cb613c522deb3bd18e35a67b2a7e5/src/nlp/builder.py#L423-L438

Likely because multiple processes step into download_and_prepare, https://github.com/huggingface/nlp/blob/6655008c738cb613c522deb3bd18e35a67b2a7e5/src/nlp/load.py#L550-L554

This can happen when launching distributed training with commands like `python -m torch.distributed.launch --nproc_per_node 4` on a new collection of files never loaded before.

I can create a PR that puts in some file locks. It would be helpful if I can be informed of the convention for naming and placement of the lock.