[distributed env] potentially unsafe parallel execution
MEMBER
You can pass the same `experiment_id` for all the metrics of the same group, and use another `experiment_id` for the other groups.
Maybe we can add an environment variable that sets the default value for `experiment_id` ? What do you think ?
2021-03-01T11:13:19Z
https://github.com/huggingface/datasets/issues/1956#issuecomment-787867739
https://api.github.com/repos/huggingface/datasets/issues/1956
MDEyOklzc3VlQ29tbWVudDc4Nzg2NzczOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/787867739/reactions
2021-03-01T11:13:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/787867739
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Ah, you're absolutely correct, @lhoestq - it's exactly the equivalent of the shared secret. Thank you!
2021-03-01T17:24:41Z
https://github.com/huggingface/datasets/issues/1956#issuecomment-788126061
https://api.github.com/repos/huggingface/datasets/issues/1956
MDEyOklzc3VlQ29tbWVudDc4ODEyNjA2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/788126061/reactions
2021-03-01T17:24:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/788126061
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you