Wait for writing in distributed metrics
MEMBER
I agree this fix the problem for the CI where the files are always created in a new and clean temporary directory.

However, in a general setting of a succession of fast distributed operation, the files could already exist from previous metrics runs but one process may still finish before another has even started in which case it would mix results from separate operations.

I feel like the most robust way to solve this is to setup a rendez-vous on the first time we write on files and where each process will test and only finish its operation when it cannot acquire a lock on all the other processes (meaning they all have started).

What do you think?
2020-09-02T13:08:36Z
https://github.com/huggingface/datasets/pull/564#issuecomment-685724313
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4NTcyNDMxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/685724313/reactions
2020-09-02T14:35:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/685724313
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
What do you think of this @thomwolf  ? I check all the locks before finalizing
2020-09-02T13:55:28Z
https://github.com/huggingface/datasets/pull/564#issuecomment-685753784
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4NTc1Mzc4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/685753784/reactions
2020-09-02T13:55:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/685753784
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok on my side @lhoestq (cannot add you as a reviewer)
2020-09-04T09:44:02Z
https://github.com/huggingface/datasets/pull/564#issuecomment-687042726
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4NzA0MjcyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/687042726/reactions
2020-09-04T09:44:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/687042726
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
The test doesn't pass if I add:
```python
            import time
            if self.process_id == 1:
                time.sleep(0.5)
```
right before `self.add_batch` in `Metric.compute`.

I'm investigating why it doesn't work in that case
2020-09-07T08:17:22Z
https://github.com/huggingface/datasets/pull/564#issuecomment-688151256
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4ODE1MTI1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/688151256/reactions
2020-09-07T08:17:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688151256
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It looks like the process 1 runs `_check_all_processes_locks` correctly and then finishes and releases its lock before process 0 even managed to to run `_check_all_processes_locks` correctly.
2020-09-07T08:51:17Z
https://github.com/huggingface/datasets/pull/564#issuecomment-688173690
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4ODE3MzY5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688173690/reactions
2020-09-07T08:51:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688173690
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Strange!
2020-09-07T08:58:03Z
https://github.com/huggingface/datasets/pull/564#issuecomment-688177886
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4ODE3Nzg4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/688177886/reactions
2020-09-07T08:58:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688177886
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
I changed the way the rendez-vous is done @thomwolf , let me know what you think.
The idea is that the master process has an additional lock `rendez_vous_lock` to tell every other process to wait for everyone to be ready before starting to write
2020-09-07T10:31:39Z
https://github.com/huggingface/datasets/pull/564#issuecomment-688233951
https://api.github.com/repos/huggingface/datasets/issues/564
MDEyOklzc3VlQ29tbWVudDY4ODIzMzk1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/688233951/reactions
2020-09-07T10:31:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688233951
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
There were CI bugs where a distributed metric would try to read all the files in process 0 while the other processes haven't started writing.

To fix that I added a custom locking mechanism that waits for the file to exist before trying to read it