[METRICS, breaking] Refactor caching behavior, pickle/cloudpickle metrics and dataset, add tests on metrics
MEMBER
(test failure is unrelated)
2020-08-20T07:47:24Z
https://github.com/huggingface/datasets/pull/518#issuecomment-677434575
https://api.github.com/repos/huggingface/datasets/issues/518
MDEyOklzc3VlQ29tbWVudDY3NzQzNDU3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/677434575/reactions
2020-08-20T07:47:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/677434575
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
As discussed with @thomwolf merging since the hyperparameter-search has been merged in transformers.
2020-08-24T16:01:35Z
https://github.com/huggingface/datasets/pull/518#issuecomment-679216421
https://api.github.com/repos/huggingface/datasets/issues/518
MDEyOklzc3VlQ29tbWVudDY3OTIxNjQyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/679216421/reactions
2020-08-24T16:01:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/679216421
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
Move the acquisition of the filelock at a later stage during metrics processing so it can be pickled/cloudpickled after instantiation.

Also add some tests on pickling, concurrent but separate metric instances and concurrent and distributed metric instances.

Changes significantly the caching behavior for the metrics:
- if the metric is used in a non-distributed setup (most common case) we try to find a free cache file using UUID instead of asking for an `experiment_id` if we can't lock the cache file this allows to use several instances of the same metrics in parallel.
- if the metrics is used in a distributed setup we ask for an `experiment_id` if we can't lock the cache file (because all the nodes need to have related cache file names for the final sync.
- after the computation, we free the locks and delete all the cache files.

Breaking: Some arguments for Metrics initialization have been removed for simplicity (`version`...) and some have been renamed for consistency with the rest of the library (`in_memory` => `keep_in_memory`).

Also remove the `_has_transformers` detection in utils to avoid importing transformers everytime during loading.