Add OpenAI's pass@k code evaluation metric
MEMBER
> The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in datasets?

It should work normally, but feel free to test it.
There is some documentation about using metrics in a distributed setup that uses multiprocessing [here](https://huggingface.co/docs/datasets/loading.html?highlight=rank#distributed-setup)
You can test to spawn several processes where each process would load the metric. Then in each process you add some references/predictions to the metric. Finally you call compute() in each process and on process 0 it should return the result on all the references/predictions

Let me know if you have questions or if I can help
2021-09-16T17:15:49Z
https://github.com/huggingface/datasets/pull/2916#issuecomment-921084432
https://api.github.com/repos/huggingface/datasets/issues/2916
IC_kwDODunzps425qIQ
https://api.github.com/repos/huggingface/datasets/issues/comments/921084432/reactions
2021-09-16T17:16:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/921084432
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Is there a good way to debug the Windows tests? I suspect it is an issue with `multiprocessing`, but I can't see the error messages.
2021-09-18T07:58:58Z
https://github.com/huggingface/datasets/pull/2916#issuecomment-922236574
https://api.github.com/repos/huggingface/datasets/issues/2916
IC_kwDODunzps42-Dae
https://api.github.com/repos/huggingface/datasets/issues/comments/922236574/reactions
2021-09-18T07:58:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/922236574
https://avatars.githubusercontent.com/u/8264887?v=4
https://api.github.com/users/lvwerra/events{/privacy}
https://api.github.com/users/lvwerra/followers
https://api.github.com/users/lvwerra/following{/other_user}
https://api.github.com/users/lvwerra/gists{/gist_id}
https://github.com/lvwerra
lvwerra
MDQ6VXNlcjgyNjQ4ODc=
https://api.github.com/users/lvwerra/orgs
https://api.github.com/users/lvwerra/received_events
https://api.github.com/users/lvwerra/repos
https://api.github.com/users/lvwerra/starred{/owner}{/repo}
https://api.github.com/users/lvwerra/subscriptions
User
https://api.github.com/users/lvwerra
public
MEMBER
Indeed it has an issue on windows.
In your example it's supposed to output
```python
{'pass@1': 0.5, 'pass@2': 1.0}
```
but it gets
```python
{'pass@1': 0.0, 'pass@2': 0.0}
```

I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want
2021-09-23T12:42:36Z
https://github.com/huggingface/datasets/pull/2916#issuecomment-925773525
https://api.github.com/repos/huggingface/datasets/issues/2916
IC_kwDODunzps43Li7V
https://api.github.com/repos/huggingface/datasets/issues/comments/925773525/reactions
2021-09-23T12:42:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/925773525
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> I'm not on my windows machine today so I can't take a look at it. I can dive into it early next week if you want

That would be great - unfortunately I have no access to a windows machine at the moment. I am quite sure it is an issue with in exectue.py because of multiprocessing.

2021-09-27T12:58:59Z
https://github.com/huggingface/datasets/pull/2916#issuecomment-927849197
https://api.github.com/repos/huggingface/datasets/issues/2916
IC_kwDODunzps43Tdrt
https://api.github.com/repos/huggingface/datasets/issues/comments/927849197/reactions
2021-09-27T12:58:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/927849197
https://avatars.githubusercontent.com/u/8264887?v=4
https://api.github.com/users/lvwerra/events{/privacy}
https://api.github.com/users/lvwerra/followers
https://api.github.com/users/lvwerra/following{/other_user}
https://api.github.com/users/lvwerra/gists{/gist_id}
https://github.com/lvwerra
lvwerra
MDQ6VXNlcjgyNjQ4ODc=
https://api.github.com/users/lvwerra/orgs
https://api.github.com/users/lvwerra/received_events
https://api.github.com/users/lvwerra/repos
https://api.github.com/users/lvwerra/starred{/owner}{/repo}
https://api.github.com/users/lvwerra/subscriptions
User
https://api.github.com/users/lvwerra
public
This PR introduces the `code_eval` metric which implements [OpenAI's code evaluation harness](https://github.com/openai/human-eval) introduced in the [Codex paper](https://arxiv.org/abs/2107.03374). It is heavily based on the original implementation and just adapts the interface to follow the `predictions`/`references` convention.

The addition of this metric should enable the evaluation against the code evaluation datasets added in #2897 and #2893.

A few open questions:

- The implementation makes heavy use of multiprocessing which this PR does not touch. Is this conflicting with multiprocessing natively integrated in `datasets`?
- This metric executes generated Python code and as such it poses dangers of executing malicious code. OpenAI addresses this issue by 1) commenting the `exec` call in the code so the user has to actively uncomment it and read the warning and 2) suggests using a sandbox environment (gVisor container). Should we add a similar safeguard? E.g. a prompt that needs to be answered when initialising the metric? Or at least a warning message?
- Naming: the implementation sticks to the `predictions`/`references` naming, however, the references are not reference solutions but unittest to test the solution. While reference solutions are also available they are not used. Should the naming be adapted?