<jupyter_start><jupyter_text>Human Eval in Google Colab[Original Openai github source](https://github.com/openai/human-eval)[Paper](https://arxiv.org/pdf/2107.03374.pdf)  Import modules<jupyter_code>from data import write_jsonl, read_problems
problems = read_problems()

num_samples_per_task = 200
generate_one_completion = {"task_id": "test/0", 
                           "prompt": "def return1():\n", 
                           "canonical_solution": "    return 1", 
                           "test": "def check(candidate):\n    assert candidate() == 1", 
                           "entry_point": "return1"}
samples = [
    generate_one_completion
    for task_id in problems
    for _ in range(num_samples_per_task)
]

write_jsonl("samples.jsonl", samples)
len(problems)
problems['HumanEval/1']
## problems given by openai

import json
with open('problems.json', 'w') as fp:
    json.dump(problems, fp)<jupyter_output><empty_output><jupyter_text>update execution.py Note**This program exists to run untrusted model-generated code. Users are strongly encouraged not to do so outside of a robust security sandbox. The execution call in execution.py is deliberately commented out to ensure users read this disclaimer before running code in a potentially unsafe manner. See the comment in execution.py for more information and instructions.**To continue, you need to manually update `execution.py`, specifically uncomment `exec(check_program, exec_globals)`<jupyter_code>!python evaluate_functional_correctness.py ../data/example_samples.jsonl --problem_file=../data/example_problem.jsonl<jupyter_output><empty_output>