# HumanEval

- Page: https://shared.btbytes.com/p/humaneval
- Date: 2023-08-26
- Tags: humaneval, paperswithcode

HumanEval is an evaluation harness for the HumanEval Problem solving dataset described in the paper ([Chen et al. 2021](https://arxiv.org/abs/2107.03374)); [arXiv](https://arxiv.org/abs/2107.03374). The code is available on [github](https://github.com/openai/human-eval)

You can see this being used to evaluate new models like here - [Beating GPT-4 on HumanEval with a Fine-Tuned CodeLlama-34B](https://www.phind.com/blog/code-llama-beats-gpt4) by Phind

> We have fine-tuned CodeLlama-34B and CodeLlama-34B-Python on an internal Phind dataset that achieved 67.6% and 69.5% pass@1 on HumanEval, respectively. GPT-4 achieved 67% according to their official technical report in March. To ensure result validity, we applied OpenAI's decontamination methodology to our dataset.

## References

- Chen, Tworek, Jun, Yuan, de Oliveira Pinto, Kaplan, Edwards, Burda, Joseph, Brockman, Ray, Puri, Krueger, Petrov, Khlaaf, Sastry, Mishkin, Chan, Gray, Ryder, Pavlov, Power, Kaiser, Bavarian, Winter, Tillet, Such, Cummings, Plappert, Chantzis, Barnes, Herbert-Voss, Guss, Nichol, Paino, Tezak, Tang, Babuschkin, Balaji, Jain, Saunders, Hesse, Carr, Leike, Achiam, Misra, Morikawa, Radford, Knight, Brundage, Murati, Mayer, Welinder, McGrew, Amodei, McCandlish, Sutskever, Zaremba (2021). [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374).