Measuring Coding Challenge Competence With APPS · HackerPapers discussions

Measuring Coding Challenge Competence With APPS — mentioned by nopinsight

Mon, 04 Mar 2024 16:51:14 GMT

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation results (more datasets than in the blog post) https://twitter.com/karinanguyen_/status/1764666528220557320 APPS dataset: https://huggingface.co/datasets/codeparrot/apps APPS dataset…

Paper and discussion history. A mention is not an endorsement.

Measuring Coding Challenge Competence With APPS — mentioned by nopinsight

Fri, 24 Mar 2023 15:12:07 GMT

By the late 2020s, it's entirely possible that a "weaker" AGI will emerge. Consequently, there may be much less need for senior developers as well. However, that could be among the least of our concerns if we cannot reliably align AI with human interests by then. Date Weakly General AI is Publicly Known https://www.metaculus.com/questions/3479/date-weakly-general... Date of Artificial General Intelligence https://www.metaculus.com/questions/5121/date-of-artificial-... The latter includes this criterion: "Able to get top-1 strict accuracy of at least 90.0% on interview-level problems found in…

Paper and discussion history. A mention is not an endorsement.

Measuring Coding Challenge Competence With APPS — HN submission

Tue, 25 May 2021 02:42:19 GMT

Can Transformers Crack the Coding Interview?

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.