Are large language models superhuman chemists? — mentioned by kjappelbaum

Wed, 06 Nov 2024 21:06:07 GMT

We've released a new version of ChemBench (https://arxiv.org/abs/2404.01475), a framework measuring LLMs' chemistry capabilities against human experts. We Evaluated 2,788 expert-curated chemistry questions across undergrad/grad topics and provide a human baseline from 19 chemistry experts (mostly MS/PhD level). Leading LLMs (like o1, Claude 3.5) significantly outperformed human experts on average Performance varies by topic: - Strong: General chemistry, calculations - Weak: Safety/toxicity, analytical chemistry Notable: Models excel at textbook problems but struggle with reasoning that…

Paper and discussion history. A mention is not an endorsement.

Are large language models superhuman chemists? — HN submission

Wed, 06 Nov 2024 21:06:07 GMT

ChemBench: Evaluating LLMs Against Expert Chemists [New Results]

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Are large language models superhuman chemists? · HackerPapers discussions

Are large language models superhuman chemists? — mentioned by kjappelbaum

Are large language models superhuman chemists? — HN submission