LamaLab · Benchmark

ChemBench

Are large language models superhuman chemists?

About ChemBench

ChemBench is a comprehensive benchmark for evaluating language and multimodal models on chemical reasoning, knowledge, and intuition — built on the Big Bench format, with scientifically sound parsing and evaluation beyond simple MCQ scoring. Run it with the ChemBench engine or Inspect AI, then submit your results to appear here.

Developed by the Lab for AI for Materials (LamaLab) at the University of Jena and the Helmholtz Institute for Polymers in Energy Applications Jena. Released under the MIT license.

Cite

@article{mirza2024large,
  title   = {Are large language models superhuman chemists?},
  author  = {Adrian Mirza and Nawaf Alampara and Sreekanth Kunchapu and
             Benedict Emoekabu and Aswanth Krishnan and Mara Wilhelmi and
             Macjonathan Okereke and Juliane Eberhardt and Amir Mohammad Elahi and
             Maximilian Greiner and Caroline T. Holick and Tanya Gupta and
             Mehrdad Asgari and Christina Glaubitz and Lea C. Klepsch and
             Yannik Köster and Jakob Meyer and Santiago Miret and Tim Hoffmann and
             Fabian Alexander Kreth and Michael Ringleb and Nicole Roesner and
             Ulrich S. Schubert and Leanne M. Stafast and Dinga Wonanke and
             Michael Pieler and Philippe Schwaller and Kevin Maik Jablonka},
  year    = {2024},
  journal = {arXiv preprint arXiv:2404.01475}
}