🧪 ChemBench Leaderboard

Welcome to ChemBench, a comprehensive benchmark for evaluating language models on Chemical Reasoning, Knowledge and Intuition.

Leaderboard Version

Dataset Version: 1.0.0

Leaderboard Version: 1.0.4

{

"headers": [
- "Model",
- "Parameters (B)",
- "Date Published",
- "Open Source",
- "Open Dataset",
- "MoE",
- "Tool Use",
- "Reasoning",
- "Overall Score",
- "Analytical Chemistry",
- "Chemical Preference",
- "General Chemistry",
- "Inorganic Chemistry",
- "Materials Science",
- "Organic Chemistry",
- "Physical Chemistry",
- "Technical Chemistry",
- "Toxicity and Safety"
],
"data": [
- [
  - "o1-preview",
  - null,
  - "2024-09-12",
  - false,
  - false,
  - null,
  - false,
  - true,
  - 0.64,
  - 0.62,
  - 0.56,
  - 0.93,
  - 0.9,
  - 0.73,
  - 0.83,
  - 0.85,
  - 0.85,
  - 0.48
  ],
- [
  - "Claude-3.5 (Sonnet)",
  - null,
  - "2024-06-20",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.63,
  - 0.57,
  - 0.58,
  - 0.83,
  - 0.84,
  - 0.71,
  - 0.83,
  - 0.77,
  - 0.85,
  - 0.44
  ],
- [
  - "Claude-3.5 (Sonnet) React",
  - null,
  - "2024-06-20",
  - false,
  - false,
  - null,
  - true,
  - false,
  - 0.62,
  - 0.58,
  - 0.6,
  - 0.87,
  - 0.8,
  - 0.68,
  - 0.84,
  - 0.76,
  - 0.8,
  - 0.41
  ],
- [
  - "GPT-4o",
  - null,
  - "2024-05-13",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.61,
  - 0.56,
  - 0.59,
  - 0.81,
  - 0.8,
  - 0.75,
  - 0.76,
  - 0.72,
  - 0.75,
  - 0.44
  ],
- [
  - "Llama-3.1-405B-Instruct",
  - 405,
  - "2024-07-23",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.58,
  - 0.51,
  - 0.54,
  - 0.79,
  - 0.77,
  - 0.65,
  - 0.76,
  - 0.71,
  - 0.7,
  - 0.42
  ],
- [
  - "Mistral-Large-2",
  - 123,
  - "2024-07-24",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.57,
  - 0.48,
  - 0.55,
  - 0.79,
  - 0.79,
  - 0.67,
  - 0.73,
  - 0.69,
  - 0.68,
  - 0.4
  ],
- [
  - "Claude-3 (Opus)",
  - null,
  - "2024-03-04",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.57,
  - 0.47,
  - 0.57,
  - 0.77,
  - 0.79,
  - 0.63,
  - 0.7,
  - 0.65,
  - 0.7,
  - 0.41
  ],
- [
  - "PaperQA2",
  - null,
  - "2024-09-11",
  - null,
  - null,
  - null,
  - true,
  - false,
  - 0.57,
  - 0.46,
  - 0.56,
  - 0.72,
  - 0.74,
  - 0.69,
  - 0.68,
  - 0.68,
  - 0.7,
  - 0.42
  ],
- [
  - "Llama-3.1-70B-Instruct",
  - 70,
  - "2024-07-23",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.53,
  - 0.41,
  - 0.52,
  - 0.69,
  - 0.77,
  - 0.67,
  - 0.66,
  - 0.64,
  - 0.65,
  - 0.38
  ],
- [
  - "Qwen-2.5-32B",
  - null,
  - null,
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.53,
  - 0.43,
  - 0.53,
  - 0.67,
  - 0.68,
  - 0.62,
  - 0.64,
  - 0.65,
  - 0.57,
  - 0.38
  ],
- [
  - "Llama-3-70B-Instruct",
  - 70,
  - "2024-04-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.52,
  - 0.41,
  - 0.53,
  - 0.6,
  - 0.66,
  - 0.63,
  - 0.63,
  - 0.59,
  - 0.62,
  - 0.37
  ],
- [
  - "Llama-3-70B-Instruct (Temperature 1.0)",
  - 70,
  - "2024-04-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.52,
  - 0.38,
  - 0.53,
  - 0.6,
  - 0.68,
  - 0.62,
  - 0.63,
  - 0.6,
  - 0.6,
  - 0.37
  ],
- [
  - "Llama-3.1-70B-Instruct (Temperature 1.0)",
  - 70,
  - "2024-07-23",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.51,
  - 0.37,
  - 0.54,
  - 0.66,
  - 0.7,
  - 0.65,
  - 0.55,
  - 0.56,
  - 0.55,
  - 0.39
  ],
- [
  - "GPT-4o React",
  - null,
  - "2024-05-13",
  - false,
  - false,
  - null,
  - true,
  - false,
  - 0.51,
  - 0.47,
  - 0.42,
  - 0.76,
  - 0.73,
  - 0.56,
  - 0.72,
  - 0.6,
  - 0.72,
  - 0.37
  ],
- [
  - "Gemma-2-9B-it",
  - 9,
  - "2024-06-27",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.48,
  - 0.32,
  - 0.55,
  - 0.54,
  - 0.55,
  - 0.52,
  - 0.56,
  - 0.48,
  - 0.52,
  - 0.34
  ],
- [
  - "Gemma-2-9B-it (Temperature 1.0)",
  - 9,
  - "2024-06-27",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.48,
  - 0.29,
  - 0.56,
  - 0.56,
  - 0.54,
  - 0.5,
  - 0.55,
  - 0.47,
  - 0.48,
  - 0.34
  ],
- [
  - "Phi-3-Medium-4k-Instruct",
  - 14,
  - "2024-05-21",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.47,
  - 0.34,
  - 0.53,
  - 0.48,
  - 0.63,
  - 0.55,
  - 0.56,
  - 0.46,
  - 0.55,
  - 0.33
  ],
- [
  - "Claude-2",
  - null,
  - "2023-07-11",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.47,
  - 0.38,
  - 0.51,
  - 0.5,
  - 0.61,
  - 0.46,
  - 0.59,
  - 0.5,
  - 0.48,
  - 0.33
  ],
- [
  - "Llama-3.1-8B-Instruct",
  - 8,
  - "2024-07-23",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.47,
  - 0.39,
  - 0.53,
  - 0.5,
  - 0.5,
  - 0.4,
  - 0.58,
  - 0.51,
  - 0.45,
  - 0.33
  ],
- [
  - "GPT-3.5 Turbo",
  - null,
  - "2023-11-06",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.47,
  - 0.38,
  - 0.53,
  - 0.49,
  - 0.54,
  - 0.48,
  - 0.59,
  - 0.4,
  - 0.4,
  - 0.31
  ],
- [
  - "Llama-3.1-8B-Instruct (Temperature 1.0)",
  - 8,
  - "2024-07-23",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.46,
  - 0.36,
  - 0.52,
  - 0.53,
  - 0.48,
  - 0.42,
  - 0.58,
  - 0.42,
  - 0.4,
  - 0.32
  ],
- [
  - "Llama-3-8B-Instruct (Temperature 1.0)",
  - 8,
  - "2024-04-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.46,
  - 0.4,
  - 0.52,
  - 0.44,
  - 0.54,
  - 0.45,
  - 0.55,
  - 0.35,
  - 0.62,
  - 0.32
  ],
- [
  - "Llama-3-8B-Instruct",
  - 8,
  - "2024-04-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.46,
  - 0.41,
  - 0.52,
  - 0.44,
  - 0.49,
  - 0.42,
  - 0.56,
  - 0.37,
  - 0.6,
  - 0.32
  ],
- [
  - "Gemini-Pro",
  - null,
  - "2024-06-07",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.45,
  - 0.39,
  - 0.5,
  - 0.48,
  - 0.47,
  - 0.5,
  - 0.57,
  - 0.45,
  - 0.48,
  - 0.31
  ],
- [
  - "Command-R+",
  - 104,
  - "2024-04-04",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.45,
  - 0.34,
  - 0.51,
  - 0.5,
  - 0.52,
  - 0.46,
  - 0.55,
  - 0.33,
  - 0.5,
  - 0.31
  ],
- [
  - "Mixtral-8x7b-Instruct",
  - 47,
  - "2023-12-11",
  - true,
  - false,
  - true,
  - false,
  - false,
  - 0.42,
  - 0.27,
  - 0.54,
  - 0.42,
  - 0.55,
  - 0.42,
  - 0.48,
  - 0.33,
  - 0.32,
  - 0.27
  ],
- [
  - "Mixtral-8x7b-Instruct (Temperature 1.0)",
  - 47,
  - "2023-12-11",
  - true,
  - false,
  - true,
  - false,
  - false,
  - 0.42,
  - 0.28,
  - 0.52,
  - 0.45,
  - 0.51,
  - 0.4,
  - 0.47,
  - 0.35,
  - 0.32,
  - 0.27
  ],
- [
  - "GPT-4",
  - null,
  - "2023-03-14",
  - false,
  - false,
  - null,
  - false,
  - false,
  - 0.41,
  - 0.43,
  - 0.16,
  - 0.7,
  - 0.7,
  - 0.61,
  - 0.68,
  - 0.64,
  - 0.7,
  - 0.41
  ],
- [
  - "Llama-2-70B Chat",
  - 70,
  - "2023-07-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.27,
  - 0.07,
  - 0.49,
  - 0.13,
  - 0.22,
  - 0.18,
  - 0.15,
  - 0.17,
  - 0.12,
  - 0.14
  ],
- [
  - "Llama-2-13B Chat",
  - 13,
  - "2023-07-18",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.26,
  - 0.09,
  - 0.48,
  - 0.11,
  - 0.27,
  - 0.1,
  - 0.15,
  - 0.15,
  - 0.1,
  - 0.1
  ],
- [
  - "olympiad",
  - null,
  - null,
  - false,
  - false,
  - false,
  - false,
  - false,
  - 0.22,
  - 0.16,
  - 0.37,
  - 0.11,
  - 0.22,
  - 0.19,
  - 0.18,
  - 0.08,
  - 0.25,
  - 0.09
  ],
- [
  - "Gemma-1.1-7B-it",
  - 7,
  - "2024-02-21",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.19,
  - 0.21,
  - 0,
  - 0.34,
  - 0.41,
  - 0.36,
  - 0.38,
  - 0.29,
  - 0.38,
  - 0.23
  ],
- [
  - "Gemma-1.1-7B-it (Temperature 1.0)",
  - 7,
  - "2024-02-21",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.19,
  - 0.21,
  - 0.01,
  - 0.35,
  - 0.41,
  - 0.36,
  - 0.37,
  - 0.3,
  - 0.38,
  - 0.22
  ],
- [
  - "Llama-2-7b-chat-hf ",
  - null,
  - null,
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.04,
  - 0.01,
  - 0,
  - 0.02,
  - 0,
  - 0,
  - 0.02,
  - 0.02,
  - 0.02,
  - 0.01
  ],
- [
  - "Galactica-120b",
  - 120,
  - "2022-11-01",
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.02,
  - 0,
  - 0,
  - 0.05,
  - 0.05,
  - 0,
  - 0.01,
  - 0.06,
  - 0,
  - 0.02
  ],
- [
  - "gsm8k",
  - null,
  - null,
  - true,
  - false,
  - false,
  - false,
  - false,
  - 0.01,
  - 0.01,
  - 0,
  - 0.01,
  - 0.04,
  - 0,
  - 0.01,
  - 0.01,
  - 0.02,
  - 0
  ]
],
"metadata": null

}

Welcome to ChemBench 🧪

ChemBench is a comprehensive toolkit for benchmarking language and multimodal models in chemistry tasks. ChemBench is designed to be modular and extensible, allowing users to easily add new datasets, models, and evaluation metrics.

Key Features 🔑

Standardized Benchmarking: Built on the proven Big Bench format
Universal Model Support: Seamless integration with LiteLLM backend
Scientifically sound parsing and evaluation: ChemBench goes beyond simple MCQ evaluation, supports floating point answers (also in scientific notation). We also support different extraction techniques, including LLM-based extraction.
Extensible: ChemBench is designed to be easily extensible. Prompts, models, extraction, and evaluation methods can be easily added and customized via a pipeline system.
Easy to use: ChemBench is designed to be easy to use. We provide a simple interface you can use to get started testing your models with just a few lines of code.
Caching and parallelization: ChemBench can cache results and can parallelize inference to speed up benchmarking.
Refusal detection: ChemBench can detect when a model refuses to answer a question, handle it, and report it in the final results.

Getting Started 🚀

Start benchmarking your models with just a few lines of code. Here's how you can get started:

from chembench.evaluate import ChemBenchmark
from chembench.prompter import PrompterBuilder
from chembench.utilis import enable_logging
from dotenv import load_dotenv

load_dotenv(".env")
enable_logging()
benchmark = ChemBenchmark.from_huggingface()

model = "openai/gpt-4"  
prompter = PrompterBuilder.from_model_object(
    model=model,
)
results = benchmark.bench(
    prompter,
)
benchmark.submit(results)

For more details, check out our documentation

Scientific Foundation 📚

ChemBench has been validated through peer review and extensive testing. For detailed methodology, check our papers:

ChemBench is open-source software licensed under the MIT license. Its development is led by the Lab for AI for Materials (LamaLab) at the University of Jena and the Helmholtz Institute for Polymers in Energy Applications Jena.

Connect With Us 🔗

GitHub Twitter