Dataset Card for NatureBench

NatureBench is a cross-discipline benchmark of 27 tasks distilled from peer-reviewed Nature-family publications, spanning 6 scientific domains. It is designed to evaluate whether AI coding agents can move beyond reproduction toward discovery: each task asks an agent to solve a real scientific machine-learning problem and is scored against the source paper's reported state of the art.

• 📄 arXiv paper: https://arxiv.org/abs/2606.24530
• 💻 GitHub code repository: https://github.com/FrontisAI/NatureBench
• 🏆 Leaderboard: https://frontisai.github.io/NatureBench/

Dataset Description

NatureBench is built on NatureGym, an automated pipeline that converts a published paper into a containerized task package comprising a task brief, the paper's dataset, a held-out test set with hidden ground truth, and an automated evaluator.

The benchmark draws 27 tasks (88 evaluated instances) from peer-reviewed Nature-family papers published between 2022 and 2025, spanning six scientific domains: cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning. Each task is scored against the source paper's reported state of the art through a SOTA-normalized relative gap, which keeps results comparable across heterogeneous metrics. Agents are evaluated in isolated containers with web search disabled, so a task must be solved from its brief and data rather than by retrieving the paper's original results, and a post-hoc validity judge screens submissions for shortcut solutions.

Dataset Structure

tasks/
└── <case_id>/
    ├── problem/
    ├── evaluation/
    ├── environment/
    ├── licenses/
    └── metadata.json

Path	Description
tasks/<case_id>/problem/	Agent-visible task descriptions and visible input data.
tasks/<case_id>/evaluation/	Evaluator and ground truth; not exposed to the agent during a run.
tasks/<case_id>/environment/	Task-specific containerized environment.
tasks/<case_id>/licenses/	Third-party license notices governing that task's data.
tasks/<case_id>/metadata.json	Task name, domain, compute-resource demand, and per-instance SOTA scores.

How to Use

NatureBench is run with the companion code at https://github.com/FrontisAI/NatureBench.

Note: Some tasks store large datasets as compressed archives (problem/data_archives/*.tar.gz). The official download script in the GitHub repository automatically extracts them into the expected problem/data/ layout. Downloading directly from HuggingFace will leave these archives unexpanded, causing inconsistencies in the task package.

# Download all 27 tasks
python run_naturebench.py --dataset-id FrontisAI/NatureBench --tasks all --download-only

# Download a specific subset
python run_naturebench.py --dataset-id FrontisAI/NatureBench --task-file task-set/gpu_low.txt --download-only

If you prefer to download manually, extract every .tar.gz under problem/data_archives/ into problem/ and remove the data_archives/ directory. See run_naturebench.py for a reference implementation.

See the GitHub repository for installation, agent configuration, and full evaluation instructions.

License

The top-level LICENSE applies only to original NatureBench contributions. Third-party data is governed by the notices in each task's tasks/<case_id>/licenses/ directory.

Citation

If you use NatureBench in your research, please cite our work:

@misc{wang2026naturebench,
  title         = {NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?},
  author        = {Yuru Wang and Lejun Cheng and Yuxin Zuo and Sihang Zeng and Bingxiang He and Che Jiang and Junlin Yang and Yuchong Wang and Kaikai Zhao and Weifeng Huang and Kai Tian and Zhenzhao Yuan and Jincheng Zhong and Weizhi Wang and Ning Ding and Bowen Zhou and Kaiyan Zhang},
  year          = {2026},
  eprint        = {2606.24530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24530}
}