Modulo Research is developing specialized datasets to help advance empirical alignment research in artificial intelligence. These datasets will not only serve as the foundation for our own follow-up studies but will also be made available to the broader AI safety research community.

You can sign up to be notified when we release future datasets.

Currently Available

FindTheFlaws is a set of datasets that include (1) long-form expert-verified correct solutions and (2) long-form flawed solutions with annotations highlighting specific errors to difficult questions in medicine, physics, chemistry and more. While several of the questions are drawn from existing benchmarks such as GPQA Diamond, it also includes the novel CELS dataset containing detailed expert annotations of LLM responses to difficult questions in surgical medicine, law, and Lojban. The repository includes the datasets presented in the paper, and the scripts used to conduct model evals using UK AISI’s Inspect library.

Datasets Under Development

03/2025 – We’re now finalizing a dataset of textual representations of the research processes followed by high-performing participants in an experiment involving an online research task — for use in improving LLM capability elicitations — and writing up the results of our associated experiments.