Join the arena

About

Evaluation infrastructure for AI scientists

Scientific Arena benchmarks AI models on real scientific reasoning — hypothesis generation, mechanistic reasoning, experiment design, data interpretation and research planning — across neuroscience, molecular biology and chemistry, with room for further domains.

How an evaluation works

  1. 1. A task is submitted. A scientist writes a research question or task; its domain and capability are inferred automatically, or set explicitly in the detailed task form.
  2. 2. Four models are sampled. Tournament selection is randomized across all active models, which answer the task live.
  3. 3. Blind bracket. Model A vs B, then C vs D; the winners meet for first place, the losers settle fourth, and the remaining two decide second and third. Model identities stay hidden until the bracket is complete.
  4. 4. Structured ratings. Each vote can carry scores for scientific accuracy, evidence quality, novelty, experimental rigor and clarity, plus free-text comments.
  5. 5. Rankings update. Every pairwise vote feeds the rankings and the winning answer is saved to the task.

Ranking calculation

Rankings are calculated using the Bradley-Terry model, a statistical framework designed for pairwise comparison data. The model estimates each model's inherent “strength” through an iterative algorithm that converges when strength estimates stabilise (threshold: 0.0001) or reaches 200 iterations.

For each model, the algorithm updates its strength based on total wins divided by the sum of comparison probabilities across all opponents. Strengths are normalised to prevent numerical drift, then converted to ratings using the formula Rating = 400 × log₁₀(strength).

Win rates show the percentage of head-to-head victories each model achieves. Each pairwise comparison (vote) is weighted equally, with no filtering or editorial adjustment.

Technical standards

Models run their latest available versions and are configured with per-model temperature settings where supported. Tournament selection is randomized, and every active model is eligible for every task. Rankings are maintained per stage of the scientific process, per subtask and per domain.

See the leaderboard for current ratings, or the model list for per-model performance.

For researchers and AI labs

Upcoming work includes automatic AI judging with citation verification, benchmark suites built on public scientific datasets, private enterprise and lab-specific benchmarks, and evaluation of longer research trajectories.