43
validated ClimaQA Gold test questions found word for word in Silver, the set the paper fine-tunes on · exact matches, so a lower bound
audit report

Evaluation

3.2e-8 Pa
difference between its forecast grader and WeatherBench 2’s published IFS HRES MSLP 72 h RMSE
reproduction report
4
task sets: era5-forecast-v1, ibtracs-storm-v1, heat-basis-v1 and climaqa-v1 · 648, 300, 300 and 350 items
README

The forecast grader reproduces WeatherBench 2’s published IFS HRES scores for mean sea level pressure, the cyclone grader NOAA NCEI’s North Atlantic figures, and the heat grader NWS Phoenix’s counts.

The ClimaQA audit found validated Gold test questions word for word in Silver, the set the paper fine-tunes on. Only exact matches count, so the figure is a lower bound.

WeatherBench 2 reproductionClimaQA audit

How it works

Each environment is a verifiers taskset with an Inspect task beside it, and both score through one contract module. Code computes every reward: forecast skill over persistence and climatology, an exact answer, or an exact match.

Four task sets

In era5-forecast-v1 the model forecasts ERA5 pressure or temperature from one grid cell’s history. In ibtracs-storm-v1 it computes a storm’s accumulated cyclone energy or peak category from the track, and in heat-basis-v1 it counts a heat policy’s trigger days from station data. The fourth, climaqa-v1, asks ClimaQA multiple-choice and cloze questions, with the leaked items taken out.

What failed and what changed

  1. GRPO on ClimaQA multiple choice

    Result
    A small GRPO run of Qwen3.5-2B on the ClimaQA multiple-choice items showed no measurable transfer: held-out reward went from 0.750 to 0.774, which is not significant (two-sided sign test, p 0.541).

    run report

Stack

  • Python
  • Inspect AI
  • verifiers
  • TRL
  • ERA5
  • IBTrACS
  • GHCN-Daily

Contact / Rourkela, Odisha, India

Write to me