Evaluation
3.2e-8 Pa
difference between its forecast grader and WeatherBench 2’s published IFS HRES MSLP 72 h RMSE
reproduction report4
task sets: era5-forecast-v1, ibtracs-storm-v1, heat-basis-v1 and climaqa-v1 · 648, 300, 300 and 350 items
READMEThe forecast grader reproduces WeatherBench 2’s published IFS HRES scores for mean sea level pressure, the cyclone grader NOAA NCEI’s North Atlantic figures, and the heat grader NWS Phoenix’s counts.
The ClimaQA audit found validated Gold test questions word for word in Silver, the set the paper fine-tunes on. Only exact matches count, so the figure is a lower bound.
WeatherBench 2 reproductionClimaQA audit
How it works
Each environment is a verifiers taskset with an Inspect task beside it, and both score through one contract module. Code computes every reward: forecast skill over persistence and climatology, an exact answer, or an exact match.
Four task sets
In era5-forecast-v1 the model forecasts ERA5 pressure or temperature from one grid cell’s history. In ibtracs-storm-v1 it computes a storm’s accumulated cyclone energy or peak category from the track, and in heat-basis-v1 it counts a heat policy’s trigger days from station data. The fourth, climaqa-v1, asks ClimaQA multiple-choice and cloze questions, with the leaked items taken out.
What failed and what changed
GRPO on ClimaQA multiple choice
- Result
- A small GRPO run of Qwen3.5-2B on the ClimaQA multiple-choice items showed no measurable transfer: held-out reward went from 0.750 to 0.774, which is not significant (two-sided sign test, p 0.541).
run report
Stack
- Python
- Inspect AI
- verifiers
- TRL
- ERA5
- IBTrACS
- GHCN-Daily