0 false answers
on 105 held-out questions · all 35 out-of-scope questions refused; 18 answerable ones refused too
held-out run
Part of a report from the live app for extreme rainfall in Mumbai over 7 days: four risk drivers with their numbers, and five IPCC AR6 citations, each to a page.
An answer from the live app: extreme rainfall in Mumbai over 7 days, rated low. Each IPCC citation is checked against its page.original
{"answer": "","citations": [],"abstain": true,"abstain_reason": "The provided excerpts do not contain information specific to Rourkela or short-term forecasts (next 7 days) for heatwaves."}
A refusal from the IPCC tool over MCP: no answer, no citations, and the reason why.original

Problem

A question about heat, extreme rain or wind at one place needs three kinds of evidence at once: a live forecast, the local record of extremes from ERA5, and what IPCC AR6 assesses for that region.

Evaluation

96%
citation validity: held-out answers citing the right page · 50 of 52
held-out run
87 / 91 / 96%
right page retrieved in the top 3 / 5 / 10 · 70 questions
eval report
≈ $0.0025
per question for the IPCC answer step; p50 4.1 s per model call
held-out run
0.70 → 1.93 °C
max-temperature forecast error, day 1 → day 7 · 13 cities
README

Both eval sets are SHA-256 frozen. The 105-question held-out set was written after the dev set existed and never used to tune anything.

Refusals are scored with the answers: correct answer, correct refusal, false refusal, false answer. Any false answer fails the build.

The agent also measures its own forecast skill per lead day and weights the confidence of every report by it.

held-out retrieval run, 2026-09-07 (JSON)evaluation section of the README

How it works

The question is parsed, geocoded and mapped to its IPCC AR6 region before the agent sees it. The agent can refuse at its first step; otherwise it fetches the forecast, searches AR6, reads the regional projection and writes the report. The flow follows the diagram in the repository README.

Before the agent

  1. questionfree text
  2. parselocation · hazard · horizon
  3. geocodeOpen-Meteo
  4. AR6 regionIPCC reference regions

The agent · LangGraph

  1. planscope checkout of scope: refusal, a valid typed output
  2. calllive forecast
  3. researchIPCC AR6 retrieval
  4. projectAR6 Ch.12 regional projection
  5. synthesizetyped, cited RiskReport

Feeds call and synthesize

ERA5 to GEV hazard statisticsstationary and drifting-location GEV · likelihood-ratio trend test · return levels with 90% bootstrap CI

Feeds research and synthesize

IPCC AR6 retrievalBM25 + dense hybrid, RRF fusion · citations validated against the retrieved pages

Hybrid retrieval

BM25 plus dense embeddings fused with reciprocal rank fusion, with every citation validated against the source page.

Hazard statistics

60+ years of ERA5 fitted with stationary and non-stationary GEV distributions, a likelihood-ratio trend test, and 90% bootstrap confidence intervals.

+0.75 °C / decade
Berlin warming trend (non-stationary GEV) · p < 0.0001
README
stationary
Delhi trend test · p = 0.46 at the preset point, 0.84 at the city point
README

Two MCP servers

Two MCP servers ship with it, weather and ipcc-rag, on protocol revision 2026-07-28 with ToolAnnotations. The IPCC server is indexed on the official MCP registry; its container image is private for now, so run it from a clone.

Operations

Records per-request telemetry and cost, with a Cloud Storage cache in production (Redis optional), a disk fallback and a prewarm script for the hazard fits.

README

What failed and what changed

  1. Rerankers and query rewriting

    Result
    Two rerankers and a query rewriter were measured on the dev set. They cost 4 to 38 seconds per query, and none beat the baseline at any k.
    Changed
    Both stay wired in the code and ship switched off.

    README

  2. Forecast confidence

    Result
    A forecast peak far out is a weaker claim than tomorrow's, and the error grows with lead time (see Evaluation).
    Changed
    Report confidence is weighted by the measured skill at the lead day asked about.

    README

  3. MCP servers without credentials

    Result
    Servers launched by an MCP client started without credentials.
    Changed
    Fixed with load_dotenv(override=False).

Stack

  • LangGraph
  • Gemini
  • FastAPI
  • MCP
  • scipy
  • Docker
  • Cloud Run

1,138 tests passed in the CI run on main · 7 skippedCI run · Nov 2026

Contact / Rourkela, Odisha, India

Write to me