76.7%
true root cause ranked first, on 90 sealed held-out incidents (deterministic engine) · in the top three: 87.8%
eval report
verdict: supportedp1: supported observed=increase supporting=1 contradicting=0 baseline ranking (deterministic, no model): kind=ranking minimum_score=1.001. cpu       suspicion=31.11  direction=increase2. latency   suspicion=1.90   direction=increase
One run of the real service on committed synthetic telemetry: the verdict on the hypothesis, with its evidence count, and the engine’s own ranking beside it.original

What is different · a generic pipeline against this agent

GENERICTHISAGENTincidentLLM reads signalsLLM names a causeanswers anywayincidentPrometheus telemetryevidence ledgercontent-addressedLLM proposesallow-listed hypothesesverifier decidesdeterministic codeSUPPORTEDREFUTEDUNKNOWN · abstainGENERICincidentLLM reads signalsLLM names a causeanswers anywayTHIS AGENTincidentPrometheus telemetryevidence ledgercontent-addressedLLM proposesallow-listed hypothesesverifier decidesdeterministic codeSUPPORTEDREFUTEDUNKNOWN · abstainGENERICincidentLLM reads signalsLLM names a causeanswers anywayTHIS AGENTincidentPrometheus telemetryevidence ledgercontent-addressedLLM proposesallow-listed hypothesesverifier decidesdeterministic codeSUPPORTEDREFUTEDUNKNOWN · abstain
Red marks what differs. Scored on a sealed held-out split, opened exactly once.

Evaluation

93.2%
root cause ranked first on the 88 development incidents · in the top three: 98.9%
eval report
0
invalid evidence citations across 178 evaluated cases and two model generations
README

Two arms on the same incidents from the public RCAEval benchmark: the deterministic engine alone, and Gemini behind the verifier. The held-out set was sealed and opened exactly once, on 2026-07-19, against a frozen commit. No tuning after.

The Gemini arm is a stress test of the verification gate, not a race with the engine: it sees signal names only, never values, so it is choosing among about 72 signals. On the held-out set it answered 38 of 90 incidents and ranked the cause first on 36.8% of those; it abstained on the rest and cited no invalid evidence.

Percent of the 90 sealed held-out incidents. An abstention counts as a miss.
armfirsttop 3abstained
Deterministic engine76.787.80
Gemini, names only15.615.657.8
Percent of the 90 sealed held-out incidents. An abstention counts as a miss.README

sealed-run protocolCI gate

How it works

Gemini proposes hypotheses over an allow-list of signals. A verifier tests each one against a content-addressed evidence ledger and returns SUPPORTED, REFUTED or UNKNOWN.

Telemetry

Reads a real Prometheus through a bounded range-query client and exposes five metrics at /metrics: job outcomes, per-stage duration, provider timeouts, tokens and verdicts.

README

Hostile input

3,000 generated hostile inputs across two parser boundaries assert that only typed errors escape. Token comparison is constant-time, poison jobs fail closed, and opaque case ids keep fault labels out of the prompt.

README

Architecture

Hexagonal architecture, PostgreSQL with SKIP LOCKED, mypy strict, 19 ADRs. Untrusted input is capped at 65,536 characters and 32 predicates before it is parsed.

What failed and what changed

  1. The held-out score is lower than the development score

    Tested
    That the engine's accuracy on the development incidents would hold on a sealed held-out set.
    Result
    On the development incidents the engine ranked the true root cause first 93.2% of the time. On the sealed held-out set, opened once against a frozen commit, it was 76.7%. Both are published side by side, with the protocol.
    Lesson
    Publishing the drop is what makes either number believable.

Stack

  • Python 3.12
  • FastAPI
  • PostgreSQL
  • Prometheus
  • Gemini
  • unittest
  • mypy --strict

363 tests run by the CI gate, no DB, network or credentials · 2 skippedCI run · Nov 2026

Contact / Rourkela, Odisha, India

Write to me