Intra-model variance · production Opus 4.7 across 5 reruns
This page isolates run-to-run LLM stochasticity. We re-ran
predict_multi_outcome_retrieval five times on the same 26 events.
Same prompt, same retrieval, fresh LLM calls. The grand mean is
0.03772 with a standard deviation of 0.00090;
the canonical file (0.03782) sits well within one standard deviation. Read this
as a noise floor of ± 0.0009 on the replay metric.
Leakage note: these reruns use unfiltered retrieval, so the ~0.038
level is best-case-with-hindsight (retrieval can surface post-resolution
articles). Our honest headline is the leakage-disciplined binary Brier
0.118 — date-capping retrieval to each event's close moves
the number from 0.038 to 0.118, a 3.1× hindsight inflation (paired bootstrap
n=26, 95% CI [−0.136, −0.030]). What this page shows is that the
±0.0009 stochastic noise is tiny next to that 0.080 leakage gap.
1. Per-run mean Brier · 5 fresh reruns + canonical
Dashed line = grand mean of the 5 variance runs. Shaded band = ± 1 σ.
If the canonical bar sits inside the band, the published number is consistent with
intra-model noise (not an outlier).
2. Per-event p_yes spread · which events are stable vs noisy
For each event, the dots show p_yes across the 5 reruns;
the bar shows max − min. Sorted by spread (most volatile at top). Events with
near-zero spread are the ones where the model converges to the same probability
regardless of stochastic effects; events with high spread are where small
wording differences in the LLM output flip our score.
3. Per-event Brier · all 5 runs overlaid
Same x-axis: events sorted by mean Brier. Each event has 5 dots
(one per rerun) jittered vertically for visibility. Most events cluster tight;
a handful have wider spread.