50,000 paired bootstrap resamples of per-event Brier improvement (baseline − model). Bars = distribution shape; shaded band = 95% confidence interval; vertical line = observed mean. The CI excludes zero, so the production model significantly improves over baseline on this dataset at α=0.05. Hover for counts at each bin.
Leakage note: this is the model-swap delta (Sonnet to Opus), computed on the unfiltered-retrieval replay where both arms got the same hindsight benefit — so the delta is valid even though the absolute level (~0.038) is best-case-with-hindsight. Our honest headline binary Brier is the leakage-disciplined 0.118: date-capping retrieval to each event's close moves the absolute number from 0.038 to 0.118 (3.1× hindsight inflation; a separate paired bootstrap, n=26, gives that gap's 95% CI as [−0.136, −0.030]).
per-event scatter · cross-model heatmap · abstain slider · summary report · home