Note · 2026-05-15

Why we publish per-sport calibration N — and why you should ignore us at low N

Calibration buckets are meaningful at N≥200 per sport. Below 50 they're noise. We tell you N on every bucket. Here's the honest reading guide.

Visit /trust/calibration and you'll see a row of color-coded chips at the top: NFL N=145 gray, NBA N=512 green, NHL N=89 gray, tennis N=24 red. Those colors carry a specific claim about what you should and shouldn't trust.

The math, briefly

Calibration measures whether predicted probabilities match actual outcomes. If we say "this pick has a 60% chance of winning," over N=200 picks at that confidence level we should win roughly 120 of them. The deviation from 120 is the calibration gap.

The catch: at small N, expected deviation is huge by chance alone. The standard error on a 60% bucket is roughly √(0.6 × 0.4 / N). At N=10: ±15pp. At N=50: ±7pp. At N=200: ±3pp. At N=500: ±2pp.

This means a "10pp gap" with N=10 is INSIDE the normal noise band. The same gap with N=500 is FIVE standard errors out — a real and serious signal of a broken model.

Our color tiers, explained

Red chip (N<50): calibration is brittle. Take any claim about model accuracy at this confidence level with significant skepticism. We show the data because hiding it would be dishonest, but we don't trust it ourselves yet.

Gray chip (50≤N<200): calibration is starting to be informative. A 5pp+ gap in a single bucket is worth noticing but not panic-worthy. Trend across multiple buckets matters more than any single number.

Green chip (N≥200): calibration is trustworthy. A 5pp+ gap here is a real model defect — we'd investigate, retune, and either fix or document the limit.

Why "all sports" is a misleading default

Combining NFL + NBA + MLB + NHL + UFC + cricket calibration into one Brier score buys you nothing. The sports have different market efficiency profiles, different signal sets, different base rates. A model that's perfectly calibrated on NFL and broken on NHL averages out to "fine" in the aggregate. That's the kind of obfuscation tout products use. We show you per-sport because we want you to find the broken sport before we ship a pick on it.

The cohort sizes today (as of writing)

NFL is approaching N=200 across the season. NBA and MLB are well above. NHL is gray-tier — usable but not green-confident. Tennis is red because the model only graded a few Slam matches before we paused the cohort for the post-RG audit. Cricket is gray with N growing through IPL.

When tennis flips to green we'll say so on this page. When NHL crosses 200 we'll say so. When any sport drops below 50 (because we changed the model and reset the cohort), we'll say that too.

What to do with this

Trust green-tier predictions at face value.
Trust gray-tier predictions but cross-check the per-bucket gaps before committing serious money.
Treat red-tier sports as exploratory. The picks are still shown but the model's track record there is too short to evaluate.
Anyone claiming "X% win rate" without showing N is telling you nothing. Apply the same standard to us as to every other product.

Live tracker: /trust/calibration. The chips update as picks resolve.