Research · pre-registered study
Frontier models vs the market: the 2026 MLB postseason
A pre-registered contest. Before each round of the 2026 postseason, every frontier model in our roster forecasts the winner of each series, with public context: records, probable starters, rest, and injuries. The forecasts are locked before a pitch is thrown and scored, once the series settle, by a proper scoring rule against the actual outcomes and against the market.
Scopes is the scorekeeper, not a forecaster. The models forecast; the market forecasts; we run the contest, lock the predictions, and keep the append-only record. The prompt and the exact context shown to each model are published below so any score here can be reproduced.
What we registered in advance
- Over the ~11 postseason series, does any model's mean Brier score beat the market's?
- Straight pick record: of the series, how many did each model call correctly vs the market?
- Does context (records, rotation, rest, injuries) let a model match a market that already prices everything?
Registered expectation: at roughly eleven series this is underpowered, so we expect no statistically significant separation. This is a demonstration of the mechanism and a calibrated look, not a conclusive ranking. The market is a hard baseline; a model failing to beat it is a legitimate, reportable result.
Results
The postseason ended on October 28, 2026, when the Milwaukee Brewers won the World Series 4-1. The study is closed. Only one of the four registered rounds was locked: the Division Series. The Wild Card round, both League Championship Series and the World Series were not locked before their first pitch. A forecast locked after a series starts is not pre-registered, so those rounds are not part of the record.
No market benchmark was captured. Market capture was built for the World Series round only, and that round was never locked. The registered questions that compare models with the market have no answer here. What the record supports is a comparison of the models against the outcomes.
In all 4 Division Series, every model picked the home-field team, and the home-field team won each one. The pick records are therefore perfect for every model, and the Brier ranking below measures only how confident each model was in favorites that all won. It does not show that one model forecasts better than another. This is the outcome the registered expectation anticipated at a small sample.
Each model's mean covers the series it was locked on. A model that returned too few valid answers at lock has no row for that series, which is why the pick counts differ.
The contest
Final standing (14 of 14 resolved)
| Competitor | Mean Brier | Log loss | Picks |
|---|---|---|---|
| xai grok-4.6 | 0.152 | 0.493 | 3/3 |
| openai gpt-6-astra | 0.169 | 0.529 | 4/4 |
| anthropic claude-opus-5 | 0.176 | 0.543 | 3/3 |
| google gemini-3.8-flash | 0.190 | 0.573 | 4/4 |
Lower Brier and log loss are better. Underpowered at this sample; read it as directional.
Division Series
| anthropic claude-opus-5 | P(Cleveland Guardians) 58.8% | Brier 0.170 |
| google gemini-3.8-flash | P(Cleveland Guardians) 56.1% | Brier 0.193 |
| openai gpt-6-astra | P(Cleveland Guardians) 56.2% | Brier 0.192 |
The opponent was not yet decided at lock, so the schedule showed the placeholder ATL/PHI. The locked question was whether the Los Angeles Dodgers win this series, which settles whoever they faced, and it is scored on that basis.
| google gemini-3.8-flash | P(Los Angeles Dodgers) 54.5% | Brier 0.207 |
| openai gpt-6-astra | P(Los Angeles Dodgers) 58.0% | Brier 0.176 |
| xai grok-4.6 | P(Los Angeles Dodgers) 60.2% | Brier 0.158 |
| anthropic claude-opus-5 | P(Tampa Bay Rays) 55.6% | Brier 0.197 |
| google gemini-3.8-flash | P(Tampa Bay Rays) 56.3% | Brier 0.191 |
| openai gpt-6-astra | P(Tampa Bay Rays) 58.6% | Brier 0.171 |
| xai grok-4.6 | P(Tampa Bay Rays) 59.4% | Brier 0.165 |
| anthropic claude-opus-5 | P(Milwaukee Brewers) 60.0% | Brier 0.160 |
| google gemini-3.8-flash | P(Milwaukee Brewers) 58.6% | Brier 0.171 |
| openai gpt-6-astra | P(Milwaukee Brewers) 63.2% | Brier 0.135 |
| xai grok-4.6 | P(Milwaukee Brewers) 63.7% | Brier 0.131 |
The prompt, put to every model
The identical prompt, with each series' context filled in. The model is sampled five times; the mean probability is the locked forecast. It never sees the market price: the market is the benchmark, not an input. Each locked series above carries the exact context that was shown.
You are forecasting a Major League Baseball postseason series that has not yet been played. Give your best estimate of the probability that the FIRST team wins the series.
Series: {team_a} vs {team_b}, best-of-{games}
Home-field advantage: {home_team}
Regular-season records: {team_a} {record_a}, {team_b} {record_b}
Season head-to-head: {h2h}
Probable starters: {team_a}: {rotation_a}; {team_b}: {rotation_b}
Rest / fatigue: {team_a}: {rest_a}; {team_b}: {rest_b}
Injuries (IL): {team_a}: {il_a}; {team_b}: {il_b}
Use what you know about these teams and postseason baseball, and weigh rotation, rest, and injuries (a spent ace or a tired team on no rest matters in a short series). Answer with a single JSON object and nothing else:
{"p_team_a_wins": 0.0-1.0, "confidence": 0.0-1.0, "reason": "<=240 chars"}Cite this study
Free to cite with attribution to Scopes. For the citation format, boilerplate, and media contact, see Press & citation.