Scopes

Research · methodology note · Sept 23, 2026

Does context make a model commit?

On the Read-off, some frontier models abstain a lot. Asked which of two probabilities is better calibrated, and given nothing but the two numbers, GPT and Grok answer indeterminate on nearly every event. We wanted to know whether that caution is fixed, or whether it changes when the model has something to reason from. So we ran a controlled test on one desk.

The test

On the weather desk, where our fair-value input is rights-clean and public domain, we run each read two ways. The base prompt (v1) shows only the two probabilities. The enriched prompt (v2) adds a short reference block: the National Weather Service daily-high forecast for that city, a public-domain fact. Same models, same desk, same events, the only difference is the reference. Each read is version-stamped so the two never mix.

ModelAbstain (numbers-only)Abstain (with reference)Committed with reference
openai gpt-6-astra100% (6/6)20% (2/10)8/10 (5 market, 3 fair)
xai grok-4.6100% (6/6)0% (0/10)10/10 (6 market, 4 fair)
anthropic claude-opus-560% (3/5)0% (0/8)8/8
google gemini-3.8-flash75% (3/4)0% (0/8)8/8

Snapshot as of Sept 23, 2026. The live version of this split is the "Does context make a model commit?" panel on the Read-off.

Correction, November 1, 2026. Three statements on this page do not match the record; the table is kept as the September 23 snapshot.Not the same events. Each model reads a flag once, so no flag was read both ways. The numbers-only reads ran on weather markets from September 12 to 23; the reads with the NWS reference began on the evening of September 19, on later markets. This is a comparison of the same models on different markets and days, not a within-event A/B.The numbers-only column covers only a few days. The with-reference column matches the record. Across every numbers-only weather read through September 23, GPT abstained on 39 of 39 and Grok on 39 of 39, so the central finding holds, but Claude abstained on 5 of 20 (25%) and Gemini on 5 of 18 (28%), not 60% and 75%.The enriched reads leaned to the market. The section below says they lean toward the fair value. In the record they chose the market more often: GPT 7 to 3, Grok 8 to 4, Claude 10 to 1, Gemini 7 to 2.Through November 1, with more reads in: numbers-only, GPT and Grok still abstained on every read (44 of 44 each); with the reference, Grok abstained on 0 of 39 and GPT on 14 of 43 (33%).

What it shows

The two most cautious models flip. GPT goes from abstaining on every numbers-only read to committing on eight of ten with the forecast in hand; Grok goes from every to none. The two that already committed sometimes now commit throughout. One public-domain fact is the difference between a model that will not answer and one that will.

It is not the models parroting the reference back. The enriched GPT and Grok reads split between choosing the market and choosing the fair value, so they are making a genuine judgment, not echoing the number they were handed.

What it does not show

This is a result about willingness to answer, not about being right. The reference we added is the same NWS forecast the weather fair value is built from, so the enriched reads lean toward the fair value; the commit-rate finding is clean, but a bias-free read on accuracy would need an independent reference (climate normals), which is the follow-up. The samples are small, the finding is on one desk with a public-domain input, and it firms up as more reads accumulate. The honest headline is narrow and real: context changes whether a cautious model participates.

Cite this study

Free to cite with attribution to Scopes. For the citation format, boilerplate, and media contact, see Press & citation.