Forecasting HN · Methodology
How every number is scored.
The rules below are fixed before a question opens. If a rule ever changes, the change is dated here and applies only to questions opened afterwards.
What counts as a question
A yes-or-no question about something that will be publicly checkable by a set date. Each one names the exact rule that decides it, the public source that settles it (a regulator's docket, an official register, a company filing), the moment forecasting closes, and when the answer should be known. A person approves a question before it opens. After the first forecast, the wording, rule, source and close date are frozen; a clarification is allowed only as a dated note on the ledger.
What a forecast is
A probability that the answer is YES, in whole percentages from 1% to 99%. We never publish exactly 50%, so every call leans one way. Each forecast is stamped with the time the ledger recorded it, not a time anyone typed. Forecasts recorded after a question closes are refused. Only forecasts made at the time count toward the score; reconstructions or back-tests may be shown, but they are labelled and never scored.
The scores
- Brier score: the squared gap between the forecast and what happened (1 for YES, 0 for NO), averaged. 0 is perfect; 0.25 is what saying 50% on everything earns; 1 is the worst possible. This is the headline number.
- Log loss: punishes confident misses much harder than Brier does. Reported alongside.
- What gets scored: the desk's standing forecast at the close, meaning the last one recorded before the deadline.
- Time-averaged Brier: the standing forecast averaged over the whole time the question was open, counting 50% before the first forecast. This stops a late, lucky update from carrying the score.
- Void questions: if the rule's own void condition is met (say, the proceeding is cancelled), the question is marked void and left out of every score. Voids stay listed.
Baselines: simple rules we have to beat
- Coin flip: 50% on everything. A record that cannot beat this is not adding information.
- Base rate: the share of earlier questions that came out YES, using only answers known before each question closed.
- First call held: the desk's first forecast, never updated. Beating it shows that the updates helped.
Comparisons are paired, on the same questions. The 95% interval comes from re-sampling questions 2,000 times. If the interval includes zero, we say the difference is not distinguishable yet, whatever the point estimate says.
When numbers are shown
- No headline score until 15 questions have resolved.
- The calibration chart is marked early below 50 resolved questions.
- The record is called measured from 150 resolved questions. Before that it is an early record, and the page says so.
Prediction markets and other outside numbers
Market prices can be a useful comparison, but each market has its own terms for showing its data. On this page they are switched off by default: no market price, and no score computed from one, appears until that market's display terms are confirmed and a person turns it on. Outside numbers are always labelled as outside references, never as our forecasts.
Integrity
- Every question event and forecast is an entry in an append-only ledger. Each entry carries a SHA-256 fingerprint of the one before it, so changing or deleting any past entry breaks every fingerprint after it.
- The current chain head has an internal anchor receipt. Publishing that anchor somewhere we cannot edit afterwards (a dated public post or a public code commit) lets anyone with a later copy of the ledger confirm nothing earlier was changed. Public anchoring status is shown in the record notes below.
- Answers are recorded by a named person with a link to the evidence. Corrections are new entries; nothing is overwritten.
- The engine's code fingerprint is printed in the footer, so a score can be traced to the exact code that produced it.
Limits, stated plainly
- A small number of questions says little. Early records swing; the intervals show how much.
- We choose the questions. A record built from easy questions would look better than it is, which is why baselines and the base rate are shown next to every score.
- This is a beta. Methods may improve; every change is dated and applies only going forward.
- Nothing here is financial, investment, legal or trading advice.