Calibration

Every duel is forecast before it is answered. This page is how those forecasts are scored, and why the obvious metric would have lied.

Prequential by construction

record_duel scores the posterior’s $P(A \text{ wins})$ and only then appends the observation. So every forecast is an out-of-sample, one-step-ahead prediction: the model has never seen the answer it is being scored on.

Why not accuracy

A running count of $p > 0.5$ outcomes is accuracy, and accuracy is not a proper scoring rule. Two failures, and the second is fatal here:

It cannot see sharpness. A model that says 0.51 every time and is right 51% of the time scores identically to one that says 0.99 and is right 51% of the time. The second is wildly overconfident, and accuracy cannot say so.

It is pinned near 50% by the acquisition rule. An information-seeking rule deliberately picks pairs near $p = 0.5$, because those are the questions worth asking. So the hit rate sits near chance by construction: a perfectly calibrated model looks like a coin flip, and the player concludes it is not learning.

The second point is what makes accuracy harmful rather than merely crude: it penalizes the search for doing its job.

hit_rate is still computed, only so a frontend could show how misleading it is next to the skill score. The app does not show it.

Brier score and skill

$$B = \frac{1}{n}\sum_{i} \big(p_i^{\text{chosen}} - 1\big)^2$$

where $p^{\text{chosen}}$ is the probability the model gave to the option the player actually picked. Lower is better; $B = 0.25$ is what always saying 0.5 scores.

Reported as skill against that baseline:

$$\text{skill} = 1 - \frac{B}{0.25}$$

Brier is proper and bounded, and it moves as sharpness improves rather than only as accuracy does, which is the property accuracy lacked.

Log-loss, and what it may not be compared across

$$\text{LL} = \frac{1}{n}\sum_i -\log p_i^{\text{chosen}} \quad \text{(nats)}$$

Baseline $\log 2 \approx 0.693$.

Comparable across time for one acquisition rule. Not comparable across acquisition rules: an information-seeking rule serves duels near $p = 0.5$, which carry the highest log-loss by construction. Comparing two rules on their own self-chosen question sets would score the willingness to ask hard questions as a failure.

check_log_loss is the version for that comparison. See below.

The selection-bias fix

The acquisition function chooses which duels get scored, which means overall skill is measured on a question set the model helped select. That is circular.

So one shown pair in ten (duel_check_every, 10, in engine.rs) is drawn uniformly at random and flagged Forecast::random_check, and calibration restricted to those is unbiased. Under a rule that chooses its pairs (BALD or Thompson), the app marks such a pair ◇ unbiased probe. Under the default rule, its deal line says every pair is dealt at random (◇ random pair).

It costs a small share of the query budget and it is the only number here that means what it says unqualified.

The share is of the pairs the player is shown, not of the pairs dealt: the app deals the next pair ahead and sometimes throws a deal away unseen, and a probe thrown away does not use up its turn.

The reliability diagram

Five buckets over $P(A \text{ wins})$ (N_BINS = 5, in calib.rs: the most a small session can fill without every bucket being noise). Each bucket reports:

Plotted, the diagonal is the claim and the dots are the reality: a single number cannot distinguish “overconfident at the top end” from “underconfident in the middle”, and the shape of the failure is what says what to do about it.

The app does not plot it. LEARNING draws every forecast instead (WasmEngine::forecasts, the list these scores are taken over), each as the probability it gave the sound picked, with the hits out of all of them and the mean confidence of its guesses (expected, the mean of $\max(p, 1 - p)$) beside the share it got right (was). Expected above was is overconfidence, the diagram’s top-end failure, in two numbers. The buckets stay in calibration() and in LEARNING’s copy as JSON.

By provenance

The same scores, split by how the answer was collected:

pub struct ProvenanceScore {
    pub provenance: String,  // "duel" | "heard_edit" | "self_report" | "perform_offer"
    pub n: usize,
    pub brier: f64,
    pub log_loss: f64,
    pub skill: f64,
}

The comparison this exists for: a hand edit kept as new through a heard duel and one kept by ticking pick the edit make the same claim in the log, and there is no reason to believe they are equally reliable. Scoring them against forecasts the model made before either answer arrived is the only way to find out which, and it costs one tag.

Empty streams are omitted, so a session whose every answer was a dealt pair carries exactly one row.

Interpreting it

The user-facing version of this table is in Reading what it learned.

Why the number can look bad

Committing to a forecast before each answer and then reporting the error against a proper scoring rule means the model can publicly fail, and early on it does.

That is what makes the number worth reading later, and it is why the app shows “not beating a coin flip yet” rather than hiding the metric until it flatters.