Majores Ventures

← All projects

dutchbook

Find and repair Dutch books in probability forecasts.

Python library and CLI · Python, NumPy, SciPy, rich · MIT · v0.1.0 · github.com/Thesimpleex/dutchbook

Abstract

Detects when probability forecasts over logically related events are incoherent, prices the guaranteed arbitrage against them exactly, and repairs them so that no outcome's Brier score gets worse. For AI-evaluation researchers, forecasting teams and students of probability.

Brier scores of forecasts before and after coherent repair
Figure 1. Brier scores of forecasts before and after coherent repair.

1Motivation

Language-model forecasters are scored one question at a time, so nobody checks whether their answers fit together. A Dutch book is a set of bets that loses in every outcome; de Finetti showed it exists exactly when forecasts are incoherent, which in finance terms is an arbitrage.

2Method

  1. Parse propositional formulas and enumerate possible worlds.
  2. Test coherence as membership in the convex hull of world vectors (linear programming).
  3. Read the Dutch book off the dual solution.
  4. Project the forecasts onto the coherent set, which provably lowers the Brier score in every world.
  5. Apply the audit to the public ForecastBench data.

3Results

Dot plot of the mean guaranteed arbitrage per unit of total stake for each model configuration, as elicited and after renormalising the four joint cells
Figure 2. Dot plot of the mean guaranteed arbitrage per unit of total stake for each model configuration, as elicited and after renormalising the four joint cells.
Cumulative distribution of the guaranteed arbitrage per tuple, as elicited and after renormalising
Figure 3. Cumulative distribution of the guaranteed arbitrage per tuple, as elicited and after renormalising.
Terminal output of dutchbook forecastbench for round 2024-07-21: a coherence leaderboard of model configurations
Figure 4. Terminal output of dutchbook forecastbench for round 2024-07-21: a coherence leaderboard of model configurations.
Geometry of accuracy dominance for the forecasts of A and not A
Figure 5. Geometry of accuracy dominance for the forecasts of A and not A.
Terminal output of dutchbook repair: forecasts before and after the projection and the per-world improvement
Figure 6. Terminal output of dutchbook repair: forecasts before and after the projection and the per-world improvement.
Self-contained HTML audit report with a per-model coherence leaderboard
Figure 7. Self-contained HTML audit report with a per-model coherence leaderboard.

4Usage

4.1 Audit forecasts

Coherence status, the bets of a Dutch book and the payoff in every possible world.

dutchbook audit forecasts.csv --spec formulas.toml

4.2 Audit a benchmark

Coherence audit of a public AI forecasting benchmark round.

dutchbook forecastbench --round 2024-07-21 --html report.html

5Installation

pip install git+https://github.com/Thesimpleex/dutchbook

Python 3.11 or later. Not on PyPI; installs from GitHub.

6Specification

Runs
Python 3.11+, CLI
Built with
Python, NumPy, SciPy, rich
License
MIT
Version
0.1.0

Source code, issues and releases: github.com/Thesimpleex/dutchbook. License: MIT.