Majores Ventures

← All projects

peekproof

Catch look-ahead leakage before your backtest lies.

Python library and CLI · Python, pandas, NumPy · MIT · v0.1.0 · github.com/Thesimpleex/peekproof

Abstract

Tests whether a feature function uses information from the future by re-running it on truncated and scrambled histories, and lints research code for thirteen common leak patterns. For quant researchers, data scientists and anyone who backtests trading strategies or validates time-series models.

Cumulative return of the same mean-reversion rule with a centered rolling window (Sharpe 5.1) and a trailing window (Sharpe 0.1) on a synthetic random walk.
Figure 1. Cumulative return of the same mean-reversion rule with a centered rolling window (Sharpe 5.1) and a trailing window (Sharpe 0.1) on a synthetic random walk.

1Motivation

Most backtests that look too good quietly read the future: a centered rolling window, a backward fill, a shift in the wrong direction, a position multiplied by the same bar's return. Nothing in a tear sheet reveals it.

On synthetic random walks with no edge, a single center=True turns a 20-day mean-reversion rule into a Sharpe ratio between 3.7 and 5.3 in all 50 worlds tested, while the honest version scatters around zero.

2Method

  1. Run the feature function on the full history.
  2. Run it again on the history cut at 64 cutoffs, and on histories whose future rows are scrambled.
  3. Compare every output row at or before each cutoff with the full run.
  4. A changed value or a vanished row is a leak; its distance to the cutoff is the leak horizon.
  5. Separately, the linter reads the source code and flags thirteen known leak patterns with a confidence level.

3Results

Sharpe ratio of the leaky and the causal variant across 50 synthetic random walks: the leaky variant clusters between 3.7 and 5.3 with median 4.5, the causal variant scatters around zero
Figure 2. Sharpe ratio of the leaky and the causal variant across 50 synthetic random walks: the leaky variant clusters between 3.7 and 5.3 with median 4.5, the causal variant scatters around zero.

4Usage

4.1 Check a feature function

Re-runs the function on truncated and perturbed histories and reports whether its output depends on future rows, which rows, and how far ahead it looked (the leak horizon).

peekproof check features:build_features --data prices.parquet

4.2 Lint research code

Reads the code without running it and flags patterns such as shift(-1), backward fills, centered windows and shuffled train/test splits, each with a one-line fix. Jupyter notebooks are linted cell by cell.

peekproof lint src/ notebooks/

5Installation

pip install git+https://github.com/Thesimpleex/peekproof

Python 3.12 or later. Runtime dependencies are pandas and NumPy only; polars is supported optionally. Not on PyPI yet.

6Specification

Runs
Python 3.12+, CLI
Built with
Python, pandas, NumPy
License
MIT
Version
0.1.0

Source code, issues and releases: github.com/Thesimpleex/peekproof. License: MIT.