whitehall bench
Run the benchmarks in src/benches/ on a device.
The shape
A file under src/benches/ is a bench file, the way a file under src/routes/ is a route. Every top-level function taking no arguments is a
benchmark. A nativeFoo beside a foo is the same benchmark in another
language, and the two are compared.
// src/benches/quadrature.wh
import $native.quadrature
fun integrate(): Double { … } // the Kotlin
fun nativeIntegrate(): Double = quadrature.integrate() // the same work, in Rustintegrate and nativeIntegrate are one benchmark with two
implementations, paired by name. Writing the second function is the statement that it is the
same work. Functions take no arguments because what size to run at is part of the benchmark.
Usage
whitehall bench [target] [options]whitehall bench # run everything, report both sides
whitehall bench --gate # exit 1 if native loses to its Kotlin twin
whitehall bench --filter integrate
whitehall bench --save-baseline # record this run
whitehall bench --json quadrature::integrate kotlin 48.21 ms native 3.11 ms 15.5× ✓
kotlin ±0.41 ms over 200 iterations +1.2% slower than baseline
native ±0.03 ms over 200 iterationsOptions
| Flag | What it does |
|---|---|
-f, --filter <text> | Only benchmarks whose id contains this |
-d, --device <id> | Which phone |
--gate | Exit 1 if a native implementation loses to its Kotlin twin |
--json | Output as JSON |
--baseline <file> | Baseline file to compare against. Default src/benches/baseline.json |
--save-baseline | Write this run to the baseline file |
--timeout <s> | Seconds to wait for the harness to finish. Default 300 |
-r, --release | Measure a release build instead of debug |
The method
Each benchmark warms up, runs many times, and reports the median and a robust spread rather than a mean, because on a phone something else always wakes up during one iteration. Measurement happens on the device: a claim about a phone needs a number from one.
What the gate asserts
A benchmark clears --gate when native is no slower than Kotlin plus the noise.
Two implementations whose medians differ by less than the measured spread have not been
shown to differ. A benchmark with only one implementation is never gated: that is a
measurement, not a claim.
--gate is a check you run before pushing, in a
project that has benches.