A grognard exercise, not a scientific model. What that means

Roadmap / 0.1.2.10

Test harness

Version 0.1.2.10, 12 Nov 2026. tag v0.1.2.10, commit 99f2e36. File docs/RESULTS_v0.1.2.10_test_harness.md.
Written at this version and not revised since. Later versions may change these numbers; see the roadmap.

Built because ten switches had outgrown the old way of testing: effects under 3 points drowned in noise, and a full on/off grid would need 1,024 combinations.

What it is

python -m forceonforce.harness <command> <scenario>; code in forceonforce/harness/.

Command What it does
effects All off, all on, remove-one and add-one for every switch, as paired differences with 95% intervals
screen Random factorial design: every run draws its own on/off setting for every switch; a regression gives main effects and all pairwise interactions. Scales to any number of switches
sensitivity Ranks every sampled parameter by how much it moves an outcome
check Invariants: determinism, bounds, neutral-when-off, isolation, expected directions
golden Frozen results; fails when model behaviour changes without the file being updated
switches Lists the switch registry (harness/switches.py: one entry per module)

The change that makes it work: common random numbers

Every module and every sampled scenario value now draws from its own named random stream, seeded from the run number. Two variants on the same run share every random number except where their settings differ. montecarlo.run_scenario uses this too, so every sweep is now paired. Results from earlier versions shift slightly (different random numbers, same model); the Iraq backtests still pass.

Noise fell 20× to 200× (median variance reduction: Alberta 22×, full annexation 99×, coercion 206×). At 3,000 runs, a single switch's effect is now readable to about ±0.2 points instead of ±2.5.

What it found (3,000 paired runs per variant; 20,000 runs for screening and for sensitivity)

1. Switch effects on the chance the US quits (points, 95% interval)

Switch (remove-one) Full annexation Alberta detachment Economic coercion
All switches +10.3 (9.2 to 11.5) +23.9 (22.3 to 25.4) +27.3 (25.7 to 28.9)
Canadian mobilization +5.9 (5.0 to 6.7) +8.3 (7.3 to 9.3) +0.7 (0.4 to 1.0)
NATO and allies +4.1 (3.3 to 4.9) +8.0 (6.9 to 9.0) +8.2 (7.1 to 9.3)
Potash +0.1 (0.0 to 0.3) +2.8 (2.2 to 3.4) +11.6 (10.4 to 12.7)
Oil 0.0 (−0.1 to 0.1) +2.3 (1.7 to 2.8) +10.2 (9.1 to 11.3)
Wildlife 0.0 (0.0 to 0.1) +1.5 (1.0 to 1.9) exactly 0
Weather −0.2 (−0.5 to 0.2) +0.5 (0.2 to 0.8) +0.1 (0.0 to 0.2)
Logistics −0.5 (−0.9 to −0.1) +0.4 (−0.2 to 1.0) 0.0 (−0.2 to 0.3)
Small arms +0.2 (0.0 to 0.3) −0.6 (−0.9 to −0.3) exactly 0
Seasonal insurgency 0.0 (−0.3 to 0.2) +0.1 (−0.1 to 0.4) exactly 0
Electricity exactly 0 exactly 0 +0.2 (0.1 to 0.4)

2. Corrections to v0.1.2.9

  • The "four switches carry it" finding holds, now with tight intervals.
  • The 2-point "effects" of wildlife and seasonal insurgency in economic coercion were noise. They are now exactly zero, run by run, which the isolation check enforces.
  • Some effects I called noise are real but small: wildlife in the Alberta detachment (+1.5; bear watch and insects wear down a thin occupier), weather in Alberta (+0.5), logistics in full annexation (−0.5; it favours the US).
  • Small arms slightly favour the US in Alberta (−0.6): with the module on, the resistance can run short of ammunition.

3. Interactions: none

Screening tested all 45 switch pairs. Four came up "significant" in full annexation, about what chance alone gives at 95% (2 expected). I retested the three largest with paired 2×2 designs: potash × oil −0.1 (−0.3 to 0.1), wildlife × arms exactly 0, mobilization × NATO −0.7 (−1.6 to 0.1). Switch effects are additive. The workflow is: screen wide, then confirm anything flagged with a paired design.

4. The parameters that decide the war are the ones with the least evidence

Swing = chance the US quits in runs where the parameter is in its top fifth, minus runs where it is in its bottom fifth.

Scenario Parameter Swing (points)
Full annexation Canadian capitulation base hazard −21
US withdrawal maximum hazard +18
Canadian resolve +6
Alberta detachment US initial public support −30
US withdrawal maximum hazard +17
Canadian capitulation base hazard −16
Economic coercion US initial public support −49
US withdrawal threshold +34
US withdrawal maximum hazard +21
Economic-pain slope on US support +13

No module parameter comes close: the largest is the potash political weight in coercion (+8). Uncertainty about the political layer is worth two to five times the largest module. Capitulation and withdrawal are judgment-set. The best use of the next research effort is those four parameters, not another module.

5. Invariants

All 50 checks pass across the three scenarios: determinism, bounds, 7 neutral-when-off checks per scenario, 3 isolation checks in coercion, and 10 direction checks (for example allied forces > materiel > sanctions > paralysis; cutting off arms lowers US deaths; a cold-trained US force takes fewer cold casualties).

How to use it when adding a module

  1. Add one entry to harness/switches.py.
  2. Run check (the new module must be neutral when off and isolated where it has no path).
  3. Run effects for its paired effect, then screen if interactions are plausible.
  4. Run golden --update and commit the file with the change. 20 tests pass, including the golden check.