Proving It

Evidence comes in rungs

Programs of this kind are usually justified with a benefits case built on assumptions, benchmarks and a projected improvement in an accuracy metric. Such cases are approved routinely and believed rarely, which is why they tend to be revisited the moment the first budget pressure arrives. The problem is not that the people who write them are careless. It is that a benefits case is a forecast of value, and the whole argument of this series is that forecasts defended as numbers are worth very little.

There is a better way to make the case, and it does not require a leap of faith at any point. Evidence can be climbed one rung at a time, each rung cheap relative to the commitment it unlocks, and each answering the specific doubt that a sensible finance director would raise about the one below it.

The priced backtest: lived history, in currency

The first rung can be built before any system is bought. Take the decisions your organization actually made over the last two years and price them against the ledger. Then take the same historical data, apply a simple priced policy, and price the decisions it would have made instead. Both numbers are in currency. Both describe events that have already happened. Neither depends on a forecast of the benefit.

Three ways a backtest lies

A backtest is the cheapest strong evidence available, and for exactly that reason it is the easiest to inflate by accident. Three errors account for most of the inflation, and a competent finance team will look for all three.

1. Censored demand. Sales history records what was sold, not what was wanted. On every day an item was out of stock, true demand was higher than the record shows, and a backtest that treats sales as demand will misprice the old decisions on precisely the items where they failed. Reconstruct the lost demand or exclude the stockout periods, and say which you did.
2. Hindsight. Anything that was not known at the moment of the decision must be kept out: a revised forecast, a price set later, an outcome already in the data. Freeze the information available at each decision date and replay from that. One leak discredits the whole result, and rightly.
3. The convenient period. Choose the window before looking at results, and make sure it contains a bad season. A backtest over a calm year proves that a policy can handle a calm year.

Two further disciplines make the result hard to argue with. Use a policy simple enough that nobody can accuse it of being fitted to the past; a newsvendor calculation against the ledger is usually sufficient and is harder to dispute than anything more sophisticated. And leave out the benefits you cannot observe, such as customers who would not have been lost, rather than estimating them generously. Present the result as a range, state what was excluded, and let the number be smaller than it could have been.

A backtest showing a sixty percent improvement should worry you more than one showing fifteen. Large numbers usually mean a leak.

Monte Carlo: the thousands of years you have not lived

A backtest has one weakness that no amount of rigor removes. It describes a single history. The year you lived was one draw from the range of years you could have had, and it may have been unusually calm or unusually harsh. A policy’s value is not how it performed in that draw but how it performs across the whole range.

A Monte Carlo simulation addresses this directly. It generates thousands of plausible years, each with its own demand, lead times, disruptions and prices drawn from distributions fitted to your data, and runs every candidate policy through all of them against the same ledger. What comes back is not one number per policy but a distribution of outcomes: the average, the bad year, the very bad year, and how likely each is.

Exhibit 3 shows what this reveals for the portfolio behind the frontier in Paper 05. The averages are close, about $0.3 million apart. The priced policy wins only 53 percent of individual years, which means a backtest on any single year has close to an even chance of making it look worse than the rule it replaces. Its value sits almost entirely in the tail: in the worst year in twenty it earns about $6 million more. It gives up some upside in the best years to do so, which is exactly the trade the frontier makes explicit, and a choice leadership should make knowingly. That is the kind of value a single history rarely shows, and the kind a finance committee most needs to see.

What a validated simulator can do for you

1. Test the futures history has not produced. A port closure, a demand shock, the loss of a supplier: run them deliberately, before they happen, and see which policy holds up.
2. Price risk, not just the average. The distribution is what makes a frontier like the one in Paper 05 possible. Without it, risk appetite is a feeling rather than a number.
3. Compare policies before committing anything. Dozens of candidates can be ruled out in an afternoon without disappointing a single customer.
4. Size the live test. The simulator says how many items and how many weeks a controlled cutover needs before the difference will stand clear of the noise.

A simulator that cannot reproduce the past has no standing to describe the future.

Running it in parallel before betting on it

A backtest establishes that a better decision was available. It does not establish that the organization can make one prospectively, and a sensible executive will make that distinction. The bridge between them is a parallel run.

For a defined period, usually one or two planning cycles, the existing process continues to make the real decisions. The policy makes shadow decisions alongside it, recorded but not executed. Each cycle, the two are compared in currency, and every divergence is examined with the planner who would have overruled it.

The parallel run does three things at once, and the second and third are usually undervalued. It produces prospective evidence rather than retrospective evidence. It surfaces the override signal before anything depends on it, so the first live cycle starts with a model that has already absorbed a season of operator knowledge. And it gives the planning team a low-stakes period in which to disagree with the system and be taken seriously, which does more for adoption than any amount of training.

Its limit is that shadow decisions are never executed, so their consequences have to be estimated rather than observed. That leaves room for the same doubts a backtest invites, and it is why the parallel run is a rung rather than the destination.

The controlled cutover: the only evidence that proves cause

Every rung so far compares the policy with history or with a hypothetical. The decisive rung compares it with the existing process, live, in the same weeks and the same market.

Split the portfolio at random into two halves, by item, store or region. Let the policy make the real decisions for one half while the existing process continues on the other, for at least one full replenishment cycle and usually a quarter. Then compare the realized cost of error in each half against the same ledger. Because the assignment was random, the season, the competitor, the supply disruption and the extra management attention all fall equally on both halves. Whatever difference remains was caused by the policy.

Exhibit 4 shows why the control group is worth the trouble. Peak season builds through the test period, and a supply disruption lands in the middle of it. Measured before and after on its own, the policy group improved by about one percent, a result that would end most programs at the next budget review. Measured against a control group that lived through the same season and the same disruption, it cut the cost of error by about a fifth. The confounders did not cancel out. They concealed the effect, and only the randomized comparison could see past them.

Two design points decide whether the result will be believed. Keep the halves from interfering with each other: if both draw on the same constrained warehouse, the policy can appear to win simply by taking capacity from the control. And size the test so that the expected difference is larger than the week-to-week noise, which in practice means hundreds of items rather than dozens. The validated simulator will tell you how many.

The six measures that matter

Conventional planning metrics are largely absent from what follows, because they measure the forecast rather than the decision. These six measure the decision.

How a working system decays

Systems of this kind rarely fail visibly. Nobody switches them off. They become ignorable, gradually, and the organization returns to committee without ever having decided to.

The decay is detectable well in advance, and Exhibit 5 shows the two curves that reveal it.

Overrides on routine decisions should decline steadily, because the model is absorbing the reasoning behind them. A routine line that refuses to fall means the models are not trusted, and the cause is almost always that the ranking is not inspectable: planners cannot see why the top option won, so they substitute their own.

Overrides on genuinely novel decisions should stay roughly flat, indefinitely. That line is the system working as designed. A novel line trending toward zero is not maturity; it is judgment being suppressed, usually because someone has begun reporting a blended override rate as a performance metric and the organization has responded rationally to it.

A single blended override number conceals both failure modes, and is the metric most organizations choose.

The governance that stops the drift

Sustaining the system requires four standing commitments. They are unglamorous, they take perhaps a day a month in total, and their absence is the difference between an architecture that compounds and one that has to be rebuilt in four years.

1. Recalibrate the ledger on a fixed schedule. Quarterly is usually right. Compare what the ledger predicted the cost of error would be against what it actually was, and adjust. A ledger that has not been revised in a year is a ledger nobody is using.
2. Review model drift explicitly, by name and by owner. Lever-response relationships decay as markets move, and they decay silently. The model steward reports on confidence ranges, not only on central estimates.
3. Keep the experiment queue funded. The tests that resolve thin evidence are the only mechanism by which response models improve. They are also the first thing cut in a difficult quarter, which is precisely when the information is most valuable.
4. Publish the spreadsheet count. Uncomfortable, trivially cheap, and the single most reliable early warning available. An organization willing to put that number on a slide each quarter will not drift far without noticing.

This closes the series. The argument across the six papers has been a single one. Recurring operational decisions are a manufacturing problem rather than an analytical one. They cannot be improved until each direction of error carries a price. The forecast is an input to them, not the point of them. Once the routine work is automated, people move to the work only people can do: arguing the economics, choosing where on the frontier to sit, and answering for that choice. And all of it can be proved in currency, one rung at a time, before anything is bet. The organization that does this acquires something more durable than a better forecast. It acquires the ability to see the distribution it has been operating inside the whole time, and to choose its position within it deliberately.