Forecasting Is Not the Point

What a forecast actually is

Every planning organization produces a forecast, and almost every one of them produces the wrong object. Not an inaccurate one. The wrong kind of object.

What circulates in the monthly cycle is a single number per item per period. What the business actually faces is a range of possible outcomes with different likelihoods attached. The single number is one summary of that range, usually the mean or something near it, and the act of reducing the range to it discards precisely the information a decision needs.

Consider what Exhibit 1 shows. The point forecast of eleven units is not wrong; it is a perfectly reasonable summary of that distribution. But a planner deciding how much to stock needs to know something the point forecast cannot express: how bad the bad case is, how likely it is, and what it would cost. Those questions are answerable from the shape and unanswerable from the number.

This is why the most common improvement program in the discipline, which is to make the point forecast more accurate, so frequently produces disappointing returns. It is an investment in refining a summary that was the wrong summary to begin with.

Why accuracy metrics select the wrong model

The second problem is more serious, because it is actively misleading rather than merely incomplete. The metrics used to choose between forecasting approaches do not measure what the business cares about, and they can and do select the worse option.

Error metrics are symmetric. Being ten units over and ten units under score identically. The business is not symmetric. As Paper 02 established, the two directions carry different costs, land in different budgets, and one of them is largely invisible. A metric that treats them as equivalent is measuring something the organization has no economic interest in.

To make these concepts concrete, we walk through the same demand data working through two different models.

The setup

One item. Ten weeks of history. The ledger from Paper 02 applies: $10 for each unit carried that was not needed (capital, storage and expected markdown) and $85 for each unit of demand not served ($40 of forgone contribution plus $45 of customer damage, the midpoint of the signed range). The stocking decision is the forecast, one for one, which keeps the arithmetic visible.

Two candidate models are on the table. Model B's errors are the exact mirror of Model A's: same magnitudes, opposite signs. This is not a trick to make the point, it is the cleanest way to remove every other explanation. Any symmetric metric will score them identically, because symmetric metrics cannot see sign.

Step one: score them the way your organization scores them

A bake-off scored this way is a coin flip. Switching to RMSE does not help, because RMSE punishes large errors without asking which direction they went. The only line that distinguishes the two models is bias, and bias is conventionally treated as a diagnostic to be eliminated rather than a lever to be set.

Step two: price the same errors with the ledger

Split each model's errors into units carried and units short, then multiply.

Two models that no accuracy metric can tell apart differ by $4,500 over ten weeks on a single item. Model B costs 44 percent less. Annualized, that is roughly $23,000 on one line of the catalog, decided by a coin flip in a model evaluation that was reported as a success.

The two models are indistinguishable to the metric and $4,500 apart in the accounts. The metric is not incomplete. It is pointing the wrong way.

Step three: the model that loses the accuracy contest

Now take Model A, the one that was tied for first, and make it deliberately worse. Add a fixed buffer of 35 units to every forecast, sized from the ledger rather than from a service target. Call it Model C.

With $85 against $10, the critical ratio is 85 ÷ 95, or 89.5 percent. The ledger is telling you to sit near the 90th percentile of demand, not at the middle of it. A buffer is not a fudge factor here. It is what the arithmetic asks for.

Read the two rankings against each other. On accuracy: A and B tied for first, C a distant and embarrassing last, 76 percent worse on MAE and nearly double the MAPE. On money: C first by a wide margin, then B, then A. The rankings are inverted. The model your forecast scorecard would reject outright is the one that costs 67 percent less than the model it would have shortlisted.

Model C is safety stock, stripped of the framing. The point is not that a buffer is novel. It is that this buffer was sized by the ledger instead of by a service target, and that the choice between A and B, which no amount of safety stock fixes, was already worth $4,500 on its own.

Model C also looks indefensible on the one summary a planning review actually reads. It forecasts 1,390 units against 1,100 of demand, a 26 percent over-forecast sustained for ten straight weeks. Any competent forecast governance process would flag that as systematic bias and order it corrected. Correcting it costs the business $6,950 over ten weeks.

What this means for how you choose

The organizations that win are not the ones with the smallest forecast error. They are the ones whose errors cost the least.

The remedy is not a better error metric. Weighted and asymmetric metrics exist, and they are an improvement, but they are still a proxy standing between you and the number you actually care about. The remedy is to stop scoring the instrument and start scoring the outcome. Every candidate model gets evaluated by replaying the decisions it would have produced against the historical record and pricing those decisions with the ledger. Same history, same ledger, one number per candidate, in currency.

That is a harder evaluation to build. It takes a few weeks rather than an afternoon, and it requires Paper 02 to have been done first. It is also the only evaluation whose result a finance director has any reason to act on.

Model the drivers rather than adjusting for them

There is a third and quite different failure, and it concerns the things the business controls.

Price changes, promotions, marketing spend, assortment decisions and service commitments all move demand. In most planning processes they are handled as adjustments: a baseline forecast is produced, and then someone uplifts it because a promotion is planned. The uplift is a judgment, it is rarely recorded with its reasoning, and its accuracy is never assessed because the counterfactual is unobservable.

The correct treatment inverts the sequence. The controllable levers are decided first, and the model computes demand conditional on those decisions. The effect of each lever becomes a fitted relationship in its own right, estimated from the organization's own history of what it did and what followed, and carrying an explicit confidence range rather than a single coefficient.

Here is what changes in the room.

Somebody asks what happens if we double the promotion. Today that question is settled by whoever argues best. Marketing has a number from the last time they ran it. Supply has a different number and a long memory of the last time they believed marketing. The meeting picks one, the promotion runs, and six weeks later nobody can separate its effect from the weather, a competitor's price move, and the fact that Easter was early.

With lever effects modeled, that question has an answer before the money is committed. Not a certain answer. A number with a range around it, which is something two functions can argue about on the merits and then check afterward.

The second change is slower and worth more. Every promotion you run becomes evidence about the next one. The uplift argument stops resetting to zero every year.

When the data is too thin to exploit

Modeling lever effects raises an honest objection: for many decisions, the historical evidence is genuinely weak. The promotion was run once, three years ago, in one region, alongside a competitor action that confounds it. No amount of statistical sophistication conjures information that is not in the record.

This objection is correct, and the response to it is one of the few genuinely novel capabilities the architecture provides. A model that carries explicit confidence ranges knows where its evidence is thin. That knowledge is actionable: it identifies exactly which deliberate experiment would be worth running, and roughly what resolving the uncertainty would be worth in currency.

The organization therefore gains a queue of proposed tests, ranked by the value of the information they would produce. A controlled price variation in two regions. A holdout on one campaign. A deliberate service-level difference across comparable accounts. Each is small, each is priced, and each converts a permanent argument into a settled fact.

What to ask your analytics team on Monday

This paper is not a brief to rebuild the forecasting function. In most organizations the forecasting function is competent and is being asked the wrong question. Four questions are usually enough to establish where you actually stand.

1. Does our forecast carry a distribution, or only a number? If the model produces one internally and the process discards it downstream, the capability already exists and is being thrown away at the interface, which is a much easier problem than it sounds.
2. How do we choose between two candidate models? If the answer is an error metric, the selection process cannot see the difference that matters, and the fix is to score candidates on the priced outcome of their decisions instead.
3. Are promotions and price changes inputs to the model or adjustments after it? If they are adjustments, the organization is systematically failing to learn from its own commercial activity.
4. Where is our data too thin to trust, and what would it cost to find out? A team that can answer this immediately is further along than most. A team that has never been asked will usually have a good answer within a fortnight.

None of these require a platform decision, and all four are answerable with people already on the payroll. The next paper sets out the machine that these components fit into, and where exactly a human being enters it.