The Baseline Problem: Measuring What Was Never Counted

The pilot ends, the results meeting convenes, and someone asks the only question that matters: compared to what? The room goes quiet in a particular way. The team measured the new process carefully — every draft faster, every ticket cheaper — but nobody measured the old one, because measuring the old one was nobody’s job while it was still just how things were done. So the improvement claim rests on a baseline reconstructed from memory, anecdote, and one manager’s strong impression that it used to take longer. This is the baseline problem, and it is the most common honest failure in AI measurement: not fraud, not hype, but an absent “before” that no amount of “after” data can repair. The work that follows a pilot judged this way is salvage — and salvage, done openly, is a respectable discipline with rules of its own.

Reconstructing a “before” you can defend

The records usually know more than the room remembers. The old process left evidence: timestamps in the ticketing system, email threads with their dates intact, invoices from the vendor who used to do it, payroll hours in the weeks before the change, the version history of the documents themselves. A baseline rebuilt from contemporaneous records — artifacts created before anyone knew a pilot was coming, and therefore created without an interest in the answer — is weaker than a planned measurement and stronger than any recollection. Where records run out, sample: take the cases that survive, measure the old way on a fair handful of them, and say plainly that is what you did. And throughout, carry the reconstruction as a range rather than a point. A baseline stated as “between six and nine hours, most likely seven” is not vagueness; it is the honest shape of the knowledge, and improvement claims sized against the wrong end of it should be discounted in the open, by the team itself, before a skeptic does it less kindly.

The rule the salvage operation teaches

Every team that has done this excavation once arrives at the same rule, usually written down in the post-mortem in tired language: next time, measure first. The operational form is simple enough to fit on the pilot’s authorization: no pilot starts until the current state has numbers on it — the time, the cost, the error rate, the queue — collected by the people who will later be asked to believe the comparison. Two subtleties complete the discipline. Define the comparison’s terms before the pilot, too: which cases count, whose hours count, whether the new tool’s supervision time is inside or outside the ledger, because terms negotiated after results arrive are negotiated by people who have already seen the results. And keep the old process measurable during the pilot — a holdout corner, a parallel week, anything that keeps the “before” alive a little longer — because conditions drift, and a baseline from a different season measures the season as much as the process. Compared to what? is not an embarrassment to be avoided. It is the whole method, asked on time.

One warning about the salvage itself: a reconstructed baseline is a foundation for learning, not for boasting. Internally, a range-estimated improvement can honestly guide the next decision — scale it, fix it, or stop it — because the alternative is deciding on nothing. Externally, in the board deck and the vendor’s case study, the same numbers acquire a certainty the excavation never produced, and that is where honest salvage turns into the hype the measurement discipline exists to prevent. Label the reconstructed figures as reconstructed, wherever they travel. The label is not a weakness in the report. It is the report’s credibility — the visible sign that this team distinguishes between what it measured, what it rebuilt, and what it merely believes, which is the entire difference between measurement and marketing.

← All Insights

© 2026 Cognosco Learning. All rights reserved.