Measure the Work, Not the Hype: AI Results You Can Defend

Every AI pilot produces a highlight reel. The demo dazzles, the volunteers are enthusiastic, the vendor’s screenshots are beautiful. Then someone senior asks the only question that matters: what actually changed, and how do you know? Most AI projects cannot answer it — not because nothing happened, but because nobody decided in advance what evidence would count. Measuring AI is not a reporting task tacked on at the end. It is the discipline that separates an experiment from a performance, and it has to be designed before the first license is switched on.

Decide the test before you run it

A measurement that can defend itself starts with a baseline: how long does the work take now, how often does it go wrong, what does a finished case cost? Baseline first, tool second — because once the tool arrives, the old way is gone and the comparison is unrecoverable. Then choose the measures before the pilot starts, while nobody knows which numbers will flatter them. Choosing afterwards is how pilots get graded on the one metric that moved. And state the failure condition out loud: what result, honestly read, would make you stop? That sentence feels harsh in the kickoff meeting and merciful in the review. It is the difference between an experiment and an installment plan.

The question underneath the baseline is the counterfactual: what would this quarter have looked like without the tool? Work speeds up for other reasons — a quiet season, a process fixed in passing, a strong hire. A defensible measurement names those rivals and deals with them honestly, even if the dealing is as simple as comparing against a team that did not change. The point is not laboratory purity. The point is being able to say, in a room full of skeptics, “here is what we attribute to the tool, and here is what we don’t.”

Count outcomes, not activity

The easiest AI numbers to collect are activity numbers: logins, prompts, documents generated. They measure enthusiasm, not value, and they rise right up to the moment the novelty wears off. Working numbers live downstream: cycle time on the real task, error and rework rates, quality held or improved, hours returned to work that matters. They also live on the cost side of the ledger, where pilots prefer not to look — licenses, the human review the output still needs, the cleanup when a confident answer turns out to be wrong. A measurement that counts only the wins is a brochure. Both sides of the ledger, or the number is hype wearing a spreadsheet.

Two design choices make all of this practical rather than aspirational. First, time-box the pilot and fix its population: a named team, a named task, a defined window. A pilot that drifts across the whole company for an open-ended quarter can never be read, because nothing in it stays still long enough to compare. Second, write the measurement plan down before the licenses arrive — one page is enough: the baseline, the measures, the failure condition, the review date, and the name of the person who will read the results cold. Plans written before results exist cannot be accused of being fitted to them, and that is precisely the accusation every AI review will eventually face. The time to answer it is at the design, not in the meeting.

Report like you will be checked

The final discipline is the review itself. Let someone who did not champion the pilot examine the numbers — fresh eyes catch the baseline that quietly moved and the definition that quietly changed. Report the flat results alongside the good ones; a null finding stated plainly buys more credibility than a win stated loudly. Then decide, in writing: scale it, fix it, or stop it — and put the date on the next review. AI results you can defend are not the ones with the biggest percentages. They are the ones that survive a hostile reading, because they were built for one from the start.

← All Insights