The Cost of Activity Metrics

The slides were moving fast. NPS scores were up. The chat was full of fire emojis and congratulations. Someone typed “incredible work team” and it got fourteen thumbs up before the next slide appeared.

I was sitting at my desk thinking one thing.

Will any of this change what I see happening with our partners?

It did not. It never does. And the honest answer to why is not that the training was bad, or that the facilitators were not skilled, or that the content missed the mark. The honest answer is that we were measuring the wrong things on purpose, and celebrating what we found.

This is the cost of activity metrics. Not the money spent on the training itself. The cost of deciding, implicitly and repeatedly, that completion is the same as change.

The Framework

We Have Known This for Seventy Years

Donald Kirkpatrick published his four-level model for evaluating training in the 1950s. Level 1 is reaction: did people like it. Level 2 is learning: did people acquire new knowledge or skills. Level 3 is behavior: did people actually do anything differently back on the job. Level 4 is results: did the organization get better because of it.

Most organizations never get past Level 1.

Not because they have not heard of Kirkpatrick. Most L&D professionals can recite the four levels from memory. They stop at Level 1 because reaction data is easy to collect, fast to present, and almost always positive. People are generally polite on a feedback form at the end of a session they just sat through. The scores come back good. The room celebrates.

What reaction data does not tell you is whether a single behavior changed the following Monday. And most organizations have quietly decided they would rather not know.

Robert Brinkerhoff spent decades studying why training fails to transfer. His finding was simple and devastating: the training event itself accounts for roughly 15% of the outcome. The other 85% is determined by what happens before the learning event and after it; the environment people return to, the manager who does or does not reinforce the new behavior, the system that does or does not make the new behavior easier than the old one. Most organizations design for the 15% and measure it enthusiastically. The 85% goes largely unexamined.

The Real Problem

The Belief Underneath the Metric

Here is the uncomfortable part. This is not ignorance. The people in that Teams meeting were not unaware that NPS scores are a measure of satisfaction, not impact. They knew. The question is why they celebrated anyway.

Activity metrics persist because they are safe. Completion rates go up when you run more sessions. Satisfaction scores go up when your facilitators are skilled and your content is well-designed. NPS scores go up when people feel good about what they experienced. None of these things require behavior to change. None of them require the organization to change. They only require the event to happen and the form to be filled out.

Impact metrics are dangerous. They require you to go back into the organization weeks or months later and ask whether anything is actually different. They require you to design the pre-work and the post-event reinforcement, not just the training day itself. They require a manager to be involved, which means a conversation about what managers are actually doing after their teams attend your program. And they risk coming back negative, which means the celebration in the chat does not happen, which means someone has to explain to leadership why the number is not good.

Measuring what is easy to measure is a choice. And the choice reveals what an organization actually believes about learning.

The Fix

What to Measure Instead

This is where most critiques of activity metrics stop. They name the problem, declare the smile sheet dead, and leave the reader with nowhere to go.

So here is the practical version.

If behavior change is the goal, you need leading indicators that show up before the 18-month cultural shift arrives. Three worth building into any program:

The first is the commitment record. At the end of any learning event, capture a specific behavioral commitment from each participant. Not a feeling, not a takeaway, a behavior: what specifically will you do differently, and by when. Route that record to whoever can hold them accountable. In a corporate context that is their manager. That commitment record becomes the agenda for the next one on one. The handoff from learning to work is no longer accidental.

The second is manager observation. If a manager cannot name one behavior they are watching for after their team attends a program, the program is not designed for transfer. It is designed for attendance. Build the manager into the design before the training day, not after it.

The third is the honest 60-day check. Not a satisfaction survey. A direct question: did you use what you learned, and what happened when you did. The answer to that question tells you more about whether your program works than every emoji in the chat combined.

The Bottom Line

The Real Cost

When organizations measure activity and call it impact, decisions get made on false data.

Programs that produce completion but not change get funded again because the scores look good. Programs that are harder to attend but actually shift behavior get cut because the numbers are smaller. The training calendar fills up with events that feel productive and accomplish little. And the L&D function spends its credibility defending metrics that everyone in the room privately suspects are not the point.

The gap between what we measure and what we actually want is not a measurement problem. It is a courage problem.

The Teams meeting will always feel good. The chat will always fill up. The question is whether anyone in the room is willing to ask the one question that matters: will any of this change what we see happening on the ground next month.