The short answer
Confidence scores show how people feel, not what they do. Measure at four levels: reaction, learning (scenario checks), behavior (use of approved tools on real work, reviewed), and results (time, rework, or incidents on a few baseline tasks). Set baselines before training and report at 90 days.
Why are confidence scores not enough?
Confidence is useful and cheap to collect, and it is also self-reported. In our own flagship rollout, participants reported confidence rising from 3.0 to 4.8 on a five-point scale. We report that as exactly what it is: a participant-reported signal.
It does not tell you whether people use Claude on real work, whether their output is better or checked, or whether any business measure moved. A team can feel more confident and change nothing, or feel unsure while quietly saving hours.
Good measurement answers three further questions: Did people learn to make good decisions? Did they change what they do? Did it matter to the business? The Kirkpatrick model is a standard way to organize them.
How do the four Kirkpatrick levels map to Claude training?
The Kirkpatrick Model defines four levels: Reaction, Learning, Behavior, and Results. Here is how to apply each to Claude training:
| Level | Question | Example Claude measures | Where the data comes from |
|---|---|---|---|
| 1. Reaction | Was it relevant and engaging? | Session ratings, "will I use this on my work?" | Short survey after each session |
| 2. Learning | Did people gain skill and judgment? | Scenario check scores by scenario; confidence, labeled as self-reported | Live scenario results; pre and post survey |
| 3. Behavior | Are they using it on real work, safely? | Active use of approved tools; share of sampled outputs that were reviewed; policy incidents | Admin usage reports; manager spot checks; incident log |
| 4. Results | Did the business outcome move? | Time on a defined task; rework rate; turnaround; error rate | Baseline vs after on the same tasks |
Levels 1 and 2 you can measure during training. Levels 3 and 4 need time and a baseline, which is why the 90-day readout matters.
How do you set a baseline before training?
A baseline is what the work looks like today, measured the same way you will measure it later.
- Pick three to five recurring tasks per department, such as a weekly report, a customer reply, or a document summary.
- Define the measure for each: minutes to a reviewed draft, number of revision rounds, or errors found in review.
- Collect a before sample from real work, using the same reviewers and criteria you will use afterwards.
- Record the context: volume, season, and who did the work.
Without this step, any later claim of time saved is an estimate. If you must estimate, label it as an estimate and say how it was derived.
What belongs in a 90-day readout?
A readout turns measurement into a decision. A useful structure:
- What we set out to do: the objectives and the baseline tasks.
- Participation: who was trained, by level and department.
- Learning: scenario results by scenario, and where people struggled.
- Behavior: active use of approved tools and reviewed-work samples; incidents and near misses.
- Results: before and after on the baseline tasks, with sample size and method stated.
- What did not work and what changed.
- Recommendations: which departments and use cases next, and what to fix first.
Our flagship engagement ended with a 90-day readout that set up Phase 2. Report use, not attendance, so the sponsor can see who is actually working differently.
How do you calculate ROI without fooling yourself?
The arithmetic is simple. The discipline is in the inputs.
ROI = (value of time recovered + avoided costs − program cost) / program costThree cautions:
- Time recovered is only value if it is used. An hour saved that disappears into nothing is not a return. Count it when it is redeployed to other work or avoids a cost.
- Use measured time, not hoped-for time. Self-reported savings tend to be generous; prefer your baseline comparison, and label estimates.
- Name the confounds. Seasonality, a novelty effect, and measuring only enthusiastic volunteers can all flatter the result. Compare like with like, and include some people who were not keen.
A purely hypothetical illustration of the method, not a result: suppose a team of 10 saves 1.5 measured hours a week each on baseline tasks for 40 working weeks, and you value an hour at a figure your finance team agrees. Multiply those inputs, subtract the program cost, divide by the program cost. Change any input and the answer changes, so show your inputs.
Our Readiness Check includes an adjustable estimate of hours recovered. It is labeled illustrative because actual time saved should come from your own pilot.
Frequently asked questions
How do you measure the ROI of AI training?
Set baselines on a few recurring tasks before training, then compare time, rework, or errors afterwards, and add behavior measures such as active use of approved tools. Report at 90 days with the method and sample stated.
What are the four Kirkpatrick levels?
Reaction, Learning, Behavior, and Results, per Kirkpatrick Partners. They move from how people felt about training to whether business outcomes changed.
Is a confidence score a good measure of AI training success?
It is a useful but self-reported signal. Pair it with scenario checks, real usage data, and baseline-versus-after task measures.
Sources and further reading
Product features, plans, and limits change often. This guide reflects the vendor documentation listed above as of November 2026. Check the vendor’s current documentation before you make purchasing or policy decisions. TechStack Consulting is an independent training firm and is not affiliated with Anthropic or OpenAI.