At the beginning of all my training design projects, I quickly and carefully prepare a Level 4 evaluation plan. (his is the type of evaluation that relates training to business results. I know that conducting Level 4 evaluation is tricky, expensive, and time consuming. Few clients are interested in this type of evaluation, but I plan for it anyway.
Why do I do this? I will explain my reasons after I give you details of a recent Level 4 evaluation design.
Evaluating a recent training design
Recently I did some training for Customer Service Representatives (CSRs) for an insurance company on how to show concern and empathy during telephone conversations with customer. Let me describe my evaluation plan.
We will use an interview protocol to collect information from customers who had a conversation with a specific CSR during the previous 48 hours. This protocol will contain a set of standardized instructions on how to ask a series of open-ended questions that begin with “Tell me about your conversation with the CSR”. There will be additional follow-up questions. The interview will end with a request for the customer to rate the CSR’s performance along the dimensions of courtesy, friendliness, competency, and promptness.
All interviews will be recorded. They will be analyzed by a trained group of listeners who will rate the CSRs’ performance based on the customers’ comments.
Evaluation design
The evaluation will use a time series design with the interviews being conducted at three different times. We selected this evaluation design because the training workshops will be offered to different groups of employees, 20 employees at a time, once a week during a 10-week period. Customers who spoke to the same 10 CSRs will be involved in each of the three sets of interviews.
Here are additional details about the timing of these interviews:
Baseline interviews. These interviews will be conducted with two customers each who had talked with each of the 10 CSRs. This interview will be conducted before any CSR received his or her training.
Mid-course interviews. These interviews will be conducted with two new customers each who had talked with each of the 10 CSRs. Of these 10 CSRs, five would have received training and the other five would not have received training.
Final interviews. These interviews will be conducted with two new customers who had talked with each of the 10 CSRs. At this time, all 10 CSRs would have received training.
Analysis
At the end of all three sets of interviews, we will have a total of 60 recorded conversations, 20 each from the baseline, midstream, and final interviews. The analysts will listen to all interviews and rate the CSRs’ performance as reflected in the customer’s comments. The recordings will be randomly presented to analysts and they will not be told whether or not the CSR had received training before the interview.
Here’s what we hope to learn from the analyses:
Comparison between the baseline and the final interviews. Positive differences will show the probable impact of training. If there is no difference (or if there is a negative difference), we will have to question the validity of the training.
Comparison between the trained and the untrained CSRs during the mid-course interviews. The trained group scoring higher than the untrained group suggests the effectiveness of the training. If there is no difference (or if there is a difference in the opposite direction) we will have to question the validity of the training.
Comparison between the mid-course and final interviews. If the first group shows an improvement in the score, we may attribute it to the impact of additional real-world practice. If there is deterioration in the scores, we may suspect that the novelty effect is wearing off or that there is not enough incentives or management support to sustain the performance improvement.
If the client is willing to provide the resources and time, we will implement this Level 4 evaluation plan. However, in my experience, less than 5 percent of the clients are willing to support this type of evaluation.
So why am I doing it?
I don’t think that planning a rigorous Level 4 evaluation is a waste of time, even if it never gets implemented. In the long run, I think that this planning enables the training design to proceed at a faster, cheaper, and better fashion by focusing our efforts on business-related outcomes and reducing the need for revisions.
Planning Level 4 Evaluation provides a concrete goal for the training design team. It helps the SMEs and writers to better understand the rationale for the training project. We can align our training content and objectives to the business results by repeatedly asking ourselves, “How would this help us perform better on our Level 4 evaluation?”
We can share the interview and the analysis procedure with participants in our training session to concretely explain the overall training goal. We can create training activities that prepare participants to get ready for this type of evaluation. All of these activities speed up our instructional design process.
Why not Level 3?
Why don’t just plan for Level 3 Evaluation (measuring transfer of training to the workplace) that has a greater probability of being implemented? Why not just record actual CSR conversations with customers and analyze them to the see how the CSRs are applying their new skills and knowledge? To me, this approach appears to be the instructional equivalent of inviting the fox to guard the chicken coop. Using Level 3 data to predict business results assumes that the behaviors that we are training for will produce desirable business results. By planning for Level 4 Evaluation, we go beyond just what we taught our trainees and actually validating the training content.
Let me repeat
Here’s my main message: At the beginning of all training design activities, plan for a rigorous Level 4 Evaluation. Even if your plans do not get implemented, you can use it to align, improve, and speed up your training design activities.