Designing and Evaluating Competency-Based Training Materials

Last week we talked about the assessments that need to be conducted to prepare for shifting from a read and acknowledge training framework to a competency based program. With those assessments in hand, it’s time to dig in and begin designing training and assessment materials.

What do the training materials look like in competency based training? The possibilities are virtually unlimited, ranging from leader's guides for classroom-based or on-the-job training to self-instructional programs delivered in computer-based or workbook formats. The materials are dictated in large part by the training situation, which includes the constraints placed on the way in which the training can be delivered.

In most cases, though, job training involving SOPS is delivered by an SME trainer in a combination on-the-job and classroom setting. This entails developing leader's and participants' materials, job aids such as checklists, and appropriate performance demonstrations to ensure competency.

The analyses you developed will form the backbone of your training. The needs analysis showed where training is needed and by who. The situation analysis showed how the training needs to be delivered. The task analysis revealed what information needs to be covered in the training.

As trainers present information to trainees, the goal is to make sure that each item identified by the analyses has been covered. Converting the task analysis to a checklist can help in making sure all the needed material is covered during training.

Evaluations

Once implemented, the new training programs need to be evaluated for four criteria.

Level 1: Reaction

What did the trainees feel about the program? Evaluation of trainees' reactions to training is usually conducted by questionnaires administered immediately after the program ends.

This type of evaluation can measure whether the program met its objectives, whether the trainees felt the training could be improved, and how well the program applies to trainees' jobs. It should not be used as the only measurement, though. Level I evaluations are most useful when they are complemented by the other levels of evaluation, which help us measure the trainee's skills and knowledge, and measure the results of the training.

Level 2: Learning

Most technical training is aimed at conveying skills and knowledge. To what degree were skills and knowledge affected by the training? This can be measured by pre- and post-tests administered before and after the training. In most cases tests contain a written component and performance demonstrations to ensure skills as well as knowledge have been conveyed. Reviewing these evaluations over time will help us establish the reliability of the program.

Level 3: Behavior

Suppose Level 1 and Level 2 evaluations show participants felt positive about the training and scored well on the post-test. Next we need to assess whether this change in behavior transferred to the job. Did participants change their job behavior as a result of the training? In what ways? If the participants did not change their behavior, was it because other factors intervened--for example, a supervisor discouraging the new behavior or a leader who failed to reward it?

This type of assessment needs to be conducted in follow up reviews of actual job performance, either through direct monitoring or measuring metrics that can capture data about compliance with the new training.

Level 4: Results

Did the training pay off? Did productivity increase? Was improvement in quality or reduction in costs noted? We should never forget that a primary objective of job training is to ensure on-the-job results.

This type of evaluation can be difficult to design and administer, which is why it is still seldom performed in industry-though this will probably change over time. In many cases it involves gathering baseline data before the training and comparing it with post-training results.

Evaluation at this level can involve looking at the cost of time spent in training, the costs of the program, and other factors that many trainers and managers would just as soon forget. But the purpose of industrial training is ultimately to produce results that impact favorably on the bottom line, and Level 4 evaluation can help make the case for training.

Next week, we’ll talk about putting all the parts of the competency based training program together and using them to help build support for additional investments in training.

Contact Information:

Dave Gallup
GMPTraining.com, Inc.
18585 Coastal Highway
Unit 10, #149
Rehoboth, DE  19971
215-870-5665
dagallup@gmptraining.com
www.gmptraining.com