Académique Documents
Professionnel Documents
Culture Documents
POWER HR FORUM
Subhasis Ghosh
Executive Director (HR), NTPC &
Honorary Secretary Power HR Forum
Introduction ............................................................................................................................. 1
2.
3.
4.
5.
6.
7.
4. Levels of Evaluation
Developing a credible and comprehensive evaluation process is much like putting together
the pieces of a puzzle. The first step of the puzzle is the evaluation framework. This
framework defines the levels at which training prorammes are to be evaluated and how
data is to be captured. The following framework is based on the work of Donald L.
Kirpatrick, with an addition of fifth level by Jack J. Phillips.
Table 1: Training Evaluation Framework
Level
Measurement Category
Learning
Application and
implementation
Business Impact
ROTI
Develop
Training
Programme
Objectives
Develop
Evaluation
Plan
STEP 1, 2 Evaluation
Planning
3
Level 5: ROTI
Level 4:
Business Impact
Level 3:
Application
Collect data
at workplace
Tangible
- Intangible
Isolate the
effects of
training
alone
Convert
Business Impact
Data to
monetary value
Level 2: Learning
Collect data
in the class
room
Based on the above evaluations levels in Table I, evaluation process is defined graphically as shown in Figure 1:
5. Evaluation Process
Programme
Cost
Calculate
ROTI
Level
1) Reaction &
Satisfaction
2) Learning
Knowledge,
skills,
behavioral
changes
Level
3) Application &
Implementation
4) Business Impact
5) ROTI
Outcomes of
application of
learning at
workplace
Monetary
benefits
Levels
Purpose
Level 1 Reaction
Satisfaction
Level 2 Learning
Ease
Usefulness
& Relatively
easy
Least
data
Target
Coverage
useful 100% (say)
80% (say)
Level 3 Application
20% (say)
5% (say)
Level 5 ROTI
Most difficult
Most useful
1% (say)
The sub steps for this particular step are given below:
1)
2)
Reaction
Learning
Timing of Data
Collection
Responsibility
of Data
Collection
It requires a one page document having Immediately at the Training
Likert scale for different measures on end of programme, Manager /
which feedback is sought. Two different within the classroom Coordinator
formats are given for illustration in
Annexure I.
Word of Caution: Data Collection at this level is very easy therefore training
managers get tempted to include as many factor as possible. But evaluation at this
level does not provide us any concrete information. Therefore we need to be
cautioned while choosing the factors to be measured and only focus on:- Factors
which are actionable and which can aid us to take a business decision.
This level requires instruments using which Pre-Post Test: Once
Training
knowledge/skill gained by participants at beginning and
Manager /
after training can be assessed. Details of other at end of
Coordinator
various instruments are shared in training programme,
Annexure II for more details.
within the classroom
Training
Manager/
Coordinator
Timing of Data
Collection
Business
Impact
Responsibility
of Data
Collection
Training
Manager /
Coordinator
Data Sources:
-Functional Heads
-Sectional Heads
-Direct Reporting Officer (in case of
senior level employees)
ROTI
ii)
ii)
business impact, following suggested strategies could be used to con vert data into
monetary values:
Output data, like additional sales can be converted to profit contribution
Cost of quality measure, such as customer complaint, can be calculated and
reported as standard value.
Employee time saved can be converted to wages saved and benefits, etc.
The programme benefit data is represented in annualized form which means the
analysis could have been for two months period and while reporting, it has to be
converted to 12 months by extrapolation.
iii)
Capture Programme Costs: Calculate all the direct and indirect costs involved in
the programme. A detailed list of various parameters are at Annexure III.
iv)
v)
10
3
4
Intangible benefits
Lessons learned
11
Page no
12
Sec 3
Determine the
Stakeholders
interested in
Training
Evaluation
Sec 2
Annexure III,
Sec A
Estimate the
cost of study
& get
approval
(Reaction/Learning/
Application/Business
Impact/ROTI)
Data
Analysis
Highlight desired
depth & breadth
of study to take
care of
staeholders
concern
Reporting
the outcome
of Evaluation
Project
Case I Training Evaluation is taken up as an Afterthought (after the specific training programme has been conducted)
7.
Stakeholder
Identification
13
Establish
level of
evaluation
Setting Objectives:
-Reaction
-Learning
-Application
-Business Impact
-ROTI
Data
collection
Capturing
Stakeholders
concern(s)
Programme Design
Data
Analysis
Case II Training Evaluation is taken up, as a planned initiative and thought of before the programme is conceived
Reporting
the
outcome of
evaluation
Annexure I
A. Sample Feedback Form I (used by NTPC)
PROGRAM TITLE
PROGRAM DURATION
Organizer
PMI
VENUE
Program coordinator
DMR PANDA
Please provide feedback on following parameters on a scale of 1-10, where 1 Not Satisfied and in
the increasing order of satisfaction 10- Highly Satisfied.
1. Please rate the extent to which Overall Program Objectives were achieved
Programme Objective:
Describe the internal assessment model for business excellence
State the coverage of nine criteria of excellence
Explain the process of assessment of submitted document from participating
stations
Evaluate and score the application document from any NTPC power station
Describe the process of generating feedback report for the particular station based
on the review of application document and plant visit
S
No
1.
Not Satisfied
Satisfied
1
2
3
Highly
4
10
2. Please rate the extent to which the following aspects of the program are met as per your expectation
S No
1.
2.
3.
4.
Dimensions
Not Satisfied
Satisfied
1
2
3
14
Highly
4
10
3. Please rate the extent to which different faculties met your expectations in terms of subject
knowledge, structure and delivery
S No
Session Topic
Faculty Name
Not Satisfied
1
Internal BE Model
Highly
Satisfied
4 5 6 7
10
C V Subrahmanyam
4. Please rate the extent to which following Administrative Aspects are met as per your expectation
S No
Dimensions
Not Satisfied
Satisfied
1
2
3
Highly
4
10
1. Hospitality Arrangements
2. Overall Administrative
Arrangements
3. In house residential arrangements
5. Kindly mention where and how you shall apply the learning at workplace?
6. Kindly give your comments regarding program and suggestions for improvement of the Program.
Date ------------------------------
Signature________________
Name ___________________
Emp No.________________
15
CSR & SD
September 29, 2014
Power HR Forum, PMI NOIDA
Please rate the extent to which Overall Program Objectives were achieved
Moderate (less than (<40%)
Partially (41-60%)
Fully (61-80%)
Up to - the mark (81 100%)
2.
3. How interesting, effective and appropriate were the case studies and examples
Moderate (less than 40%)
Relevant (41-60%)
Greater extent (61-80%)
Up to the mark (81 100%)
4. Please rate the extent to which the program will help in your official capacity.
Little (less than 40%)
Moderate (41-60%)
Greater extent (61-80%)
Fully (81 100%)
5. Please rate how this programme was coordinated?
Average (less than 40%)
Good (41-60%)
Very good (61-80%)
Excellent (81 100%)
(Not necessary to mention your name and other details)
16
Annexure-II
Test of Learning
The aim of the Learning Evaluation (Level-2) is to check if the learning of the participants
has reached to some desired level after attending the training programme. There are two
options in this regard:
1) Pre-post test
2) Post test
Further, each of these categories may include following methods:
a) MCQ (multiple choice questions)
b) True/False type
c) Subjective assessment
d) Demonstration through role play / working on models
e) Analysis of case studies
f) Simulation
The job of the training manager at this stage is to follow the learning objectives finalized
during the step 1. Then based on those objectives, determine the mode he/she going to use
for assessing the change in learning achieved during training programme as expla ined
below:
A) Pre-Post Training Test and Post Test:
Pre-Post Test is preferred where participant has some background knowledge of the
training programme contents or participants are already working in the same domain
and the intent of training programme is to acquire advanced skills. Then, training
manager can make out clearly improvement in the skill levels of employees.
Table 1: Comparison of Test Methods
Parameters
Pre-Post Test
Post Test
Level of skill to be achieved
Degree of learning Certain
level
of
enhancement
proficiency has to be
achieved; say 70 % at
end of training.
Criticality of learning
High
Lower
Location of proficiency
Within the classroom
Within the classroom
Extra time and cost
More
Less
Seriousness/objectivity/engagement High
Lower
of participants
17
18
Usually objective test is most widely used which is a good compromise between
outcome achieved and input required.
(C) How to decide the Number and Difficulty Level of Questions in Pre-Post/Post Training
Tests
To get the more accurate picture of the knowledge/skills acquired during the training
programme, inclination is on to ask more number questions, for a clearer assessment.
But this has to be restricted to limit ourselves and be more focused. When we are going
for objective type of questions, one rule of thumb is to ask 16 questions for a day of
training programme or 4 questions per session. Say a training programme is scheduled
for 3 days, then the number of questions come down to 45. In case of subjective type of
questions 6 questions per day should be asked. A combination of objective as well
subjective can also be used.
While deciding the level of questions to be asked in the Pre-Post/Post Training Tests, the
objectives of the training programme as pre-decided need to be considered and
accordingly decided.
19
Annexure III
Direct costs are expenses an organization can easily specify for an object or
services acquired. Companies describe these as expenses where one gets what
one pays. An organization can have its own list of direct cost depending on the
resources available with them. A list specifying indirect costs incurred in a
training programme is given below:1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
ii)
Indirect cost involves costs which are spread out for usage for different
purposes and cannot be calculated directly for the product/service acquired. A
list specifying indirect costs incurred in a training programme is given below:1.
2.
3.
20
21