Vous êtes sur la page 1sur 24

Plan to Evaluate the PROF2001:

Enterprise Competency Management


(ECM) System for Improved Professional
Development Training for Supervisors
Rozy Parlette
HRD 585
May 6, 2014

Table of Contents
A.

Rationale Statement............................................................................................ 2

B.

Purpose Statement............................................................................................... 3

C. Audience.............................................................................................................. 3
Primary Stakeholders.............................................................................................. 3
Secondary Stakeholders.......................................................................................... 3
Tertiary Stakeholders............................................................................................... 4
D. Key Questions...................................................................................................... 4
E.

Evaluation Design................................................................................................ 5
Mode of Inquiry....................................................................................................... 5
Nature of Evaluation................................................................................................ 5
Evaluation Design................................................................................................... 5
Method of Data Collection....................................................................................... 6
Drafts of Instruments.............................................................................................. 8
Pre-test................................................................................................................. 8
Post-test............................................................................................................. 10
Questionnaire for Employees of Supervisor.......................................................11
Questionnaire for Organizational Leaders of Supervisor(s)................................12

F.

Data Analysis..................................................................................................... 14

G. Evaluation Management Plan.............................................................................15


H. Evaluation Constraints......................................................................................... 18
I. Reporting the Results............................................................................................ 19
J. Evaluation Budget................................................................................................. 20

A. Rationale Statement
A U.S. Intelligence Community (IC) Agency with a mission to protect U.S. national security
systems and to produce foreign signals intelligence information conducts a climate survey
annually. Agency employees are dispersed across the world and apply their skills to gather and
analyze foreign intelligence, protect our nation's information systems, advance research, and
solve a number of other challenges not found in the private sector. The work they do is not only
important for policymakers and military leaders, it is critical to helping to keep the world safe.
According to the outcomes of a recent climate survey conducted by the IC agency, workforce
morale is low and performance ratings are no longer a good indication of performance because
90% of the workforce is given the highest rating possible in their annual evaluations.
The results of the climate survey also revealed that employees feel like they dont know what
they need to do to be successful and their supervisors dont know how to or dont want to
talk to their employees about their performance and/or how to improve performance. Overall,
there is little understanding of the workforces true skill strengths and gaps. All of these problems
are exacerbated by the fact that the Agency is responsible for responding to mission-critical
national security events on a 24-by-7 basis and their workforce require specialized skills to do
their jobs.
In reaction to these issues, the agency is in the process of implementing a competency-based
assessment tool (Enterprise Competency Management, ECM) that identifies skill strengths and
gaps, requires a discussion between a supervisor and employees to de-conflict competency
proficiency ratings, and is linked to training to fill skill gaps. The desired results for the adoption
of the ECM professional development system is to help civilian supervisors and agency leaders
to: 1) capture the strengths and gaps of the workforce to measure technical health; 2) conduct
meaningful and better-informed development discussions; and 3) determine how to best spend
training dollars to fill workforce skill gaps.
To aid in the organizational change management efforts, the Associate Director for Education
and Training (ADET) was enlisted to develop, design, implement, and evaluate the supervisorspecific, ECM training for the agency. The training is a three hour workshop entitled
PROF2001: ECM System for Improved Professional Development, which has been designed
to help all civilian supervisors in the agency who are currently responsible for completing
employee performance evaluations to:
1) articulate the characteristics and purpose of ECM;
2) discuss each of the tools in the ECM system;
3) complete all the steps included in an e-Competency assessment;
4) use the assessment results to understand employee competency strengths and gaps as
well as the training and development activities needed to fill gaps; and
5) utilize best practices for conducting a professional development discussion focused on
employee strengths and development areas.

The evaluation is being conducted to identify possible enhancements that could improve the
trainings effectiveness in meeting its objectives and the needs of the target audience. The results
of the evaluation will also be considered in relationship to the trainings costs to determine if it
should be continued. The primary factors leading to the evaluation are the desire to deliver an
effective and impactful training that addresses the issues identified in the workforce climate
survey and ensures adoption of the ECM professional development system by civilian
supervisors; and in turn their employees. As such, it is critical that the training be perceived as
valuable, worth the time investment, and easily and quickly applicable to the day-to-day
supervisory routine.
The evaluation for this training is not required by the Agency, nor any other external body, but is
being commissioned by the IC Agencys Chief of Staff (CoS) and overseen by the ADET and
executed by ECM Program Team to determine the extent to which the training meets the needs
and expectations of the target audience; supervisors.

B. Purpose Statement
The purpose of the evaluation study is to determine the extent to which the training meets the
needs and expectations of the target audience which includes all civilian supervisors in the
agency who are currently responsible for completing employee performance evaluations and
addresses the issues identified in the workforce climate survey and ensures adoption of the ECM
professional development system by civilian supervisors; and in turn their employees.
The results will be used to determine if the training should continue to be offered and, if
continued, to identify enhancements of the content, facilitators, delivery method, or activities to
meet course objectives.

C. Audience
Primary Stakeholders
The primary stakeholders include the IC Agency administration, who commissioned the
evaluation, and the ADET. The ADET is accountable for generating the anticipated results and
reporting the training outcomes of this evaluation to the Agency administration. The ADET will
determine if the training is effective and what, if any, changes need to be made to improve the
training. They will be in charge of re-designing the course curriculum to reflect the needed
improvements and will oversee the execution of the improvements. The Agencys CoS will fund
the training and this evaluation.

Secondary Stakeholders
The secondary stakeholders are the ECM course facilitators. The facilitators will be responsible
for delivering the ECM training to the supervisors. They will need expert knowledge of the Final
Development Document: Facilitators Guide, the guidelines of PROF2001: ECM System for
Improved Professional Development, and the ECM training system. Other secondary
3

stakeholders are the supervisors providing the civilian employee training. They will be
responsible for delivering the training to their staff. In order for the supervisors to be successful,
they must have an in-depth understanding of all components of the ECM professional
development program. They will use the information learned in the professional development to
improve their staff relations and staff productivity. Also included in the secondary stakeholders
are the civilian trainees receiving the ECM training from their supervisors. They will benefit
from the training by gaining more positive workforce perceptions of professional development,
management, and morale. A more positive job outlook by employees will lead to more
productivity and better job performance.

Tertiary Stakeholders
The tertiary stakeholders include other U.S. government agencies that may face the same
workforce problems. They could include agencies that may be contemplating the use of a similar
competency management system. The other agencies could gather data on the effectiveness of
ECM and use it as a model to design their own system to address the issues of professional
development. If the ECM system is successful, it could significantly improve staff relations and
training within many U.S. governmental agencies. Staff from other U.S. government agencies
could also be tertiary stakeholders as any improvement in employee relations, morale, and
training in one agency may lead to improvements in their agency. In addition, the American
people can be included as tertiary stakeholders. The American people benefit when the
government operates efficiently. This includes agencies that ensure Americas safety and use
citizens tax dollars to fund government operations.

D.Key Questions
Q1. How can the training effectiveness be measured?
Q2. In what ways and to what extent the supervisors are able to apply what they have learned
from the course?
Q3. In what ways have the employees and organizations of supervisors who have completed
PROF2001 benefited from the professional development?
Q4. What changes could improve the PROF2001 training to better transfer new knowledge,
skills, and attitudes to the supervisors work setting?

E. Evaluation Design
Mode of Inquiry
Our evaluation will employ a mixed-method approach and focus on collecting both qualitative
and quantitative data to answer the key questions of the evaluation. A mixed-method approach is
the best for our study because: mixing both qualitative and quantitative data helps to offset the
weakness that either method may have; using both qualitative and quantitative data gives a more
comprehensive account of whats going on in the training program; both qualitative and
quantitative approaches enhance the integrity and credibility of an evaluation outcome because
one type of data can bolster the other and help to better interpret findings; and quantitative data
is already accessible in the climate survey results.

Nature of Evaluation
Our evaluation is summative in nature because it seeks to determine whether a training program
that has already been launched is delivering on its objectives. Essentially, we are evaluating the
overall performance or efficacy of the training course, which is appropriate for a summative
evaluation. More specifically, we will be conducting a Kirkpatrick Level 3 evaluation to
determine whether the PROF2001 learners (supervisors) are able to demonstrate the new
knowledge, skills, and attitudes in their work setting.

Evaluation Design
The specific design our study represents is a mixed methods approach consisting of:

Pre-testPost-test Control Group Design to assess the PROF2001 course impact on the
attitudes and behaviors of supervisor-level participants. The sample size will consist of at
least 50 participants. To provide greater internal validity to the study, we will assign two
groups, one that participates in the training, and one that does not. Group 1 and Group 2
will both complete a Pretest about ECM at the same time. Then, Group 1 will complete
the training, while Group 2 does not. At the end of the course, Group1 and Group 2 will
take the Posttest on ECM at the same time to help determine whether the training helps
improve a supervisors ability to transfer the their new knowledge, skills, and attitudes in
their work setting

Questionnaires for employees and organizational leaders of supervisors who have


completed PROF2001 to determine whether supervisor knowledge, skills, and behaviors
towards professional development have changed.

Interviews with supervisors that have completed the PROF2001 course conducted via
phone which seeks to measure transfer of learning.

Method of Data Collection


Key Evaluation Question(s)

Q1. How can the training


effectiveness be measured?

Q2. In what ways and to what


extent the supervisors are able to
apply what they have learned
from the course?
Q3. In what ways have the
employees and organizations of
supervisors who have completed
PROF2001 benefited from the
professional development?
Q2. In what ways and to what
extent the supervisors are able to
apply what they have learned
from the course?

Method of Data Collection


Pre-test-Post-test: Conduct Pretest and Post-test with two
groups, one that participates in
the training, and one that does
not. Both groups complete a
Pretest about ECM at the same
time. Then, Group 1 will
complete the training, while
Group 2 does not. At the end of
the course, Group1 and Group
2 will take the Posttest on ECM
at the same time.

Questionnaire for Employees


of Supervisors who have
Completed PROF2001

Questionnaire for
Organizational Leaders of
Supervisors who have
Completed PROF2001

Sampling Approach

Instrument Validity

The sample size will consist


of at least 50 participants in
The data triangulation of
total.
Group 1 Random sample of
25 supervisors needed
Group 2 Random sample of
25 supervisors needed

All employees in the


management chain of
supervisors who have
completed the
PROF2001course

supervisor/participant tests with


employee and organizational leader
feedback helps to validate the
evaluation
Pilot testing
Create audit trail

The data triangulation of


supervisor/participant tests with
employee and organizational leader
feedback helps to validate the
evaluation
Create audit trail

Organizational leaders (at the The data triangulation of


chief of staff and Deputy level)
supervisor/participant questionnaire
in the management chain of
with employee and organizational
supervisors who have
leader feedback helps to validate the
completed the
evaluation
6

Key Evaluation Question(s)

Method of Data Collection

Q3. In what ways have the


employees and organizations of
supervisors who have completed
PROF2001 benefited from the
professional development?
Q2. In what ways and to what
extent the supervisors are able to
apply what they have learned
from the course?
Q4. What changes could
improve the PROF2001 training
to better transfer new knowledge,
skills, and attitudes to the
supervisors work setting?

Sampling Approach

PROF2001course

Post-Training Phone Interviews


with Supervisor-level
PROF2001 Graduates

Instrument Validity

Create audit trail

The sample size will consist of


at least 100 participants who
The data triangulation of
completed the PROF2001
supervisor/participant responses
course within the timeframe of
with employee and organizational
April - June 2014. The
leader feedback helps to validate the
interviews will be set up via
evaluation
email calendar invites,
Checking accuracy in data recording
conducted via phone call, and
and coding for survey responses
collected in a spreadsheet that
will be coded.

Drafts of Instruments

Pre-test

Sender: ECM Program Team


Receiver: Random Sampling, Group 1 & Group 2 supervisors
Purpose: To measure supervisor pre-knowledge and record behaviors and attitudes
before training and create a control group response.
Format: Email with a link to survey on intranet site.

When: At the same time, one week before the training course is offered to Group 1.

1. What tools do you use for guiding your employees professional


development today? (choose all the options that apply)
A. I use the tools that my organization provides (internship programs,
etc.).
B. I point my employees to the ADET website.
C. I suggest my employees use the individual development plan (IDP)
tool.
D. I point my employee to all the tools listed above.
E. I dont use any tools for guiding my employees professional
development today.
2. What do you think ECM is? (choose all the options that apply)
A. ECM is the same as the e-IDP.
B. ECM is a new, competency-based professional development
program for the Agency.
C. ECM is an online system that helps to consolidate disparate
development programs in an Enterprise-wide system.
D. ECM is another way to evaluate the performance of my employees.
E. Both B and C.
3. How do you determine your employees skill strengths and
development areas? (choose all the options that apply)
A. I use the Annual Contribution Evaluation (ACE) to guide my
determination.

B. I use the other (non-ACE) tools/forms that the Agency provides me


to guide my determination (i.e., 360 assessments, IDP).
C. I just go with my gut to determine strengths and development
areas.
D. I rarely think about my employees strengths and development
areas.

4. What are your impressions of conducting development discussions with


your employees? (choose all the options that apply)
A. I enjoy conducting development discussions with my employees and
do so on a regular basis.
B. I only engage in development discussions if I have to and even then
I dont feel like I have the right tools for a productive discussion.
C. I never conduct development discussions and would rather my
employees learn about their shortcomings around the water cooler.
5. How would you rate your proficiency in managing professional
development for your team today?
A. 0 No Proficiency
B. 1 Basic: demonstrates familiarity with concepts and processes
C. 2 Intermediate: demonstrates understanding of concepts and
processes
D. 3 Advanced: demonstrates broad understanding of concepts and
processes
E. 4 Expert: demonstrates comprehensive, expert understanding of
concepts and processes

Post-test

Sender: ECM Program Team


Receiver: Random Sampling, Group 1 & Group 2 supervisors
Purpose: To measure supervisor post-knowledge and record behaviors and attitudes after
training and create a control group response.
Format: Email with a link to survey on intranet site.

When: At the same time, the last day the training course is offered to Group 1.

1. What are the key tools involved in the ECM system?


A. ACE, e-Competency, e-IDP, e-ITP, e-JQS
B. ACE, e-Competency, e-IDP, e-ITP, e-JQS, PHP
C. e-Competency, e-IDP, e-ITP, e-JQS
D. e-Competency, e-IDP, e-ITP, e-JQS, Leadership Navigator
2. What are the key benefits of the ECM system?
A. Professional Development and Technical Health
B. Professional Development, Performance, and Technical Health
C. Professional Development, Agency Goal Setting, and Leadership
Development

3. What is your role in the ECM system?


A. Help your employees to understand their performance objectives
and meet them.
B. Help your employees to understand their job requirements,
determine skill strengths and gaps, offer development opportunities to
fill gaps, encourage them to work with mentors, approve requested
training, and conduct regular development discussions.
C. Review your employees e-IDP and approve training in their e-ITP.
6. How might you determine your employees skill strengths and
development areas today?
Free text field.
7. What are your impressions of conducting development discussions with
your employees?
Free text field.

10

Questionnaire for Employees of Supervisor

Sender: ECM Program Team


Receiver: All employees in the management chain of supervisors who have completed the
PROF2001course

Purpose: To measure whether the employees of the supervisor-level PROF2001


graduates are seeing benefits of the course in their professional development experience.
Format: Email with a link to a survey on intranet site. Keeping it quick because these employees
work in a fast-paced, demanding mission.

When: Send 3 months following course completion

1. Your Name. ____________________________________


2. Since my supervisor returned from the PROF2001 course, I have
benefitted from improved professional development discussions.
A. Strongly
E. Strongly
F. Not
B. Disagree
C. Neutral
D. Agree
Disagree
Agree
Applicable
3. Please provide a specific example of your supervisor improving on your
professional development discussions
Open field
4. Since my supervisor returned from the PROF2001 course, I have
benefitted from a better understanding of my strengths and
development areas.
A. Strongly
E. Strongly
F. Not
B. Disagree
C. Neutral
D. Agree
Disagree
Agree
Applicable
5. Please provide a specific example of your supervisor helping you to
better understand your strengths and development areas.
Open field

6. Since my supervisor returned from the PROF2001 course, I have


benefitted from an improved understanding of how to take action on
strengths and development areas.
A. Strongly
E. Strongly
F. Not
B. Disagree
C. Neutral
D. Agree
Disagree
Agree
Applicable
7. Please provide any other feedback youd like to share with regards to
your supervisors professional development activities following his/her
return from the PROF2001 course.
Open field

11

Questionnaire for Organizational Leaders of Supervisor(s)

Sender: ECM Program Team


Receiver: Organizational leaders (at the chief of staff and Deputy level) in the management
chain of supervisors who have completed the PROF2001course

Purpose: To measure whether the organizations of the supervisor-level PROF2001


graduates are seeing benefits of the course in their organizational objectives. Since
organizational leaders will likely have multiple supervisors attending training at the same time
(due to the phased, organizationally-driven approach to implementation) we will address changes
in their overall supervisory cadre.
Format: Email with a link to a survey on intranet site. Keeping it quick b/c they are senior
leaders with very little bandwidth.

When: Send 3 months following a cadre-sized course completion.

1. Your Name. ____________________________________


2. Since my supervisor cadre returned from the PROF2001 course, my
organization is now better able to capture the strengths and gaps of
the workforce to measure technical health.
A. Strongly
E. Strongly
F. Not
B. Disagree
C. Neutral
D. Agree
Disagree
Agree
Applicable
3. Please provide a specific example of your supervisor cadre better
capturing the strengths and gaps of the workforce to measure technical
health
Open field
4. Since my supervisor cadre returned from the PROF2001 course, my
organization is now better able to determine how to best spend
training dollars to fill workforce skill gaps.
A. Strongly
E. Strongly
F. Not
B. Disagree
C. Neutral
D. Agree
Disagree
Agree
Applicable
5. Please provide a specific example of your supervisor cadre better
determining how to best spend training dollars to fill workforce skill
gaps.
Open field

6. Was there any support you needed to provide to facilitate this?


Open field

7. Which aspects of the course have proved to be of most value to your


12

delegate in his/her work? If at all possible, provide specific examples of


improved results:
Open field

8. Please provide any other feedback youd like to share with regards to
your supervisor cadres professional development activities following
its return from the PROF2001 course.
Open field

13

Post-Training Phone Interviews with Supervisor-level PROF2001 Graduates

Sender: ECM Program Team


Receiver: At least 50 of the supervisors who have completed the PROF2001course
Purpose: To measure whether the supervisors have transferred the knowledge, skills, and
attitudes learned in PROF2001 to their work setting and uncover any barriers
organizational or individual to successful transfer
Format: Phone call scheduled via email calendar invite. No more than 15 minutes.

When: Send 3 months following course completion.

1. How would you rate your proficiency in managing professional


development for your team today?
A. 0 No Proficiency
B. 1 Basic: demonstrates familiarity with concepts and processes
C. 2 Intermediate: demonstrates understanding of concepts and
processes
D. 3 Advanced: demonstrates broad understanding of concepts and
processes
E. 4 Expert: demonstrates comprehensive, expert understanding of
concepts and processes
2. Did this course enhance your knowledge / skill? Please give specific
examples of enhanced knowledge / proficiency.
3. Have you received the necessary support back at work to apply the
knowledge, skills, and attitudes taught in the course? Please give
specific examples.
4. Which aspects of the course have proved to be of most value to you in
your work? Please give specific examples.
5. Are there any aspects of the course that you now feel should have been
handled differently?

14

15

F. Data Analysis
For all quantitative analysis, we include descriptions of: total number of people in the population,
the number of people in the sample, the number and percentage of respondents, the number and
percentage of missing responses, any recordings or re-scorings that were performed in translating
responses into numerical scales, a measure of central tendency, and a measure of dispersion. For
all qualitative analysis we will derive categories from the current data set to code the data. We
will ensure that our categories are mutually exclusive and we will keep notes about the
justification for why a response goes in one category over another. We will include a
miscellaneous category for data that dont fit in the categories we determine. Both quantitative
and qualitative data analysis will be captured in a final outcomes report that will articulate an
Executive Summary, Background, Objectives, Sampling Approach for each Data Collection
Method, Participants, Data Collection Methodology, Themes & Recommendations, and
Conclusion.
Method of Data
Collection

Pre-test-Post-test

Questionnaire for Employees


of Supervisors who have
Completed PROF2001

Data Analysis Approach


The pre-test and post-test will be administered in person at the
Agencys Assessment Center via a computerized adaptive test (CAT).
The results of all the tests will be exported from the CAT system into
a spreadsheet. In the spreadsheet we will check for correct answers to
the tests and calculate a frequency distribution to uncover disconnects.
We will compare # of correct answers to the pre-test and post-test
between Group 1 and Group 2 to determine whether taking the course
was helpful for acquiring new knowledge, skills, and attitudes toward
professional development. We will look at percentage of correct
answers and use that as a determining factor for success and then
include the findings in the final outcomes report.
Questionnaires will be emailed to participants with a link to survey on
an intranet site. The survey tool has the capability to export responses
to a spreadsheet. For the quantitative (Likert scale) questions in the
questionnaire, we will measure central tendency and dispersion.
For the qualitative questions in the questionnaire, we will derive

categories from the current data set and code the data and
measure central tendency.
Questionnaire for
Organizational Leaders of
Supervisors who have
Completed PROF2001

Questionnaires will be emailed to participants with a link to survey on


an intranet site. The survey tool has the capability to export responses
to a spreadsheet. For the quantitative (Likert scale) questions in the
questionnaire, we will measure central tendency and dispersion.
For the qualitative questions in the questionnaire, we will derive

categories from the current data set and code the data and
measure central tendency.
16

Method of Data
Collection

Data Analysis Approach

Post-Training Phone
Interviews with Supervisorlevel PROF2001 Graduates

Interviews will be conducted by two facilitators, who will split up the


interview schedule. Interviewers will collect data during the phone
call by typing responses as he/she conducts the interview. All
feedback will be aggregated and transcribed into a single comment
spreadsheet in preparation for content analysis. Facilitators meet to
discuss primary impressions and categorize content and include the
findings in the final outcomes report.

G.Evaluation Management Plan


Tasks

Responsible
Parties

Finalize Evaluation Plan


ECM Program
Team
Receive approval of the evaluation plan from ADET
ECM Program
Team & ADET
Receive approval of the evaluation plan from CoS
ECM Program
Team & CoS
Develop Evaluation Instruments
Develop the Pre-Test
ECM Program
Team
Develop the Post-Test
ECM Program
Team
Develop the Questionnaire for Employees of
ECM Program
Supervisors who have Completed PROF2001
Team
Develop the Questionnaire for Organizational Leaders
ECM Program
of Supervisors who have Completed PROF2001
Team
Develop the Facilitators Guide for Post-Training Phone
ECM Program
Interviews with Supervisor-level PROF2001 Graduates
Team
Review and approve the Pre-Test
ADET
Review and approve the Post-Test
ADET
Review and approve the Questionnaire for Employees
ADET
of Supervisors who have Completed PROF2001
Review and approve the Questionnaire for
Organizational Leaders of Supervisors who have
ADET
Completed PROF2001
Review and approve the Facilitators Guide for PostTraining Phone Interviews with Supervisor-level
ADET
PROF2001 Graduates
Prepare Populations
Determine random sample of 25 supervisors for
ECM Program
Group1 Pre- and Post-Test and create an email
Team
distribution for the group
Finalize the evaluation plan

17

Start Date

End
Date

Tasks

Responsible
Parties

Determine random sample of 25 supervisors for Group


ECM Program
2 Pre- and Post-Test and create an email distribution
Team
for the group
Pull data from HR database to determine employees
and organizational leaders of supervisors who have
ECM Program
completed PROF2001 and create an email distribution
Team
for each group
Program and Test Evaluation Instruments
Program Pre-Test in the CAT system
ECM Program
Team
Program Post-Test in the CAT system
ECM Program
Team
Program Questionnaire for Employees of Supervisors
ECM Program
who have Completed PROF2001 in internal survey
Team
tool
Program Questionnaire for Organizational Leaders of
ECM Program
Supervisors who have Completed PROF2001 in
Team
internal survey tool
Pilot the Pre- and Post-Test with a class of participants
ECM Program
already scheduled for the month of May 2014
Team
Review outcomes from Pre- and Post-Test Pilot and
ECM Program
make revisions based on findings
Team
Schedule Pre- and Post-Test timeslots in Agency
ECM Program
Assessment Center
Team
Develop and Send Communications
Develop and send series of email communiques for
ECM Program
random sample of 25 supervisors for Group1 Pre- and
Team
Post-Test
Develop and send series of email communiques for
ECM Program
random sample of 25 supervisors for Group 2 Pre- and
Team
Post-Test
Develop email for Questionnaire for Employees of
ECM Program
Supervisors who have Completed PROF2001
Team
Develop email for Questionnaire for Organizational
ECM Program
Leaders of Supervisors who have Completed
Team
PROF2001
Develop and send email invitations for Post-Training
ECM Program
Phone Interviews with Supervisor-level PROF2001
Team
Graduates
Collect Data
Conduct Group1 and Group 2 Pre-Test
ECM Program
Team
Conduct Group1 and Group 2 Post-Test
ECM Program
Team
18

Start Date

2 weeks
before the 3
month mark

End
Date

Tasks
Email Questionnaire for Employees of Supervisors
who have Completed PROF2001
Email Questionnaire for Organizational Leaders of
Supervisors who have Completed PROF2001
Pull data from Learning database to determine
PROF2001 graduates from April June 2014
Conduct Post-Training Phone Interviews with
Supervisor-level PROF2001 Graduates (from April
June 2014)

Responsible
Parties

Start Date

ECM Program
Team
ECM Program
Team
ECM Program
Team

3 months
later
3 months
later
June 2014

ECM Program
Team

Analyze Data
Analyze Group1 and Group 2 Pre-Test results
ECM Program
Team
Analyze Group1 and Group 2 Post-Test results
ECM Program
Team
Analyze Questionnaire for Employees of Supervisors
ECM Program
who have Completed PROF2001 results
Team
Analyze Questionnaire for Organizational Leaders of
ECM Program
Supervisors who have Completed PROF2001 results
Team
Analyze Post-Training Phone Interviews with
ECM Program
Supervisor-level PROF2001 Graduates results
Team
Report Findings
Report initial findings to ADET in quarterly meetings
ECM Program
Team & ADET
Develop final outcomes report
ECM Program
Team
Receive approval of the final outcomes report from
ECM Program
ADET
Team & ADET
Present final outcomes report to CoS
ECM Program
Team & CoS

19

Incrementall
y to hit each
supervisor at
their 3
month mark
June, July,
August

End
Date

H. Evaluation Constraints
The constraints for this evaluation are based on the Program Evaluation Standards developed by
the PgES3 Development Task Force. The standards used to categorize the evaluation constraints
are feasibility, propriety, accuracy, and utility.

Feasibility Constraints
This evaluation has been commissioned by a U.S. Intelligence Community Agency whose
organizational funding is based on a budget approved by the U.S. government. During the
course of this evaluation budgetary spending cuts may obstruct the completion of the evaluation.
Moreover, a change in leadership during the evaluation process may affect the evaluation if the
new director finds the evaluation too costly. Time constraints could have an impact on the
evaluation if the schedule provided by the agency does not allow for adequate implementation of
the evaluation process. To mitigate feasibility constraints, the evaluators will develop an
evaluation contract that will serve as a framework for the deadlines, undertakings, and the staff
needed for different evaluation responsibilities.

Propriety Constraints
Due to the political nature of the organization commissioning the evaluation, it is important that
we diminish political influences that may affect the evaluation process. The evaluators have an
obligation to ensure the findings of the evaluation are reported completely and with transparency.
20

The information produced by the evaluation must not be used to further any particular political
agenda. The findings should not be used to reward or penalize any employees or initiate
termination of employees positions within the agency. At the onset of the evaluation, it is
critical to develop the intended outcomes of the training evaluation. During this initial meeting,
the stakeholders and the evaluators will determine if there will be any political repercussions of
the evaluation findings. The evaluation design will incorporate the use of focus groups. It is
vital that any conflicts of interest held by the participants are understood and limited. Due to the
sensitive information that may be discussed during the focus groups, the evaluators must
eliminate any situations where there is potential for impartiality that may negatively affect the
focus groups discussions.

Accuracy Constraints
The results of the climate survey previously conducted at agency have shown that the
performance ratings have not been a good indicator of performance because 90% of the staff has
received the highest possible rating for annual evaluations. There is a concern about the
truthfulness and accuracy of the ratings given by the supervisors. For the evaluation to be
effective, all participants must be completely truthful on the tests and surveys. The evaluation
will utilize a mixed-method mode of inquiry to answer the evaluation questions. This will
include qualitative data collection through the use of open-ended questions. The evaluators must
ensure accurate coding when analyzing the information provided in the surveys. Any
misinterpretation could invalidate the evaluation findings.

Utility Constraints
There is a critical need for the evaluation to be continuously supported by the Director of
Training and Education and the Agencys Chief of Staff. The evaluation will need to be
perceived as advantageous to the success of the Agency. If it is deemed ineffective, they may
discontinue its process. Moreover, a regime change within the organization could present
challenges if the new leaders find the evaluation inessential. The evaluators must maintain clear
communication with the primary stakeholders to ensure the evaluation purposes are consistently
aligned with the stakeholders needs.

I. Reporting the Results


As the primary stakeholders the Director of Training and Education and the Agencys Chief of
Staff will receive the concluding report at the end of the 12-week evaluation cycle. This report
will summarize the outcomes of the evaluation. The final report will be delivered to each of the
primary stakeholders through hardcopy and email. The outcomes of the final report will
concentrate on the inclusive effect of the ECM training on the organization. The report will
summarize the extent to which the ECM training meets the needs of the stakeholders by
providing evidence which will allow the stakeholders to make judgments about the training and
its effectiveness and benefits to supervisors and employees. The report will include succinct
21

answers to the evaluation questions. One week after the reports have been sent, a meeting will
be held with all stakeholders to discuss the findings of the evaluation. This meeting will consist
of a Power Point presentation reporting the evaluation findings and will provide the primary
stakeholders with an opportunity to discuss any matters concerning the evaluation and its
informational data. If warranted by the findings, the evaluation team will present improvements
that could be made to the ECM training program so that any future training may provide better
knowledge, skills, and attitudes to the supervisors and their employees.
The secondary stakeholders will receive an abridged form of the final evaluation findings
including the design, procedure, and the outcomes of the evaluation. The supervisors will
receive specific information that will assist them in understanding the benefits, if any, of the
ECM training. The civilian trainees will also receive an email summarizing the evaluation
findings.
Other U.S. government agencies that are part of the tertiary stakeholders will have access to the
evaluation through a link on the agencys website. Agencies that may want to use ECM as a
model for their own training programs could use the evaluation results to gage the effectiveness
of the training. The results of the evaluation could also accessible to the American people via a
link on the U.S. government website.

J. Evaluation Budget
Budget Type
Personnel
Supplies and
Equipment
Printing and
Copying
Travel
Facilities
Hotel
Training

Explanation of Cost
Evaluation Team (2) ($30.00 per hour x 480 hours)
Evaluation Timeframe: 40 hours per week
Computers, projectors, paper, folders

Cost Estimate
$28,800

Printing and binding hard copies of the final reports

$200.00

Flight costs for one evaluator to fly to Baltimore


The evaluators will use the facilities provided by the
client
12 weeks accommodation for one evaluator in Baltimore
The training program has already been paid for by the

$600.00
$0.00

22

$1500.00

$4800.00
$0.00

Program
Food

CoS
Lunch meetings, refreshments for focus groups and final $500.00
report meeting
Total Cost of Evaluation $36,400.00

23