Qualitative Metaevaluation Form Using The Program Evaluation Standards, 3rd Edition

MEDT 8480

Evaluator: Group 7
Metaevaluator Group 8: Eden Clark, Jastacia Harris, April 9, 2017
: Michelle Rogers & Teresa White
NAME DATE

Instructions: Rate the relevance of each standard as it currently applies to the present evaluation effort. Then provide
feedback to the evaluator on each standard by highlighting where the evaluator addresses compliance with the standard
statement in the evaluation plan, and the extent that the standard is being met. If in your opinion the evaluator has not
or has insufficiently addressed the standard, indicate so and provide constructive feedback or a suggestion as to how to
improve the plan. Finally, in the Overall Comments section found below, summarize your feedback as it pertains to the
evaluations Utility, Feasibility, Propriety, Accuracy, and Evaluation Accountability. Try to come up with an overall
statement of the evaluation plans merit taking into consideration the context of the evaluation, the relevance of the
evaluation standards to the evaluation effort, and the extent that the standards are adhered to and met by the
evaluator in the evaluation plan. NOTE: This particular assignment is to provide students an experience with an
important and often overlooked aspect of evaluation: metaevaluation. Your metaevaluative conclusions and feedback
will in no way negatively impact the course grade of the particular evaluator; however, they should be a constructive, yet
fair assessment so as to help improve the overall evaluation effort.

When addressing each standard, consider the whole evaluation plan including appendices. This is particularly true of the
Accuracy Standards. Are the evaluation instruments (if any) and methods sufficient to answer the evaluation questions?
If you have additional feedback (e.g., formatting, editing, APA), please comment in the Additional Feedback section.

Overall Comments:
Overall, we think that your Evaluation Plan contains several strengths and only a few weaknesses. We think that the
background information is adequately described, and your evaluation purpose is clearly described with understandable
goals that would derive from the conclusion of your evaluation. It appears that the overall need was that teachers
lacked the confidence to implement and use the Chromebooks effectively. We also think that your three evaluation
questions clearly relate to your plan and help point you towards your final goal: determining whether Chromebook Carts
Teacher Professional Learning Community is beneficial to students and teachers. Finally, we think you have a strong
sense of how you are going to collect your data.
Another strength of the plan is the Analysis section which is well-thought out. While looking specifically at the Accuracy
standard, we do think that the evaluation plan can improve on clearly determining who the stakeholders are, clarifying
how the evaluators are qualified by describing who they are, and predetermining the questions that are going to be
provided in the questionnaires and interviews in order to avoid biased questions. Another suggestion is to include the
standards/benchmarks you will use for each of your evaluation questions in the Standards section. As you continue to
develop and finalize your plan, do not forget to include any new instruments that you describe in your text in the
Appendix area.
We had three main concerns:
If teachers and students get used to using the chromebooks within every subject,what affects will it have on
instruction when they are unable to use the chromebooks during GMAS testing?
How will a student (who has been in a chromebook classroom) be affected if he/she ends up in a teachers
classroom the next year who does not have the chromebooks?
An Ed Tech Specialist and Program Lead were both mentioned in the program logic model. Are they your
clients? If so, the Background and Description of the Program section would be a great place to insert this
information.
Additional Feedback:
We found a few grammatical errors (commas and subject verb agreement. After the plan was printed, the running head
and page numbers were off the top of the page. Also there was additional space before the References heading on the
reference page. A formatting suggestion is to type out long phrases such as Professional Learning Community and
Sutton Middle School the first time they are mentioned in the plan with the abbreviation PLC or SMS in parenthesis after
it so that the abbreviation can then be used throughout the remainder of the plan. The future tense is not used
consistently through the plan, especially in the Analysis section. Finally, a suggestion to include more of the information
from the Assumptions section of the Program Logic Model into the text in the Background and Description of the
Program section.

STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
UTILITY
U1 Evaluator Evaluations should be conducted by This standard is not met. There was not a description of the evaluation
Credibility qualified people who establish and team in the Background and Program Evaluation section.
maintain credibility in the evaluation
context. X

U2 Attention to Evaluations should devote attention to This standard is met. The stakeholders who are invested in the program
Stakeholders the full range of individuals and groups and affected by its evaluation include the SMS administration, teachers
invested in the program and affected and students. The district itself is a stakeholder too because it provided
by its evaluation. X the technology and would benefit from the information from the
evaluation. This information was readily available in the Background
and Description of the Program section.

U3 Negotiated Evaluation purposes should be This standard is partially met. The purpose of the program to
Purposes identified and continually negotiated determine if the PLC has a positive impact on teachers and students
based on the needs of stakeholders. who are using Chromebooks to enhance their learning is stated in the
X Background and Description of the Program section and clarified in
the Evaluation Purpose section; however, negotiations in the program
are not mentioned. Negotiations can be implemented and clarified
once evaluators incorporate a contract.

U4 Explicit Values Evaluations should clarify and specify This standard is partially met. Some information is provided in the
the individual and cultural values Background and Description of the Program section. Additional text in
underpinning purposes, processes, and the Evaluation Purpose section that incorporates the Contextual
judgments. X Statement, Assumptions and External Factors from the program logic
model will provide additional information.
U5 Relevant Evaluation information should serve This standard is met. The evaluation questions that you have described
Information the identified and emergent needs of in the Evaluation Purpose section and data sets that you describe in
stakeholders. the Methodology section will provide relevant program information to
X determine the if the goals of the PLC including increased student
achievement on the GMAS have been met.

U6 Meaningful Evaluations should construct activities, This standard has been partially met. As you stated In the
Processes and descriptions, and judgments in ways Methodology section for Q1, the time you have built in between
Products that encourage participants to teacher questionnaire and personal interviews will give teachers time to
rediscover, reinterpret, or revise their X uncover questions they feel were left unanswered by the PLC but were
understandings and behaviors. unaware of immediately (p. 6). Q3 also allows time between the pre
and post tests. Could the the observations to collect data for Q2
possibly be structured to allow allow teachers and students additional
time?
U7 Timely and Evaluations should attend to the This standard is partially met. Your Evaluation Plan and Analysis
Appropriate continuing information needs of their sections very clearly describe the timeframe for data collection and
Communicating multiple audiences. analysis. However, it would be helpful and informative for your client
and Reporting X and stakeholders to add information about how the results of the plan
will be determined and when the results should be ready for
stakeholders in the Standards/Benchmarks for Evaluation Questions
section.

U8 Concern for Evaluations should promote This standard has been partially met. In the Evaluation Questions
Consequences responsible and adaptive use while section Q1, you mention the importance of teachers anonymity so they
and Influence guarding against unintended negative have no fear of repercussions (p. 6). You can incorporate additional
consequences and misuse. X information about how the report will be used and guard against misuse
when you incorporate the contract into your plan.
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
FEASIBILITY
F1 Project Evaluations should use effective This standard was met. Within your evaluation, you have put in place a
Management project management strategies. process that allows teachers sit through trainings and then having
teachers re-deliver to their students. This plan will prevent failure when
X you using the Chromebooks.

F2 Practical Evaluation procedures should be This standard was met because all of the procedures of the Chromebook
Procedures practical and responsive to the way PLC Program Evaluation Plan are practical and responsive to how the
the program operates. program operates.
X Technology specialists will teach teachers how to effectively
use and implement chromebooks within their instruction
Teachers will participate in monthly face to face meetings
Teachers will re-deliver and educate their students on how to
use the chromebooks effectively
Teachers will integrate chromebooks within their instruction

F3 Contextual Evaluations should recognize, monitor, This standard was met. Your evaluation proves that you recognized,
Viability and balance the cultural and political monitored, and balanced the cultural and political interests and needs
interests and needs of individuals and of teachers when it comes to using the Chromebooks. Since the
groups. X teachers confidence levels on how to use and implement chromebooks
within their instruction was low, hands on training will be provided to
them through trainings offered by the technology specialist.

F4 Resource Use Evaluations should use resources This standard was met. The evaluation team is using its resources
effectively and efficiently. effectively and efficiently; however, you might consider combining some
of the questionnaires so that teachers do not have as many to complete.
X Throughout this evaluation it is evident that the goal is to have teachers
and students use Chromebooks effectively and efficiently.
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
PROPRIETY
P1 Responsive and Evaluations should be responsive to This standard is met. In the Description of Evaluation for Participants,
Inclusive stakeholders and their communities. you state who the participants are and which content and grade levels
Orientation they represent. In your description of Q1 in the Evaluation Questions
X section, you show a sensitivity and understanding for the teachers who
are participating by allowing them the opportunity to submit
anonymous responses. The only suggestion is to consider a way to
combine the different types of questionnaires you propose for the
teachers so that they do not have to complete as many separate ones.
P2 Formal Evaluation agreements should be This standard has not been met yet, but will be addressed when you
Agreements negotiated to make obligations explicit incorporate the contract from Dr. Westine into your plan. It will be
and take into account the needs, important to communicate with the administration at SMS and any
expectations, and cultural contexts of X district level administration required about the students and teachers
clients and other stakeholders. who will be participating in the evaluation and about the access that you
will need to students GMAS scores.

P3 Human Rights Evaluations should be designed and
and Respect conducted to protect human and legal
rights and maintain the dignity of
participants and other stakeholders.
This standard is partially met. In the Evaluation Questions section on
page 6, you have designated that an unbiased evaluator from the team
will conduct the personal interviews with the teachers. Additionally, in
X the same section on page 8, you described how student names will be
removed from GMAS data. Two suggestions: state if the questionnaires
for students and teachers for Q2 will also be anonymous and how
students identities will be protected in the pre and post tests for Q3.

P4 Clarity and Evaluations should be understandable This standard is partially met. In the Description of Evaluation
Fairness and fair in addressing stakeholder Participants section, you state that the 14 teacher participants were
needs and purposes. randomly chosen. You might consider clarifying how were they were
X randomly chosen since there are 14 Chromebook carts and 14 teachers
participating in the PLC. In the same section on page 5, you mention
that each teacher has 5 sections with approximately 30 students each.
Will students in all 5 sections participate? In the Evaluation Questions
section on page 8, you mention that the pre and post test will also be
administered to students in classrooms of teachers who were not part
of the PLC. You might consider including how those teachers and
classes were selected.
P5 Transparency Evaluations should provide complete This standard has not been met yet, but can be addressed when you
and Disclosure descriptions of findings, limitations, incorporate the contract into your plan. Protecting the identities of the
and conclusions to all stakeholders, teaches and students when reporting the findings will be important.
unless doing so would violate legal and X
propriety obligations.

P6 Conflicts of Evaluations should openly and This standard was not addressed. Information about your evaluation
Interest honestly identify and address real or team, clarification of internal and external evaluators, and possible
perceived conflicts of interest that may conflicts of interest can be included in the last part of the Background
compromise the evaluation. X of the Program and Evaluation section.

P7 Fiscal Evaluations should account for all This standard was not addressed in the plan; however, it is not an area
Responsibility expended resources and comply with of concern in this evaluation effort because there is no budget.
sound fiscal procedures and processes X
 STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS
THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
ACCURACY
A1 Justified Evaluation conclusions and decisions This standard was met. You have done a great job describing the
Conclusions should be explicitly justified in the program and the programs needs for evaluation. In the Background
and Decisions cultures and contexts where they have and Description of the Program, you gave good description on the
consequences. X distribution of Google Chromebooks and the school's goals with
implementing the Chromebooks. In the Evaluation Purpose, you
clearly identified the program that is being evaluated, Chromebook
Carts Teacher Professional Learning Community, and why the program
needs to be evaluated. According to this section, you clearly stated that
evaluation PLC will help in identifying what is working, what is not
working, and what needs to be changed in order to help teachers and
students benefit from the program.
A2 Valid Evaluation information should serve This standard was met. In the standards section, you have provided a
Information the intended purposes and support strong, detailed statement on how you are meeting the expectation of
valid interpretations. this standard. According to the Evaluation Plan, the evaluators will use
X their information to serve the intended purpose and support valid
interpretations by collecting qualitative and quantitative data that are
useful and unbiased. As stated in the evaluation plan, by using the data,
evaluators will be able to determine if the PLC has proved to be
beneficial to both teachers and students. This is made highly clear in
your Evaluation Plan.
A3 Reliable Evaluation procedures should yield This standard is partially met. It is said that you are collecting both
Information sufficiently dependable and consistent qualitative (questionnaires and interviews) and quantitative (Pre ,Post
information for the intended uses. tests and GMAS Scores) data. You are then going to compare your data
X with the data from classrooms who are not utilizing Chromebooks. The
quantitative data will be dependable; however, we are concerned that
the data from questionnaires and the interview will be biased because
we do not have access to the questions nor do we know how the
evaluators are planning on providing unbiased questions. Perhaps
putting the questions in the appendix would help to ensure that you are
adequately meeting this standard.
A4 Explicit Evaluations should document This standard is partially met. We think you did a great job determining
Program and programs and their contexts with the evaluation purpose which is to determine if the PLC has a positive
Context appropriate detail and scope for the impact on teachers and students who are using Chromebooks to
Descriptions evaluation purposes. X enhance their learning. We do feel like more detail could be used when
describing the Program (for example, who are the stakeholders, how is
PLC being implemented, who is running PLC and how, etc.) in order to
better clarify why the program works or what can be improved.

A5 Information Evaluations should employ systematic This standard has been met. The evaluators of this program have a clear
Management information collection, review, understanding of how they are going to derive at their conclusion on if
verification, and storage methods. the program works or not. In the Evaluation Questions section, you
X did a great job at clarifying your essential questions and determining
what data (qualitative or quantative) you are going to use and you
justified it. You stated that you are going to use electronic
questionnaires, interviews, and quantitative data collected from Pre and
Post Test, as well as data collected from the GMAS and then compare
this data to the students/classrooms who are not using Chromebooks.
While reviewing this data, we think you will have a clear, systematic way
of collecting enough information to conclude if the program benefits its
users or not.

A6 Sound Designs Evaluations should employ technically This standard has been met. As stated above, your way of collecting
and Analyses adequate designs and analyses that data in order to analyze the data is clear and appropriate for the
are appropriate for the evaluation evaluation purposes. We cannot think of a better way to collect this
purposes. X data rather than interview, use questionnaires, and compare tests
scores from students who utilizes the Chromebooks to students who do
not. This data will provide adequate evidence to suggest if PLC is
benefitting its teachers, who in turn, help students improve their
education and technical skills.
A7 Explicit Evaluation reasoning leading from This standard was partially met. In your Evaluation Plan, you
Evaluation information and analyses to findings, determined how the data was going to be collected, processed, and
Reasoning interpretations, conclusions and reported; however, it is not exactly clear on how the qualitative data is
judgments should be clearly and X going to be assessed. We would suggest implementing a specific way
 completely documented. that leads to an unbiased review of the questionnaires and interviews.

A8 Communicatio Evaluation communications should This standard was partially met. As previously stated, we think in order
n and have adequate scope and guard for your Evaluation Plan to demonstrate your adequate use of
Reporting against misconceptions, biases, communication leading to unbiased results, we think that it would be
distortions, and errors. X beneficial to put the questions of your survey and interview in your
appendix so that all evaluators can ensure unbiased and error-filled
questions.

STANDARDS STANDARD STATEMENTS RELEVANCE TO QUALITATIVE FEEDBACK/COMMENTS

THE EVALUATION
(CHECK ONE)
H F C H
A A L I
R I E G
D R A H
L L R L
Y Y L Y
Y
EVALUATION ACCOUNTABILITY
E1 Evaluation Evaluations should fully document This standard was met. Within your evaluation you have identified and
Documentation their negotiated purposes and documented the purpose, procedures, data, and outcomes of
implemented designs, procedures, Chromebook PLC Program Evaluation Plan.
data, and outcomes. X Purpose- To have teachers and students use chromebooks effectively
and efficiently
Procedures-
Technology specialist will teach teachers how to effectively
use and implement chromebooks within their instruction
Teachers will participate in monthly face to face meetings
Teachers will re-deliver and educate their students on how to
use the chromebooks effectively
Teachers will integrate chromebooks within their instruction
Data- Questionnaires, observations, summative assessments
Outcomes- Increased use of technology in Chromebook Cart
Classrooms.
E2 Internal Evaluators should use these and other This standard was partially met. The standards were mentioned, but
Metaevaluatio applicable standards to examine the you did not examine their accountability to your evaluation clearly.
n accountability of the evaluation Each team will have more opportunity for internal metaevaluation when
design, procedures employed, X submitting a final draft.
information collected, and outcomes.

E3 External Program evaluation sponsors, clients, This standard has now been met with the submission of this external
Metaevaluatio evaluators, and other stakeholders metaevaluation.
n should encourage the conduct of
external metaevaluations using these X
and other applicable standards.