Evaluation of Far West Laboratorys Determining Instructional Purposes Program

A Proposal Submitted to Far West Laboratory for Educational Research and Development by

Mesler Educational and Training Consultants (METC)

Introduction In March, 2011, the Far West Laboratory for Educational Research and Development (FWL) issued a request for proposal for an evaluation on the Determining Instructional Purposes (DIP) training program. This document is a proposal of services from Mesler Educational and Training Consultants (METC) of Boise, Idaho in response to the Far West Laboratory (FWL) request for proposal. METC will provide analysis and recommendations to FWL so an informed decision may be made regarding the current training programs continuation and possible expansion into the retail world. The Determining Instructional Purposes (DIP) Training Program FWL was created in 1966 and funded by Congress as one of 20 Regional Educational Laboratories (REL). The assignment of FWL was to bridge the gap between research and practice. As time passed, FWL developed mini-courses on a variety of subjects to improve teacher skills, education, and effectiveness in the classroom (as described in Wikipedia). The DIP program was created to provide training for school administrators and graduate level students in educational administration in skills linked to planning effective school programs. The DIP training package consists of a Coordinators Handbook, and three training modules (Setting Goals, Analyzing Problems, and Deriving Objectives), with each unit containing four to six modules that address a limited number of instructional objectives. Each module contains related reading materials, practice activities, and feedback for the activities. The units may be either used as stand-alone or in combination, and may be presented as a condensed short-term workshop, or in individual sessions extended over time. Units 1 and 3 require 10 to 15 hours to complete, while Unit 2 takes 12 to 18 hours. A coordinator needs to have experienced the training units individually or as a member of a training group, but does not need to have any other prior content knowledge in the area. The Coordinators Handbook provides all of the information needed to conduct the training. The coordinator must organize, guide, and monitor activities while the participants use the materials and procedures in the units. Evaluation Method The evaluation itself will be tasked to find out if the intended outcomes of program effectiveness, impact, and quality meet the criteria for continued use and expanded retail distribution. METC will use a combination of evaluation methods to determine the

success of the DIP, suggest any improvements, research comparative programs, and then present the final evaluation to the senior administration of FWL to assist them in determining the viability of retailing their DIP products. The evaluation of DIP involves participation from members of the following ladder:

Information Developer/ Retailer

Far West Laboratory DIP Package

Customer

School District Admininstration

Universities

Learner

School Administrators

Graduate Level Educational Administration Students

Type of Training

Individual

Training Group

Training Management

District Trainer

Self-Directed

Unit Training Coordinator

The members of the ladder will be the participants and sources of data in the evaluation, and will be included in the analysis of the DIP training package, in order to establish and recommend the appropriate training that meets the educational objective of each training unit. The package will be evaluated as three individual units, and then as a whole to see how effective they work when combined. Information Developer/Retailer: In order to start the evaluation process, it will be necessary for FWL to provide copies of the units, including all additional materials and activities, and any post-instruction information obtained by FWL. METC will meet with FWL training program administration and developers to discuss any feedback received prior, during, or after training is completed, either from educational customers, or employees of FWL.

The Customer: The current instructional goals for each unit will be studied and used as a guideline to determine whether adjustments need to be made to the training units to meet the Customers goals. The items in each unit, including reading materials and activities, will be compared to the goals to ensure the applications from the unit information are appropriate to meet the goals. Any records of new skills attained by the learners, as well as the feedback received from the training program will be reviewed. METC will conduct short surveys and interviews with Customers in order to assess buyer attitude and evaluate the desire to see the training program continue, as well as grow, within the Customers domain. The Learner and Type of Training: Evaluation of the program from the Learners viewpoint will take place in various ways. First, a survey will be sent to learners who have completed and those who have not completed the training program. This survey will focus on attitudes, opinions, benefits and weaknesses of the program as it is experienced, and also look for long-range benefits that may be found since their training ended. Questions in the survey will be distinct for those learners who trained individually, as well as those who trained in a group. Secondly, using a list of learners registered for summer sessions, a sample will be derived from two groups those who plan to train individually, and those who will participate in training group session or sessions. For these samples, a pretest/posttest of knowledge and skills will be administered for depth of knowledge gained from the training, and later on, they will also complete the follow-up survey as those who previously completed the training. Thirdly, names of learners that have completed the program will drawn randomly and METC will complete personal interviews, either in-person, online, or by phone, regarding long-term benefits or weaknesses in the program. The Unit Coordinator/District Trainer: Using information from FWL, METC will create a rubric for best practices as a Training Coordinator. It will be used to assess the training programs live monitoring methods. METC will observe at least three live sessions of group training over the summer, with at least two different Coordinators/Trainers, in order to witness how the materials are used during training and what questions/issues arise during that time. This observation will enable METC to evaluate training efficiency and effectiveness, as well as the quality of instructional materials. If possible, the METC observer will watch from another vantage point, but if necessary, the observer will be located in the training area. All coordinators will complete a survey on attitudes about the instructional materials and their role as coordinator in the process. METC will interview each coordinator that checks a box on the survey saying he/she would like to share more feedback about the training program. In addition, after a close analysis of the information gained at this juncture, METC will conduct training sessions and collect feedback from the learners in the way of online assessments, quizzes, and surveys in order to evaluate the accuracy of METCs content and instructional improvements that will be recommended to FWL.

The Competition: METC will also complete a general price/quality comparison of the FWL DIP training program to other programs of comparable type, whether programs suggested by FWL to METC, or those found through research during the evaluation process. Task Schedule Task # 1 Task Description Meet with FWL to present and discuss METC evaluation proposal Review notes from first meeting and set up a second meeting with FWL to present any new/modified proposal recommendations, due to discussion in first meeting. Get a list of all stakeholders at FWL that should be invited to participate in meetings and/or feedback sessions. Submit proposal for data collection to FWL for their approval on collaboration and participation. FWL provides 5 hard copies of all DIP training units, materials, and 3 hard copies of the Coordinators Manual to METC. METC meets with FWL stakeholders for customer/employee feedback. Review the proposal plan and training program with METC group and assign tasks to appropriate team members Create evaluation criteria and materials for all testing, surveys, interviews, and observations Upon FWL approval, METC initiates short surveys and interviews with University/School District Administration personnel about the effectiveness and viability of the DIP training program. METC initiates Learner surveys and interviews by appointment, upon FWL approval. University/School District Administration personnel surveys Responsible Group METC/FWL Deadline May 3, 2011

METC/FWL

May 10, 2011

METC/FWL

May 17, 2011

METC/FWL

May 17, 2011

METC

May 26, 2011

METC

June 3, 2011

METC/FWL

June 7, 2011

METC

June 17, 2011

10

11

11 12

13

14 15

and interviews completed. Learner surveys and interviews are completed. METC reviews input and writes an analysis summary of the information Learner Samples take Pretest before training and Posttest after training is complete for skill comparison. METC initiates training observations and send out coordinator surveys Learner Samples have completed training. METC observations are complete and coordinator surveys are collected. METC reviews input from observations and coordinator surveys, and write an analysis summary of the information. METC initiates DIP training sessions and collects real-time data, upon FWL approval. METC DIP training is completed. METC reviews all collected data and analyses and compares to the existing DIP training program. METC performs an analysis of materials, activities, skill acquisition, and attitudes from the METC-conducted DIP package. METC composes a summary for FWL. METC meets with FWL to present evaluation results and improvement recommendations. Write final report and submit to FWL.

METC/FWL

June 20, 2011

METC

July 8, 2011

METC

July 15, 2011

METC METC

July 18, 2011 July 29, 2011

METC

August 5, 2011

METC/FWL METC

August 16, 2011 August 26, 2011

METC Project Team Kris Mesler, METCs evaluation project administrator, has a Masters degree in Educational Technology and is an evaluation specialist. She will be directly involved in the training sessions, and coordinate planning and implementation of project tasks.

Michelle Klonowski, instruction specialist at METC, will monitor and evaluate DIP training sessions, as well as conduct learner interviews. She will be responsible for approval of content instruction for the training sessions. Michelle also speaks Spanish, and will do any translation of materials that is necessary. Kyla Dordan, METC data analysis specialist and instructor, will conduct the METC DIP training and is responsible for taking the data collected in reporting form and distributing it to staff members prior to analysis meetings. She is also the administrative assistant for the project team on this evaluation. Claudia Mehlhaff, an METC instructor and specialist in instructional design, will conduct the interviews of university and school district administration personnel. She will also participate in the development of the survey and interview materials. Kurt Betts, METC IT data specialist, will design all of the online surveys and assessments, as well as hard copies and disseminate them to participants as required in the evaluation design. He will organize the data into reporting form.

Budget Budget Item

Personnel: 10 hrs/wk x 12 wks + benefits

Description
Kris Mesler Michelle Klonowski Kyla Dordan Claudia Mehlhaff Kurt Betts Internet, Telephone, and Videoconferencing

Expense
$830 x 12 wks = $9960 $700 x 12 wks = $8400 $600 x 12 wks = $7200 $600 x 12 wks = $7200 $750 x 12 wks = $9000

Totals

$41,760

Communications

$100/wk x 12 wks = $1200

$1200

Photocopying Summaries, training packages, Office Supplies Airline Hotel Car Rental Food

Supplies and Materials

~ $150/wk x 12 wks = $1800 $375 x 4 mtgs x 2 admin = $3000 $125/night x 4 nights = $500 $70/day x 4 days = $280 $125/day x 8 days = $1000 Budget Total =

$1800

Travel for personnel to four in-person Meetings

$4780 $49,540