Whitepaper:

UNESCO Exam Development

Microsoft Certified Educator

The Exam Development Process:

Microsoft Certified Educator
Introduction
This paper provides information about how the certification exam content that aligns with the
Technology Literacy approach of the UNESCO Competency Framework for Teachers (UNESCO ICT-CFT)
was created.
Microsofts Assessment and Certification Exam (ACE) team worked with a global exam development
vendor to create this exam. The Microsoft development team consisted of a Senior Psychometrician, a
Senior Content Development Manager, and an experienced Project Manager. The vendor development
team consisted of two Lead Content Development Managers, a Senior Editor, and Subject Matter
Experts (SMEs) with deep experience in the integration of ICT in teaching and learning, recruited from
around the world.

Exam Item Development Methodology & Process

Microsoft is committed to creating high quality, legally defensible, and psychometrically sound exams.
To support this goal, our methodology follows industry standards and best practices and incorporates
input and expertise from a geographically diverse set of experts. SMEs were identified by UNESCO
education sector and ICT-CFT partners while others were recruited from academic institutions around
the world.
The exam item development process proceeded as follows:

SMEs attended an in-person item writing session in Montreal, Canada. This session was
facilitated by an experienced exam content development vendor and lasted for two weeks.

Every morning, each SME drafted 3-4 exam items that aligned with the exam specification.
Items were then reviewed by the Content Development Manager to ensure that they mapped
appropriately to the objective and adhered to Microsofts item writing guidelines.

In the afternoon, two types of reviews occurred: Peer and Group.

o

During Peer Review, each SME reviewed another SMEs items and offered suggestions,
comments and revisions for improvements. These changes were incorporated before
the item was sent for Group Review.

During Group Review, the facilitator reviewed the items with all SMEs. The items
technical accuracy, cognitive level, real world value, and alignment to the content
domain were discussed as appropriate, and the item was modified accordingly. All
modifications were made by the facilitator after consensus was reached with the SMEs.

On the final day of the item development session, each SME was assigned a pool of items to
which they added the item rationale. This process resulted in an additional review of the item
based on comments and queries by the SME writing the rationale; all of these questions and
comments were discussed using a process similar to the Group Review described above.

If SMEs could not reach consensus on the technical accuracy, appropriateness, and relevance of
the item, the item was rewritten by the SMEs using a collaborative process.

Beta Test Phase: Testing Items for Psychometric Soundness

The purpose of the beta test phase is to ensure that the items are fair and psychometrically sound prior
to scoring candidates. Psychometrically sound items are those that are neither too easy nor too
difficult, differentiate between high and low performers, and are reliable measures of the content
domain; only psychometrically sound items will be included on the live version of the exam. Because this
psychometric information can only be obtained by having candidates sit the exam, an unscored beta
exam is administered to gather the necessary data. This data ensures the quality of resulting
certification exam.

Process
The beta exam pool is administered to an invitation-only audience that is similar to the target audience.
Ensuring that the beta audience and target audience are similar is required to obtain meaningful
psychometric results.
The beta exam will be available for interested candidates until the live, scored version is published.

Scoring
Candidates who take the beta exam will not receive a score or any information on their passing status
until the psychometric analysis occurs. The process of obtaining sufficient data and psychometric
analyzing this data typically takes approximately 6-8 weeks; however, because the psychometric analysis
cannot begin until sufficient data has been collected, this estimate is dependent on the amount of time
required to have 500 beta candidates take this exam.
(Note that 500 is the minimum number needed for meaningful psychometric analysis for this
certification exam; the required number of beta participants may increase or decrease for other
certification exams depending on the number of items administered during each delivery of the beta
exam.)
Once sufficient data has been collected, the results will be analyzed. Items that are not psychometrically
sound will be removed from the item pool. At this time, all beta tests will be scored. Certifications will
be awarded to beta candidates as appropriate based on those scores.

What Candidates Should Expect During the Exam

Each delivery of the exam, including the beta exam, will contain approximately 50 items. Note that this
is subject to change based on desired exam time and the psychometric performance of the content over
time. The item pool contains both case study items and multiple choice items.
Candidates will be given 150 minutes to complete the beta exam to ensure that they have sufficient
time to complete the exam. We will use the information obtained during beta to set the final exam time.
Our target exam time for the live exam is 120 minutes, but the amount of time provided will be based
on the time that it takes to complete the number of items (estimated at 50) needed for a valid and
reliable exam.

Sample Questions
The case study exam format uses scenarios that simulate how teachers might use ICT resources as they
prepare for and conduct their classes. A case study model enables us to test a teachers ability to
analyze and synthesize information to make decisions. Case Studies contain separate sections (e.g.,
Learning and Teaching Environment; Student Objectives; Professional Development Objectives);
candidates answer several questions based on information provided in the case study.
The following samples are representative of items candidates can expect to see associated with each
case study. Please note that these items are samples; disclosing actual items would compromise the
security of the exam.
Sample 1
Your students have accomplished the learning objectives and submitted their work.

You need to show the work to parents at the next parent teacher conference. You want to display the
work in a continuous loop for three hours, on a computer screen.

Which resource should you use to display the student work?
Word processing software
Web publishing software
Presentation software
Desktop publishing software
Picture editing software

Sample 2
What ICT skills do your students need to develop to meet the learning objectives?
How to use a database to store information.
How to carry out effective Internet searches.
How to download files to a removable storage device.
How to edit photos in an image-editing package.

Global Relevancy
Ensuring global relevancy has been a critical goal throughout the project to date. The UNESCO ICT-CFT
and all design specifications were created with the input of SMEs from around the world.
The SMEs chosen to develop the exam items were also recruited from an international pool. This was
intentional and intended to ensure global representation and perspectives. The item development team
consisted of three experts from Europe, one expert from North America, one expert from Africa, and
one expert from Asia.

Local Contextualization
The development team adhered to the Microsoft Globalization standards that are currently applied to
all Microsoft Certification exams. Compliance with these standards ensures a consistent and predictable
candidate experience across all countries while also maintaining the intended reliability of the
certification credential. Modifying exams at a local level is not currently supported as doing so may
compromise the validity of the certification.