Quality Assurance Netherlands Universities

From learning outcomes to testing

Steps
Intended learning outcomes of the programme Intended learning outcomes of the course Develop a teaching and learning strategy to enable students to achieve learning outcomes Design a assessment method to test if the learning outcomes have been achieved

From curriculum to course

Intended learning outcomes for the programme | V Intended learning outcomes for courses or modules
How to assess LOs for each course How to assess if the students have reached the LOs of the whole programme

 When the student has passed each course can you be sure that she also achieved the LO of the programme?

Smart

Specific Measurable Acceptable Realistic Time limited or defined

Smart formulated LOs are easier to assess

Test forms

Multiple choice tests Essay question exams Papers, essays, reports Oral exams, case study assessments Presentation Posters Theses What kind of tests can you add to this list?

Tests and learning outcomes

LO of a module in Economics On successful completion of this module students should be able to recognise the main indicators of stock market timing

What kind of test would you use to assess this LO?

What kind of questions

This LO concerns knowledge of facts Verbs connected to facts are name, describe, cite, define, identify
So a possible test question could be: Name the main indicators of stock market timing.

LO and testing

Another LO of the same module is: Criticise budgetary decisions using economic criteria On what level in Blooms taxonomy is this question?

What kind of questions or tests

This LO is on the highest level of the taxonomy it concerns evaluation What kind of test is appropriate? What would you prefer?

Validity and representativity

Tests have to be valid. What does that mean There are several kinds of validity: Content validity or representative
- The test questions need to be representative for the LO and content of the course
It has no use to ask a question about stock markets for an Information Sciences course

Criterium validity
- The results of a student on a test should be coherent with the results/ scores of the same student on the same kind of tests

validity
Concept validity
- When a student scores 60% on a test,does this mean that she in reality achieved 60% of what was aimed with the course?

More about different kind of tests

The choice for a certain kind of test can be determined by: The intended learning outcomes (Blooms taxonomy) The number of candidates The amount of knowledge,skills and competences you want to test Size of the course Time available for the test
- To answer one mc question takes < 1 minute - To answer one essay question 10-60 minutes

objectivity
Construction
- The construction of multiple choice tests takes a lot of effort when you want to do it right - You have to make sure that all the intended learning outcomes are covered by the tests ( and not only what you like most about the course) - Can you test insight, or application,skills with a mc test

Assessment
- The assessment of essay questions, papers etc. takes a lot of effort when you want to do it right

How to assess?

Quality assurance principles related to assessment - Validity, I discussed this before - Objectivity, we want to be sure that you will not favour your favourite students - Reliability, is the test distinguishing between the good and the bad students

How to?

- Statistics (test the tests) - Four eyes- applied to construction of the test as well as the assessment - External reviewers, not the teachers themselves but independent reviewers - Answer models that describe what kind of answer you expect on an essay question, what elements should be part of it, which concepts should be named and how you weigh these aspects in the final mark - Assessment forms to be used for theses, reports etc.

Tools
Test matrix
a refined list of all LO of the course or the programme. This can be useful when there are a lot of items a student has to know and when you can not test them all in one exam. A test matrix can categorize these items and guarantee that all items have a fair chance to come into the test.

Rubrics
- Rubrics can help to when it is not easy to establish a final mark e.g. for scoring a presentation or a report. They can be seen as refined assessment forms or answering models.

More about Test Policy

Back to the question: How to assess if the students have reached the LOs of the whole programme? What if all the courses are assessed by multiple choice exams? What if all the courses ask the students to give an oral presentation and never a paper or another kind of written report. What about group work how to assess that?

Some issues of test policy in NL

Both bachelor and master programmes require a kind of final thesis by the student, which should indicate that the graduate has achieved the LO of the programme

bachelor theses: group work with guarantees that the individual input of every group member is visible; specific assessment forms are used for this.
Programmes aim at a variety of test forms in the curriculum and make sure that all kind of academic skills are tested during the programme

Responsibility
Boards of Exams are responsible for the quality of the exams, they monitor the tests, and regularly review the graduation theses to check if they fulfill the requirements

How to ensure quality

Did the graduates achieve the intended learning outcomes Is the institute in control Does the programme reach its objectives

Any questions?

Literature: Marjorie Devine & Nevart Yaghlian: Test Construction Manual,Construction of Objective Tests. http://www.cte.cornell.edu/documents/Test%20Constructio n%20Manual.pdf John Biggs: Aligning teaching and assessing to course objectives. Teaching and Learning in Higher Education: New Trends and Innovations. University of Aveiro, 2003

Test policy in your university

What kind of tests are used in your degree programme? (Name at least three) How do you know that the graduates have achieved the intended learning outcomes? Who is responsible for the quality of exams?