SOLO TAXONOMY

PRESENTED BY AZMAT SIDDIQUE

Regional Programme Manager-II DSD Punjab, Lahore

Background to the new exam system

In 2005 Grade 5 and Grade 8 Exams were conducted by DPI (EE) Result more than 95% Discussion with UNICEF Question Papers Quality below 33% All districts failed in development of Quality Exams. papers

Background to the new exam system

International Tender for development of papers No one was ready to help Special request to UNICEF Two Australians Provided Support Dr.Ken Dr.Ted Redden

A Key Question
How do Exams, and particularly Grade 5 and 8 large scale exams, fit into the general Assessment issue?

What are the strengths of such exams

ASSESSMENT CURRICULUM DYNAMIC

5

ASSESSMENT

CURRICULUM

TEACHER TRAINING

CLASSROOM PRACTICES

How do we Do all this?

6

We design quality exam papers with questions reflecting learning outcome statements (supported by SOLO) We have a new form of analysis of exam results (Rasch analysis) We provide a extensive set of reports to allow judgments to be made by the education stakeholders: Children, Parents, Teachers/Principals and Officials

SOLO Taxonomy
Structure of

Observed

Learning

Outcomes

Levels of responses in SOLO TAXONOMY

There are a number of levels at which we can classify a student response to an examination question
Pre-structual Uni-structural . Multi structural Relational . Extended abstract

Uni-structural

This set of responses uses only one relevant element of data from the stimulus item. A feature of responses at this level is the desire to close quickly and to ignore inconsistencies that may result from the response

Multi structural

The learner at this level can use multiple data elements, but the elements are not integrated. Hence the response can consist of a number of discrete closures. Typical of these responses would be the following of strict algorithmic procedures that involve a number of steps. However, if a single step was forgotten, or an error made, the respondent would be unable to reconstruct the algorithm. This lack of an overview of the data elements and their relationships makes the response patterns inherently unstable and thus considerable variability may be expected from children responding at this level.

Relational

A relational response reflects the ability to integrate the elements and operations of the question in a way that enables an overview of the stimulus item Children using an algorithm at this level would be able to check for errors and inconsistencies, and would be able to reconstruct missing elements of the algorithm. Features of responses at this level include the ability to reverse operations and the set of elements used are internal to the system

Extended abstract

The use of data elements external to the system is a feature of an extended abstract response. The generalisation of the elements takes account of new and more abstract features.

INDICATORS FOR CLASSIFICATION OF RESPONSES

Level of Indicators Outcome 1 2 3 4

Prestructural Unistructural Multistructural Relational Use of irrelevant information, or no meaningful response Focuses on one relevant aspect only. Focuses on several relevant aspects, but they are not co-ordinated together. The several parts are integrated into a coherent whole: details are linked to conclusion; meaning understood

Extended Abstract

Answer generalises the structure beyond the information given: Higher order principles are used to bring in a new and broader set of issues

Properties of SOLO

Rote learned responses have no level For every new task you will need to progress through each level. You may use experiences from one task to assist in moving through levels in another related task. You cannot respond at level N unless you have moved through level N-1 Teachers sometimes use strategies that lead to level reduction. That is assist students to get the answer right but do not improve understanding.

Example 1
We can use SOLO to develop a marking scale for open ended responses

Imagine the essay topic:

What I did on the weekend

What I did on the weekend

Uni-structural

On the weekend I went to grandmas house

What I did on the weekend

Multi-structural I went to grandmas house and I watched football and I played with my friends and I had ice-cream on Sunday and I helped with the harvest.

What I did on the weekend.

Relational We were going to the beach but it started to rain so we decided to go to grandmas house. It was lucky we decided not to go to the beach because the road flooded during the storm and we would not have been able to get home until the next day.

Questions leading towards Uni Structural Response

Add 5+6 Make a fraction Using 2 and 3 Which is solved first a) [ ] b) { } c) ( ) d) ______

Questions leading towards Multi Structural Response

Solve 2+3 x 5

Write fraction for shaded portion In which order brackets should be solved a) [ ], ( ), { } b) { }, ( ), [ ] c) [ ], { }, ( ) d) ( ), { }, [ ]

Questions leading towards Relational Responses

A person went to market he purchased 3 apples at the rate of 5 rupees per apple. After purchase he has 2 rupees in his pocket . Find the amount with which he went to market?
Solution for the expression 10 + [{ ( 5 + 7) x 2 } + 3 ] is a) 32 b) 47 c) 37 d) 67

Outcome, Solution, Structure of Essay

Prestructural:

Misses the point Completely incorrect solution Inappropriate or few issues identified. No framework for discussion and little relevant material selected. Poor structure to the essay. Irrelevant detail and some misinterpretation of the question. Little logical relationship to the topic and poor use of examples.

Outcome, Solution, Structure of Essay

Uni structural:
State, Recognise, Recall, Quote, Note, Name
Correct answer to simple algorithmic problem requiring substitution of data into formula. Correct solution of one part of more complex problem.

Poor essay structure. One issue identified and this becomes the sole focus; no framework for organizing discussion. Dogmatic presentation of a single solution to the set task. This idea may be restated in different ways. Little support from the literature.

Outcome, Solution, Structure of Essay

Multi structural:
Explain, Define, List, Solve, Describe, Interpret

Correct solution to multiple part problem requiring substitution of data from one part to the next. Poorly structured project report or practical report on open task.
Essay poorly structured. A range of material has been selected and most of the material selected is appropriate. Weak introduction and conclusion. Little attempt to provide a clear logical structure. Focus on a large number of facts with little attempt at conceptual explanations. Very little linking of material between sections in the essay or report.

Solution, Structure of Essay

Between Multi structural & Relational

Solution to multiple part problem with most parts correctly solved but some errors. Reasonably well structured project or practical report on open task.

Essay fairly well structured. Some issues identified. Attempt at a limited framework. Most of the material selected is appropriate. Introduction and conclusion exists. Logical presentation attempted and successful in a limited way. Some structure to the argument but only limited number of differing views and no new ideas.

Outcome, Solution, Structure of Essay

Relational:
Apply, Outline, Distinguish, Analyse, Classify, Contrast, Summarise, Categorise

Elegant solution to complex problem requiring identification of variables to be evaluated or hypotheses to be tested.Well structured project or practical report on open task.
Essay well structured with a clear introduction and conclusion. Framework exists which is well developed. Appropriate material. Content has logical flow, with ideas clearly expressed. Clearly identifiable structure to the argument with discussion of differing views.

Outcome, Solution, Structure of Essay

Extended Abstract:
Create, Synthesise, Hypothesise, Validate, Predict, Debate, Theorise

Solution to problem which goes beyond anticipated answer. Project or practical report dealing with real world ill-defined topic.
Well structured essay with clear introduction and conclusion. Issues clearly identified; clear framework for organizing discussion; appropriate material selected. Evidence of wide reading from many sources. Clear evidence of sophisticated analysis or innovative thinking.

Marking Scheme in SOLO For G5 & G8

Level of Response Extended Abstract Structure Marks

10 8 6-7

Relational

Multi structural

Uni structural
Pre structural

5 2-4 1 0

31

New form of analysis

CRITERION AND NORM REFERENCED APPROACH
MARKS FOR EACH QUESTION IN EVERY SUBJECT

ASSESS WHAT STUDENTS IN DIFFERENT PERFORMANCE BANDS ACTUALLY "KNOW" IN EACH SUBJECT

QUEST SOFTWARE

ASSESS CHANGES IN PERFORMANCE OVER TIME WITHIN AND BETWEEN STUDENT COHORTS

ASSESS CONSTRUCT VALIDITY

LOCATE STUDENT ABILITY AND QUESTION DIFFICULTY ON THE SAME SCALE

COMPILE FREQUENCY DISTIBUTIONS ACROSS COMPETENCY BANDS

A way of thinking about these issues. Implications for test design

Situation 1

Students

Worse
Easy Q

x
Q Q

Better
Hard

Questions Situation 2 Students

Worse
Easy

X X XXX X Q QQ Q Q

Better
Hard
32

Questions

Situation 3 Students Worse Easy Questions Situation 4 Students Worse X Q Q Q Q Q QQ XX X XXX X X X X Q QQ X XX Q Q Q X XX X Q Q X Q Q X Q

Better Hard

Better Hard
33

Questions

Easy

Balance of item difficulties

Balance of items for target group

Low ability

High Ability
Medium Ability

Difficulty of Urdu Paper

Low ability

66% Medium Ability

High Ability

Difficulty of Maths Paper

Low ability

66%

High Ability
Medium Ability

Performance of Districts on Common questions

Sr.N Subj Versi Versi Versi Versi Versi Versi o. ect on-1 on-2 on-3 on-4 on-5 on-6

Math s

THANKS