Overreach in High-Stakes Testing:

The case of Chiles National University Ad

mission Test (PSU)
Mladen
Koljatic
Monica Silva
Richard P.
Phelps

9th ITC Conference, San Sebastian, Spain

July 5, 2014

1.Types of admission tests

2.The PSU context and problems
3.Lessons to be learned
4.The role of international testing
associations

1. Types of admission tests

Aptitude Tests or Achievement Tests in th

Where school
inputs vary widely
e Context
of Admission?
academic achievement tests are likely to
measure the opportunity to learn more than
the ability to learn. [Developing] nations
concerned about picking their future talent
must consider the possibility that an aptitude
testmay be more able to overcome the
local differences in school quality
(Heyneman, 1987, p. 253)

SOURCE: Phelps, Standardized Testing Primer,

2007

Aptitude Tests or Achievement Tests in th

e Context of Admission?
Achievement tests are fairer to students because
they measure accomplishment rather than illdefined notions of aptitude; they can be used to
improve performance; they are less vulnerable to
charges of socioeconomic bias; they are more
appropriate for schools, because they set clear
curricular guidelines and clarify what is important for
students to learnthe movement away from
aptitude tests is an appropriate step for U.S.
students, schools and universities (Atkinson, 2001)

Purpose of Admission Tests

To predict college success
without providing an unfair or
unjustified advantage to any particular
group of test-takers

SOURCE: Phelps, Standardized Testing Primer,

2007

2. The PSU context and problems

School Context: 2 curriculums

Curricular

ScientificHumanistic
(SH)

TechnicalProfessional (TP)

Municipal

Private-Subsidized

Private- Paid

Branch
Type of School

The context of Chiles PSU

PSU focused on measuring the SH curriculum
Students who attend the TP tracks
the poorest in the nation
over 90% wish to pursue tertiary education
social m obility hig hly related to univ ersity
education in Chile
* Source: Mineduc, 2009

The context of Chiles PSU

PSU test scores used

Admission
Financial aid
Government subsidies to universities

v ery hig h- stakes associated to test

scores

The context of Chiles PSU

Centralized admission system to all public

funded universities (+ some private universities)
PSU is run by a department within the largest
public university (U. de Chile)
Admission system requires:
PSU test scores
High-school GPA (HSGP)
Class ranking

No other criterion is considered for application

to higher education institutions

11

History of Chiles PSU

From 1966-2003 : aptitude-type test (similar to SAT)

From 2004- the present: PSU
Achievement tests: measure 100% of SH
curriculum
Mathematics
Language
Science (Chemistry, Physics and Biology)

Social Sciences (History, Geography and

Economics)
Change of tests aimed at evaluating outcomes of an
educational reform

Project to Develop the PSU TESTS

Supported and promoted by the

Ministry of Education and council of
rectors of public universities (CRUCh)
Developed using public funds
Project led by an economist and a
social psychologist with no training or
expertise in the development of highstakes tests
13

Multiplicity of Purposes for the PSU

Measure the implementation of a new curricul

um
Fairly measure mastery of one of the two natio
nal curricula (SH)
Incentivize high schools to implement the new
curriculum
Incentivize high school students to study more
Predict overall success of students in universiti
es
Predict success across very different types of u
niversity programs
Reduce socio-economic disparities in performa
nce

Multiplicity of Purposes for the

PSU
Evidence of the tests technical
quality should be provided for each
purpose (St.13.2)
as well as adequate monitoring of
potential negative consequences
(St.13.1)

15

How the PSU Runs

Ministry of
Education

CRUCH

CRUCHS
COMMITTEE OF
TECHNICAL
ADVISORS FOR
PSU

U. de Chile

Main funder of the PSU since 2007.

It holds the purse for CRUCh
universities, since they are funded
through public monies
Council of Rectors of Chilean
Universities
"owners" of the PSU
designated by CRUCh as
supervisors and official evaluators
of the PSU PRESIDED BY
CREATORS OF THE TEST
agency responsible for developing
test items, test assembly, tests
administration, test scoring, for
CRUCh and associated
universities

Source: adapted from the Pearson Report

(2013)

Eight evaluative reports from CRUCh between 200

Official
4-2010: Chilean Evaluations of the PSU

Pseudo-evaluations: overly optimistic reports aimed t

o construct a positive image of the test
Do not report any analyses for the TP group
When reporting gaps, omit the TP data from the analy
ses (without saying so)
Deny the existence of a growing score gap between pr
ivate paid and municipal school groups, especially th
ose attending the TP track
CRUCh denies access to data bases to independent re
searchers

International Evaluations of the PSU

ETS (2005)
Pearson (2013)

19

Efforts to suppress information

2005: ETS Report not released, according t

o U. Chile because of secrecy clause impose
d by ETS
2012: New audit. CRUCh intended:

Auditors to report to expert team chosen by CRUCh

to curtail analyses for TP students
... to control and edit final report

20

Efforts to reveal information

2007: U. de Chile sued to release ETS r
eport
A group of faculty and students repr
esented by a pro-transparency nonp
rofit organization go to court
judge rules in favor of U. de Chile

21

Efforts to reveal information

From 2007: 5 years of pressure by

university student leaders, faculty and
leaders of NGOs for new audit

2012: Funded by the Ministry of Education

(maximum amount suggested by CRUCh
s CTA: US$400,000)

ETS report is finally released in 2012

thanks to TRANSPARENCY LAWS
passed in the country
22

PSU

needs
to be
revised
to advance
equity
Pearsons
Final
Report
of the
PSU (2013):
for
TP groups
Recommendations
reduction in contents
or study alternatives for the equitable as
sessment of CH and TP students
PSU is below predictive standards of other inte
rnational tests
belying claims from CRUChs experts as t
o the predictive capacity of the PSU

Pearsons Final Report of the PSU (2013): S

ERIOUS DEFICITS

Inadequate use of PSU cut scores to assign financial he

lp and scholarships
Inadequate use of scores to pass judgment on the qual
ity of schools

24

Pearsons Final Report of the PSU (2013): S

ERIOUS
Faulty DEFICITS
piloting of test items
DIF analyses does not include factors as
socioeconomic status, high school curric
ular branch (S&H and TP)
Inadequate difficulty level of the PSU Ma
th Test (detected in ETSs 2005 audit)
Actual reporting of PSU Science score de
emed untenable: Science PSU requires
not one, but three scores

25

Pearsons Final Report of the PSU (2013): S

ERIOUS DEFICITS
Inadequate

level of analysis expected when equat

ing a high-stakes assessment: misleading informati
on regarding year-to-year equating and pilot calibr
ations. THESE ACTIVITIES ARE NOT TAKING PLACE EVEN T
HOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDIC
ATE THEY ARE. (p.43)
PSU

equating is below international standards

ETC

26

Pearsons Evaluation of the PSU (201

3)
Comprehensive
Pearson

met with stakeholders other than CRUCh

Incorporated some analyses for the TP group (at no extr
a cost)
Specified new studies that needed (e.g.,, to guide reducti
on of contents)
However
Recommendations are not prioritized
Highly technical report, not aimed at policy makers
Did not look at effects on enrollment
27

Pearsons Final Report of the PSU (201

3):
Revealed how easy it is to mislead t
he public when it comes to technical
matters

CRUCh deceived the public as to the qualit

y of the tests, in terms of prediction and fa
irness
CRUCH does NOT reduce contents in the P
SU, against Pearson's reccomendation
CRUCh's new commitee of technical advis
ors (CTA) is controlled by an associate of t
he PSU test-makers

Aftermath of Pearsons Report...

Past Problems = Present Problem

s
Vested

interests
No real changes in the governance sy
stem
Lack of transparency
No accountability

Main problems remain unsolved

29

Aftermath of Pearsons Report

Students

demand an end to the PSU elimination of the test

Only

HSGPA and class ranking as selection criteria

Moratorium not only to the PSU but to other educational tests as well

30

3. Lessons to be learned

31

Changes in tests

Some nations that are considering

moving towards achievement
tests might benefit from looking
at the Chilean experience to
avoid a proliferation of purposes,
test corruption and abuse

32

Changes entail risks

-Without transparency and full access
to data it is tempting to those in power
to produce pseudo-evaluations that
construct an overly positive image of
the quality of the new test
-There is a need for a governance
scheme that guarantees independent
evaluations in order to protect
applicants rights to be tested by good
and fair tests

Data access is not only a problem in

developing nations

The SAT item-level information is not

frequently available to independent
researchers who are not affiliated with
ETS or the College Board
Source: Santelices & Wilson, 2010.
34

Governance Issues in Order to Prevent Test Ab

use
Test-developing agency should be
professional and independent
Account for funds received and quality of
work
Report to external boards of experts that
can demonstrate independence from the
agency
Periodically audit tests for quality and
fairness
Audit reports & data bases should be
public

4. The role of international testing asso

ciations

Changes in tests
Question to ask:
How can international organizations such
as ITC play a more active role in
promoting good consulting practices by
test experts in developing nations?

Is there a larger role for testing associat

ions such as ITC in the promotion of go
od practices worldwide?

The ITC Guidelines on Adapting Tests

The ITC Guidelines on Test Use
The ITC Guidelines on Computer-Based
and Internet-delivered Testing
The ITC Guidelines on Quality Control in
Scoring, Test Analysis and Reporting of T
est Scores
The ITC Guidelines for Test Review?
The ITC Guidelines for International Consult
ing?

Overreach in High-Stakes Testin

g:
The case of Chiles National Univ
ersity Admission Test (PSU)
Mladen Koljatic
Monica Silva
Richard P. Phelps

9th ITC Conference

San Sebastian, Spain
July 5, 2014

Predictive validities of the

PSU
(CRUCh vs Pearson
estimates)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU,

January 2013; CTA
41

Predictive validities: ACT

and PSU

SOURCE: ACT, Research Summary Services, 1997_1998; Pearson,

Final Report Evaluation of the Chile PSU, January 2013

Predictive validities: SAT

and PSU

SOURCE: Pearson, Final Report Evaluation of the Chile PSU,

January 2013

Predictive validities: SAT

and PSU
(BUSINESS MAJORS)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU,

January 2013
44

Predictive validities: SAT

and PSU
(Education)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January

2013

Percentage-point drop in enrollment of municipal s

chool graduates attending two top universities
UC

U. de Chil

e
PAA (adm. 2003)
17
32
__________________________________
PSU (adm.2004)
PSU (adm.2005)
PSU (adm.2006)
PSU (adm.2007)
PSU (adm.2008)
PSU (adm.2009)
PSU (adm.2010)
Source: official institutional info

17
16
14
13
11
11
12

29
25
20
----46

Drop in enrolled students from Municipal

Schools, other CRUCh Universities

Fuente: E. Simonsen, 2009

Estimated average 10% drop in e

nrollment from students from muni
cipal schools at top CRUCh Unive
rsities in three years NO measu
res taken