1

Evaluation Process

Evaluation Process
Michelle M. Fraley, Ph.D.
CUR/ 528
October 26, 2015
Professor Siddeeqah Johnson

2
Evaluation Process
Evaluation Process
The Agreement for the Sustainable Reform of the Puerto Rico Police Department
(PRPD), also known as the Consent Decree, requires the former University College of
Criminal Justice (UCCJ), now the Puerto Rico Police Academy, to conduct cyclic needs
assessments of the pre-service education and training program (United States of America,
v. Department of Justice of the Commonwealth of Puerto Rico, 2013). The pre-service
education and training program for the PRPD cadets must consist of at least 900 hours
of instruction that comports with generally accepted policing practice with respect to
quality and content, and that (United States of America, v. Department of Justice of
the Commonwealth of Puerto Rico, 2013, p. 46). Requirement 120 of the Consent
Decree states, The PRPD and the UCCJ shall conduct subsequent regular reviews, at
least semi-annually, and submit their findings and recommendations in written reports to
the Superintendent. (United States of America, v. Department of Justice of the
Commonwealth of Puerto Rico, 2013, p. 49).
The Consent Decree does not specify a pre-service education and training program
evaluation; however, the pre-service education and training program evaluation could be
designed using Kirkpatrick four-level model. Kirkpatricks model includes the following
to evaluate a program: reaction (level 1), learning (level 2), behaviors (level 3), and
results (level 4) (Abdulghani, Shaik, Khamis, Al-Drees, Irshad, Khalil, & Isnani, 2014).
During the federal public hearing held on 22 and 23 October 2015, stakeholders are
demanding an evaluation of the program, namely the Civil Rights Commission present at
the hearing.

3
Evaluation Process
Level 1 focuses on the cadets perceptions of the pre-service education and training
program; level 1 measures cadets satisfaction of the training received (Abdulghani et al.,
2014). Level 2 aims at the acquired knowledge, skills that result in changes in behavior;
level 2 measures what knowledge and cognitive skills were learned to change attitudes
and behaviors (Abdulghani et al., 2014). Level 3 focuses on the workplace; level 3
measures whether the learned knowledge, skills and attitudes transferred to the
workplace to reflect positive changes in behavior and job performance (Abdulghani et
al., 2014, p. S25). Finally, level 4 aims at the organizational benefits of the training; level
4 evaluates the results of the improved performance of the police officers (Abdulghani et
al., 2014).
Training is one of eleven areas of compliance in the Consent Decree. Within the
training compliance area, there are four major categories: pre-service education and
training, field training program, in-service training, and training records. To do a holistic
training program evaluation implementing Kirkpatricks four-level model, pre-service
education and training, field training program, and in-service training must be integrated.
The Investigation of the Puerto Rico Police Department (2011) document
highlights events from 2007 to 2011, showing reasonable cause to believe that PRPD
officers engage in a pattern and practice of excessive use of force and unlawful searches
and seizures against Fourth Amendment rights, and unreasonable force and other
misconduct against First Amendment rights (p. 5). If the Investigation of the Puerto Rico
Police Department (2011) document is used as the source document to a level 4
evaluation, the outcomes of the PRPD integrated training program have been a failure.

4
Evaluation Process
The Investigation of the Puerto Rico Police Department (2011) report used both
qualitative and quantitative methods involving both objective and subjective measures.
In reviewing Royse, Thyer, and Padgett (2010), the integrated training program
evaluation process falls under scenario 4: Evaluation for Accountability (p. 14). The
purpose of the evaluation of the integrated training program is to ensure quality,
consistency and compliance with applicable law, PRPD Policy and this Agreement
[Consent Decree] (p. 46). The level of evaluation is statewide, meaning the
Commonwealth of Puerto Rico. The evaluation includes several stakeholders such as the
United States and local Justice Departments, the American Civil Liberties Union
(ACLU), the Civil Rights Commission, underrepresented communities such as Lesbian,
Gay, Bisexual, Transvestite, Transsexual (LGBTT), Dominican Republic, handicapped,
and homeless, to name a few, and all the people who live and/or have property in Puerto
Rico.
Data collection procedures vary based on the level. For level 1 data collection,
Likert scale surveys at the end of each course as well as at the end of the pre-service
education and training program would work well. The overarching direction behind the
questions during a level 1 evaluation is how did participants feel about the course?
(Abdulghani et al., 2014, p. S25). The cadets are in a controlled environment at the
Puerto Rico Police Academy that would facilitate a qualitative approach following the
quantitative survey as part of the procedure for analyzing the data. At the end of the preservice education and training program, the data would be analyzed at a higher level to
integrate all thirty-plus courses. Table 1 depicts results of a mixed approach using an
explanatory method collecting quantitative data, then qualitative data (Creswell, 2008).

5
Evaluation Process
Data collection procedures for level 2 include rubrics to measure improvement in
knowledge, skills, and attitudes. The rubrics are part of scenario driven training during
both pre-service education and training and in-service training. The procedures for
analyzing the data is done at the Puerto Rico Police Academy for both types of training
done in Gurabo, Puerto Rico. However, the procedures for analyzing the data for the inservice training is done at each of the 13 police command areas. Once the Puerto Rico
Police Department develops an automated tool, the procedures for analyzing the data can
be integrated throughout the entire Puerto Rico Police Department. The overarching
direction behind the questions during a level 2 evaluation is to what extent participants
(cadets and police officers) improve knowledge, skills, and change attitudes as a result of
training? (Abdulghani et al., 2014, p. S25).
Level 3 data collection at the Puerto Rico Police Department has not been done
under the Consent Decree because not enough time has elapsed since the time the inservice training was conducted based on the newly approved General Orders. The
overarching direction behind the questions during a level 3 evaluation is to what extent
did participants (police officers) change their behavior in the workplace as a result of
training? (Abdulghani et al., 2014). The Puerto Rico Police Department Forms, such as
Informe de Uso de Fuerza (PPR-854), would be used as the data collection tool for
quantitative analysis while the follow-on determinations from the Use of Force Board
would provide the qualitative analysis. Analysis of the data would involve correlation of
behavior with training, both content and instructors.
Level 4 data collection, during the next five years will most likely be best suited to
be performed under the purview of the Technical Compliance Advisor because the need

6
Evaluation Process
to measure the main outcomes of the Consent Decree. The overarching direction behind
the questions during a level 4 evaluation is what is the organization benefit as a result of
training? (Abdulghani et al., 2014). Data collection procedures will include both
qualitative Island wide surveys as well as face-to-face interviews with stakeholders
represented in the newly established Community Interaction Councils under requirements
209 through 213 (United States of America, v. Department of Justice of the
Commonwealth of Puerto Rico, 2013).
In addition to the program evaluation, as the policies and procedures are created or
updated, the Puerto Rico Police Academy would update the training programs.
Concurrently, the Puerto Rico Police Academy would analyze the data gathered using the
Puerto Rico Police Department Forms to capture patterns of behavior that require
modifications to the pre-service education and training program and in-service training
program. The transfer of the collection of data from the operational units to the Puerto
Rico Police Academy does not exist until the automated tool is implemented.
Preparing the report is a twofold effort. As previously mentioned Requirement 120
of the Consent Decree mandates a report to the Superintendent with the findings and
recommendation. Before submitting the findings and recommendations to the
Superintendent, key stakeholders are invited to collaborate to obtain their feedback. The
first phase of their feedback is in the form of a table describing the courses and hours of
instruction with a remarks column capturing the recommended change to the
Superintendent (Table 1). Once the Superintendent approves the recommendation for
implementation, the second phase of stakeholder feedback starts with the review of every
course of the pre-service education and training program. Finally, the Puerto Rico Police

7
Evaluation Process
Academy submits input to the semi-annual TCA and the United States Department of
Justice report. The report is eventually published as a public document.

8
Evaluation Process
Table 1. Pre-service education and training curriculum 2015-2016

Criminologa

45

Secuencial
Curricular
2015 2016
Horas Contacto
Recomendadas
24

Destrezas de
Comunicacin:
Redaccin de
Informes y
Formularios
Curso Bsico de Uso
de Fuerza

45

45

45

45

Fundamentos
Psicolgicos de las
Relaciones Humanas
y Desarrollo Personal

45

45

Curso

Secuencial
Curricular
2014 2015
Horas Contacto

Justificacin
Se recomienda la disminucin
de 45 a 24 horas de contacto.
El curso de 45 horas contiene
unos temas que no son
relevantes como lo son la
historia de la criminologa y
las principales escuelas de
pensamiento. Tambin
contiene temas que se ofrecen
en otros cursos como lo son los
programas preventivos de la
Polica de Puerto Rico
(Programa Liga Atltica,
Programa de Vuelta a la Vida
y Consejos Comunitario de
Seguridad) que se ofrecen
como parte del curso de Polica
Comunitario.
Cambios internos para dar ms
nfasis a la redaccin y menos
nfasis a informes y
formularios que se cubren en
otros cursos.
Atemperar con las nuevas
rdenes Generales 600-601
(Uso de Fuera), 603 (Armas
Impacto) y 605 (Uso de
Fuera).

Translation: Spanish to English:

Curso=course
Secuencial Curricular 2014-2015 horas contacto=Curriculum 2014-2015 contact hours
Secuencial Curricular 2015-2016 horas contacto recomendadas= Curriculum 2015-2016
recommended contact hours
Justificacin=justification

9
Evaluation Process

tica Policial

45

Secuencial
Curricular
2015 2016
Horas Contacto
Recomendadas
32

Derecho Penal

45

45

Derecho Civil y
Constitucionales
Curso Bsico en Uso
y Manejo de Armas

45

45

44

44

Eficiencia Fsica
(Primer Bimestre)

24

45

Leyes Especiales

45

45

Curso

Secuencial
Curricular
2014 2015
Horas Contacto

Justificacin
Se recomienda la disminucin
de 45 a 24 horas de contacto.
El curso de 45 horas contiene
unos temas que cubre la
Oficina de tica
Gubernamental durante la
semana de inmersin y hay
algunos temas como la teora
de Desarrollo Moral y
Pensamiento Crtico, Uso de
Fuerza y Derechos Civiles y
Constitucionales, que se
cubren en otros cursos.
Cuando la poltica pblica de
la PPR sea aprobada, este
curso se ampliar para cubrir
temas como el Cdigo de
Conducta de la PPR.
Cambios internos para incluir
el nuevo cdigo penal con
fecha efectiva de marzo de
2015.
Esperando recomendaciones
de la ACLU.

Se recomienda aumentar de 24
a 45 horas contacto Las 24
horas de contacto no fueron
suficiente para que los cadetes
pasaran con un 70% cada uno
de los tres eventos.

10
Evaluation Process

Curso

Secuencial
Curricular
2014 2015
Horas
Contacto

Procedimiento Criminal

45

Registro y Allanamiento

45

Administracin de Leyes de
Transito
Curso Intermedio en Uso y
Manejo de Armas
Curso Intermedio en Defensa
Personal, Tcnicas y
Mecnicas de Arresto

45

Introduccin a la
Mediacin y Solucin de
Conflictos

Secuencial
Curricular
2015 2016
Justificacin
Horas
Contacto
Recomendadas
40
Se recomienda la disminucin
de 45 a 40 horas contacto. Las
Reglas del 229 al 234 se
discuten en el curso de
Registro y Allanamiento y se
eliminan de este curso.
40
Se recomienda la disminucin
de 45 a 40 horas contacto. Se
elimin temas repetitivos que
se discuten en el curso de
Derechos Civiles y
Constitucionales.
45

28

28

45

45

45

32

Atemperar con las nuevas

rdenes Generales 600-601
(Uso de Fuera), 602
(Dispositivo de Control
Elctrico) y 605 (Uso de
Fuera).
Se recomienda la disminucin
de 45 a 32 horas contacto
porque se eliminara el Modelo
S.A.R.A. ya este tema se cubre
en el curso de Polica
Comunitario. Tambin se
eliminara los temas como la
comunicacin eficaz,
comunicacin no verbal y
escuchar activamente ya que se
han discutido en el curso de La
Conducta Organizacional en el
Contexto Policial.

11
Evaluation Process

Curso

Secuencial
Curricular
2014 2015
Horas
Contacto

Violencia
Domestica/Hostigamiento
Sexual/LGBTT

32/4/0

Certificacin en Uso y
Manejo de Gas Pimienta

Eficiencia Fsica (Segundo

Bimestre)

24

Secuencial
Curricular
2015 2016
Justificacin
Horas
Contacto
Recomendadas
45
Se recomienda aumentar de 32
(Violencia Domestica) y 4
(Hostigamiento Sexual) a 45
horas contacto total para
incorporar la comunidad
LGBTT segn recomendado
por la Comisin de Derechos
Civiles.
8
Atemperar con las nuevas
rdenes Generales 600-601
(Uso de Fuera), 604 (Agentes
Qumicos) y 605 (Uso de
Fuera).
45
Se recomienda aumentar de 24
a 45 horas contacto Las 24
horas de contacto no fueron
suficiente para que los cadetes
pasaran con un 75% cada uno
de los tres eventos

12
Evaluation Process

Cursos
Polica Comunitario

Secuencial
Curricular
2014 2015
Horas
Contactos
45

Introduccin a la Justicia
Juvenil

45

Ley de Evidencia con

Testimonio en Corte

45

La Conducta
Organizacional en el
Contexto Policial

45

Secuencial
Curricular
2015 2016
Justificacin
Horas
Recomendadas
24
Se recomienda la disminucin
de 45 a 24 horas contacto. El
curso de 24 horas es una
orientacin al trabajo Policial
orientado a la Comunidad
explicando el Modelo
S.A.R.A., los programas
preventivos de la Polica de
Puerto Rico (Programa Liga
Atltica, Programa de Vuelta
a la Vida y Consejos
Comunitario de Seguridad) y
la aplicacin de la teora de
motivacin de Abraham
Maslow, Jerarqua de
Necesidades. Cuando la
poltica pblica de la PPR sea
aprobada, este curso se
ampliara para cubrir temas y
actividades prcticas
relacionadas al rol del polica
comunitario
32
Se recomienda la disminucin
de 45 a 32 horas contacto. La
Ley 37, Ley 95 y Ley 88 son
bastantes cortas y se cubren
en 32 horas contacto.
45
40

Se recomienda la disminucin
de 45 a 40 horas contacto. El
curso de 45 horas contiene
unos temas que se cubren en
el curso de Fundamentos
Psicolgicos de las
Relaciones Humanas y
Desarrollo Personal.

13
Evaluation Process

Investigacin Criminal y
Manejo de Escenas

45

Secuencial
Curricular
2015 2016
Horas
Contacto
Recomendadas
45

Manejo y Protocolo en
Situaciones de Crisis

45

45

Situaciones Simuladas en
Destrezas Operacionales

45

45

Curso

Secuencial
Curricular
2014 2015
Horas
Contacto

Justificacin

14
Evaluation Process

Cursos
Tcnicas de Entrevista e
Interrogatorio

Secuencial
Curricular
2014 2015
Horas
Contactos
No estaba
integrado el
curso

Secuencial
Curricular
2015 2016
Justificacin
Horas
Recomendadas
40
Se recomienda integrar el
curso de Tcnicas de
Entrevista e Interrogatorio
porque es fundamental tener
el conocimiento de las
tcnicas de entrevistas y como
se interroga a un sospechoso y
sus vctimas.
4

Ciberntica en la
Investigacin Criminal

Igual Proteccin y No
Discrimen
(Personas Sin Hogar) y
Trata Humana
Ingls Bsico Policial

4/0

No estaba
integrado el
curso

20

16

24

45

NIMS 100

NIMS 700

1110

1158

Patgenos
Sanguneos/Respuesta en
Caso de
Emergencias/Primeros
Auxilios
Eficiencia Fsica (Tercer
Bimestre)

Total de Horas Contacto

Se recomienda integrar del

Curso de Trata Humana segn
recomendado por la Comisin
de Derechos Civiles.
Se recomienda integrar el
Curso de Ingls Bsico
Policial para aprender la
terminologa policial utilizada
en el mbito laboral Policiaco
y facilitar cursos virtuales en
ingls (FEMA, HIDTA, etc.)
Se recomienda la integracin
del Curso de Primeros
Auxilios.
Se recomienda aumentar de
24 a 45 horas contacto. Las
24 horas de contacto no
fueron suficiente para que los
cadetes pasaran con un 80%
cada uno de los tres eventos.
Adiestramiento virtual
(FEMA)
Adiestramiento virtual
(FEMA)

15
Evaluation Process
References
Abdulghani, H. M., Shaik, S. A., Khamis, N., Al-Drees, A. A., Irshad, M., Khalil, M. S.,
& Isnani, A. (2014). Research methodology workshops evaluation using the
Kirkpatrick's model: Translating theory into practice. Medical Teacher, 36S24S29. doi:10.3109/0142159X.2014.886012
Creswell, J. W. (2008). Educational research: Planning, conducting and evaluating
quantitative and qualitative research (3rd ed.). UpperSaddle River, NJ: Pearson
Royse, D., Thyer, B., & Padgett, D. (2010). Program evaluation: An Introduction
(5th ed.). Belmont, CA: Cengage Learning.
United States Department of Justice Civil Rights Division (September 5, 2011).
Investigation of the Puerto Rico Police Department. Retrieved from
https://drive.google.com/file/d/0BztrH26ENotejhpNDkzVVVHUjQ/view?pli=1
United States of America, Plaintiff, v. Department of Justice of the Commonwealth of
Puerto Rico, Defendant. Civil Action No. 11-1339 (SEC) (2013). Retrieved from
http://www.ada.gov/puerto_rico_doj.htm