Developing an Early Warning System: Identifying

Factors that may Predict High School Completion

by 6th Grade Students

(Revised 3/25/2009 1:46 PM)

Prepared by

Kenneth T. Wilburn, Ph.D.

University of North Florida

Marcia Lamkin, Ed.D.

University of North Florida

Dax M. Weaver, MPH

President, Health-Tech Consultants, Inc.

Prepared For

The Community Foundation of Jacksonville

For additional information, contact

Ms. Pam D. Paul, Vice President, Community Initiatives

The Community Foundation in Jacksonville

121 West Forsyth St., Suite 900, Jacksonville, FL 32202

(904) 356-4483, www.jaxcf.org

 Early Warning System  2 

Executive Summary

Purpose

The primary purpose of this study was to identify specific academic and social factors
that may be used as “flags” to identify Duval County Public Schools (DCPS) 6th grade students
who are at high risk of not completing high school. A secondary purpose was to understand
how these factors may or may not be aligned with those identified by Neal, Balfanz, and other
researchers (Neal and Balfanz, 2006).

Four questions guided our 6th grade analysis: (1) Are significant numbers of students
showing the signs of disengagement previously identified in our examination of 9th grade
students who fail to complete high school? (2) Can we identify a set of indicators that flag sixth
graders who have high odds of failing to complete high school? (3) From this set of indicators,
can we identify a practical set of indicators routinely collected and reported at the individual level
that flag sixth graders who have odds of failing to continue in school until graduation? (4) Can
we then identify a set of those indicators that may be influenced by school and community
programs?

Data

The data used in this analysis were provided by the DCPS Department of Instructional
Research and Accountability. The data were gathered from the district’s data warehouse and
provided to the research team in the form of several Excel spreadsheets. The data included
selected demographic, academic, attendance, and disciplinary information from each student
enrolled in grades 6–12 during the 2001–2002 through the 2007–2008 school years.
Approximately 60,000–65,000 students were enrolled in grades 6-12 during each school year
with each grade’s enrollment averaging between 10,000–12,000 students.

Since the focus of this analysis was 6th grade students, we began by identifying all 6th
grade students for the 2001–2002 school year and then tracking that cohort of students through
each school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093)
students remained for inclusion in the analysis after the elimination of withdrawals, transfers,
and incomplete data sets.
 Early Warning System  3 

Methods and Procedures

Once the data set had been cleaned of cases with missing data or incorrectly coded
data, an initial factor analysis was conducted to determine whether variables fell into coherent
categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance
or mathematics standardized test scores, has enough weight to serve as a valid predictor.
Therefore, this process of combining various data sets was repeated until a consistent set of
categories was identified. Once a coherent set of predictors was identified, linear regression
(stepwise) and logistic regression analyses were conducted on each identified factor in order to
estimate the strength of each factor.

Results
A significant number of 6th grade students displayed the signs of disengagement, such
as poor academic achievement and excessive absences, previously identified in our study of 9th
grade students and in studies conducted by other researchers. Five factors proved to be of the
greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate
on-time from high school in 2007-2008. Each flag was a statistically significant flag (p< .05)
even after controlling for the other flags. The five flags listed below, individually and in
combination, identified 81.52% of the non-graduates.
ƒ Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate
on-time scored below Level 3 on the Florida Comprehensive Assessment Test.
ƒ Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate
on-time scored below Level 3 on the Florida Comprehensive Assessment Test.
ƒ Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to
graduate on-time failed a math course.
ƒ Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who
failed to graduate on-time failed an English/Language Arts course.
ƒ Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time
missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among
the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th
grader who failed to graduate on-time. However, our tracking of students indicated that the
number of Class 3 and 4 violations increase significantly as students get older and as more
 Early Warning System  4 

students become over-age for their assigned grade. Consequently, the conduct violations from
subsequent years have more predictive power than those from Grade 6. While the yield for this
factor was relatively small as a single factor and no greater when considered in combination
with test scores or grades, we nevertheless consider this to be an important flag to monitor as it
becomes a key indicator in later grades.

Results from our study indicated

1. Significant numbers of 6th grade students show signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school.

2. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth
graders who have high odds of failing to complete high school.

3. From this set of flags, we can identify a practical set of indicators routinely collected and
reported at the individual level that flag sixth graders who have odds of failing to
continue in school until graduation.

4. These indicators are factors that may be impacted by school and community programs.

In conclusion, as we have learned from our previous studies of first-time 9th grade
students and focus groups with students who had previously dropped out of school, students do
not graduate from high school for a multitude of reasons. Many of those reasons are not linked
directly to academic achievement, attendance, and behavior at school. However, our research
supports the work by Balfanz (2006) and others (Balfanz, Hertzog, Neild, and MacIver, (2007))
that there is ample, readily available data for early identification of students with the greatest risk
of failing to graduate high school. 
 Early Warning System  5 

Table of Contents

Page

INTRODUCTION ……………………………………………………………………………...… 7

BACKGROUND……………………………………………………………………………..…... 8

A Descriptive Study of First-time 9th Grade Students

Focus Groups with Dropouts, Parents and Service Providers

METHODS AND PROCEDURES……………………………………………………………… 12

Guiding Questions

Outcome Variable

Data Source

Sample and Measures

Data Analysis Procedures

FINDINGS………………………………………………………………………………………... 17

Identifying Warning Flags

Attendance

Standardized Test Scores

Course Grades

Discipline

Status Variables

Final Selection of Identification Flags

Validity of the Flags

Limitations of the Study

SUMMARY……………………………………………………………………………..………… 26

References…….…………………………………………………………………………………. 30

Appendix: Demographic Characteristics of the Study Population…………………………. 31

 Early Warning System  6 

List of Tables Page

Table 1. School Related Variables of the Study Population  14 

Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags  19 

Table 3. Double Warning Flag Combinations and Counts for all Duval 6th Grade 19 
Students in 2001–2002 Who Failed to Graduate in 2007-2008 
 Early Warning System  7 

Introduction

This study is the third in a series sponsored by The Community Foundation of

Jacksonville Florida (CF) designed to examine issues related to Duval County Public School

(DCPS) students who fail to graduate from high school. The first study, Where Have All the 9th

Graders Gone? (Wilburn, Weaver, and Wilburn, 2008), was a descriptive study of a cohort of

first-time 9th grade students. The study was designed to provide the local community with a

more detailed understanding of the non-completion problem in Jacksonville. The second study,

Kids Say the Darndest Things (Roush and Wilburn, 2008), was a qualitative analysis of focus

group data which had been collected by The Community Foundation and their community

partners in the Learning To Finish Project. These focus groups were conducted with 16–24

year old students who had been identified by the school district as being of high risk of dropping

out, currently enrolled in a school or community sponsored dropout prevention program, or had

dropped out of school and were incarcerated in the Duval County jail for a criminal offense. As

with the descriptive study, the primary goal of the effort was to gain a richer understanding of

students who have dropped out of the public school system. This study used the information

learned in the first two studies to develop an early warning system that could be used by the

community to identify the students, at the end of the 6th grade, who are the most at risk for

failing to complete high school.

The report will be organized into the following four sections:

• Background. This section provides a review and summary of the 9th grade descriptive

and focus group studies and how the outcomes from those studies were used to inform

the 6th grade predictive study.

 Early Warning System  8 

• Methods and Procedures. This section describes the methods and procedures used to

conduct the 6th grade predictive study.

• Findings. This section reports the results from the predictive analysis and their

application to the development of an early warning system as well as the limitations of

the study.

• Summary. The final section reviews the main points of the study.

Background

In this section we provide a summary of the two studies we conducted as precursors to

beginning the process of developing flags for identifying 6th grade students with a high

probability of not completing high school.

A Descriptive Study of First-time 9th Grade Students

The first study was designed with the overall goal of developing a better understanding

of issues surrounding the graduation crisis and the dropout problem found in DCPS. Simply

stated, the design of the study was to follow three cohorts of students from the time they first

entered the 9th grade through their 4th or 5th years of high school. Using data provided by the

DCPS Department of Research and Accountability, we were able to select a study population

that met our criteria for inclusion in the analysis. As recommended by Balfanz (2007), we sought

to study cohorts of 9th grade students who had been enrolled in DCPS during the previous

grades, were enrolled in the 9th grade for the first time, and were engaged in an academic

program for which a standard high school diploma was a reasonable expectation. Additionally,

we targeted a study population that was reflective of the overall 9th grade population in gender,

race, and socioeconomic status.

 Early Warning System  9 

Graduation and Dropout Rates. The analysis indicated that by the end of the 3rd year of

high school, approximately 488 students (2.5%) of the original first-time 9th grade cohort had

received their diploma. By the end of the 4th year, the number of students who had graduated

increased to 11,842 students (60.74%), and by the end of the 5th year, the students who had

received a standard high school diploma was 11,901 (61.04%). While approximately 61% of the

first-time 9th grade students eventually graduated, 95% of those did so in the customary 4th year

of high school. Only a small percentage (4.1%) of the students graduated early, and an even

smaller percentage (0.005%) graduated after a 5th year of high school. In addition to tracking

graduates and non-graduates, we also identified the number and characteristics of students

labeled “dropped out” (i.e., did not withdraw or transfer but just failed to show up the next school

year). Our calculations indicate that the number of students who dropped out increased with

each succeeding cohort. The 2002 cohort had an estimated dropout rate of 25.5%, followed by

29.5% for the 2003 cohort, and rising to a high of 31.3% with the 2004 cohort. When examined

as a whole, the estimated dropout rate for the combined three student cohorts was

approximately 28.2%. What this means in terms of students lost is that over the 4 years of high

school almost 2,000 students left each cohort without receiving diplomas.

Focus Groups with Dropouts, Parents and Service Providers

In 2007-2008, under the direction of Dr. Shannon Perry, chair of the projects qualitative

focus group, seven focus groups met to explore the dropout situation in Jacksonville.

Information from these sessions were transcribed and provided for analysis to Dr. Connie

Roush, a qualitative research specialist with the Brooks College of Health at the University of

North Florida. In this report, qualitative themes were identified with highlighted responses from

each focus group and quotes provided to illustrate some of the major points. The major themes

identified were participants’ attitudes and beliefs about the dropout situation; the influence of the

family, school, and community environments; specific issues related to peer pressure; the
 Early Warning System  10 

culture of violence; the need to know and care for students; and accountability for school

attendance. Each group of participants approached the focus groups from a different point of

view or attitude shaped partially by their own beliefs about what spurs school disengagement

and dropout.

Major Themes. Each group reinforced the belief that getting to know a student takes

continuous contact over time and a lot of listening. This lays the groundwork for a trusting and

caring relationship necessary to (as one student said) “…attract us to school and keep us

there.” Listening was a topic most often addressed by the middle school and incarcerated

youth while all groups offered wisdom about the caring relationship and the time that it takes to

evolve. Another theme our analysis identified was the power of the need to “fit in” to school

culture and/or the “wrong crowd.” Whether it is with other students in school or with groups in

the neighborhood, peer pressure is extremely strong. A third theme identified was the “culture

of violence” that is so much a part of students’ lives. Many students described their

disengagement from school as being “sucked in” to using and selling drugs and the violent

lifestyle that goes with it. While the family emerged as a critical theme, it was seen by many

teens as equally as positive and negative as a factor. For many, the discussion centered around

the barriers put up by family to be supportive. In a smaller but equally powerful part of the

discussions, students spoke about the people who motivated and inspired them to stay in

school. In the course of reports on all previous themes, the influence of the school environment

on disengagement and dropout is evident. There was a great deal of concern by all

participating groups regarding the family’s communication with the school, conditions that

prompt children and teachers to stay at a school, and the need for a variety of educational

options. While all emphasized that the lines of communication must be open for parents,

students, and the school to work together, few could provide any positive examples. Most of the

examples were negative in nature describing the problems and consequences of poor
 Early Warning System  11 

communication. In regard to perceptions of the community, little time in the focus groups

examined this in particular. However, most comments conveyed the message that the

community was not aware and did not understand why students are leaving school and was not

doing anything to address the problem. During the focus groups, participants explored the issue

of school attendance from multiple perspectives. Students described “skipping school” and

being “kicked out” of class; parents were concerned about timely notification of absences; and

all participants discussed the need for accountability. One other topic of concern was the

enforcement of attendance policies and the potentially negative consequences for students with

special circumstances.

Taking Action. Students, parents, and teachers discussed the different programs that

can lead to graduation, the need to “graduate on time,” and the pros and cons of graduation

options (the usual high school diploma, the G.E.D. pathway, and the certificate of graduation).

They also discussed the experience of transferring to an alternative program (specifically the

Pathways program). All three groups agreed that the high school diploma was the most

desirable of the three. According to one teacher, the high school certificate has a major impact

on the community as well as the students because it limits their employment opportunities.

Another teacher stated that it basically indicates, “I went to school for 12 years but I did not earn

a high school diploma.” When comparing the G.E.D. to the diploma, one parent told her child,

“Some places won’t take the G.E.D., so it is better for you to get your high school diploma. It will

open more doors.” From the perspective of a student, programs to help students stay in school

were described this way:

“When we get funding we try to make the kid fit into the grant so we aren’t actually

working on what needs to be worked on. The people who are giving the money or

disbursing the money do not understand the population to which they are giving. Schools
 Early Warning System  12 

try to make the money fit in the programs but it doesn’t work” (Roush and Wilburn, 2008,

p. 17).

Methods and Procedures

Guiding Questions

Four questions guided our 6th grade analysis:

1. Are significant numbers of students showing the signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school?

2. Can we identify a set of indicators that flag sixth graders who have high odds of failing to

complete high school?

3. From this set of indicators, can we identify a practical set of indicators routinely collected

and reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation?

4. Can we identify indicators within that set that may be influenced by school and

community programs?

Outcome Variable

The outcome variable used was whether or not the sixth graders in the 2001–2002

cohort graduated from high school on time or after one extra year. We chose this outcome, in

part, because of our desire to replicate the study reported by Balfanz, Herzog, Neild, and

MacIver (2007) which previously identified a set of sixth grade predictor variables. In addition,

we selected this variable because in the previous analysis of 9th grade students, the vast

majority of students who earn a diploma did so on time or within one additional year. Therefore,

extending the time frame would be of little significance. It is also important to note that we
 Early Warning System  13 

selected to use graduating from high school as opposed to dropped out of school since each

student who receives a standard high school diploma is clearly identified with a high degree of

accuracy in the district’s data system; however, students who fail to graduate can do so for a

multitude of reasons (e.g., transfers, withdrawal, failing to report to school) that are not captured

with a high degree of accuracy by the district’s data system.

Data Source

As with previous studies, the data used in this analysis were provided by the DCPS

Department of Instructional Research and Accountability. The data were gathered from the

district’s data warehouse by a district staff member and provided to the research team in the

form of several Excel spreadsheets. The data included selected demographic, academic,

attendance, and disciplinary information from each student enrolled in grades 6–12 during the

2001–2002 through the 2007–2008 school years.

Sample and Measures

Since the focus of this analysis was 6th grade students, we began by identifying all 6th

grade students for the 2001–2002 school year and then tracking that cohort of students through

each school year including 12th grade (2007–2008 school year). We created a longitudinal

dataset designed to follow the performance of students enrolled in Grade 6 during the 2001-

2002 school year. The dataset included attendance, demographic components, math and

reading course grades, and math and reading standardized test scores per year. These

students were traced through the 2007-2008 school year, the year during which students

normally complete Grade 12 and graduate or finish a program. Those still enrolled as DCPS

students at the end of the 2008 school year were categorized as either “Graduated with

diploma,” “Received certification of completion,” or “Transferred/withdrew.” These categories

were added to the dataset.

 Early Warning System  14 

In our data file 10,516 students were identified as being in Grade 6 for the 2001–2002

school year. From this initial population, 423 students were deleted due to incomplete data sets

which resulted in a data set of 10,093 students for our study population. This sample of

students in Grade 6 consisted of White (47.7%), African-American (44.3%), Hispanic (3.7%),

Asian-American (2.8%), and Multiracial (1.3 %) students. Slightly more students were male

(51.5%) than female (48.5%) and slightly more than eight percent (8.3%) of these sixth graders

required services as English Language Learners. For the purpose of this study, students in the

two most severe classifications of special education (those who were not expected to graduate

or to earn a certificate of completion) were eliminated from the sample. The remaining ESE

students (those who spent at least part of the school day in regular education classes) were

handled as regular education students.

No data about free or reduced lunch eligibility were available for about half the students

in the Grade 6 population. Data for 5,015 of the students indicated that 94% qualified for free or

reduced lunch (about half of the total Grade 6 sample). Of the complete study population

(10,093), only about 1% was younger than expected in the Florida public schools, while 41.5%

were of “normal” age for the grade level and 58.4% were over the expected age indicating that

they had been retained at some earlier grade level. At the end of the 2008 school year, 66.7%

(6,733) of the students graduated by 2008 while 33.3% (3,360) did not.

Table 1 provides an overview of the study population in regard to the school related

variables such as program participation, disciplinary infractions, course grades and

standardized test scores.

Table 1. School Related Variables of the Study Population

Group Variable Completers Non-completers

Drop-Out % Enrolled in Dropout Prevention 12.4 28.2

Program Program
 Early Warning System  15 

% Not Enrolled in a Drop-Out 87.6 71.8

Prevention Program

Discipline % Serious Violation (Code 3/4) 6.4 11.3

% No Serious Violation (Code 3/4) 93.6 88.7

Standardized Reading 2002 Mean Score 1721.67 1698.17

Test Scores
(FCAT Math 2002 Mean Score 1724.63 1741.99
Development
Scale Score) Reading 2003 Mean Score 1811.22 1784.85

Math 2003 Mean Score 1819.72 1826.28

Reading 2004 Mean Score 1880.48 1722.11

Math 2004 Mean Score 1907.80 1772.70

Reading 2005 Mean Score 1904.81 1698.39

Math 2005 Mean Score 1941.38 1793.39

End of Language Arts 2002 Mean Grade 2.57 1.98

Course
Grade GPA Math 2002 Mean Grade 2.47 1.83

Language Arts 2003 Mean Grade 2.55 2.09

Math 2003 Mean Grade 2.42 1.86

Language Arts 2004 Mean Grade 2.48 2.11

Math 2004 Mean Grade 2.23 1.81

Language Arts 2005 Mean Grade 2.39 2.28

Math 2005 Mean Grade 2.10 1.87

As presented in Table 1, there are significant differences in academic related

characteristics between those 6th grade students who graduated from high school and those

who did not. For example, for those students who were enrolled in a dropout prevention

program, only 12.4% graduated high school while 87.6% of those who were not in the program

earned their diploma. Among students who committed a serious conduct violation, a much

larger percentage did not complete high school. There were also significant differences

between completers and non-completers on the Developmental Scale Scores of the Florida
 Early Warning System  16 

Comprehensive Assessment Test (FCAT). When compared over the four years students were

tracked, those who completed high school had higher reading scores each of the four years and

higher math scores two of the years. While the differences were not always great enough to be

significant, the final course grade point averages (GPA) in language arts and mathematics were

also higher for high school completers for each of the four years.

Data Analysis Procedures

Once the data set had been cleaned of cases with missing data or incorrectly coded

data, an initial factor analysis was conducted on the 23 different variables available in the Duval

dataset to determine whether variables fell into coherent categories (i.e., factors) as predictors

of completion. One of the reasons for beginning with this data reduction procedure was to

determine if one piece of data, such as attendance or mathematics standardized test scores,

had enough weight to serve as a valid predictor or would combinations of variables serve as

stronger predictors. The process of combining various data sets was repeated until a consistent

set of categories was identified. Once a coherent set of predictors was identified, linear

regression (stepwise) analyses were conducted on each identified factor in order to estimate the

strength of each factor.

The factor analysis yielded eight variables strong enough to be included in the regression

analysis. These are as follows:

1. End-of-sixth-grade standardized test scores in reading and mathematics were used.

Developmental scores on the Florida Comprehensive Achievement Test (FCAT) were

converted to one of five individual levels in each student record, and the five levels were

employed in statistical testing.

 Early Warning System  17 

2. Although the nature of the courses varied from student to student, end-of-sixth-grade

course grades in English and math were included. Letter grades were converted to

numeric equivalents for the purpose of statistical testing.

3. Students who had incurred serious infractions of the conduct code (major violations or

violations of zero tolerance policies) were identified and labeled dichotomously from

students who had not incurred such infractions.

4. Absences during Grade 6 were calculated and included. No distinctions were made

among reasons for absence: a simple total of days missed was initially used. Based on

indications from previous studies, in subsequent analyses, absences were divided into

two categories: less than 20% of the school days in 2001-2002 and more than 20% of

the school days that year.

5. Graduation status for each year up to and including the routine six years of high school

were aggregated and used as a single variable. Students were designated as

“Graduated with diploma,” “Received certification of completion,” or

“Transferred/withdrew” by the end of the 2007-2008 academic year.

6. Participation in dropout prevention programs during Grade 6 only was included. Duval

County maintains seven different prevention programs: educational alternatives, dropout

retrieval, disciplinary, alternatives to expulsion, teen pregnancy, Department of Juvenile

Justice, and neglected or delinquent. Students were labeled by the specific program in

which they participated or as non-participants in any of these programs.

7. Demographic variables such as race, gender, and date of birth were included. Dates of

birth were subsequently converted to age in months and then divided into three

categories: under expected age, normal expected age, and over expected age.

8. The need for services as English Language Learners was identified and coded

dichotomously (services/no services) into the dataset.

 Early Warning System  18 

Findings

Identifying Warning Flags

As we worked through the identification procedure, it became apparent that the most

valuable warning flags from an educational standpoint would be those that could be impacted by

the school and/ or community based educational programs. These are (1) measures of

academic learning, (2) school attendance, (3) student behavior, and (4) program placement.

Consequently, some demographic variables (e.g., race, gender, primary language) were

dropped from further analysis.

Table 2 shows the yield of each of the eight preliminary predictors. Seven of these

independent flags met our test of being within the influence of the educational program and

accounting for at least 10% of the students who failed to graduate, but one did not.

(1) Over-age for 6th grade

(2) Earned a low FCAT score (Level 1 or 2) in math

(3) Earned a low FCAT score (Level 1 or 2) in reading

(4) Failed a 6th grade math course

(5) Failed a 6th grade English course

(6) Was enrolled in a Dropout Prevention Program

(7) Absence from school more than 20% of the time

(8) Receiving a serious (Code 3 or 4) disciplinary referral (This flag only identified 6.7% of

the non-graduates. It is possible that this flag has some value but failed to meet the two

pronged test because of the low number of students in the overall population who met
 Early Warning System  19 

Table 2. Graduate and Non-graduate Percentages Identified by Preliminary Flags

Absence VCode Low Low Over- DPP Failed 6th Failed 6th
>20% FCAT FCAT age for grade grade
th
Math Reading 6 Grade math English
course course
Graduated 158 96 4099 3157 3470 341 1097 860
on time
Did not 409 207 2313 1878 2423 517 1016 946
graduate
Yield: % 12.2 6.7 68.8 55.9 72.1 15.4 30.2 28.1
of non-
graduates
flagged
N = 10,093 6th grade students/3,360 Non-graduates

One of the lessons learned from our qualitative analysis of interviews with students who

had dropped out of school was that often there was no single factor that prompted students to

leave school. As Balfanz has reported, “examining the occurrence of multiple flags and their

impact on students’ graduation chances provides additional insight into the process and impact

of student disengagement at the start of the middle grades” (Balfanz, 2007, p. 229). With this in

mind, in our next step we set out to determine how our predictors might be used in combinations

to better flag potential 6th grade non-graduates. Using the three strongest flags (i.e., being over

age, FCAT scores, and course grades) we reanalyzed the data to examine the power in

identifying non-graduating students when using two factors. These results are presented in

Table 3.

Table 3. Double Warning Flag Combinations and Counts for all Duval 6th Grade Students in
2001–2002 Who Failed to Graduate in 2007-2008

Risk Category Did not graduate n Valid %

FCAT Scores
 Early Warning System  20 

Low FCAT Math + Overage 1753 52.17

Low FCAT Math + Low FCAT Reading 1676 49.88

Low FCAT Reading + Overage 1548 46.07

Course Grades

Failed Math Course + Low FCAT Math 1638 48.75

Failed Math Course + Overage 1618 48.15

Failed English Course + Overage 1349 40.15

Failed English Course + Low FCAT Math 1340 39.88

Failed Math Course + Low FCAT Reading 1313 39.08

Failed English Course + Low FCAT Reading 1115 33.18

Failed Math Course + DPP 415 12.35

Failed English Course + DPP 378 11.25

Absence

Absence>20%+Older 381 11.34

Absence>20% + Failed English Course 358 10.65

Note: Yield < 10% for all other combinations

Attendance

Our analysis confirmed the information previously reported by Balfanz (2007) that

attending school less than 80% of the time increases the chance that students will not complete

high school. In our study population, approximately 11% of the 6th grade students who failed to

graduate were absent from school for more than 36 days. However, as reported in our previous

study of 9th grade students (Wilburn and Weaver, 2007), attendance was not as strongly

associated with failing to graduate as academic factors. While useful, in combination with other

factors, independently attendance does not provide a meaningful yield as a predictor variable.
 Early Warning System  21 

Standardized Test Scores

Consistent with findings from other studies, academic achievement identified the highest

number of 6th grade students who failed to graduate on time (Balfanz & Boccanfuso, 2007;

Balfanz 2007). However, unlike other studies, in this case standardized test scores were the

most reliable predictor of failure to graduate. When used independently, having a low (Level 1

and Level 2) Florida Comprehensive Assessment Test score in Reading or Math accounted for

approximately 62% percent of the 6th grade students who failed to graduate. While being over-

age identified the highest number of future non-graduates among the 6th grade study population,

in most cases the reason that a student is overage is because that have been retained for failing

to make an acceptable score on the FCAT. Consequently, being over-age may simply be a

proxy for poor FCAT performance over time. In addition, it takes more than one year to be

“overage” while FCAT score are available on a year to yearly basis.

Course Grades

In our work with the data provided by the school district for our descriptive study of 9th

grade graduation rates and this study, we found many inconsistencies in the course grade data

sets. In our efforts to identify early warning flags for 6th grade non-completers, we found the

data to be so inconsistent that we initially removed the course grade variable form our analysis.

However, because it is information that is readily available, we have included it in this analysis.

When taken independently, failing one 6th grade math or English course identified approximately

36.2% to 28.1%, respectively, of the students who eventually failed to complete high school.

Because of inconsistency in course grades from year to year, we believe that standardized test

scores provide the most valid and reliable predictor of high school completion for sixth grade

students, and we strongly recommend caution in using of course grades as an independent

warning flag for identifying students. This point may be illustrated by the fact that course grade
 Early Warning System  22 

is the only academic factor that is stronger when combined with other factors. For example

when failing a math course is combined with the low FCAT math score factor, the number of

students identified increases from a math course alone yield of 30.2% to combined yield of

48.7%. A similar gain is also realized when the two factors of a failed English course and a low

FCAT reading score are combined. For this reason we do not recommend using course grades

as an independent predictor.

Discipline

The discipline codes used in this study were those that represented the most serious

(i.e., Class 3 and 4) offences that a student could receive. For example, students’ behaviors that

involve physical assault, weapons and the use and/or possession of illegal drugs fall into these

categories. In the vast majority of cases, when a student receives this type of disciplinary

referral, he/she is automatically suspended from school and/or removed from their home school

and assigned to one of the District’s alternative schools for a period of 45 to 90 days.

Consequently, the number of Class 3 and 4 Code of Conduct violations occurring at the 6th

grade was not very large. As a result, this flag did not identify a large number of the 6th graders

who eventually failed to complete high school. From our work in tracking the 6th grade student

as well as our previous analysis of 9th grade students who failed to graduate, we know that the

number of Class 3 and 4 violations increase significantly as students get older and as more

students become over-age for their assigned grade. Consequently, the conduct violations from

subsequent years have more predictive power than those from Grade 6. While the yield for this

factor was relatively small (6.7%) as a single factor and no greater when considered in

combination with test scores or grades, we nevertheless consider this to be an important flag to

monitor.

Status Variables (Dropout Prevention Program/Over-age)

 Early Warning System  23 

Being either in the District’s Dropout Prevention Program (DPP) or over-age for the 6th

grade significantly reduces the probability that a student will successfully complete high school.

Our analysis indicates that of the 858 6th grade students enrolled in one or more of the District’s

Drop Out Prevention Programs, 341 (40%) graduated from high school on time while 60% failed

to complete high school. Obviously, for a student to be placed in one of these programs as early

as the 6th grade, he/she exhibits a number of our warning flags such as poor attendance, low

FCAT scores, and poor course grades. Consequently, the probability for completing high school

is low (i.e., 4 out of 10) for this student group. While the value of this factor as an early warning

flag for identifying large numbers of 6th grade students is limited, 15% of the 6th graders who

failed to complete high school were in the DPP. We believe it would be useful to examine the

program as to a better understanding of what initiatives may be successful in working with high

risk students.

On the other hand, being over-age for sixth grade appears to account for a large number

on non-high school completers. In our study population, 72% of the 6th grade students who

were over-age failed to complete high school. While this initially appears to be a strong flag for

identifying those who will not graduate from high school, as indicated in Balfanz’s 2007 study,

this is primarily because a high percentage of over-age students scored at Level 1 or 2 on the

FCAT, failed a 6th grade math or English course, and attended less than 80% of the time. The

few over-age students who did not exhibit any of the other flags tend to graduate at the same

rate as other 6th grade students. Consequently, being over-age is not the issue.

Final Selection of Identification Flags

Our final action in the identification of variables (i.e., predictive flags) was to narrow the

number of flags to those that independently or in combination with other flags provided the

strongest practical tool for early identification of 6th grade students who had the highest
 Early Warning System  24 

probability of not graduating from high school. As discussed above, some of the variables, such

as being over-age and/or enrolled in a dropout prevention program, were actually products of

other variables such as poor academic achievement or attendance. While a case could be

made for including the discipline code flag in the final selection as it seems to be such a reliable

predictor in later grades, it also was very duplicative in that over 90% of the 6th grade students

who had a Class 3 or 4 code of conduct violation also had one of the other flags such as poor

attendance or failing course grades. Therefore, we limited our final selection of flags to five

school related factors:

1. FCAT math scores

2. FCAT reading scores

3. Failing a 6th grade math class

4. Failing a 6th grade English class

5. Greater than 20% days absent from school

By using these five factors and eliminating duplicate records, we were able to identify

approximately 82% of the students who failed to graduate.

Validity of the Flags

In order to estimate the validity of our final five flags while controlling for the other flags,

we conducted a multivariate logistic regression to estimate the predictive power of each flag.

The analysis showed that, all else being equal, 6th grade students who missed more than 20%

of school attendance days were 6.07 times more likely not to graduate than students who

missed less than 20% of school attendance days (confidence interval (CI) = 5.03 to 7.32).

Students who failed English/Language Arts in the 6th grade were 2.86 times more likely not to
 Early Warning System  25 

graduate than students who passed Language Arts (CI = 2.15 to 3.17). Sixth grade students

who failed a math course were 2.38 times more likely not to graduate than those who passed

their math courses (CI = 2.15 to 2.62). Those 6th grade students who scored below 3 on the

FCAT reading test were 1.62 times more likely not to graduate than students who scored 3 or

above (CI 1.49 to 1.77). Finally, those 6th grade students who scored below 3 on the FCAT

math test were 1.71 times more likely not to graduate than those who scored 3 or above (CI =

1.56 – 1.88). Each flag was a statistically significant flag (p< .05) even after controlling for the

other flags. Our five flags, individually and in combination, identified 81.52% of the non-

graduates. Among the 10,093 total study population, 49% (4,975) graduated with none of the

five flags while 17% (1,706) had no flags but still did not graduate.

Limitations of the Study

It is important to recognize that, as with all investigations of this type, these numbers

provide a description of the 2001-2002 6th graders who had not graduated by the end of 2008.

In the course of our data analysis, we conducted both linear and logistic regressions with the

common wisdom being that a logistic regression is the better test to measure predictors on a

dichotomous variable, resulting in the graduation variable being yes/no on graduation. Though

none of the 4,975 students in our 10,093 member 6th grade cohort had any of the five flags and

82% of the non-graduates had one or more of the flags, it is not valid to state that the

combination of factors has a predictive power. The regression calculates mathematical

possibilities, not actual counts. This is evidenced by the fact that 1,706 of the 6th graders had

no flag but had not graduated by the end of the 2008 school year.

In generalizing our results to other populations, it is also important to note that we did not

control for race in the regressions that we ran. Since our 6th grade population was divided so

evenly between Black and White, there seemed no need to do so. It is common practice to only

control for such a factor if there are large imbalances inside the group. Consequently, caution
 Early Warning System  26 

should be exercised when applying our results to populations that have racial imbalances within

the study cohort.

Summary

Purpose

The primary purpose of this study was to identify specific academic and social factors

that may be used as “flags” to identify DCPS 6th grade students who are at high risk of not

completing high school. A secondary purpose was to understand how these factors may or may

not be aligned with those identified by Neal, Balfanz, and other researchers (Neal and Balfanz,

2006).

Data

The data used in this analysis were provided by the DCPS Department of Instructional

Research and Accountability. The data were gathered from the district’s data warehouse and

provided to the research team in the form of several Excel spreadsheets. The data included

selected demographic, academic, attendance, and disciplinary information from each student

enrolled in grades 6–12 during the 2001–2002 through the 2007–2008 school years.

Approximately 60,000–65,000 students were enrolled in grades 6-12 during each school year

with each grade’s enrollment averaging between 10,000–12,000 students.

Since the focus of this analysis was 6th grade students, we began by identifying all 6th grade

students for the 2001–2002 school year and then tracking that cohort of students through each

school year through the 12th grade (2007-2008). Ten thousand ninety-three (10,093) students
 Early Warning System  27 

remained for inclusion in the analysis after the elimination of withdrawals, transfers, and

incomplete data sets.

 Early Warning System  28 

Methods and Procedures

Once the data set had been cleaned of cases with missing data or incorrectly coded

data, an initial factor analysis was conducted to determine whether variables fell into coherent

categories (i.e., factors) as predictors of completion. No one piece of data, such as attendance

or mathematics standardized test scores, has enough weight to serve as a valid predictor.

Therefore, this process of combining various data sets was repeated until a consistent set of

categories was identified. Once a coherent set of predictors was identified, linear regression

(stepwise) and logistic regression analyses were conducted on each identified factor in order to

estimate the strength of each factor.

Four questions guided our 6th grade analysis:

1. Are significant numbers of students showing the signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school?

2. Can we identify a set of indicators that flag sixth graders who have high odds of failing

to complete high school?

3. From this set of indicators, can we identify a practical set of indicators routinely collected

and reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation?

4. Can we identify indicators within that set that may be influenced by school and

community programs?

Results

A significant number of 6th grade students displayed the signs of disengagement, such

as poor academic achievement and excessive absences, previously identified in our study of 9th
 Early Warning System  29 

grade students and in studies conducted by other researchers. Five factors proved to be of

greatest value in identifying members of the 2001-2002 6th grade cohort who did not graduate

on-time from high school in 2007-2008. These factors were:

ƒ Standardized Math Test Scores: 69% of the 6th grade students who failed to graduate

on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ Standardized Reading Test Scores: 56% of the 6th grade students who failed to graduate

on-time scored below Level 3 on the Florida Comprehensive Assessment Test.

ƒ Failing Course Grades in Mathematics: 30.2% of the 6th grade students who failed to

graduate on-time failed a math course.

ƒ Failing Course Grades in English/Language Arts: 28.1% of the 6th grade students who

failed to graduate on-time failed an English/Language Arts course.

ƒ Excessive Absences: 12.2% of the 6th grade students who failed to graduate on-time

missed more the 20% of the scheduled school days.

The number of Class 3 and 4 Code of Conduct violations occurring at the 6th grade among

the study population was not very large (3%). As a result, this flag only identified 6.7% of the 6th

grader who failed to graduate on-time. However, our tracking of students indicated that the

number of Class 3 and 4 violations increase significantly as students get older and as more

students become over-age for their assigned grade. Consequently, the conduct violations from

subsequent years have more predictive power than those from Grade 6. While the yield for this

factor was relatively small, as a single factor and no greater when considered in combination

with test scores or grades, we nevertheless consider this to be an important flag to monitor.

Results from our study indicate:

1. Significant numbers of 6th grade students show signs of disengagement previously

identified in our examination of 9th grade students who fail to complete high school.
 Early Warning System  30 

2. We can, with a reasonable degree of validity, identify a set of indicators that flag sixth

graders who have high odds of failing to complete high school

3. From this set of flags, we can identify a practical set of indicators routinely collected and

reported at the individual level that flag sixth graders who have odds of failing to

continue in school until graduation

4. These indicators are factors that may be impacted by school and community programs.

In conclusion, as we have learned from our previous studies of first-time 9th grade

students and focus groups with students who had previously dropped out of school, students do

not graduate from high school for a multitude of reasons; and many of those reasons are not

linked directly to academic achievement, attendance, and behavior at school. Even though the

issues is complex and no one single factor or group of factors can account for every child’s

decision to leave school, we believe that it would be unconscionable to ignore the early warning

flags readily available to even the most casual observer. 

 Early Warning System  31 

References

Balfanz, R., Hertzog, L., Neild, R. C. and Mac Iver, D. J. (2007). Preventing student

disengagement and keeping students on the graduation path in urban middle-grades

schools: Early identification and effective interventions. Educational Psychologist. 42(4),

223-235.

Balfanz, R. (2006). Unfulfilled promise: The dimensions and characteristics of Philadelphia’s

dropout crisis, 2000 – 2005. Philadelphia, PA: Philadelphia Youth Transition’s

Collaborative.

Roush, C., Wilburn, K. T. and Weaver, D.M. (January, 2009). Learning to finish focus group

data: qualitative analysis report. Jacksonville, FL: The Community Foundation.

StatSoft, Inc. (2007). Electronic Statistics Textbook. Tulsa, OK: StatSoft. WEB:

http://www.statsoft.com/textbook/stathome.html.

Wilburn, K. T. and Weaver, D. M. (November, 2008) Where Have All the 9th Graders Gone? A

Descriptive Study of Three First-Time 9th Grade Student Cohorts. Jacksonville, Florida:

The Community Foundation.

 Early Warning System  32 

Appendix

Demographic Characteristics of the Study Population

Group Variable Completers (%) Non-completers (%)

Gender Female 51.8 46.8

Male 48.2 53.2

Race/Ethic Group Asian American 3.6 2.6

Black 45.2 43.4

Hispanic 3.9 4.0

American Indian 0.1 0.2

Multiracial 1.0 1.1

White 46.2 48.8

Primary Language Non-English Speaking 9.2 7.0

English Speaking 90.8 93.0

Socioeconomic Status Free Lunch 23.3 20.5

(School Lunch
Program) Public Assistance 5.1 6.7

Reduced Lunch 9.3 6.7

Not Eligible 2.7 1.5

No Information 59.6 64.5