Vous êtes sur la page 1sur 44

See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/268075671

The Validity of the Multi-Informant Approach


to Assessing Child and Adolescent Mental
Health

Article in Psychological Bulletin July 2015


DOI: 10.1037/a0038498

CITATIONS READS

53 783

7 authors, including:

Andres De Los Reyes Tara M. Augenstein


University of Maryland, College Park University of Maryland, College Park
75 PUBLICATIONS 2,628 CITATIONS 11 PUBLICATIONS 88 CITATIONS

SEE PROFILE SEE PROFILE

Sarah A. Thomas Deborah A G Drabick


Alpert Medical School - Brown University Temple University
12 PUBLICATIONS 236 CITATIONS 49 PUBLICATIONS 901 CITATIONS

SEE PROFILE SEE PROFILE

Available from: Andres De Los Reyes


Retrieved on: 15 October 2016
Psychological Bulletin 2015 American Psychological Association
2015, Vol. 141, No. 4, 858 900 0033-2909/15/$12.00 http://dx.doi.org/10.1037/a0038498

The Validity of the Multi-Informant Approach to Assessing Child


and Adolescent Mental Health

Andres De Los Reyes and Tara M. Augenstein Mo Wang


University of Maryland University of Florida

Sarah A. Thomas Deborah A. G. Drabick, Darcy E. Burgers,


University of Maryland and Jill Rabinowitz
Temple University
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Child and adolescent patients may display mental health concerns within some contexts and not others
(e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may
assist mental health professionals in tailoring treatments to patients needs. Consequently, clinical
assessments often include reports from multiple informants who vary in the contexts in which they
observe patients behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that infor-
mants reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspon-
dence among reports as indicating that patients display concerns in some contexts and not others? We
meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspon-
dence estimates, and observed low-to-moderate correspondence (mean internalizing: r .25; mean
externalizing: r .30; mean overall: r .28). Informant pair, mental health domain, and measurement
method moderated magnitudes of correspondence. These robust findings have informed the development
of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predic-
tions about the incremental and construct validity of these assessments. In turn, we critically evaluated
research on the incremental and construct validity of the multi-informant approach to clinical child and
adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and
provide recommendations for best practices in using and interpreting multi-informant assessments in
clinical work and research. This article has important implications for developing personalized ap-
proaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target
the specific contexts where patients display concerns.

Keywords: construct validity, incremental validity, informant discrepancies, multiple informants,


Operations Triad Model

Child and adolescent mental health patients lead complex lives Luthar, Cicchetti, & Becker, 2000; Sanislow et al., 2010). How-
(i.e., collectively referred to as children unless otherwise spec- ever, not all contexts elicit mental health concerns to the same
ified). Indeed, mental health concerns arise out of an interplay degree (e.g., Carey, 1998; Kazdin & Kagan, 1994; Mischel &
among biological, psychological, and sociocultural factors that Shoda, 1995). Therefore, patients may display concerns within
pose risk for, or offer protection against, developing maladaptive some contexts, such as home and school, but not others, such as
reactions to environmental or social contexts (e.g., Cicchetti, 1984; within peer interactions. In fact, these contextual variations in
displays of mental health concerns occur within a variety of mental
health domains including social anxiety, attention and hyperactiv-
This article was published Online First April 27, 2015. ity, and conduct problems (e.g., Bgels et al., 2010; Dirks, De Los
Andres De Los Reyes and Tara M. Augenstein, Comprehensive Assess- Reyes, Briggs-Gowan, Cella, & Wakschlag, 2012; Drabick,
ment and Intervention Program, Department of Psychology, University of Gadow, & Loney, 2007, 2008; Kraemer et al., 2003). Thus, iden-
Maryland; Mo Wang, Department of Management, University of Florida; tifying the specific contexts in which patients display concerns
Sarah A. Thomas, Comprehensive Assessment and Intervention Program, may facilitate treatment planning and boost treatment efficacy
Department of Psychology, University of Maryland; Deborah A. G. Drab- (e.g., De Los Reyes, 2013; National Institute of Mental Health
ick, Darcy E. Burgers, and Jill Rabinowitz, Department of Psychology, [NIMH], 2008).
Temple University. The most prevalent strategy for assessing contextual variations
This work was partially supported by a predoctoral National Research
in mental health is the multi-informant assessment approach
Service Award to Sarah A. Thomas from the National Institute on Drug
Abuse (F31-DA033913).
(Kraemer et al., 2003). Specifically, this approach involves taking
Correspondence concerning this article should be addressed to Andres reports from informants who share close relationships with the
De Los Reyes, Comprehensive Assessment and Intervention Program, patients about whom they are providing reports, or at minimum
Department of Psychology, University of Maryland, Biology/Psychology spend a significant amount of time observing patients behavior
Building, Room 3123H, College Park, MD 20742. E-mail: adlr@umd.edu (Achenbach, 2006). By considering reports from informants who
858
VALIDITY OF MULTI-INFORMANT ASSESSMENT 859

vary among each other in the specific contexts in which they dence in reports of childrens mental health published over the last
observe patients behavior (e.g., home vs. school vs. peer interac- 25 years (1989 2014). Second, we use this updated quantitative
tions), mental health professionals may gain an understanding as to review as a backdrop for discussing recent work on the conceptual
how consistently or inconsistently patients display concerns across rationale for conducting multi-informant assessments (De Los
contexts (Dirks et al., 2012). For child patients, these informants Reyes, Thomas, Goodman, & Kundey, 2013). We expand upon
most often include parents, teachers, and patients themselves this conceptual work by developing specific predictions about the
(Hunsley & Mash, 2007). Further, trained raters might also com- incremental and construct validity of the multi-informant ap-
plete reports, such as clinical interviewers and independent ob- proach. Third, we review research on the incremental validity of
servers of patients behavior on standardized clinical tasks (e.g., multi-informant assessments, or work examining whether a multi-
structured social interactions) or unstandardized home or school informant approach yields reports that, relative to one another,
observations (Groth-Marnat, 2009). incrementally contribute information in the prediction of relevant
Clinicians may use informants reports to make decisions about criterion variables (e.g., diagnostic status or treatment response;
mental health care, such as assigning diagnoses and planning see Dawes, 1999; Garb, 2003; Hunsley & Meyer, 2003). Fourth,
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

treatment (e.g., Hawley & Weisz, 2003). Researchers may also use we summarize recent work on the construct validity of multi-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

these reports to draw conclusions from empirical work on such informant assessments. By construct validity, we mean research
topics as identifying treatments that successfully ameliorate mental examining whether it is valid to interpret patterns of convergence
health concerns (i.e., evidence-based treatments; Weisz, Doss, & and divergence among multiple informants reports as accurate
Hawley, 2005). In both practice and research settings, collecting reflections of contextual variations in childrens mental health
multiple informants reports generates a great deal of information concerns (e.g., Borsboom, Mellenbergh, & van Heerden, 2004;
about patients concerns. However, the individual reports often Campbell & Fiske, 1959). Lastly, we synthesize work on multi-
yield inconsistent conclusions (i.e., informant discrepancies; informant assessments to provide recommendations for both future
Achenbach, 2006; De Los Reyes & Kazdin, 2004, 2005, 2006; research and best practices for using and interpreting multi-
Goodman, De Los Reyes, & Bradshaw, 2010).1 For instance, a informant assessments administered in clinic settings.
female adolescent patient in a pretreatment assessment may be
identified as experiencing low positive mood based on a parent
or teacher report whereas the adolescent self-reports her mood as Correspondence in Multi-Informant Clinical
elevated. Child Assessments
Informant discrepancies often create considerable uncertainties One of the most normative observations in assessments of child
in delivering services to patients and drawing conclusions from patients is that of low correspondence between informants reports
research (De Los Reyes, Kundey, & Wang, 2011). A key reason about patients mental health (Achenbach, 2011). Over five de-
for these uncertainties originates from the near-exclusive focus in cades ago, Lapouse and Monk (1958) were the first to report about
mental health research on whether informant discrepancies reflect this phenomenon. Their primary aim was to estimate the preva-
measurement error or reporting biases (e.g., De Los Reyes, 2011; lence of childrens mental health concerns in an epidemiological
Richters, 1992). Consequently, what remains unclear is whether study of 482 6- to 12-year-old children representative of the
multi-informant approaches to assessment validly capture contex- community from which they were sampled (i.e., Buffalo, New
tual variations in displays of patients mental health concerns, and York). To estimate prevalence, the researchers relied on structured
which informants ought to be included in mental health assess- clinical interviews administered to mothers about a single child
ments. As we explain below, these decisions might involve con- from the household. In this study, the researchers also wanted to
sidering data from multiple assessment literatures, chief among address what they termed the problem of validity with respect to
them research on multi-informant clinical child assessments and mothers interview responses. Thus, the researchers conducted a
their (a) correspondence, (b) incremental validity, and (c) construct secondary study using a separate convenience sample of 193
validity. Prior work has addressed each of these issues as they children ages 8 12 and their mothers, recruited from local outpa-
relate to clinical child assessment broadly (e.g., Achenbach, Mc- tient hospitals and pediatricians offices. In this sample, research-
Conaughy, & Howell, 1987; Johnston & Murray, 2003; Mash & ers conducted structured clinical interviews with the mothers and
Hunsley, 2005), and within assessments in other clinical literatures the identified child separately and simultaneously, using two dif-
(e.g., Achenbach, 2006; Campbell & Fiske, 1959; Dawes, 1999; ferent interviewers. To assess correspondence between mother and
Garb, 2003; Hunsley & Mash, 2005). However, with regard to
child reports, researchers examined responses on a subset of be-
multi-informant clinical child assessments, these literatures have
advanced largely in isolation of one another.
1
We use the term informant discrepancies to generally denote studies of
differences and similarities among informants reports. Yet, researchers
The Present Review have used other terms to characterize this research (e.g., informant agree-
ment and informant correspondence). Importantly, these terms are not
The purpose of this article is to review and synthesize research interchangeable and, where possible, we use different terms to denote
on using and interpreting multiple informants reports in clinical disparate lines of research. That is, these terms distinguish examinations of
child assessments. We expand the literature on this topic in five divergence between reports (discrepancies) from examinations of conver-
ways. First, as an update to the seminal meta-analysis on cross- gence between reports (agreement and correspondence), and researchers
often use disparate statistical techniques within and between studies (e.g.,
informant correspondence of clinical child assessments carried out statistical interactions between reports to study divergence vs. correlations
by Achenbach, McConaughy, and Howell (1987), we conduct a between reports to study convergence; De Los Reyes, Salas, Menzer, &
quantitative review of 341 studies of cross-informant correspon- Daruwala, 2013).
860 DE LOS REYES ET AL.

havioral domains assessed in the interview, including fears, night- anxiety symptoms (Rapee, Barrett, Dadds, & Evans, 1994), and
mares, bed wetting, restlessness, and repetitive behaviors (e.g., disruptive behavior assessments taken from representative nonclinic
tics, thumb sucking, skin picking). Using percent agreement to samples (Offord et al., 1996). Further, the Lapouse and Monk (1958)
assess correspondence, the authors reported agreement between finding of greater endorsements among childrens self-reports relative
46% (amount of food intake) and 84% (bed wetting). Broadly, the to mothers reports was corroborated by comparisons between mother
authors observed greater correspondence between reports of rela- and child behavioral checklist reports from 25 countries (Rescorla
tively more observable behaviors (e.g., bed wetting, stuttering, et al., 2013). Since the Achenbach et al. (1987) review, researchers
thumb sucking) than relatively less observable behaviors (e.g., have expanded the range of domains examined for cross-informant
fears and worries, nightmares, restlessness). Further, with two correspondence (e.g., anxiety, conduct problems, hyperactivity,
exceptions (bed wetting and overactivity), lack of correspondence mood; De Los Reyes, 2011). Further, studies conducted after the
was primarily due to children self-endorsing behaviors that the Achenbach et al. (1987) review collectively examined to a greater
mother did not endorse about the child. extent variations in cross-informant correspondence across multiple
Highlighting the robust nature of cross-informant correspondence (a) measurement and scaling methods (e.g., behavioral checklists,
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

patterns, in their seminal meta-analysis of 119 studies of cross- clinical interviews, symptom rating scales); (b) patients developmen-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

informant correspondence in reports of childrens mental health, tal periods; and (c) informant pairs (e.g., greater focus on childrens
Achenbach et al. (1987) reported, similar to Lapouse and Monk self-reports for assessing internalizing concerns; e.g., Achenbach,
(1958), that correspondence levels exhibited a range from low-to- 2006; De Los Reyes & Kazdin, 2005).
moderate in magnitude (i.e., rs ranging from .20 to .60). Importantly, In light of the continued attention to estimating cross-informant
Achenbach et al. (1987) noted that the low correlations they observed correspondence, have the findings of Achenbach et al. (1987)
among informants reports of childrens mental health did not neces- stood the test of time? To address this question, we conducted a
sarily indicate that such reports carried poor psychometric qualities, as meta-analysis of the last 25 years of cross-informant correspon-
they also observed satisfactory testretest reliability estimates for the dence studies for assessments of childrens internalizing and ex-
informants reports they examined. In fact, levels of correspondence ternalizing mental health concerns. We focused on these domains
systematically varied as a function of three key factors. First, pairs of because they encompass the most commonly assessed and treated
informants who observed children in the same context (e.g., pairs of childhood concerns (i.e., anxiety, attention and hyperactivity, con-
parents or pairs of teachers) tended to exhibit greater levels of corre- duct, mood; Hunsley & Mash, 2007; Weisz et al., 2005). Further,
spondence than pairs of informants who observed children in different we took two approaches to identifying studies for our review. First,
contexts (e.g., parent and teacher). Similarly, a second key finding we sampled studies included in meta-analyses of cross-informant
was that greater levels of cross-informant correspondence occurred correspondence published since Achenbach et al. (1987), which is
when informants provided reports about younger children relative to a variant of the second-order meta-analytic approach (e.g., Butler,
reports about older children. Younger children may be relatively more Chapman, Forman, & Beck, 2006; Cuijpers & Dekker, 2005;
constrained than older children in the contexts in which they display Lipsey & Wilson, 1993; Mller & Jennions, 2002; Peterson, 2001;
mental health concerns (e.g., De Los Reyes & Kazdin, 2005; Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). Sec-
Smetana, 2008). Additionally, Achenbach et al. (1987) conducted ond, we searched for studies published in the years (i.e., 2000
their meta-analysis at a time when cross-informant correspondence 2014) following the meta-analyses sampled in our review. Impor-
studies predominantly focused on comparing adults reports of chil- tantly, Achenbach et al. (1987) reviewed 119 studies published
drens concerns (e.g., parent vs. teacher; mother vs. father; see Table over roughly a quarter-century (i.e., 1960 1986). Similarly, as we
2 of Achenbach et al., 1987). Thus, this age effect might have also explain below, we reviewed 341 studies published in the most
reflected the idea that less variation among informants in contexts of recent quarter-century (i.e., 1989 2014). Thus, we were well-
observation pointed to greater correspondence between informants positioned to assess whether more recent work replicated the
reports, particularly for adult informants reports of childrens con- findings of Achenbach et al. (1987). To this end, we focused our
cerns. Third, Achenbach et al. (1987) observed larger correspondence review on studies that examined correspondence among parent,
levels between informants reports of childrens externalizing (e.g., teacher, and child reports of childrens mental health. We focused
aggression and hyperactivity concerns) versus internalizing (e.g., anx- on these three informants because these are the reporters on which
iety and mood) concerns. This third finding may have reflected mental health professionals most commonly rely when adminis-
greater correspondence between reports of directly observable con- tering and interpreting the outcomes of clinical child assessments
cerns, relative to concerns that are internally experienced by the child (e.g., Hunsley & Mash, 2007; Kraemer et al., 2003).
and thus, relatively less observable in nature.
Method
Meta-Analysis of the Last Quarter-Century
of Cross-Informant Correspondence Studies Literature Review
Since the seminal work of Lapouse and Monk (1958) and We identified meta-analyses and empirical studies published
Achenbach et al. (1987), researchers have conducted many addi- since Achenbach et al. (1987). We conducted two searches. First,
tional cross-informant correspondence studies. As in earlier work, to identify relevant meta-analyses we searched via Google Scholar
more recent studies find that low-to-moderate levels of cross- of all peer-reviewed scholarly work citing the Achenbach et al.
informant correspondence characterize such varied assessment set- (1987) review (N 3,978 citations; search conducted March 2,
tings and mental health domains as inpatients depressive mood 2014). We searched within these cited articles using the search
symptoms (Frank, Van Egeren, Fortier, & Chase, 2000), outpatients terms informant and quantitative review. We augmented this
VALIDITY OF MULTI-INFORMANT ASSESSMENT 861

search with an additional Google Scholar search of cited articles from .15.40 depending on informant pair), as this information did
using the terms meta-analysis OR quantitative review OR sys- not allow us to code moderator variables. Similarly, we excluded
tematic review, and conducted this same literature search using studies that only reported correspondence metrics as an average or
the Web of Science search engine. Combined across searches, we range across informants reports of different mental health do-
identified 1,799 articles. This search yielded an initial set of four mains. Finally, studies needed to report cross-informant correspon-
quantitative reviews (i.e., Crick et al., 1998; Duhig, Renk, Epstein, dence estimates for measures of mental health on the internalizing
& Phares, 2000; Meyer et al., 2001; Renk & Phares, 2004), to (e.g., anxiety and mood) and/or externalizing (e.g., aggression and
which we applied the inclusion and exclusion criteria described hyperactivity) spectrum. We excluded studies focused on related
below. As an additional check, we conducted Google Scholar constructs, namely studies on risk and protective factors of mental
searches using the above-listed terms of all work citing these four health (e.g., emotion regulation, parenting practices, personality
quantitative reviews, yielding an additional set of 858 articles. This traits, resiliency, self-esteem).
search yielded no additional quantitative reviews. For meta-analyses identified in our search, employing inclusion
Second, to identify empirical articles of cross-informant corre- criteria led to our excluding the Crick et al. (1998) and Renk and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

spondence conducted in the years (i.e., between 2000 and 2014) Phares (2004) reviews. We excluded Crick et al. (1998) because it
This document is copyrighted by the American Psychological Association or one of its allied publishers.

following the meta-analyses included in our review, we searched did not list the individual studies the authors used to calculate
via Google Scholar for all peer-reviewed scholarly work citing the cross-informant correspondence estimates. We excluded the Renk
Achenbach et al. (1987) review between 2000 and 2014 using the and Phares (2004) review because it focused on correspondence
search terms internalizing symptoms/problems/difficulties OR between reports on a construct (i.e., social competence) that fell
externalizing symptoms/problems/difficulties (N 1,440 cita- outside of the spectra of internalizing and externalizing mental
tions; search conducted September 9, 2014). Additionally, we health concerns. Thus, we identified relevant studies via quantita-
searched the reference lists of narrative reviews of cross-informant tive review based on two meta-analyses that met our inclusion
correspondence research published since 2000. criteria: Duhig et al. (2000) and Meyer et al. (2001). Across studies
identified via these meta-analyses and our search of individual
studies published between 2000 and 2014, collectively we coded
Inclusion and Exclusion Criteria
effect sizes on a final sample of 1218 data points taken from 341
To be included in our quantitative review, meta-analyses and studies published between 1989 and 2014. A complete study list can
studies must have (a) focused on informants reports of children at be retrieved online at https://sites.google.com/site/caipumaryland/
or under the age of 18 years; (b) examined correspondence be- Home/people/director.
tween informants reports of childrens mental health concerns
(i.e., internalizing and/or externalizing concerns); (c) examined Data Extraction and Covariates Coded
correspondence between reports completed by pairs of parents,
From Each Study
teachers, and/or children (i.e., motherfather, parent child,
parentteacher, teacher child); and (d) been published in English. Two doctoral graduate students served as coders for our quan-
In addition to these criteria, included meta-analyses must have titative review, and received coding training by the first author and
provided a list of individual studies used to calculate metrics of fifth author through discussion and practice coding. After the two
cross-informant correspondence. We employed this criterion to coders completed study coding, the fifth author served as an
ensure that articles examined in our review were published after independent assessor on 50% of the studies coded. This indepen-
1987 (i.e., not included in the original review by Achenbach et al., dent assessment resulted in 100% interrater agreement on effect
1987). Further, knowledge of individual studies within each of the sizes coded (i.e., effect size magnitude and metrics), and Cohens
reviews allowed us to identify any articles examined by more than Kappa coefficients of 1.0 in terms of interrater agreement on
one review and multiple articles that examined the same sample or coding of each of the covariates described below (i.e., child age,
cohort of children. This step allowed us to ensure that effects informant pair, measurement method, mental health domain).
observed did not occur because reviews examined the same studies Outcomes representing mean estimates of cross-informant cor-
or sample(s) across studies. respondence were presented as Pearson r correlations, Cohens
Among individual empirical studies identified either via the Kappa coefficients, Cohens d, Hedges g, means and standard
reference lists of meta-analyses or our additional search for indi- deviations of informants reports, or odds ratios that we converted
vidual studies published between 2000 and 2014, to be included in to r for the meta-analyses. In addition to mean estimates of
our review studies must have provided sufficient data to code cross-informant correspondence, we also coded for four covari-
estimates of cross-informant correspondence. Specifically, we re- ates, consistent with effects observed by Achenbach et al. (1987):
quired studies to report between-informant correspondence metrics (a) mental health domain (internalizing vs. externalizing); (b)
(e.g., Pearson correlations for dimensional measures or Kappa informant pair (motherfather, parent child, parentteacher,
coefficients for categorical measures) on measures completed by teacher child); (c) child age (younger [10 years and younger] vs.
one or more informant pairs (e.g., motherfather, parent child, older [11 years and older]); and (d) measurement method (cate-
parentteacher, teacher child). Measures completed by these in- gorical vs. dimensional). We coded studies based on measurement
formants must have assessed the same construct at the same time method because recent work indicates that, relative to categorical
point (e.g., measures about internalizing problems completed by scaling, dimensional scaling results in measures that evidence
parent and child when the child was 10-years-old). We excluded greater estimates of reliability and validity (Markon, Chmielewski,
studies that only reported correspondence metrics as an average or & Miller, 2011). Thus, we would expect greater levels of cross-
range across multiple informant pairs (e.g., correlations ranged informant correspondence for reports taken using dimensional
862 DE LOS REYES ET AL.

scales (e.g., behavioral checklists), relative to categorical scales the data points and reflects real differences among studies (Bo-
(e.g., diagnostic interview endorsements). renstein et al., 2009; Higgins, Thompson, Deeks, & Altman, 2003).
To explore the impact of the categorical variables or covariates on
Data-Analytic Strategy this observed variability among studies, we conducted ancillary,
subgroup analyses to determine effect sizes (rs) for each level of
As described below, we observed significant differences be- the categorical variable (e.g., separately for studies that used
tween cross-informant correspondence estimates for reports of categorical vs. dimensional assessment approaches or examined
internalizing concerns versus externalizing concerns (see Table 1). younger vs. older children). For these analyses, we calculated rs
Thus, we performed two primary meta-analyses, one for internal- and associated p values for each level of the categorical variable
izing concerns and one for externalizing concerns, using published (Borenstein et al., 2009). We then examined the Q statistic,
or calculated rs to estimate the precision of the mean for all which is based on the weighted sum of squares for each level of
included studies, using Comprehensive Meta-analysis Version 2 the covariate and thus provides an index of dispersion (Boren-
(Biostat, Englewood, NJ, n.d.) software. Because the studies in- stein et al., 2009), to determine whether the magnitude of
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

cluded in the meta-analyses varied in methodology and design, we correspondence differed between levels of the covariates.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

calculated a random-effects model. In addition, for some studies


and samples, we observed multiple effect sizes that varied across
Results and Discussion
our moderator variables (e.g., multiple effect sizes reported for
different informant pairs, categorical vs. dimensional approaches).
We accounted for this nesting in the data by calculating (a) effect Overall Effect Sizes
sizes for each cohort or sample, and then (b) an overall effect size. Combined across 341 studies and 1,218 data points, we
Specifically, we computed effect sizes and variances for each observed an overall cross-informant correlation of .28 (95% CI
cohort. Next, we calculated a weighted mean for each study cohort [.22, .33]; p .001). These estimates are virtually identical to
(i.e., effect sizes drawn from the same sample). We based these the mean cross-informant correlations reported by Achenbach
weights on the inverse of the total variance associated with each of et al. (1987). In Table 1, we report cross-informant correlations
the data points. Lastly, we computed a weighted mean of the effect for reports of childrens internalizing and externalizing con-
sizes for each of the study cohorts, which were based on both cerns. Consistent with Achenbach et al. (1987), we observed
within-cohort error and between-cohort variance, to produce an low-to-moderate and statistically significant magnitudes of
overall summary effect (Borenstein, Hedges, Higgins, & Roth- cross-informant correspondence for both domains. As seen in
stein, 2009). This method allowed us to capitalize on the multiple Table 1, we observed nonoverlapping 95% CI for cross-
sources of variance present both within and across studies, rather informant correspondence estimates for reports of childrens
than alternative methods that would have resulted in lost sources of internalizing concerns versus externalizing concerns. Overall,
variance (e.g., taking a simple average of correspondence esti- informant correspondence rates tended to be larger for reports
mates for a study that included multiple informant pairs). of childrens externalizing concerns, relative to reports of chil-
Additionally, we addressed the issue of statistical stability in drens internalizing concerns. Thus, below we report effects of
two ways. First, we individually excluded each study from the covariates on levels of cross-informant correspondence sepa-
analysis and recalculated the pooled r and 95% confidence interval rately for reports of childrens internalizing concerns and ex-
(CI). If an individual study contributed heavily to the pooled r, we ternalizing concerns.
would observe a change in the magnitude or significance of the
pooled r. Second, given the possibility of publication bias (i.e.,
Heterogeneity in Effect Sizes and Stability of Results
significant findings are more likely to be published), we calculated
Orwins Fail-safe N (Borenstein et al., 2009), which provides an Consistent with Achenbach et al. (1987), we observed an
index of the number of data points necessary to make the overall overall large variance in results for internalizing and external-
effect size trivial (i.e., defined as an r of .10). izing concerns, respectively (I2 99.25 and 99.40), suggesting
Following determination of the combined effect size across heterogeneity among studies in effect sizes. That is, the per-
studies, we considered heterogeneity or observed variance across centage of total variability that is attributable to heterogeneity
studies. Specifically, we calculated tau, a metric that is sensitive to among the data points included in the meta-analysis is approxi-
the unit of measurement and acts as the standard deviation of the mately 99% for reports of internalizing and externalizing concerns.
summary effect, and I2, which provides an index of the proportion Further, removal of any individual study from the analysis did not
of observed variability that is attributable to heterogeneity among affect relations between magnitudes of cross-informant correspon-

Table 1
Meta-Analytic Findings Regarding Cross-Informant Correspondence in Reports of Childrens Mental Health

Mean cross-informant correlations


Symptom type # of data points Overall effect size (95% CI) P-C P-T T-C M-F

Internalizing 697 .25 [.24, .25] (p .001) .26 (n 372) .21 (n 155) .20 (n 62) .48 (n 109)
Externalizing 521 .30 [.30, .31] (p .001) .32 (n 198) .28 (n 162) .29 (n 53) .58 (n 109)
Note. Correlations are in the Pearson r metric. P-C ParentChild; P-T ParentTeacher; T-C TeacherChild; M-F MotherFather.
VALIDITY OF MULTI-INFORMANT ASSESSMENT 863

dence and covariates (rs with each individual study removed practices for clinical child assessments. Specifically, the Achen-
ranged from .27 to .28 for reports of internalizing concerns; r with bach et al. (1987) review consisted of a relatively small proportion
each individual study removed was .36 for reports of externalizing of studies comparing parent and teacher reports to childrens
concerns; all ps .001). Using the respective effect sizes for these self-reports (see Table 2 of Achenbach et al., 1987). Since 1987, a
data points, defining a trivial effect size as an r of .10, and a proliferation of evidence-based assessment research focused on
threshold of p .05, Orwins Fail-safe N was 1,875 for reports of understanding childrens perspectives on their own mental health
internalizing concerns and 2,292 for reports of externalizing con- concerns, and this increased focus resulted in an increased number
cerns. This indicates that one would have to include in the meta- of options available for taking self-reports of mental health, par-
analysis over 1,800 data points of reports of childrens internaliz- ticularly for older children and adolescents (for reviews, see Klein,
ing concerns and well over 2,000 data points of reports of Dougherty, & Olino, 2005; McMahon & Frick, 2005; Silverman &
childrens externalizing concerns with a mean r of .00 before the Ollendick, 2005). Consequently, we based our cross-informant
cumulative effects would become trivial. correspondence estimates for reports of childrens mental health
concerns on a sample of studies in which over 50% compared
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

childrens self-reports with reports from other informants (see


Effects of Covariates
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Table 1). For some mental health domains and contexts (e.g.,
To explore whether the covariates could account for some of the worry and anxiety displayed within peer interactions; covert de-
variability among the effect sizes for each of the data points (697 linquent behaviors displayed within peer interactions), children
for internalizing, 521 for externalizing), we conducted ancillary may be in a unique position to observe displays of these concerns,
analyses where we calculated separate r and p values for each level relative to parents and teachers (e.g., Comer & Kendall, 2004;
of the categorical covariates. Because the data points within stud- McMahon & Frick, 2005). Thus, it may be that the null effects of
ies and cohorts often differed in these covariates (e.g., child age, child age on magnitudes of cross-informant correspondence that
measurement method), these supplementary analyses considered we observed in fact reflected a greater use since Achenbach et al.
the data points independently. Beyond the effects of mental health (1987) of self-reports to assess childrens mental health concerns.
domain reported previously, we observed significant effects for
two other covariates. First, pairs of mother and father informants
(i.e., informants who observe child in the same setting) yielded Conceptual Foundations of Multi-Informant
larger magnitudes of cross-informant correspondence, relative to Mental Health Assessments
all other informant pairs (i.e., parent child, parentteacher,
teacher child). We observed this effect for both reports of inter- Operations Triad Model
nalizing concerns (rs: .48 vs. .24; Q 72.42, p .001) and
externalizing concerns (rs: .58 vs. .30; Q 96.26; p .001). Collectively, our meta-analytic findings indicated that clinical
Second, informants completing reports on a dimensional scale child assessments tend to yield low-to-moderate levels of cross-
tended to yield higher magnitudes of correspondence, relative to informant correspondence, with higher levels of correspondence
informants completing reports on a categorical scale. We observed occurring when informants complete reports about mental health
this effect of measurement method for both reports of internalizing concerns that both informants either have relatively greater oppor-
concerns (rs: .29 vs. .06; Q 38.54, p .001) and externalizing tunities to observe (e.g., externalizing vs. internalizing) or observe
concerns (rs: .37 vs. .06; Q 30.86, p .001). Further, we within the same context (e.g., motherfather vs. parentteacher).
observed nonsignificant effects of child age on levels of corre- The main findings from the Achenbach et al. (1987) review, as
spondence for reports about younger children (i.e., 10 years and well as the stability of these findings in research published since
younger) versus older children (i.e., 11 years and older) for both their review (see Table 1), have greatly informed the development
reports of internalizing concerns (rs: .32 vs. .26; Q 1.42, p .23) of theoretical principles about using and interpreting multi-
and externalizing concerns (rs: .38 vs. .35; Q 0.48, p .49). informant assessment outcomes. Indeed, an underlying assumption
In sum, we made three findings consistent with previous reviews of the multi-informant approach is that informants each carry a
(Achenbach et al., 1987; Markon et al., 2011). First, informants unique and valid perspective of the patients about whom they
reports of relatively more observable or externalizing concerns tended provide reports (De Los Reyes, 2013). Thus, to maximize the
to evidence greater levels of cross-informant correspondence, relative clinical value offered by the multi-informant approach, the infor-
to reports of internalizing concerns. Second, informant pairs for which mants selected to provide reports ought to differ in their opportu-
both informants observed the child in the same setting (i.e., mother nities for observing patients mental health concerns (e.g., obser-
father) tended to yield the highest levels of correspondence, relative to vations that vary across home and school contexts; Kraemer et al.,
all other informant pairs. Third, when informants completed reports 2003). Therefore, if patients contextually vary in where they
using measures that tended to evidence greater reliability and validity display concerns, then discrepancies among informants reports
estimates (i.e., dimensional), we observed greater levels of cross- should, in part, reflect these contextual variations.
informant correspondence, relative to when informants completed These basic assumptions underlying the multi-informant assess-
reports using categorical measures. ment approach have been elaborated upon and codified in recent
One finding from our review that was inconsistent with the theoretical work. Specifically, the idea that integrating multiple
Achenbach et al. (1987) meta-analysis was our null finding re- distinct pieces of information (e.g., research evidence and clinical
garding the effects of child age on magnitudes of cross-informant expertise) improves clinical decision-making relative to relying on
correspondence. These inconsistencies may reflect changes since limited information (e.g., only clinical expertise) is a foundational
Achenbach et al. (1987) with regard to evidence-based assessment principle of evidence-based practice (e.g., Sackett, Rosenberg,
864 DE LOS REYES ET AL.

Gray, Haynes, & Richardson, 1996). Yet, much of multi-informant ment conditions within which one draws inferences of the
assessment research has focused on whether informant discrepan- veracity of multiple methodologically distinct research obser-
cies reflect error or rater biases (for a review, see De Los Reyes, vations based on whether the observations yielded similar con-
2013), and this research often conflicts with the key rationale for clusions (Garner, Hake, & Eriksen, 1956). Under this concept,
taking a multi-informant approach (De Los Reyes, Thomas et al., one draws stronger inferences from studies within which ob-
2013). To address these inconsistencies between theory and inter- servations converged on a common conclusion (e.g., multiple
pretations of multi-informant assessments, researchers developed informants reports all supported treatment efficacy), and in
the Operations Triad Model (OTM; Figure 1; De Los Reyes, turn, one draws weaker inferences from studies within which
Thomas et al., 2013). The OTM addresses a key issue underlying observations diverged toward different conclusions (De Los
these inconsistencies, namely, that research methodology in clin- Reyes, Thomas et al., 2013).
ical psychology has largely relied on the concept of converging To expand upon the converging operations concept, the OTM
operations to interpret research conclusions and assessment out- delineates conditions for two additional research concepts for
comes in practice. Converging operations is a set of measure- interpreting discrepancies among informants reports. First, the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Figure 1. Graphical representation of the research concepts that comprise the Operations Triad Model. The top
half (Figure 1a) represents converging operations: a set of measurement conditions for interpreting patterns of
findings based on the consistency within which findings yield similar conclusions. The bottom half denotes two
circumstances within which researchers identify discrepancies across empirical findings derived from multiple
informants reports and thus discrepancies in the research conclusions drawn from these reports. On the left
(Figure 1b) is a graphical representation of diverging operations: a set of measurement conditions for interpreting
patterns of inconsistent findings based on hypotheses about variations in the behavior(s) assessed. The solid lines
linking informants reports, empirical findings derived from these reports, and conclusions based on empirical findings
denote the systematic relations among these three study components. Further, the presence of dual arrowheads in the
figure representing diverging operations conveys the idea that one ties meaning to the discrepancies among empirical
findings and research conclusions and thus how one interprets informants reports to vary as a function of variation
in the behaviors being assessed. Lastly, on the right (Figure 1c) is a graphical representation of compensating
operations: a set of measurement conditions for interpreting patterns of inconsistent findings based on methodological
features of the studys measures or informants. The dashed lines denote the lack of systematic relations among
informants reports, empirical findings, and research conclusions. Originally published in De Los Reyes, Thomas et
al. (2013). Annual Review of Clinical Psychology. Copyright, 2012 Annual Reviews. All rights reserved. The
Annual Reviews logo, and other Annual Reviews products referenced herein are either registered trademarks or
trademarks of Annual Reviews. All other marks are the property of their respective owner and/or licensor.
VALIDITY OF MULTI-INFORMANT ASSESSMENT 865

OTM includes a concept, diverging operations, developed for These arrows depict the strength of relations between informants
interpreting instances in which multiple informants reports di- reports and patients behavior. That is, solid-line arrows linking
verge from each other for meaningful reasons. An example of informants reports to patients behavior indicate stronger relations
diverging operations may involve a circumstance in which a cli- between reports and behavior, relative to the dashed lines that
nician expects two informants reports about a patients mental indicate weaker relations between reports and behavior. Thus,
health concerns to meaningfully diverge because (a) the infor- Figures 2a and 2b depict examples of diverging operations, in that
mants observe the patients behavior within completely different the parent and teacher reports differ because the patient displays
contexts (e.g., home vs. school), and (b) the patient displays mental concerns in either the home context or school context, but not both
health concerns to a greater degree in one context than the other contexts. In contrast, Figure 2c depicts an example of converging
(e.g., home to a greater extent than school). Second, the OTM also operations, in that the parent and teacher reports corroborate each
includes conditions for a research concept, compensating opera- other because the patient displays concerns across home and
tions, reflecting those circumstances in which multiple informants school contexts.
reports diverge from each other because of measurement error or In Figure 2, we also describe the strength of relations between
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

some other methodological process. For instance, two informants parent and teacher reports and an independently administered
This document is copyrighted by the American Psychological Association or one of its allied publishers.

reports may yield divergent information because they completed criterion variable; an example of such a variable might be a
reports using distinct measures (e.g., different item content and consensus diagnostic assessment by an experienced clinical team.
scaling) and the measures may have varied in their psychometric We include this criterion variable in Figure 2 to illustrate the
properties (e.g., internal consistency estimates). Thus, within in- incremental validity of parent and teacher reports. Here, we use the
stances that reflect compensating operations, methodological fea- same solid-line/dashed-line approach as with the links between
tures of the assessment process provide a parsimonious account for informants reports and measures of actual patients behavior
why informants reports yielded different outcomes. In sum, the that we used to illustrate construct validity. That is, in illustrating
OTM promotes an evidence-based approach to testing whether relations between parent and teacher reports and the criterion
meaningful data can be gleaned from interpreting the consistencies variable, solid-line (relative to dashed-line) arrows linking infor-
and discrepancies observed among multiple informants reports. mants reports to each other indicate stronger relations between
reports. Further, solid-line (relative to dashed-line) arrows linking
Predictions About the Incremental and Construct an informants report to the validity criterion indicate stronger
Validity of Multi-Informant Assessments incremental validity for the solid-line informants report, relative
to the dashed-line informants report. Therefore, in Figure 2a
Mental health professionals can make a strong conceptual case parent (and not teacher) reports incrementally predict variance in
for taking a multi-informant approach to assessing child patients. the criterion. In contrast, in Figure 2b teacher (and not parent)
Further, the OTM may facilitate linking the concepts underlying reports incrementally predict variance in the criterion. In Figure
administering multi-informant assessments with interpreting their 2c, the overlap between parent and teacher reports results in
outcomes (see Figure 1). Yet, unknown is how to use the OTM to neither report incrementally predicting variance in the criterion.
make specific predictions about the incremental and construct If low correspondence among reports validly reflected contex-
validity of multi-informant assessments. That is, when multi- tual variations in patients disruptive behavior, the patterns of
informant assessments yield low correspondence among infor- assessment outcomes would be likely to reflect two predictions,
mants reports, how should the evidence appear if the low corre- each of which are graphically depicted in Figure 2. The first
spondence reflects contextual variations in patients mental health? prediction is that each informants report should build upon the
An illustrative example of multi-informant assessment outcomes other in terms of predicting or relating to relevant clinical metrics
may be helpful.2 For the purposes of this illustration, we focus on (e.g., consensus diagnoses, referral status, treatment response).
interpreting reports from informants who differ in the contexts in The uniqueness of informants reports cannot merely reflect
which they observe patients, specifically parents at home versus noise and must reveal something important about patients clin-
teachers at school. We also focus on assessments of a mental ical presentations, prognoses, and/or responses to treatment proto-
health domain for which patients concerns may significantly vary cols (see also Youngstrom, 2008). Thus, in Figure 2 one can see
across home and school contexts, namely preschoolers disruptive how conducting a study on the incremental validity of the multi-
behavior (for a review, see Wakschlag, Tolan, & Leventhal, 2010). informant approach depends heavily on the contexts in which
To assess a sample of patients experiencing disruptive behavior, patients displayed concerns. Specifically, a sample of patients
consider a research team conducting a multi-informant assessment whose concerns were specific to the home context may result in
that involved collecting parent and teacher reports. In Figure 2, we teacher reports that did not yield incrementally valid information
present graphical depictions of the outcomes of these assessments. relative to parent reports (Figure 2a). Conversely, a sample of
Figure 2 demonstrates links among parent and teacher reports of patients who exhibited school-specific concerns may yield parent
disruptive behavior and actual disruptive behavior as displayed reports that did not evidence incremental validity relative to
by the patient within home and/or school contexts. We include this
measure of actual disruptive behavior in Figure 2 to illustrate the
2
construct validity of parent and teacher reports. Specifically, we The predictions we highlight below may also apply to interpreting
graphically depict three different patients, or patients who display distinctions between observer reports (e.g., parent, teacher, and clinician
reports) and patient self-reports, with the one caveat that the contexts in
concerns in home but not school (Figure 2a), school but not home which patients observe their own behavior may overlap with those of other
(Figure 2b), and across home and school (Figure 2c). For each informants reports (see De Los Reyes, Bunnell et al., 2013; Kraemer et al.,
example, arrows link informants reports to patients behavior. 2003).
866 DE LOS REYES ET AL.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Figure 2. Graphical depiction of incremental and construct validity predictions that follow from the OTM and
conceptual rationale for taking a multi-informant approach to mental health assessment.

teacher reports (Figure 2b). Thus, for multiple informants reports Clinical severity. First, one interpretation of the patterns of
to be considered incrementally valid relative to each other, patients mental health concerns illustrated in Figure 2 is that consistencies
in a sample should be heterogeneous in terms of the contexts in and inconsistencies between informants reports reflect individual
which they displayed concerns (i.e., a sample of Figure 2a, 2b, and differences in severity of patients clinical presentations, and not
2c patients). That is, if a sample of patients exhibited individual necessarily contextual variations in patients concerns. Impor-
differences in the context(s) in which they displayed concerns, tantly, the literature is quite equivocal as to whether cross-
then each kind of informant who provided reports about these contextual consistency in patients mental health concerns can be
patients (e.g., parents and teachers) would have had the opportu- treated as a proxy for high levels of clinical severity. Indeed, prior
nity to observe at least a subgroup of patients concerns in the work identified relations between informant discrepancies and
sample. Without an informant having the opportunity to observe context-specific displays of mental health concerns, even when
patients concerns, one cannot expect that informant to provide accounting for participants clinical severity (e.g., De Los Reyes,
incrementally valid reports about such concerns. Bunnell, & Beidel, 2013; De Los Reyes, Henry, Tolan, & Wak-
The second prediction is that patterns of cross-informant corre- schlag, 2009). Additionally, studies of patients from a variety of
spondence should reflect contextual variations inherent in patients clinical domains (e.g., attention and hyperactivity, conduct prob-
concerns. For instance, a pattern of parents reporting high levels of lems, social anxiety) are quite inconsistent as to whether patients
disruptive behavior for which teachers reports do not corroborate evidencing concerns across contexts also evidence significantly
should indicate that patients exhibited disruptive behavior to a
greater clinical severity or impairment levels than patients who
greater extent at home than school (Figure 2a). Alternatively,
display concerns within specific contexts (e.g., Bgels et al., 2010;
consistently high levels of disruptive behavior reported by both
Dirks et al., 2012; Drabick et al., 2007). In contrast, studies of
parents and teachers should indicate that patients experienced
informant discrepancies in assessments of large samples of psy-
concerns across contexts (Figure 2c).
chiatric inpatients (Carlson & Youngstrom, 2003) and outpatients
(Thuppal, Carlson, Sprafkin, & Gadow, 2002) found that patients
Considering Constructs Other Than Context When whose concerns were endorsed by at least two informants evi-
Interpreting Informants Reports denced greater levels of clinical impairment, relative to patients
One key issue warrants comment. Specifically, a great deal of whose concerns were endorsed by a single informant. Overall, in
research has focused on testing whether low versus high cross- informant discrepancies studies, one ought to statistically account
informant correspondence reflects constructs other than contextual for the clinical severity of patients concerns when examining the
variations in displays of patients concerns (for reviews, see relations between informants reports and contextual variations in
Achenbach, 2006; De Los Reyes & Kazdin, 2005; Kraemer et al., patients concerns.
2003). Thus, drawing inferences about patients concerns from Informants perspectives and rater biases. Perhaps the most
multi-informant assessments ought to involve examining plausible frequently studied set of constructs in informant discrepancies
rival hypotheses for why informants reports may converge or research are those that reflect variations in informants perspec-
diverge. tives about patients concerns and possible biases in informants
VALIDITY OF MULTI-INFORMANT ASSESSMENT 867

reports. Research on these informant characteristics has been ex- Reyes & Kazdin, 2005). However, a number of studies have found
tensively reviewed in prior work (e.g., De Los Reyes, 2013; De no such support (e.g., Conrad & Hammen, 1989; De Los Reyes,
Los Reyes & Kazdin, 2005). Consequently, we will focus on the Goodman, Kliewer, & Reid-Quiones, 2010; De Los Reyes,
evidence regarding the most often-studied characteristics. For in- Youngstrom et al., 2011a; Hawley & Weisz, 2003; Weissman
stance, informants may vary in their perspectives as to which et al., 1987). Further, most depression distortion studies do not
mental health concerns in child patients warrant care (Brookman- include independent ratings of childrens behavior in the same
Frazee, Haine, Gabayan, & Garland, 2008; Hawley & Weisz, situation (see also Richters, 1992), and interactions between infor-
2003; Jensen-Doss & Weisz, 2008; Yeh & Weisz, 2001). In fact, mants (e.g., parents and teachers) and the subject of the assessment
informants may vary in perceiving that a childs mental health (e.g., children receiving treatment) are often context-dependent
concerns distress the child and thus warrant care, even when all (e.g., occur exclusively in home and/or school contexts). Conse-
informants converge on endorsing that concerns exist (e.g., Phares quently, observing a child behaving differently with their de-
& Compas, 1990; Phares & Danforth, 1994). Consequently, infor- pressed parent at home than they did with teachers at school might
mants may vary considerably in their perspectives of patients be a logical consequence of the nature of interactions between
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

mental health concerns, and these discrepant perspectives may patients and informants.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

impact such aspects of mental health care as access to services and Among those studies that have used experimental designs and
therapeutic engagement (see also Hawley & Weisz, 2003; Yeh & constrained other possible confounding factors (e.g., context of
Weisz, 2001). Similarly, informants from different ethnic or racial observation), researchers have observed, at best, modest support
backgrounds may vary in whether or which behaviors reflect for the depression distortion hypothesis. For instance, Jouriles
mental health concerns and warrant care, although research on the and Thompson (1993) experimentally induced negative mood
relation between such backgrounds and informant discrepancies states in parents before they viewed a previously recorded task
has yielded inconsistent findings (see De Los Reyes & Kazdin, involving a cleanup activity with their child, and had parents and
2005; Duhig et al., 2000). Further, low-to-moderate correspon- independent observers rate the childs behavior during the task.
dence levels and directional differences in reporting (e.g., whether The researchers observed nonsignificant differences between par-
children self-report greater concerns than parents report about ents and independent observers reports of childrens behavior
children) similarly characterize cross-informant correspondence in
during the task. Other studies using mood induction procedures
reports examined in over 20 countries (Rescorla et al., 2014;
have also failed to support the depression distortion hypothesis
Rescorla et al., 2013).
(Youngstrom, Kahana, Starr, & Schwartz, 2004).
Few characteristics in the informant discrepancies literature
The depression distortion hypothesiss strongest empirical
have been given more research attention than the impact of infor-
support comes from a quasi-experimental study examining the
mants mental health concerns on their reports of childrens mental
relation between mothers depressive mood symptoms and their
health. Broadly, the conceptual rationale for this work can be
reports of childrens behavior during completion of a frustrating
encapsulated in what researchers have termed the depression
task (Youngstrom et al., 1999). Youngstrom, Izard, and Ackerman
distortion hypothesis: When an informant experiences low mood,
(1999) controlled for the context on which informants based their
this causes the informant to attend to, encode, and thus rate
reports by comparing mothers reports of children completing a
childrens behavior with greater negative descriptors, relative to
use of positive or neutral descriptors (see Richters, 1992; Young- frustrating task with independent observers reports of children
strom, Izard, & Ackerman, 1999). Importantly, depression during the same task. The researchers observed statistically sig-
distortion effects should not be seen as mutually exclusive from the nificant depression distortion effects, and yet these effects only
previously described characteristics. For instance, depression dis- accounted for between 2%20% of incremental variance in moth-
tortion effects may result in depressed informants reporting greater ers reports, relative to independent observers reports. Conse-
levels of patients mental health concerns than nondepressed in- quently, one would be hard-pressed to discard mothers reports
formants. These effects potentially impact patients access to care based on even the highest observed figure of incremental variance
if mental health professionals encounter discrepant reports, and (i.e., 20%), as even this figure constitutes a minority of the vari-
these discrepant reports result in uncertainties when making clin- ance in the mothers report that could possibly be afflicted with
ical decisions. Among the informants providing reports, parents rater bias. Overall, the lack of strong support for depression
reports have been the most frequently studied in terms of distortion effects indicate that these effects cannot fully account
depression distortion effects, perhaps because a common risk for the presence of informant discrepancies.
factor of childrens mental health concerns is a family history of Measurement error. A construct related to that of infor-
such concerns (e.g., Goodman & Gotlib, 1999). Thus, within clinic mants perspectives and rater biases is measurement error. Mea-
samples of child patients, parents providing reports of patients surement error has key implications for research on the validity of
concerns may often experience a key characteristic thought to bias the multi-informant approach. Indeed, measurement error trun-
informants reports. cates the ability of informants reports to predict criterion vari-
A key concern with research on the depression distortion ables, thus hindering the ability of multi-informant assessments to
hypothesis is that the hypothesis does not have strong empirical evidence incremental and construct validity (see Dirks, Boyle, &
support on its behalf. As mentioned previously, many studies have Georgiades, 2011; Dirks et al., 2012). Thus, if measurement error
tested the depression distortion hypothesis, with some studies explained informant discrepancies, then mental health profession-
finding that informants experiencing depressive symptoms provide als could not validly infer that discrepancies among informants
reports that indicate greater levels of childrens mental health reports reflect changes in displays of patients mental health con-
concerns, relative to reports taken from other informants (De Los cerns across contexts.
868 DE LOS REYES ET AL.

The psychometrics literature points to three important compo- cents (De Los Reyes, Ehrlich et al., 2013). Researchers randomly
nents of measurement error that have implications for the multi- assigned parents and adolescents to the order in which they com-
informant assessment approach: (a) transient error, (b) random pleted two different assessments of parental knowledge. Within
error, and (c) systematic error. The first, transient error refers to one assessment protocol, parents and adolescents received training
characteristics of the rater that might hinder measurement reliabil- on how to use the number of contexts in which they tend to
ity (Schmidt & Hunter, 1996). In many respects, the literature observe behaviors indicative of parental knowledge to provide
reviewed previously on the depression distortion hypothesis Likert-type ratings on survey reports about such knowledge (i.e.,
represents a case of research on transient error in multi-informant greater number of contexts greater ratings; De Los Reyes,
assessment. Second, randomly distributed error variance, particu- Ehrlich et al., 2013). Within a second protocol, parents and ado-
larly within the individual informants reports in a multi-informant lescents received instructions to complete the assessments that did
assessment, may pose challenges to reliably assessing discrepan- not involve any training (i.e., a typical questionnaire completion
cies among informants reports. This is because assessments of protocol). Interestingly, the training received by parents and ado-
discrepancies between informants reports can only be as reliable lescents increased the differences between their reports, relative to
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

as the individual informants reports from which one assesses the no-training condition. These findings suggest that parents and
This document is copyrighted by the American Psychological Association or one of its allied publishers.

discrepancies (e.g., Rogosa, Brandt, & Zimowski, 1982; Rogosa & adolescents differ in reports about parental knowledge, in part,
Willett, 1983). Third and with regard to systematic error, recent because they view behaviors indicative of parental knowledge in
work indicates that multi-informant assessments may yield con- different contexts.
sistent differences between informants reports across multiple In contrast to the findings of De Los Reyes, Ehrlich et al. (2013),
measures (De Los Reyes, Alfano, & Beidel, 2010, 2011; De Los in another study clinicians with experience in assessing and treat-
Reyes, Bunnell et al., 2013; De Los Reyes, Goodman et al., 2008, ing child patients read vignettes of hypothetical children receiving
2010; De Los Reyes, Youngstrom et al., 2011a, 2011b). Yet, the a screening evaluation for conduct disorder (De Los Reyes &
presence of systematic error may result in giving consistent dif- Marsh, 2011). The clinicians read vignettes that described contex-
ferences between informants reports the appearance of meaning- tual features in a childs life that the researchers randomly manip-
ful differences (see also De Los Reyes, 2013). ulated to reflect either risks for conduct disorder (e.g., associating
Overall, measurement error may pose challenges to validly with deviant peers) or no such risks. The vignettes also described
interpreting the outcomes of multi-informant assessments as re- the child as evidencing a single symptom within the fourth edition
flecting contextual variations in patients concerns. Thus, research- of the Diagnostic and statistical manual of mental disorders
ers have examined components of the measurement process that (DSMIV) diagnostic criteria for conduct disorder (American Psy-
either constrain levels of such error or improve interpretations of chiatric Association [APA], 2000). Researchers prompted clini-
informants reports. Two research literatures appear particularly cians to make likelihood ratings of the children that reflected their
pertinent in this regard. First, even subtle changes in measurement impressions of whether the children would meet DSMIV criteria
scaling can have profound effects on informants reports. For for conduct disorder if a clinician were to administer a full diag-
example, on a positively valenced self-report measure of life nostic evaluation to the child. Further, the clinicians made judg-
success, scaling items on a 5 to 5 range produced greater ments for 30 hypothetical children, in order to make likelihood
proportions of respondents rating high life success than the pro- judgments for all 15 DSMIV conduct disorder symptoms rated in
portion observed when the scale ranged from 0 to 10 (for a both high-risk and low-risk contexts. Thus, all clinicians in the
review, see Schwarz, 1999). Along these lines, a key element of sample made ratings about the same hypothetical children and
best practices in multi-informant assessment involves ensuring that conduct disorder symptoms, were exposed to the same rating
assessors hold such measurement features as item content, scaling, instructions, and had access to the same information about the
and response labeling constant across multiple informants reports. hypothetical childrens environments. Overall, clinicians made
In this way, an assessor can decrease the likelihood that discrep- greater likelihood ratings for children described in high-risk con-
ancies among reports are the result of methodological artifacts of texts versus those described in low-risk contexts. These findings
the measurement process (for a review, see De Los Reyes, Thomas suggest that clinicians tended to apply contextual information to
et al., 2013). In fact, when researchers hold these measurement their judgments about the childrens conduct disorder symptoms.
factors constant across reports, and the measures informants com- However, clinicians exhibited little correspondence in terms of the
plete exhibit the same factor structures and similar levels of specific symptoms for which they applied contextual information.
internal consistency, informants nevertheless provide reports that For instance, whereas high-risk contextual information might have
correspond with each other at low-to-moderate magnitudes (e.g., influenced one clinicians judgments about a childs truancy, the
Achenbach & Rescorla, 2001; Baldwin & Dadds, 2007). same high-risk contextual information might have had little influ-
At the same time, holding measurement factors constant across ence on another clinicians judgments about the childs truancy.
informants reports does not guarantee that the resulting discrep- These findings have since been replicated when examining judg-
ancies reflect contextual variations in patients concerns. Indeed, ments about attention-deficit/hyperactivity concerns, as well as
informants may nonetheless react differently to the same measure- judgments completed by laypeople (Marsh, De Los Reyes, &
ment instrument, and the reasons for these different reactions may Wallerstein, 2014).
or may not reflect differences in the contexts in which informants The findings of De Los Reyes, Ehrlich et al. (2013) and De Los
observe patients behavior. For instance, in recent work research- Reyes and Marsh (2011) indicate that holding measurement factors
ers administered assessments of parental knowledge of adoles- constant alone does not guarantee interpretability of any one
cents whereabouts and activities (i.e., a risk factor for adolescent informants report or the discrepancies between informants re-
delinquency) to a community-based sample of parents and adoles- ports. Thus, a second literature involves identifying and using
VALIDITY OF MULTI-INFORMANT ASSESSMENT 869

validity criteria for assessing context-specific displays of mental plement them in vivo within the laboratory as well as routine clinic
health concerns. Using context-specific criterion measures, mental settings (see also Thomas, Aldao, & De Los Reyes, 2012).
health professionals might improve understanding of the reasons For example, in one study researchers recruited adolescents who
why informants provide discrepant reports. For example, to assess met DSMIV criteria for social anxiety disorder as well as adoles-
cross-contextual variations in preschool childrens disruptive be- cents who did not meet criteria for any mental disorder (Anderson
havior, researchers recently developed the Disruptive Behavior & Hope, 2009). In this study, adolescents engaged in a series of
Diagnostic Observation Schedule (DB-DOS; Wakschlag et al., laboratory social interaction tasks totaling 20 minutes (e.g., one-
2010). This behavioral observation measure consists of assess- on-one social interaction and public speaking tasks with trained
ments of childrens disruptive behavior as displayed within inter- confederates), to assess adolescents self-reported changes in
actions between children and adult authority figures. The adult arousal in reference to arousal indices of physiological habituation
authority figures consist of the assessed childs parent and an taken during these interactions (i.e., wireless, ambulatory heart rate
unfamiliar clinical examiner. Assessors hold the nature of the monitors). Regardless of diagnostic status, adolescents experi-
interactions constant across adult child interactions. For instance, enced physiological habituation to the social interactions (i.e.,
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

the parent prompts the child in one interaction to help with clean- sharp increase in heart rate at beginning of task, followed by
This document is copyrighted by the American Psychological Association or one of its allied publishers.

ing up toys, and in a separate interaction the clinical examiner gradual decrease in heart rate over course of task). Yet, adoles-
administers a similar cleanup prompt to the child. The consistency cents self-reports varied as a function of diagnostic status. Spe-
in the structure of adult child interactions on the DB-DOS allows cifically, adolescents experiencing social anxiety disorder were
one to assess in an analogue sense disruptive behavior as displayed more likely than adolescents not meeting criteria for any mental
in home contexts (e.g., with parental adults) and/or nonhome disorder to self-report stable and high levels of arousal from
contexts (e.g., with nonparental adults such as teachers). Thus, the pre-to-post tasks. This observation is consistent with key compo-
DB-DOS might serve as an independent assessment of whether nents of exposure-based therapies for social anxiety. Indeed, these
patients display disruptive behavior consistently across contexts or therapeutic approaches often involve training patients to subjec-
within specific contexts. Such an assessment could serve as a tively perceive decreases in physiological arousal within and
criterion for assessing the validity in interpreting patterns of in- across social situations that at the outset of therapy patients find
formants reports (e.g., parent and teacher reports) as reflective of anxiety provoking (e.g., Beidel et al., 2007). Overall, these three
contextual variations in patients concerns. Below, we describe a examples illustrate how mental health professionals might use
study that took exactly this approach. independent behavioral, survey, and/or physiological assessments
Another example of a context-sensitive validity criterion comes to use and interpret informants clinical reports of patients con-
from research on the social interactions that tend to elicit displays cerns.
of childrens behavioral and emotional concerns (Hartley, False-positive and false-negative reports. A set of constructs
Zakriski, & Wright, 2011). In this work, researchers administered related to measurement error but with different implications for the
behavioral checklists for informants (e.g., parents and teachers) to multi-informant approach involves concepts drawn from judgment
complete about children. The informants also completed measures and decision-making research, namely signal detection theory
about how the same children tend to react (e.g., aggressive behav- (SDT; for a review, see Swets, Dawes, & Monahan, 2000). Spe-
ior) within interactions with peers (e.g., peer bosses child around) cifically, let us assume that the parent and teacher reports illus-
and adult authority figures (e.g., adult giving child instructions). trated in Figures 2a and 2b reflected error-free representations of
This second measure allows researchers to examine whether con- childrens behavior, and the parents and teachers systematically
vergence between informants reports about childrens behavioral varied in terms of exclusive access to observations of childrens
and emotional concerns signals similarities in the kinds of inter- behavior as displayed at home and school, respectively. If so, then
actions encountered across contexts (e.g., child encounters teasing parents and not teachers would endorse every instance of disrup-
by peers at home and school). Alternatively, one could examine tive behavior displayed at home and not school, whereas teachers
whether divergence between informants reports about childrens and not parents would endorse every instance of disruptive behav-
concerns reflects the idea that interactions that tend to elicit chil- ior displayed at school and not home. Stated another way, both
drens concerns are present in one context (e.g., school) and not parents and teachers would evidence perfect rates of positive
another (e.g., home). endorsements of true cases of disruptive behavior as displayed
The previous examples of context-sensitive validity criteria re- within their own context of observation. Parents and teachers
lied on behavioral observations or survey methods. One final would also refrain from endorsing disruptive behavior that only
example leverages the time- and context-sensitive properties of occurred outside of their observational context.
physiological arousal. Specifically, individuals experiencing social Within SDT, researchers refer to true cases of endorsement as
anxiety vary in that some experience social anxiety in different true positives and true cases of nonendorsement as true negatives
ways, depending on the context (e.g., Bgels et al., 2010). Direct (see also Swets, 1992). However, one should not expect any
assessments of physiological arousal may distinguish social anxi- measure (e.g., informants report), to provide an error-free repre-
ety patients experiences within and across these contexts (e.g., sentation of the construct being assessed. Thus, SDT delineates
arousal during public speaking vs. arousal in anticipation of public concepts for representing incorrect positive endorsements for non-
speaking; De Los Reyes et al., 2015). These assessments have cases (i.e., false positives) and incorrect nonendorsements for true
great potential as validity indicators to interpret informants re- cases (i.e., false negatives). For instance, in a case in which a
ports (De Los Reyes & Aldao, 2015). This is because technology parent endorsed disruptive behavior that a teacher did not endorse,
now allows mental health professionals to assess arousal using if in reality the patient did not display any disruptive behavior in
wireless, ambulatory devices (e.g., heart rate monitors), and im- any context, the parent report would reflect a false positive. Con-
870 DE LOS REYES ET AL.

versely, both parent and teacher might fail to endorse disruptive could also structure an assessment similar to that described in
behavior in a case in which the patient actually did display dis- Figure 3 to gauge the context-sensitivity of teachers reports (i.e.,
ruptive behavior in both contexts; in this case both reports would independent school observation). Thus, one might incorporate
reflect false negatives. methods from SDT with the OTM to improve use and interpreta-
As in addressing measurement error, one might address false tion of multi-informant assessments.
positive and false negative outcomes with an independent, context- Informants cognitive abilities and social desirability. Lastly,
sensitive assessment of patients concerns. Figure 3 provides an how might one interpret the unique qualities of informants re-
illustration of this approach. For example, consider a hypothetical ports, beyond that they reflect contextual variations in patients
sample of children in which teachers and parents completed stan- concerns, levels of clinical severity, and/or rater biases? That is,
dardized behavioral checklists of childrens disruptive behavior, discrepancies between adult informants such as parents and teach-
and researchers also administered an independent observation of ers might originate from patients concerns varying across con-
childrens disruptive behavior in the home context. In Figure 3, the texts. However, they may also vary in light of the relationship that
left-most distribution of scores relates to teacher reports, the center the patient shares with the informant and the context in which this
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

distribution to parent reports, and the right-most distribution to the relationship develops. For instance, a parent may provide a report
This document is copyrighted by the American Psychological Association or one of its allied publishers.

independent home observation. The x-axis denotes a standardized based on extensive experiences with the patient, and although
score (i.e., T score) derived from each assessment to represent these experiences may develop over the course of several years,
childrens disruptive behavior. Further, we included a demarcation they may not be perceived within the context of normative child-
at the T score 65 to denote the clinical cutoff at which research- hood behavior (De Los Reyes et al., 2009). Conversely, a teachers
ers would identify cases of disruptive behavior. The y-axis denotes relationship with the patient may develop within the context of a
the frequency for scores at a given T score level. Given standard- single academic year and a large classroom of other students. That
ization of the disruptive behavior scores, all three assessment is, a teachers report may reflect a relatively limited time range of
modalities in the example follow a normal distribution. experiences with a patient. However, the teacher may have the
In Figure 3, the independent assessment occurred in the home opportunity to calibrate their report based on normative classroom
context. Thus, one would expect greater overlap between identi- behavior (see also Drabick et al., 2007).
fied cases of disruptive behavior concerns using the independent When researchers compare adult and child informants, infor-
assessment and identified cases using parent report (i.e., relative to mant discrepancies may be attributable, in part, to differences in
teacher report). Thus, using the independent assessment as a ref- developing cognitive abilities (e.g., Smetana, Campione-Barr, &
erence point, researchers could examine whether patients identi- Metzger, 2006; Spear, 2000). In the context of patient care, dif-
fied via the independent assessment were more likely to be those ferences between adult reports about patients and patient self-
patients positively endorsed by parents relative to teachers. Stated reports may occur as a result of social desirability concerns on the
another way, the independent assessment allows the research team part of the patient, although studies yield inconsistent support for
to identify the proportions of parent and teacher reports evidencing these concerns as contributors to informant discrepancies (for a
true positives and negatives, as well as false positives and nega- review, see De Los Reyes & Kazdin, 2005). These differences
tives. With these proportions, one could estimate how sensitive between adult informants and child informants may indicate, for
(i.e., proportion of true positives relative to false positives) and instance, that adult reports reliably and validly reflect patients
specific (i.e., proportion of true negatives to false negatives) the concerns to a greater extent than patient self-reports.
parent reports were in detecting home-specific observations of Recent evidence indicates that differences between parent re-
patients concerns, relative to an informant who observed patients ports and child reports may also reflect meaningful variation
in a context other than home (see also Youngstrom, 2013). One between their perceptions, in addition to variations in cognitive

Figure 3. Graphical depiction of T score distributions of teacher reports (left, labeled T), parent reports
(center, labeled P), and independent home-based assessments of disruptive behavior (right, labeled H), in a
hypothetical sample of young children.
VALIDITY OF MULTI-INFORMANT ASSESSMENT 871

abilities and social desirability concerns. For instance, in struc- blotches of black ink set against a plain white background; for a
tured diagnostic interview assessments of child anxiety, interviews review, see Hunsley & Lee, 2010). Patients responses are thought
based on parent reports and child reports exhibit greater correspon- to reveal important aspects of their personality functioning from a
dence when reports are about directly observable anxiety behav- psychoanalytic perspective, and thus scoring responses often in-
iors displayed in nonschool contexts (e.g., behavioral avoidance volves use of complex procedures that require a substantial amount
displayed at home) relative to internal anxiety behaviors such as of expertise on the part of the assessor (i.e., knowledge of psy-
worry displayed in school contexts (Comer & Kendall, 2004). choanalytic theory and practice; see Blais, Hilsenroth, Castlebury,
Further, recent work in an age- and gender-matched sample of Fowler, & Baity, 2001). However, Rorschach responses can also
clinic-referred adolescents and community control adolescents be scored using less onerous procedures that require little expertise
found that clinic-referred adolescents self-reported lower levels of to carry out (e.g., assessing the match between a patients
social anxiety relative to their parents reports, and adolescents response about the shape of an inkblot and the actual shape of the
self-reports exhibited little to no correspondence with objective inkblot). Further, personality functioning can be assessed with
measures of psychophysiology (i.e., during a baseline psychophys- paper-and-pencil inventories that can be administered, scored, and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

iological assessment; De Los Reyes, Aldao et al., 2012). Yet, in interpreted by personnel with little training. Consequently, Dawes
This document is copyrighted by the American Psychological Association or one of its allied publishers.

this study adolescents nonetheless provided internally consistent (1999) argues that a proper evaluation of incremental validity of the
self-reports that evidenced convergent validity and differentiated Rorschach involves assessing whether scores that require a great deal
clinic-referred from community control adolescents. This study is of expertise to obtain hold incremental value above-and-beyond the
an important contribution to the literature on social desirability. explanatory value of scores that require less expertise to obtain.
Indeed, clinic samples of anxiety patients comprise key settings In clinical child assessments, the multi-informant approach poses
within which patients social desirability has been of primary incremental validity issues similar to that of the Rorschach and
concern to mental health professionals (e.g., Dadds, Perrin, & delineated by Dawes (1999). In essence, do clinical child assessments
Yule, 1998; DiBartolo, Albano, Barlow, & Heimberg, 1998; Grills require the collective expertise of multiple informants for one to
& Ollendick, 2003; Rapee et al., 1994; Silverman & Rabian, obtain an adequate sense of the context(s) in which a patient displays
1995). Importantly, De Los Reyes, Aldao et al. (2012) did not mental health concerns? Indeed, the multi-informant approach is a
directly assess social desirability. Yet, the authors made a key comprehensive assessment procedure that incurs considerable re-
observation that researchers often interpret as indicating social sources in terms of time needed to administer, score, and interpret
desirabilityrelative underreporting via patient self-reports rela- instruments, and staffing and equipment needed to carry out the
tive to adult informants reports about patientswhile also finding assessments (see also Yates & Taub, 2003). A less onerous and
support for the internal consistency and validity of scores taken comprehensive assessment might effectively address the goals of
from patient self-reports. Thus, informant discrepancies alone assessing a patient, such as use of a single informants report. Thus,
should not be interpreted as reflecting developmental differences demonstrating that the multi-informant approach yields incremental
in cognitive abilities or social desirability in informants reports. value over, for instance, a single informants report involves testing
Rather, mental health professionals should evaluate whether these whether each informant observes patients concerns in a particular
discrepancies relate to independent assessments of informants context that other informants have relatively fewer or no opportunities
cognitive abilities or social desirability (see Table 2). to observe. That is, the expertise of each informant should result in
reports from which scores reflect different facets of patients concerns
(i.e., different contexts in which patients display concerns; see also
Incremental Validity in Multi-Informant Mental Smith et al., 2003).
Health Assessments Criterion variables and criterion contamination. A second
core idea driving our examination of the incremental validity of
Theoretical and Methodological Approaches to multi-informant clinical child assessments follows from ideas about
Incremental Validity expertise advanced by Dawes (1999). Specifically, as in other cases of
validity (e.g., construct and predictive), the incremental validity of
A full account of the validity of multi-informant assessments scores from a given measure may vary as a function of the criterion
involves examining the unique contribution of one informants variable(s) examined (e.g., diagnostic status and treatment response)
report over anothers, or the incremental validity of each report in and the other measures to which it is compared (e.g., Haynes &
the prediction of clinical outcomes (see Figure 2). Yet, despite Lench, 2003). These two factors give rise to two additional challenges
repeated appeals for more investigations of incremental validity in to using and interpreting multi-informant assessments. The first chal-
psychological assessment, a dearth of information in this area lenge is to design incremental validity studies involving multiple
remains (Hunsley & Meyer, 2003). Nevertheless, prior work on predictor variables (e.g., multiple informants reports). Predictors
incremental validity in such areas as test construction (Haynes & should evidence bivariate relations with the criterion variable(s) but
Lench, 2003; Smith, Fischer, & Fister, 2003), adult psychopathol- share little overlapping variance with each other (see also Blais et al.,
ogy (Garb, 2003), and use of projective tests (e.g., Dawes, 1999) 2001). For instance, use of parent reports and teacher reports to
has key implications for our discussion of incremental validity. In predict clinicians impressions of patients behavior across home and
particular, two ideas informed our examination of the incremental school contexts capitalizes on use of the context-specific expertise of
validity of multi-informant clinical child assessments. each informant (i.e., home-specific vs. school-specific) to predict
Expertise in assessment. First, the Rorschach Inkblot Test is clinicians reports.
a projective test that involves an assessor soliciting patients open- However, addressing challenges with overlapping variance among
ended, verbal descriptions of ambiguous stimuli (i.e., a series of predictor variables raises a second challenge. Specifically, if the
872 DE LOS REYES ET AL.

Table 2
Summary of Exemplary Findings Regarding the Incremental Validity of Cross-Informant Reports When Assessing Childrens
Psychosocial Functioning

Symptom/Disorder Author group (year) Informants Description

ADHD Power et al. (1998) P, T Researchers recommended use of a single parent report or teacher report to
rule out ADHD in children, but to combine parent report and teacher
report to identify children meeting criteria for an ADHD diagnosis.
Smith, Pelham, C, T On average, adolescent self-reports contributed less than a 1% unique
Gnagy, Molina, & variance above teacher ratings in the assessment of ADHD. In contrast,
Evans (2000) teacher reports contributed, on average, 8% unique variance relative to
self-report. However, teacher reports evidenced domain-specific
incremental validity (i.e., higher unique variance contributed by teachers
for behaviors displayed in the classroom vs. outside the classroom).
ADHD subtypes and Owens & Hoza (2003) P, T When assessing inattentive or hyperactive/impulsive subtypes of ADHD,
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

ODD combining parent and teacher reports did not provide any additional
This document is copyrighted by the American Psychological Association or one of its allied publishers.

information above relying on parent report or teacher report alone. In


contrast, when assessing ADHD, combined type, or ODD, teacher reports
alone and the combination of teacher reports and parent reports proved
superior to parent reports alone in predicting a diagnosis.
Anxiety Villab, Gere, C, P Adding parent reports to child self-reports (i.e., on the MASC) increased
Torgersen, March, prediction of childrens anxiety disorder diagnoses. Specifically, parent
& Kendall (2012) reports incrementally predicted childrens separation anxiety disorder
diagnoses, whereas child self-reports incrementally predicted childrens
social phobia diagnoses.
Bipolar Disorder Youngstrom et al. C, P, T Across estimates of diagnostic efficiency for six different rating scales, parent
(2004a) reports evidenced superior prediction of a pediatric bipolar disorder
diagnosis relative to both self-reports and teacher-reports. Additionally,
combining self-reports or teacher reports with parent reports did not
significantly enhance the prediction of a diagnosis of pediatric bipolar
disorder.
ODD Angold & Costello C, P Children tended to self-report fewer oppositional symptoms relative to
(1996) parents reports about children. However, children self-reported certain
oppositional symptoms (i.e., frequent loss of temper) at greater levels
relative to parents reports about children. Researchers recommended both
self-reports and parent reports whenever possible when considering an
ODD diagnosis.
Relational Adjustment Ladd & Kochenderfer- C, P, T, Peer No single informants report of victimization consistently predicted relational
Ladd (2002) adjustment across Grades 24. The informant whose report explained the
most variance in relational adjustment varied as a function of grade: (a)
teacher reports in Grade 2, (b) teacher reports and self-reports in Grade 3,
and (c) a combination of self-reports and teacher reports in Grade 4.
Overall, parent reports explained little unique variance.
Social Skills and Kwon, Kim, & T, Peer Teacher-reported social skills for children in Grades 35 did not uniquely or
Academic Competence Sheridan (2012) significantly predict social status indicators (e.g., likeability and popularity)
above information obtained from peer reports. In contrast, both peer-
reported and teacher-reported social skills demonstrated incremental
validity in predicting academic competence and childrens positive school
functioning.
Note. C Child; P Parent; T Teacher; ADHD attention-deficit/hyperactivity disorder; MASC Multidimensional Anxiety Scale for Children;
ODD oppositional defiant disorder; MDD major depressive disorder; CD conduct disorder.

outcomes of incremental validity studies vary depending on the cri- criterion consisted of clinicians diagnostic judgments about patients,
terion variable, then these studies should be designed such that dif- the study might suffer from criterion contamination if prior to forming
ferences in findings among the predictor variables should to the their judgments, clinicians had access to information from the predic-
greatest extent possible reflect true differences among predictors in tors evaluated for incremental validity (see also Bossuyt et al., 2003;
their incremental validity. For instance, when comparing reports Whiting et al., 2011).
within a multi-informant assessment, one should ensure that measure-
ment content be held constant across the informants reports. Yet, Incremental Validity in Multi-Informant Clinical
holding these measurement factors constant across reports may do
Child Assessments
little to address a key issue raised by Garb (2003): criterion contam-
ination. In incremental validity research, criterion contamination re- We evaluate the incremental validity of multi-informant clinical
fers to differences in incremental validity among predictors that arises, child assessments in light of key theoretical and methodological
in part, because the criterion variable was based on information from contributions from prior work. As in the incremental validity
one or more of the predictors. For instance, if a studys validity literature reviewed previously, the incremental validity of reports
VALIDITY OF MULTI-INFORMANT ASSESSMENT 873

about child patients may be influenced by measurement method- assessments may vary as a function of the validity criterion or
ology, the mental health domain assessed, and criterion contami- predictors (e.g., Johnston & Murray, 2003; Pelham et al., 2005).
nation (e.g., Johnston & Murray, 2003; Pelham, Fabiano, & Mas- How might the choice of the validity criterion and predictors
setti, 2005). In Table 2, we illustrate key findings from prior work. influence conclusions about the incremental validity of multi-
Importantly, few incremental validity studies exist for internalizing informant assessments? We address this question in two ways.
concerns such as anxiety, relative to externalizing concerns such as First, how do multi-informant assessments perform if one holds
attention and hyperactivity and conduct problems (for reviews, see measurement method, item content, and an independently assessed
Dirks et al., 2012; Silverman & Ollendick, 2005; Tulbure, Szen- correlate constant across informants reports? To address this, we
tagotai, Dobrean, & David, 2012). Thus, we focus our attention on present in Table 3 findings reported by Achenbach and Rescorla
the incremental validity of multi-informant assessments of chil- (2001) for cross-informant correspondence for parent report,
drens externalizing concerns, and where possible we incorporate teacher report, and child self-report forms of the Achenbach Sys-
work on childrens internalizing concerns. tem of Empirically Based Assessments (ASEBA), as well as
Measurement method of informants reports evaluated for relations among these reports and childrens referral status.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

incremental validity. Across both clinical and research settings, Briefly, the ASEBA battery consists of parent, teacher, and child
This document is copyrighted by the American Psychological Association or one of its allied publishers.

informants may provide reports via a variety of measurement reports that exhibit an identical format (i.e., behavioral checklist)
methods, including rating scales and structured or semistructured and response scaling (i.e., response values of none, some, and a
interviews. Under certain circumstances (e.g., diagnosing child- lot), with some minor modifications to item content to match each
hood attention-deficit/hyperactivity disorder [ADHD]), research informants perspective. Each form assesses syndromes reflecting
supports the construct validity of these approaches (Pelham et al., various childrens mental health domains that cluster around two
2005). Yet, scores from an informants report may evidence con- broadband problem domains: internalizing (e.g., anxiety, mood,
struct validity and still fail to incrementally contribute information somatic complaints) and externalizing (e.g., aggressive and rule-
beyond that contributed by other measures within an assessment breaking behaviors). Further, Table 3 reports relations between the
battery (i.e., incremental validity; Johnston & Murray, 2003). For informants reports and a single correlate, namely childrens re-
example, in a review of evidence-based assessments for ADHD, ferral status (i.e., differentiating clinic referred vs. nonreferred
Pelham, Fabiano, and Massetti (2005) examined the incremental children).
validity of rating scales and structured interviews for the assess- In Table 3 we report two sets of findings regarding the ASEBA
ment of childhood ADHD. Despite evidence of the construct forms. Specifically, we present statistical estimates reported by
validity of each measure, Pelham et al. (2005) found rating scales Achenbach and Rescorla (2001). Specifically, we report Pearson r
to be the most incrementally valid for assessing ADHD, and use of correlation coefficients among parent, teacher, and child reports,
a DSMIV-based structured interview did not contribute unique which we derived from Table 9-2 from Achenbach and Rescorla
information beyond rating scales. Yet, as we discuss below, issues (2001). As reported in Achenbach and Rescorla (2001), each value
with criterion contamination hinder the interpretability of prior denotes a significant relation between informants reports, p .05.
work on the incremental validity of measures used in clinical child Consistent with meta-analytic findings reviewed previously, levels
and adolescent assessments. of informant correspondence (a) ranged from low-to-moderate in
Mental health domain or focus of assessments. Another magnitude, and (b) were larger for correspondence on reports of
factor that can affect the incremental validity of a predictor is the externalizing relative to internalizing problem behaviors. Based on
domain being assessed or the goal of assessment. The incremental these cross-informant correspondence estimates, we calculated the
validity of an informants report may vary dramatically when common variance shared by multiple informants for the internal-
assessing externalizing symptoms versus internalizing symptoms, izing, externalizing, and total problem scores on the ASEBA
or when the goal of the assessment is to reach a single diagnosis forms. We made these calculations based on regression common-
versus ruling out a diagnosis (see Table 2). For instance, returning ality analyses illustrated by Cohen, Cohen, West, and Aiken
to Pelham et al.s (2005) review, research suggests that a single (2003) and Nimon (2010). These estimates (ranging from 0.8% to
teacher rating of ADHD sufficiently captures the presence of 3.0%; Table 3) reveal that the proportion of variance in problem
ADHD for research purposes. However, ratings from both parents scores that is captured by all informants is very small.
and teachers are important if the goal of the assessment is to reach We also report the percentages of variances in informants
a clinical diagnosis, given the diagnostic requirement of symptom reports that are explained by childrens referral status. We derived
displays across multiple contexts. Similarly, rating scales may these percentages of variance from Table 10-3 from Achenbach
suffice for identifying ADHD symptoms, and yet ruling out diag- and Rescorla (2001). Specifically, the authors conducted multiple
noses other than ADHD (i.e., for purposes of differential diagno-
sis) requires gathering additional information regarding other men-
3
tal health domains (e.g., anxiety and conduct problems; Pelham et Additional factors such as the patients age and gender, cultural char-
al., 2005).3 As with incremental validity research on measurement acteristics of the assessment setting, and the base rates of the symptoms
may affect the generalizability of information regarding the incremental
methods (e.g., rating scales vs. structured interviews), we discuss validity of assessment methods. Therefore, these other issues must be taken
below issues regarding how criterion contamination hinders the into account before generalizing findings from one study to new samples or
interpretability of prior work examining incremental validity and patient populations (Johnston & Murray, 2003). An example of this effect
use of multiple informants reports (e.g., parents vs. teachers) and can be seen in work examining the incremental validity of multiple infor-
mants reports of peer victimization in the prediction of relational adjust-
measurement of multiple mental health domains. ment in elementary school-age children (Ladd & Kochenderfer-Ladd,
Criterion variables and criterion contamination. Crucially, 2002). In this study, the unique information contributed by reporters varied
estimates of incremental validity for multi-informant clinical child as a function of childrens grade level (see Table 2).
874 DE LOS REYES ET AL.

Table 3
Cross-Informant Correspondence and Relations Between Referral Status and Informants
Reports, Based on Parent, Teacher, and Youth Forms of the Achenbach System of Empirically
Based Assessments (Achenbach & Rescorla, 2001)

Correlation estimates of cross-informant correspondence

Psychopathology domain CBCL-YSR CBCL-TRF YSR-TRF


Internalizing problems .48 .21 .17
Externalizing problems .56 .36 .28
Total problems .54 .35 .21
Estimated common variance attributable to multiple informantsa
Psychopathology domain
Internalizing problems 1.3%
Externalizing problems 3.0%
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Total problems 0.8%


Estimated variance in informant accounted for by referral status,
This document is copyrighted by the American Psychological Association or one of its allied publishers.

converted from point biserial Pearson r correlations


Psychopathology domain CBCL YSR TRF
Internalizing problems 26% 11% 14%
Externalizing problems 33% 17% 19%
Total problems 36% 15% 26%
Estimated correlation with referral status for a single informant
Psychopathology domain CBCL YSR TRF
Internalizing problems .51 .33 .37
Externalizing problems .57 .41 .44
Total problems .60 .39 .51
Estimated maximum possible correlation with referral status for
multiple informantsb
Psychopathology domain
Internalizing problems .72
Externalizing problems .85
Total problems .88
Note. CBCL Child Behavior Checklist (i.e., Parent Report Form); YSR Youth Self-Report; TRF
Teacher Report Form.
a
Estimates based on regression commonality analyses illustrated by Cohen et al. (2003) and Nimon (2010),
using cross-informant correlations reported above. b By maximum, we mean that we estimated an upper limit
correlation between informants reports and referral status, assuming unique variance among the informants
reports.

regression analyses using referral status as a predictor variable across multiple informants and the percentage of variance ac-
along with a series of childrens demographic variables as predic- counted for by referral status for the parent, teacher, and child
tor variables (i.e., age, socioeconomic status, ethnicity). In separate reports (Cohen et al., 2003). By maximum, we mean that we
analyses, each of the parent, teacher, and child reports served as estimated an upper limit correlation between informants reports
dependent variables. Following these analyses, the authors con- and referral status, assuming unique variance among the infor-
verted the regression coefficients representing unique effects of mants reports.
childrens referral status on informants reports to Fishers z, In comparing the coefficients observed for individual and multi-
averaged these Fishers zs across gender and age groups, and informant correlations, a clear pattern of findings emerges. Spe-
converted these final Fishers zs to Pearson rs. The percentages of cifically, one observes greater magnitudes of correlations between
variances reported in Table 3 are squared estimates of these referral status and an approach based on the multi-informant
Pearson rs. These values denote significant differentiation between parent, teacher, and child reports, relative to an approach based on
referred and nonreferred children for all informants reports (p use of any one of the individual parent, teacher, or child reports.
.01), with lower scores for nonreferred children relative to referred Overall, when using reports that hold item content and scaling
children. That is, each report (parent, teacher, child) significantly constant across informants and in relation to assessments of chil-
related to childrens referral status. Based on these percentages of drens referral status, multiple informants reports of childrens
variances estimated by Achenbach and Rescorla (2001), we esti- internalizing and externalizing concerns may each relate to referral
mated correlations between referral status and the individual in- status. However, in each case (i.e., relations between referral status
formants reports, as well as a total correlation between referral and internalizing, externalizing, or total broadband problems), a
status and the three informants reports. Specifically, to estimate multi-informant assessment approach yields greater correlations
correlations for individual informants, we took the square root of with referral status than an approach based on any one individual
the percentage of variance accounted for by referral status sepa- informants report. In light of our findings, our approach to illus-
rately for parent report, teacher report, and child report. To esti- trating the incremental validity of multi-informant assessments
mate maximum possible correlations for the multi-informant re- evidenced a number of key strengths. Specifically, each infor-
ports, we took the square root of the sum of the common variance mants report was assessed using the same reporting format (i.e.,
VALIDITY OF MULTI-INFORMANT ASSESSMENT 875

survey checklist) and item content, and incremental validity was clinicians were kept blind to all clinical information collected apart
evaluated using a criterion measure that evidenced relatively little from their own clinical ratings or tools used to make clinical decisions
criterion contamination. Yet, one limitation is that the findings in (e.g., diagnostic interview information; for an exception, see Carlson
Table 3 were based on the scale scores reported by Achenbach and & Blader, 2011). That being said, studies using clinicians reports as
Rescorla (2001) for the complete parent, teacher, and child a criterion variable when clinicians also had access to the multi-
ASEBA scales (Leslie Rescorla, personal communication, Sep- informant assessments evaluated for incremental validity would have
tember 10, 2014). Although these scales share many items in been likely to compound the interpretative problems raised by clini-
common, each contain items that the other scales do not (e.g., cians having knowledge of the identity of the referral source (e.g.,
teacher report contains items to assess attentional concerns that the Bossuyt et al., 2003; Whiting et al., 2011).
other reports do not contain). Consequently, we suspect the cross- Regardless of the mechanisms underlying reliance on a single
informant correspondence estimates reported in Table 3 are con- informant to make clinical decisions, this practice has important
servative and would be slightly higher if these estimates were implications for clinical decision-making. That is, presumably
based strictly on items common across the informants instru- approaches to clinical decision-making that rely on one infor-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

ments. Nevertheless, previous work examining cross-informant mants report more so than other informants reports most often
This document is copyrighted by the American Psychological Association or one of its allied publishers.

correspondence specifically on ASEBA items that were common take the form of disjunctive decision-making strategies (i.e., OR
across informants reports found low-to-moderate correspondence rule; for a review see Piacentini, Cohen, & Cohen, 1992), whereby
between informants reports (e.g., De Los Reyes, Youngstrom the informant relied upon most, for instance, to plan treatment for
et al., 2011a, 2011b). a patient might also be identifying mental health concerns that go
In light of the findings in Table 3, a second question arises: uncorroborated based on other informants reports. A strength of
What happens when a validity criterion suffers from criterion this approach is that it can increase access to mental health services
contamination, as described by Garb (2003)? Indeed, in some of for patients with impairing symptoms because only a single infor-
the incremental validity studies in Table 2, the validity criterion mant would have to report concerns for the patient to receive care.
consisted of clinicians diagnoses based, in part, on information However, relying on a single report introduces considerable mea-
provided by multiple informants, indicating nonindependence be- surement error in decision-making and may lead to the false
tween the informants reports and the criterion measure. In addi-
positive test outcomes described previously (e.g., Hunsley, 2003;
tion, in some of the studies in Table 2, when clinicians gathered
Swets et al., 2000; Youngstrom, Findling, & Calabrese, 2003).
information from multiple informants they also knew which infor-
Yet, to what extent do the data indicate that such clinical decisions
mant referred the patient for clinical services and which informants
occur?
served as collateral informants. Importantly, in these studies
In Table 4, we summarize the findings of several studies on the
clinicians would have been likely to encounter instances in which
relation between informant discrepancies and decisions that clini-
informants reports pointed to inconsistent conclusions regarding
cians make regarding ratings of childrens functional impairment,
patients concerns and whether these concerns warranted addi-
diagnosis, treatment response, and treatment planning. For four of
tional clinical services (see also Yeh & Weisz, 2001).
the studies described in Table 4 (De Los Reyes, Alfano et al.,
Two lines of research indicate that the designs of the incremen-
2011; DiBartolo et al., 1998; Grills & Ollendick, 2003; Young-
tal validity studies described in Table 2 may compromise inter-
pretations of research on the incremental validity of multi- strom, Findling, & Calabrese, 2004b), an overall pattern of find-
informant assessments. First, mental health professionals may ings emerges: When correspondence between parent reports and
view specific informants as optimal for assessing specific mental child self-reports is low, clinicians tend to make diagnostic deci-
health concerns, such as viewing teachers as better informants sions and ratings of treatment response that correspond more with
than parents for assessing child hyperactivity (e.g., Bird, Gould, & parent reports than child self-reports. For three studies (Brown-
Staghezza, 1992; Loeber, Green, Lahey, & Stouthamer-Loeber, Jacobsen, Wallace, & Whiteside, 2011; Hawley & Weisz, 2003;
1989; Loeber, Green, & Lahey, 1990). Kramer et al., 2004), whether clinicians corresponded in their
Second, compounding issues about optimal informants is the clinical decisions more with parent reports or child self-reports
fact that a single informant (e.g., parent) usually initiates a pa- depended on the measurement method or problem domain as-
tients clinical services (Hunsley & Lee, 2010). Among other sessed. For instance, given high disagreement among parents,
pieces of clinical information, the referral informant provides an patients, and clinicians in terms of which problems to target in
assessor with a referral question (e.g., Does the patient evidence therapy, clinicians decisions regarding therapeutic targets agreed
concerns with depressive mood symptoms?; Garb, 1998). This more with parent reports than patient self-reports when the target
referral question typically informs assessors initial hypotheses as concerned a patient problem (e.g., childrens anxiety, mood, at-
to the patients concerns and thus the key goal of the assessment tention concerns; Hawley & Weisz, 2003). However, in this same
(Croskerry, 2003). Thus, when clinicians make decisions based on study, clinicians agreed more with patient self-reports than parent
multi-informant assessment outcomes, their decisions may align reports when the target concerned problems with the family or the
with referral informants reports more so than other informants environment.
reports, even without compelling evidence to indicate the validity When interpreting the findings summarized in Table 4, it is
of this approach. Note that clinicians prior knowledge of which important to further scrutinize the studies and the quality of the
informant served as the referral source and which informants informants reports. Indeed, one possible interpretation of these
served as collateral informants should be distinguished from cli- findings is that clinicians rightfully identified parents as the most
nicians having access to the outcomes of multi-informant assess- valid informants for the particular mental health concerns as-
ments prior to making clinical decisions. Studies rarely note whether sessed. A key characteristic of two of the studies in Table 4 is that
876 DE LOS REYES ET AL.

Table 4
Summary of Studies on the Relation Between Informant Discrepancies in Parent and Child Reports of Childrens Mental Health and
Clinical Decision-Making

Did clinicians decisions correspond


Mean patient to a greater extent with one
Author group age in years informant relative to other
(year) (age range) Description informants?

Brown-Jacobsen, 12.3 (718) When assessing childrens anxiety symptoms, both parent- and child self- Yes, but clinicians decisions
Wallace, & reported symptoms were significantly related to the clinicians corresponded more with parent
Whiteside diagnostic impression. For parent and child questionnaire reports, reports relative to child self-
(2011) considered simultaneously, only parent reports uniquely related to the reports, depending on the
clinicians diagnostic impression of the child. Conversely, when only measurement methodology
the parent or the child endorsed an anxiety disorder diagnosis, the
clinician made a diagnosis approximately 30% of the time, with the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

prevalence based on parent-reported symptoms (28.6%) slightly higher


This document is copyrighted by the American Psychological Association or one of its allied publishers.

than child-reported symptoms (26.8%).


De Los Reyes, 11.6 (716) When assessing pre- and post-treatment social functioning in a sample of Yes (Parent Child)
Alfano, & children, clinician ratings of treatment improvement more strongly
Beidel (2011) agreed with parent-reported post-treatment functioning, relative to
childrens self-reported functioning.
DiBartolo, Albano, 13.9 (817) Overall, clinicians weighted parent reports more strongly than they did Yes (Parent Child)
Barlow, & child self-reports when determining the severity of child social phobia.
Heimberg
(1998)
Grills & Ollendick 10.7 (717) When assessing childrens anxiety symptoms, a clinicians consensus Yes (Parent Child)
(2003) diagnoses more highly correlated with parent reports, relative to child
self-reports.
Hawley & Weisz 11.5 (717) When asked to identify target problems in outpatient treatment, low Yes, but clinicians decisions
(2003) agreement was found within the childparenttherapist triad. More than corresponded more with parent
three fourths of the participating triads disagreed on target problems at reports or child self-reports,
the beginning of treatment. Therapists exhibited significantly higher depending on the problem
agreement with parents versus children on the majority of target domain
problems except for family and environmental problems where the
therapists agreed more with child self-report than parent report.
Kramer et al. 13.7 (1118) Parentadolescent agreement on reports of functional impairment was Yes, but clinicians decisions
(2004) low, and clinician agreement was variable. Overall, clinicians rated corresponded more with parent
impairment as more severe when both the parent and adolescent reports or adolescent self-reports,
endorsed the problem; however, when a problem was reported a single depending on the problem
informant, the weight given to the report by the clinician varied by domain
problem type. For example, for problems endorsed by a parent alone,
clinicians generally weighted the severity as equal to a problem
endorsed by both the parent and adolescent. In contrast, some areas of
impairment such as sibling issues or school work were rated as less
severe when only reported by the adolescent versus by both the parent
and adolescent, whereas adolescent-only reports of illegal behavior
were rated more serious than parent-only reports.
Youngstrom, 14.1 (1117) Clinicians ratings of adolescent manic symptoms were more strongly Yes (Parent Child)
Findling, & correlated with parent reports relative to adolescent self-reports.
Calabrese
(2004b)

the measures administered to parents and children were identical to Jacobsen et al., 2011; De Los Reyes, Alfano et al., 2011; DiBartolo
or slightly modified versions of those used in our illustration of the et al., 1998; Grills & Ollendick, 2003). With child anxiety con-
incremental validity of multi-informant assessments (i.e., Hawley cerns, it is important to note that decades of research attest to
& Weisz, 2003; Youngstrom et al., 2004b). One of the other the reliability and validity of child self-reports for assessing these
studies administered functional impairment measures to parents concerns (for reviews, see Silverman & Ollendick, 2005; Tulbure
and adolescents that share many of the characteristics of the et al., 2012). As mentioned previously, the lack of correspondence
measures used in our Table 3 illustration (e.g., similar item con- between parent reports and patient self-reports of child anxiety
tent, focus on last 6 months of functioning, Likert-type scaling; does not necessarily indicate that patients provide self-reports of
Kramer et al., 2004). Thus, differential measurement validity be- poor quality (e.g., Comer & Kendall, 2004; De Los Reyes, Aldao
tween parent and child reports may not parsimoniously explain et al., 2012). In fact, even when parent reports and child self-
findings from Hawley and Weisz (2003) and Youngstrom and reports of child anxiety yield low correspondence, these reports
colleagues (2004b). may all exhibit identical factor structures (Baldwin & Dadds,
The remainder of the studies described in Table 4 focused on 2007). Most crucially, recent work suggests that children provide
assessments of child anxiety symptoms and diagnoses (Brown- incrementally valid self-reports of anxiety, relative to parent re-
VALIDITY OF MULTI-INFORMANT ASSESSMENT 877

ports, in the prediction of child anxiety disorder diagnoses (Villab 5). Here, findings may reflect differential psychometric properties
et al., 2012). among measures, rather than evidence supportive of the incremen-
Overall, when mental health professionals encounter discrepant tal validity of reports.
assessment outcomes from multiple informants reports, they make Second, studies have not adequately controlled for criterion
clinical decisions that tend to correspond with a single informants contamination (see Table 5). For instance, many studies evaluate
report to a greater extent than the reports of other informants incremental validity using clinicians diagnostic decisions (i.e., via
involved in the assessment. Sometimes in the cases of child pa- structured interviews administered to informants) as the key cri-
tients, this informant may be a nonparental informant (e.g., pa- terion by which to compare informants reports. Yet, these crite-
tient); however, most often this informant is the parent, or the rion measures were derived through clinical interviews made by
informant who mental health professionals rely on most to initiate evaluators who had access to the reports of the same informants
childrens clinical services. Consequently, study design consider- being evaluated for incremental validity (see Table 5). Based on
ations in incremental validity research preclude firm conclusions the findings reported in Table 4, this is an important consideration:
as to the support for use of the multi-informant approach in clinical In the presence of discrepant informants reports, clinical evalua-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

child assessments. To further illustrate these issues, in Table 5 we tors tend to make clinical decisions that most closely correspond
This document is copyrighted by the American Psychological Association or one of its allied publishers.

describe the characteristics of the incremental validity studies with the informant initiating clinical services. In these studies, one
reviewed in Table 2. These characteristics highlight three issues interpretation is that the referral informant (e.g., parent) provided
concerning prior work. First, studies on use and interpretation of incrementally valid information not because of the validity of their
multiple informants reports require, at a minimum, informants to reports per se, but because they informed assessors hypotheses
complete reports that hold measurement content, scaling, and regarding the referral question and thus rationale and structure of
response options constant across reports to control the methodol- the clinical evaluation (Croskerry, 2003; Garb, 1998).
ogy through which informants provide reports (De Los Reyes, Third, contextual variations in patients mental health concerns
Thomas et al., 2013). However, at times incremental validity may influence the extent to which a given informants report is
studies involve informants completing reports via measures that found to be incrementally valid (see Figure 2). For example,
contain different items, scaling, and response options (see Table consider the idea that a given mental health concern, such as social

Table 5
Summary of Characteristics in Studies Examining the Incremental Validity of a Multi-Informant Approach to the Assessment of
Mental Health

Contextual
Informants examined for variations in
Descriptive information incremental validity Criterion measures assessed behavior
Criterion Independent
Use of the same independent from assessment of
measure format informants prevalence of
to gather Information sources examined for Objective contextual variations
Mental health informants used for criterion incremental criterion in participants
domain Author group (year) Informants reports measures validity useda behavior

ADHD Power et al. (1998) P, T Combined diagnostic X X X


interview and T
report
Smith, Pelham, Gnagy, C, T / Independent X
Molina, & Evans (2000) observers
ADHD subtypes
and ODD Owens & Hoza (2003) P, T P, T X X X
Anxiety Villab, Gere, Torgersen, C, P Diagnostic interview X X X
March, & Kendall (2012)
Bipolar Disorder Youngstrom et al. (2004a) C, P, T / Diagnostic interview X X X
ODD Angold & Costello (1996) C, P Diagnostic interview; X X X
P and C interview
reports of
impairment,
parental burden,
and service use
Relational Ladd & Kochenderfer-
Adjustment Ladd (2002) C, P, T, Peer / C, P, T, Peer X X X
Social Skills and
Academic Kwon, Kim, & Sheridan
Competence (2012) T, Peer X C, T, Peer / X X
Note. study utilized characteristic described by column header; / study utilized characteristic described by column header for some but not all
measures; X study did not utilize characteristic described by column header; C Child; P Parent; T Teacher; ADHD attention-deficit/
hyperactivity disorder; ODD oppositional defiant behaviors.
a
Criterion based on objective measure (e.g., record reviews, behavioral observations, or behavioral measures).
878 DE LOS REYES ET AL.

anxiety (Bgels et al., 2010), may manifest itself differently from they contextually vary in where they display concerns, then one
one patient to the next (see Figures 2a, 2b, and 2c). Further, should observe large discrepancies among the reports of infor-
consider that a researcher conducts an incremental validity study mants who observe patients behavior in different contexts. Alter-
using three informants reports, namely parents, teachers, and natively, if an independent assessment reveals that patients display
peers. In this study, patients in the sample only displayed mental concerns consistently across contexts, then one should observe
health concerns within a single context (i.e., home), and thus only high correspondence among informants reports of patients con-
one of the three informants (i.e., parents) primarily observed cerns. These patterns would, in effect, support the construct valid-
patients in this context. The results of this study may yield incre- ity of interpreting multi-informant assessments as reflecting the
mental validity evidence in support of parent reports and not the extent to which patients contextually vary in displays of their
teacher and peer reports, but only because parents had access to concerns. To this end, we review research taking exactly this
observing the behaviors assessed, and not teachers and peers. approach to understanding the construct validity of scores taken
Unfortunately, although some of the incremental validity studies in from multi-informant clinical child assessments.
Table 5 included independent measures (i.e., beyond the infor-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

mants reports examined) of patients mental health concerns, none


This document is copyrighted by the American Psychological Association or one of its allied publishers.

assessed and reported the prevalence rate of context-specific dis- Construct Validity in Multi-Informant Clinical
plays of patients mental health concerns. Stated another way, Child Assessments
previous incremental validity studies have not reported estimates
Until recently, the best evidence in support of the construct
of which patients displayed their concerns within specific contexts
validity of multi-informant assessments came from the Achenbach
(e.g., home or school) or across multiple contexts. Without the
et al. (1987) meta-analysis, as described previously. However,
availability of these data, one cannot address the methodological
these findings provide an incomplete picture of construct validity.
confound that the studys findings in support of a given infor-
That is, similar to the incremental validity research reviewed
mants report may have arisen because of a lack of variation in the
contexts in which patients displayed concerns (cf. Figures 2a and previously, strong construct validity evidence would involve stud-
2b). Consequently, findings from such a study might not general- ies that compare the points of convergence and divergence be-
ize to other samples of patients who display qualitatively distinct tween informants reports to independent and contextually sensi-
contextual variations in mental health concerns. In sum, study tive measures of patients concerns (e.g., DB-DOS; see Wakschlag
design issues preclude our ability to discern whether multi- et al., 2010). With this independent assessment, one can examine
informant clinical child assessments demonstrate incremental va- whether an informants report relates most strongly with behav-
lidity. ioral observations of patients within the observational context that
most closely matches their own context of observation (e.g., parent
at home vs. teacher at school; see Figure 2). Interestingly, recent
Construct Validity of Multi-Informant Mental research has expanded upon work by Achenbach et al. (1987) by
Health Assessments examining multi-informant assessments in reference to indepen-
dent assessments of contextual variations in patients mental health
concerns.
Theoretical and Methodological Approaches to
Two studies support the construct validity of the multi-
Construct Validity informant approach. First, a study of 327 preschool children aged
Our perspective on the construct validity of multi-informant 35 years found that children varied in the contexts in which they
assessments draws from the definitions provided by Borsboom, displayed disruptive behavior symptoms (De Los Reyes et al.,
Mellenbergh, and van Heerden (2004). We also draw from core 2009). Specifically, on the DB-DOS described previously, 29.4%
principles of convergent and divergent validity, and the multitrait/ of the children displayed disruptive behavior exclusively within
multimethod (MTMM) matrix (e.g., Campbell & Fiske, 1959; interactions with parental adults, and 15% displayed disruptive
Westen & Rosenthal, 2003, 2005). Specifically, in a multi- behavior exclusively within interactions with nonparental adults
informant assessment of a given domain of patients concerns (i.e., unfamiliar clinical examiners). Further, 8.8% of the children
(e.g., aggressive behavior), scores from any one informants report displayed disruptive behavior across interactions with parental and
may validly reflect levels of concerns on that domain. However, nonparental adults, and 46.8% did not display disruptive behavior
one informants report alone cannot estimate whether patients within any of these interactions. Thus, De Los Reyes, Henry,
display concerns across contexts (e.g., Kraemer et al., 2003); Tolan, and Wakschlag (2009) examined a heterogeneous sample of
knowledge of multiple contexts requires multiple informants. If we children whose disruptive behavior was specific to home rather
were to use MTMM matrix terminology (see Campbell & Fiske, than nonhome contexts (Figure 2a), specific to nonhome rather
1959), our focus is on assessments in which (a) multiple infor- than home contexts (Figure 2b), or nonspecific or displayed across
mants complete reports about the same domain (i.e., mono-do- home and nonhome contexts (Figure 2c), in addition to children
main); (b) each informants report reflects observations within a who were unlikely to display disruptive behavior in any of the
single context (i.e., mono-context); and (c) patterns of scores contexts. Further, in this study, parents reports of childrens
across multiple informants reflect contextual variations in patients disruptive behavior more closely matched how children behaved
concerns (i.e., multicontext). By examining patterns of scores from within interactions with parental adults than with nonparental
these mono-domain/multicontext assessments, one can assess adults, whereas teachers reports more closely matched how chil-
the construct validity of multi-informant assessments. Specifically, dren behaved within interactions with nonparental adults than with
if an independent assessment of patients concerns reveals that parental adults.
VALIDITY OF MULTI-INFORMANT ASSESSMENT 879

Second, findings consistent with those of De Los Reyes et al. might take the form of studies constructed to address research
(2009) have been observed in a study of early adolescents (N questions such as, For mental health domain X and patient Group
123; mean age 13.30), which focused on links between parents Y, do multi-informant assessments consisting of parent, teacher,
and teachers reports of aggressive behavior and the social expe- and child reports provide incrementally valid information, relative
riences that tended to elicit these behaviors (Hartley et al., 2011). to use of any one informants report? Third, our review of the
Specifically, Hartley, Zakriski, and Wright (2011) found that as construct validity of multi-informant assessments focused on stud-
parents and teachers provided increasingly similar reports of chil- ies examining the match between informant discrepancies in
drens aggressive behavior, the similarities increased between the reports of patients concerns and independent, context-sensitive
kinds of social experiences in which parents and teachers reported assessments of patients behavior. Two studies support the ability
observing the children exhibiting aggressive behavior (e.g., peer of multi-informant assessments to reflect contextual variations in
interactions or receiving instructions from adult authority figures). patients behavior for parent and teacher reports of (a) preschool-
Overall, when assessing children for displays of observable mental ers disruptive behavior, and (b) early adolescents aggressive
health concerns (i.e., aggressive and disruptive behavior), prelim- behavior. Thus, studies support the construct validity of the multi-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

inary evidence indicates that informants who differ in where they informant approach, and yet for a limited set of informants, mental
This document is copyrighted by the American Psychological Association or one of its allied publishers.

observe children (i.e., parents and teachers) provide reports that health domains, and patients developmental periods.
map onto contextual variations in such concerns.
Recommendations for Clinical Practice
Discussion
A key goal of this article was to provide individuals conducting
Main Findings clinical work guidance on when the evidence supports interpreting
the outcomes of multi-informant assessments as reflections of
In this article, we reviewed research on the validity of the contextual variations in patients mental health concerns. To this
multi-informant approach to clinical child assessment. We con- end, we identified evidence indicating that at times, multi-
ducted a quantitative review of 341 studies reporting estimates of informant clinical child assessments pinpoint the specific contexts
cross-informant correspondence in reports about childrens mental in which patients display mental health concerns. However, cur-
health (see Table 1), representing the last quarter-century of re- rently we do not have best practices for using and interpreting
search on the topic. We used the findings from this review as a multi-informant assessments in clinical work. In line with these
backdrop for discussing recent conceptual work on how multi- gaps in best practices, we can make three recommendations for
informant assessments may reveal information about the specific applying the multi-informant approach to clinical work.
contexts in which patients display mental health concerns. Guided Using multi-informant assessments to personalize patient
by this conceptual work, we made specific predictions about the care. First, a key implication of the work reviewed above is that
incremental and construct validity of multi-informant assessments. practitioners may observe significant within- and between-patient
Based on these predictions, we evaluated whether the evidence variations in terms of the specific contexts in which patients
supports using multi-informant assessments to measure context- display concerns (see Figure 2). This points to a key idea: Not all
specific displays of childrens concerns. We sought to inform patients require or may be suited for the same mental health
future research, as well as make recommendations for clinical assessment procedures administered in a single way (e.g., NIMH,
practice on using and interpreting multi-informant assessments. 2008). In line with this view, providing proper patient care may
We made three findings. First, the strongest evidence in support involve practitioners seeking to personalize assessments to opti-
of the multi-informant approach comes from studies of cross- mally fit or address patients unique needs (see Figure 2).
informant correspondence. Our quantitative review indicated that In an effort to innovate personalized approaches to mental
multiple informants reports share little variance with each other health care, the NIMH recently released a strategic plan for its
(see Table 1). Further, moderators of cross-informant correspon- funding priorities, which includes broadening outcome measure-
dence indicated that informants reports tend to correspond at ment so as to tailor assessments and interventions to patients
higher levels when informants (a) provide reports about concerns unique needs (Strategy 3.2; NIMH, 2008). Consequently, if a
that are relatively easy to observe (e.g., externalizing vs. internal- clinician wishes to administer a mental health assessment that is
izing concerns); (b) observe childrens concerns from the same personalized to the unique needs of each patient, the work re-
setting (e.g., mother and father reports vs. parent and teacher viewed above indicates that this assessment should be able to
reports); and (c) complete reports using dimensional measures capture information about the specific contexts in which the pa-
(i.e., relative to categorical measures). Second, research on the tient requires care. In this respect, future work may inform the
incremental validity of multi-informant assessments is relatively development of contextually sensitive assessments that improve
underdeveloped relative to research on cross-informant correspon- patient care.
dence. Importantly, our review should not be taken as an indication For example, as mentioned previously social anxiety patients
that multi-informant assessments cannot be incrementally valid. vary in that some experience social anxiety across social contexts
Rather, incremental validity studies generally have been con- and others within specific contexts (i.e., public speaking; Bgels
structed to address research questions such as, Which specific et al., 2010). This creates the potential for patients to display
informant(s) provide incrementally valid reports within assess- anxiety in some contexts but not others. Consequently, a practi-
ments of mental health domain X and patient Group Y? In fact, tioner ought to administer treatment planning assessments person-
there is a dearth of literature systematically evaluating which sets alized to the unique needs of the patient. For instance, assessments
of informants provide incrementally valid reports. Such work can be tailored to classify different patients who display social
880 DE LOS REYES ET AL.

anxiety mostly during public speaking (i.e., performance anxiety), the contextually sensitive observational assessments developed to
or both preceding (i.e., anticipatory anxiety) and during public assess disruptive behavior among preschool children (Wakschlag
speaking. Based on these assessments, treatments can be tailored et al., 2010). Let us assume that the data confirmed the practitio-
to patients experiencing predominantly performance anxiety (e.g., ners hypotheses: observations of greater parent-reported levels of
behavioral exposure) and/or anticipatory anxiety (e.g., emotion hyperactivity, relative to teacher report. If so, the independent
regulation strategies). Thus, practitioners might leverage assess- assessment would allow the practitioner to make an informed
ment methods to identify and implement treatment techniques decision as to whether the parent and teacher reports varied for
tailored to each patient. Such assessments may result in cost- meaningful reasons. For instance, the independent assessment
effective symptom reduction and improved functioning, relative to might have revealed the presence of higher levels of hyperactivity
existing treatments that may or may not effectively target the and less effective behavior management strategies at home relative
unique needs of each patient. to school. The key point to be taken from this example is that
Developing new multi-informant assessment procedures. assessment procedures guiding clinical decision-making would
Second, new methods for using and interpreting multi-informant greatly benefit from a priori predictions of patterns of convergence
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

assessments might assist in developing the personalized assess- and divergence between informants reports. These predictions can
This document is copyrighted by the American Psychological Association or one of its allied publishers.

ment approaches we just described. To this end, recent theoretical inform the gathering of independent qualitative and quantitative
work on interpreting multi-informant assessment outcomes in re- data about the contexts in which patients display concerns. In turn,
search (e.g., OTM; De Los Reyes, Thomas et al., 2013) may assist these data and predictions may result in greater certainty when
in future efforts toward developing evidence-based practices in interpreting the outcomes of multi-informant assessments, result-
interpreting multi-informant assessment outcomes. Specifically, a ing in increased reliability and validity in clinical decision-making.
key feature of the OTM is based on the idea that practitioners have Integrating the OTM with approaches from evidence-based
a great deal of evidence indicating that multiple informants re- medicine. Developing personalized assessment procedures that
ports will often yield discrepant outcomes and in predictable ways leverage multiple informants reports could be further facilitated
(e.g., Achenbach et al., 1987; De Los Reyes & Kazdin, 2005; by incorporating knowledge from other literatures focused on
Kraemer et al., 2003). As a result, the OTM guides practitioners patient-centered models of clinical decision-making. Specifically,
toward (a) hypothesizing patterns of convergence and divergence assessment methods within evidence-based medicine (EBM) often
among informants reports, and then (b) constructing assessments involve forming likelihood judgments for an assessment case,
that directly test these hypothesized patterns. based on whether one or more pieces of information collected by
For example, a practitioner with a referral question involving an an assessor support or fail to support a given conclusion about that
initial assessment for a childs hyperactivity might look to the patients clinical presentation (for a review, see Youngstrom,
research literature on typical rates of reporting discrepancies be- 2013). As an example, consider a practitioner who collects clinical
tween parent and teacher hyperactivity reports. The practitioner information about a patient who may evidence concerns with
could also conduct a short unstructured phone interview with the oppositional defiant disorder (ODD). The difference between
parent and teacher before conducting the assessment. Taking pre- EBM assessments and traditional psychological assessment proce-
cautions not to ask leading questions that might bias informants dures is that in EBM, one assigns specific values to data derived
reports (Groth-Marnat, 2009), the practitioner could probe for from information sources, based on whether these data support one
similarities and differences between the home and school contexts, outcome, relative to an alternative outcome.
such as in how parent and teacher manage the childs behavior For instance, to assess a patient for the presence versus absence
within and across contexts. Using this contextual information as a of an ODD diagnosis, a practitioner might begin by consulting
guide, the practitioner would predict the patterns of information to prior research on risk and associated features of ODD. Based on
be gleaned from parent and teacher reports. The practitioner would prior research, the practitioner would pose a clinical question to
then structure the assessment to make sense of the parent and inform gathering clinical information about the patient, such as,
teacher reports. What are the base rates of inconsistent parenting practices among
Specifically, based on the qualitative data the practitioner might the parents of patients diagnosed with ODD and among nonpa-
expect the patients hyperactivity to vary by contextual demands. tients? The practitioner would administer to the parent a short
These variations might reflect that adult authority figures within survey assessment of inconsistent parenting. The practitioner
the home and school contexts (i.e., parents vs. teachers) differ in would compare the outcomes of this survey against known base
the strategies used to manage the patients behavior, with more rates of inconsistent parenting. Essentially, the practitioner exam-
effective management strategies present in the school context. ines whether the parents inconsistent parenting outcomes
These variations might result in the practitioner predicting infor- matched the outcomes identified in the literature for parents of
mants reports across contexts to vary in levels of hyperactivity, patients diagnosed with ODD, relative to an alternative outcome
with greater levels of hyperactivity reported in the home context, (i.e., outcomes of inconsistent parenting for parents of nonpa-
relative to the school context. The result may be collecting reports tients). The resulting comparisons, referred to as diagnostic like-
of childrens hyperactivity from parents and teachers. To corrob- lihood ratios (DLRs) are essentially estimates of the ratio of true
orate the meaning behind convergence or divergence between positives-to-false positives (e.g., ratio of the base rate of inconsis-
parent and teacher reports, the practitioner might also administer tent parenting practices among the parents of ODD patients/ratio
independent observational assessments of the childs behavior. for parents of nonpatients). These DLRs carry with them interpre-
These observational assessments might include having the child tative conventions similar to odds ratios (OR) calculated within
complete a set of structured and unstructured tasks in the home and logistic regression (see Straus, Glasziou, Richardson, & Haynes,
school contexts. These observational assessments might resemble 2011). That is, DLRs at or below 2 carry little value in terms of
VALIDITY OF MULTI-INFORMANT ASSESSMENT 881

guiding clinical decisions, DLRs of approximately 5 carry rela- ods taken from EBM hold promise for improving the use and
tively greater value, and DLRs of 10 or above may ensure sound interpretation of multi-informant assessments.
confidence in making the particular clinical decision in question
(e.g., ruling in an ODD diagnosis; see also Youngstrom, 2013).
Research and Theoretical Implications
With this ODD example, if the parenting information about the
patient yielded a high DLR, the practitioner might conclude that Addressing methodological issues raised by incremental va-
the patient is quite likely to meet diagnostic criteria for ODD and lidity research. This review has important research and theoret-
begin treatment for the condition. The practitioner may conduct ical implications. Specifically, future multi-informant assessment
additional assessments for the presence of commonly comorbid research would benefit greatly from addressing the methodological
conditions (e.g., ADHD; see Drabick & Kendall, 2010). confounds we discussed previously with regard to prior incremen-
Within an EBM assessment, multiple informants reports might tal validity research. We have three recommendations for address-
prove particularly useful for treatment planning. To continue with ing particularly pressing methodological issues. First, future incre-
our previous example, perhaps DLRs for the presence of an ODD mental validity studies should involve use of measures completed
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

diagnosis were based primarily on useful, easy-to-collect informa- by multiple informants that hold item content, response labeling,
This document is copyrighted by the American Psychological Association or one of its allied publishers.

tion about inconsistent parenting. However, the information that and scaling constant, in order to rule out methodological differ-
informed initial diagnosis did not yield sensitive data on whether ences among reports when drawing inferences about incremental
the patient evidences ODD concerns specific to particular contexts validity. Relatedly, as in research on preschool disruptive behavior
(e.g., home only vs. home and school). Yet, knowledge about the (i.e., DB-DOS; Wakschlag et al., 2010), future research should
context(s) within which the patient evidences concerns would focus on developing measures that can assess contextual variations
greatly facilitate treatment planning. In this sense, a multi- in patients mental health concerns, independently from the infor-
informant assessment consisting of behavioral checklists adminis- mants completing reports about patients. Using these measures,
tered to the patients parents and teachers could yield great utility researchers can gauge, for instance, whether findings supporting
in revealing context-specificity in displays of the patients ODD the incremental validity of parent but not teacher reports were
concerns. Here, the clinical question might be, What is the likeli- driven by the propensity of patients in the sample to display mental
hood that the patients concerns are pervasive across contexts or health concerns only observable by parents (Figure 2a).
specific to the home context? To address this question, the prac- Second, a benchmark or standard does not exist for identifying
titioner would administer survey reports to the patients parent and an incrementally valid informant for assessing particular mental
teacher. Additionally, the practitioner would consult prior work to health concerns for particular patient groups. That is, mental health
identify estimates of patients exhibiting disruptive behavior in researchers have developed a number of guidelines for making
interactions with parental and nonparental adults, perhaps based on evidence-based decisions, most notably with regard to identifying
data from the DB-DOS (e.g., De Los Reyes et al., 2009). Within efficacious treatments (for a review, see Roth & Fonagy, 2005).
these estimates, the practitioner would identify the base rates of However, there does not exist specific criteria for determining how
patients exhibiting cross-contextual displays of disruptive behavior much incremental validity (e.g., predictive power, represented in
on the DB-DOS who also evidenced disruptive behavior across effect size metrics) is sufficient to conclude that a given infor-
parent and teacher reports versus only parent report (i.e., 29% vs. mants report should be included in an assessment, let alone
10%; see Table 5 of De Los Reyes et al., 2009). These estimates whether multiple informants reports ought to be collected versus
would translate to a DLR of approximately 2.9. Based on this DLR only a single report. This is an important consideration, in that one
figure, if the patient in fact evidenced disruptive behavior concerns tends to observe small effects for the incremental validity of a
across parent and teacher reports, the data would point to the given informants report in relation to other informants reports
patient displaying concerns across contexts. Treatment would then (Hunsley & Meyer, 2003).
progress based on the implementation of techniques at both home Third, incremental validity research would benefit from basic
and school. efficacy studies of whether informants can provide incrementally
One additional issue warrants comment. In this EBM example, valid reports when predicting validity criterion measures con-
the only data collected on behalf of the patient were the parent and structed blind or independent of all of the informants reports. This
teacher reports. That is, the practitioner relied on parent and work would focus on asking the question, All things being equal,
teacher report data to make clinical decisions (i.e., determine are scores taken from multi-informant assessments incrementally
whether patient displayed concerns across home and school con- valid, relative to scores taken from any one informant? However,
texts). However, the practitioner turned to independent behavioral like treatment efficacy research, this type of incremental validity
assessment data collected in prior published work to construct the research might involve use of validity criterion measures that do
DLR that was used to interpret the parent and teacher reports (De not reflect clinical indices or decisions made in real-world
Los Reyes et al., 2009). The value in this approach is that the settings. Thus, in tandem with basic efficacy studies, we recom-
practitioner consulted prior published work to interpret the survey mend a focus on incremental validity effectiveness research. This
reports they collected from the patients parent and teacher, and form of incremental validity work would focus on asking the
gauge whether the reports contained information that should guide question, Are scores taken from multi-informant assessments in-
decision-making. This process is quite similar to current practices crementally valid in predicting real-world clinically meaningful
in interpreting clinic data, such as when a mental health profes- criterion measures, relative to scores taken from any one infor-
sional consults published clinical cutoff values for a symptom mant? Essentially, these incremental validity studies would share
scale to determine whether a patient evidences clinically relevant many elements with prior incremental validity work, most notably
mental health concerns. Thus, this illustration highlights that meth- the focus on such validity criterion measures as consensus diag-
882 DE LOS REYES ET AL.

noses and treatment response. However, the difference between independent measures. First, previous work illustrates measures
prior research and the efficacy and effectiveness variants of incre- for assessing contextual variations in patients reactions to author-
mental validity research we propose would be in systematically ity figures (e.g., DB-DOS; see De Los Reyes et al., 2009). Impor-
ruling out the effects of criterion contamination on the outcomes of tantly, a great deal of work points to the relevance of assessing
incremental validity studies. patients interactions with authority figures such as parents and
Using basic mental health research to inform incremental teachers for a number of highly prevalent mental health concerns,
validity research. Much of our discussion of the multi- such as attention and hyperactivity, depressive mood, and disrup-
informant approach makes a fundamental assumption that mental tive behavior (APA, 2013). This indicates the potential for re-
health concerns manifest such that patients display signs of these searchers to develop versions of the DB-DOS tailored to assessing
concerns within some contexts and not others (Figures 2a and 2b) contextual variations in concerns displayed by patients of varying
or across contexts (Figure 2c). The importance of understanding clinical presentations. Further, deficits in communicating and in-
these contextual variations may be relevant to some (e.g., social teracting with others in social settings (i.e., social competence
anxiety, attention and hyperactivity), but not all, mental health deficits) characterize multiple mental health domains assessed and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

concerns. Indeed, displays of some concerns may rarely vary diagnosed among patients across the life span, including attention
This document is copyrighted by the American Psychological Association or one of its allied publishers.

across contexts (e.g., schizophrenia). and hyperactivity, schizophrenia, and social anxiety (APA, 2013).
Our findings highlight the importance of basic research on Along these lines, recent work with adult social anxiety patients
contextual variations in mental health for informing the use and indicates that the administration of laboratory tasks across varying
interpretation of multiple informants reports, particularly with social interaction contexts (e.g., one-on-one social interactions and
regard to incremental validity. Indeed, we discussed previously public speaking) allow for the detection of patients who display
how lack of knowledge on contextual variations in mental health social competence deficits across social interactions or only within
concerns may hinder our ability to draw conclusions on the gen- specific interactions (Beidel, Rao, Scharfstein, Wong, & Alfano,
eralizability of findings from incremental validity studies. That is, 2010; De Los Reyes, Bunnell et al., 2013). Thus, we encourage
the composition of contextual variations in mental health concerns future work on whether independent assessments tailored to assess
in a sample may often dictate the extent to which any one infor- contextual variations in social competence deficits improve inter-
mant provides incrementally valid reports (see Figure 2). Conse- pretability of multiple informants reports of patients of varying
quently, future research must link our basic understanding of developmental periods and clinical presentations.
contextual variations in patients mental health concerns with Second, much of our discussion about using contextually sen-
understanding and interpreting findings in incremental validity sitive independent measures to test the validity of the multi-
research. informant approach involved use of structured behavioral obser-
We previously described observational measures used in multi- vations of patients behavior. This limited view of independent
informant research on preschool disruptive behavior that have assessments extends from the fact that tests reviewed previously of
revealed individual differences among patients in the specific the construct validity of the multi-informant approach relied on
contexts in which they displayed disruptive behavior concerns (De these or similar approaches (De Los Reyes et al., 2009; Hartley et
Los Reyes et al., 2009). We recommend that future incremental al., 2011). However, other modalities may prove as useful, if not
validity studies implement independent, contextually sensitive more useful, than behavioral observations. For instance, rather
measures such as these to estimate contextual variations in pa- than take behavioral ratings of patients performance within clin-
tients concerns. Using these measures, researchers may identify, ically relevant tasks, one could assess patients physiology using
for example, whether the sample was predominantly represented wireless, ambulatory devices (e.g., heart rate monitors; for a re-
by patients who displayed mental health concerns exclusively view, see De Los Reyes & Aldao, 2015). The principles underlying
within the home context (Figure 2a). In assessments of preschool- this approach would function quite similarly to that of previous
ers disruptive behavior, such a patient sample would be likely to examples. Specifically, one might assess changes in physiological
result in parent reports evidencing incrementally valid reports to a arousal across different clinically meaningful contexts to test
far greater extent than other informants reports, such as teachers. whether these contextual changes in physiology match the dif-
Yet, we previously discussed research indicating that preschoolers ferences observed between multiple informants reports. For ex-
may evidence mental health concerns within and across a variety ample, one might assess a patients physiological arousal when
of contexts (e.g., Dirks et al., 2012). Thus, samples of patients completing a frustrating task indicative of the home context (e.g.,
displaying heterogeneity in mental health concerns within and discussing a topic that tends to elicit conflict between parent and
across contexts may yield distinct estimates of incremental validity patient) and also of the school context (e.g., completing a difficult
for parent and teacher reports (see Figures 2a and 2b). Conse- mathematics problem in a noisy room). Here, one might test
quently, the availability of contextually sensitive measures inde- whether patients who display relatively high levels of arousal
pendent of the informants providing reports about patients would during the math problem task and relatively low levels during
provide researchers with important data to rule out important the conflict discussion task also tend to be the patients whose
methodological confounds. Few measures exist for assessing con- teachers report mental health concerns that go uncorroborated by
textually sensitive variations in patients mental health concerns in reports completed by parents.
a way that is independent from the informants providing mental Venturing outside use of tasks administered in traditional clinic
health reports (e.g., using trained laboratory observers or clinical or research settings, there exist a variety of modalities with great
evaluators). This lack of contextually sensitive behavioral assess- potential for testing the validity of the multi-informant approach.
ments may pose challenges to future research. To this end, we have For instance, mental health professionals often seek to integrate
two recommendations on the development and testing of these multi-informant assessments of mental health with such modalities
VALIDITY OF MULTI-INFORMANT ASSESSMENT 883

as estimates of environmental versus genetic contributions to men- mental health, self-reports and clinician ratings may be the most
tal health (Baker, Jacobson, Raine, Lozano, & Bezdjian, 2007; often used sources, yet recently researchers also have focused on
Bartels, Boomsma, Hudziak, van Beijsterveldt, & van den Oord, collecting reports from collateral informants such as spouses or
2007; Derks, Hudziak, Van Beijsterveldt, Dolan, & Boomsma, adult caregivers, as well as trained raters (e.g., clinical interviewers
2006), neurobiological assessments (e.g., electroencephalography; and laboratory observers; Achenbach, 2006). Further, meta-
Curtis & Cicchetti, 2007), salivary assays of the cortisol awaken- analytic reviews of adult mental health assessments have revealed
ing response (Bitsika, Sharpley, Sweeney, & McFarlane, 2014), correspondence levels at low-to-moderate magnitudes, similar to
social network analyses (Neal, Cappella, Wagner, & Atkins, the correspondence levels observed in assessments of children
2011), geographical assessments of neighborhood risks (e.g., high- (Achenbach, Krukowski, Dumenci, & Ivanova, 2005).
crime and economically disadvantaged areas; Odgers, Caspi, As with clinical child assessments, commonly used informants
Bates, Sampson, & Moffitt, 2012), and performance-based neuro- in adult mental health assessments may vary in their unique
psychological tests (e.g., tests of executive functioning; Silver, perspectives on adult patients mental health concerns. For exam-
2014). Future research might involve constructing batteries con-
ple, in multi-informant assessments using patients self-reports and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

sisting of multiple independent assessments that vary in their


clinicians reports about patients, reports might also contextually
This document is copyrighted by the American Psychological Association or one of its allied publishers.

propensity to reflect one informants report to a greater extent than


vary. That is, patients may base their reports on how they behave
other informants reports. For example, a research team interested
in work and home contexts; clinicians may take this information
in assessing externalizing concerns (e.g., disruptive behavior dis-
into account, but their reports also may reflect their trained obser-
orders and ADHD) might predict that geographic assessments of
neighborhood disadvantage near the vicinity of patients homes vations of patients behaviors in clinic settings (Achenbach et al.,
reflect parents reports about patients concerns to a greater extent 2005; Groth-Marnat, 2009). Therefore, when assessing adult pa-
than teachers reports about patients. Conversely, performance- tients, informants selected to provide reports may differ in their
based assessments of patients executive functioning may include opportunities for observing concerns.
use of structured tasks akin to school activities (e.g., completing To this end, there exists preliminary support for the incremental
in-class assignments), and thus these assessments may reflect and construct validity of multi-informant approaches to assessing
teachers reports about patients to a greater extent than parents adult mental health. For example, in the assessment of personality
reports about patients. Armed with this battery, the research team pathology, collateral informants reports may yield incremental
could examine whether the discrepancies between parent and information over-and-above patients self-reports (e.g., Galione &
teacher reports represent ecologically valid reflections of contex- Oltmanns, 2013; Miller, Pilkonis, & Clifton, 2005; Ready, Wat-
tual variations in patients mental health concerns. For example, son, & Clark, 2002). Further, one study supports the construct
instances in which parent reports indicate disruptive behavior validity of multi-informant approaches to assessing adult social
concerns that go uncorroborated based on teacher reports may anxiety. Specifically, in this study patients completed a series of
reflect patients who evidence both a relatively high likelihood of social interaction tasks from which independent raters assessed
living in an economically disadvantaged neighborhood and scores patients social competence deficits (i.e., one-on-one interactions
in the normative range on standardized tests of executive function- and a public speaking task; De Los Reyes, Bunnell et al., 2013).
ing. In sum, multimodal independent assessments may result in a Patients and clinicians also completed reports about patients in-
richer understanding of whether mental health professionals can ternalizing symptoms. Consistent with previous work with chil-
use the outcomes from multi-informant assessments to draw infer- dren (De Los Reyes et al., 2009; Hartley et al., 2011), patients and
ences about the context(s) in which patients display mental health clinicians were more likely to agree than disagree that the patient
concerns. experienced moderate-to-high internalizing symptoms when pa-
Expanding construct validity research on multi-informant tients exhibited social competence deficits consistently (i.e., rather
assessments to additional clinical domains and developmental than inconsistently) across social tasks.
periods. Research on the construct validity of multi-informant
Despite promising findings about the use of multi-informant
assessments has yielded some promising findings. Yet, the find-
approaches in adult mental health assessments, this area of
ings are limited in their relevance to assessments focused on a
work is far underdeveloped relative to multi-informant ap-
circumscribed set of informants, mental health domains, and pa-
proaches to clinical child assessments. Indeed, the quantitative
tient developmental periods. Clearly, the mental health domains
review conducted by Achenbach, Krukowski, Dumenci, and
for which patients concerns may contextually vary far outnumber
the domains within which the available evidence supports the Ivanova (2005) evaluated a sample of 108 studies that provided
construct validity of the multi-informant approach. Thus, we rec- sufficient information to estimate levels of cross-informant
ommend that in the child literature, future research should focus on correspondence, accounting for 0.2% of the 51,000 articles
testing the construct validity of multi-informant assessments: (a) of identified for the review. Additionally, little work exists to
childrens mood and anxiety concerns; (b) that incorporate infor- estimate the consistency of low-to-moderate levels of corre-
mants other than parent and teacher reports (e.g., clinician, patient, spondence among different developmental periods in adulthood
peer reports); and (c) that focus on developmental periods other (e.g., emerging adulthood and mid- to late-adulthood; for an
than preschool and early adolescence. exception, see van der Ende et al., 2012). Thus, future research
Importantly, mental health researchers have made inroads in should focus on construct validity as it relates to (a) assess-
recent years on understanding multi-informant approaches to as- ments of other concerns such as mood; (b) use of collateral
sessing adult mental health (e.g., Oltmanns & Turkheimer, 2009; informants such as spouses and adult caregivers (i.e., in the case
van der Ende, Verhulst, & Tiemeier, 2012). When assessing adult of elderly patients); and (c) specific developmental periods.
884 DE LOS REYES ET AL.


Concluding Comments Allen, J. P., Manning, N., & Meyer, J. (2010). Tightly linked systems:
Reciprocal relations between maternal depressive symptoms and mater-
We provided a conceptual overview of the multi-informant nal reports of adolescent externalizing behavior. Journal of Abnormal
assessment approach, and synthesized cross-informant correspon- Psychology, 119, 825 835. http://dx.doi.org/10.1037/a0021081
dence, incremental validity, and construct validity research litera- American Psychiatric Association. (2000). Diagnostic and statistical man-
tures to assess whether the extant data support taking this approach ual of mental disorders-text revision (4th ed.) (DSMIVTR). Washing-
to assessing childrens mental health. Findings from the last 25 ton, DC: American Psychiatric Association.
years of research on cross-informant correspondence in reports of American Psychiatric Association. (2013). Diagnostic and statistical man-
childrens mental health (see Table 1) essentially mirror findings ual of mental disorders (5th ed.) (DSM-V). Washington, DC: American
Psychiatric Association.
from the preceding 25 years (Achenbach et al., 1987). These robust
Anderson, E. R., & Hope, D. A. (2009). The relationship among social
findings across 50 years of research have allowed us to develop
phobia, objective and perceived physiological reactivity, and anxiety
theoretical principles for interpreting the outcomes of multi- sensitivity in an adolescent population. Journal of Anxiety Disorders, 23,
informant assessments (De Los Reyes, Thomas et al., 2013). We 18 26. http://dx.doi.org/10.1016/j.janxdis.2008.03.011
used these theoretical principles to critically evaluate research on
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Angold, A., & Costello, J. (1996). The relative diagnostic utility of child
This document is copyrighted by the American Psychological Association or one of its allied publishers.

the incremental and construct validity of the multi-informant as- and parent reports of oppositional defiant behaviors. International Jour-
sessment approach. In so doing, we identified a series of limita- nal of Methods in Psychiatric Research, 6, 253259. http://dx.doi.org/
tions in the research designs of previous studies testing the incre- 10.1002/(SICI)1234-988X(199612)6:4253::AID-MPR1703.3.CO;
mental and construct validity of multi-informant approaches to 2-O

assessing childrens mental health. In turn, we must limit our Armistead, L., Klein, K., Forehand, R., & Wierson, M. (1997). Disclosure
conclusions to stating that the best evidence in support of the of parental HIV infection to children in the families of men with
hemophilia: Description, outcomes, and the role of family processes.
validity of the multi-informant approach exists largely for assess-
Journal of Family Psychology, 11, 49 61. http://dx.doi.org/10.1037/
ments of observable mental health concerns (e.g., aggressive and
0893-3200.11.1.49
disruptive behavior). The issues we raised in our review merit
Arseneault, L., Kim-Cohen, J., Taylor, A., Caspi, A., & Moffit, T. E.
further study in patients exhibiting varied mental health concerns (2005). Psychometric evaluation of 5- and 7-year-old childrens self-
(e.g., anxiety, attention problems and hyperactivity, depressive reports of conduct problems. Journal of Abnormal Child Psychology, 33,
symptoms) and across a wider range of developmental periods 537550. http://dx.doi.org/10.1007/s10802-005-6736-5

(e.g., childhood, adolescence, emerging adulthood, older adult- Asher, R. A., & Wakefield, J. A., Jr. (1990). The relationship between
hood). Future research on multi-informant assessments may in- parents personality and their ratings of their preschool childrens be-
form the development of contextually sensitive assessment para- haviors. Personality and Individual Differences, 11, 11311136. http://
digms that mental health professionals can use to personalize dx.doi.org/10.1016/0191-8869(90)90024-L

assessment techniques and interventions to fit patients unique Auerbach, J. G., Gross-Tsur, V., Manor, O., & Shalev, R. S. (2008).
needs. Identifying when multi-informant assessments reveal infor- Emotional and behavioral characteristics over a six-year period in youths
with persistent and nonpersistent dyscalculia. Journal of Learning Dis-
mation about the specific contexts in which patients display mental
abilities, 41, 263273. Retrieved from http://eric.ed.gov/?idEJ791343
health concerns may improve how mental health professionals use
Auger, R. W. (2004). The accuracy of teacher reports in the identification
this information to make clinical decisions, provide patient care, of middle school students with depressive symptomatology. Psychology
and evaluate the outcomes of such care. in the Schools, 41, 379 389. http://dx.doi.org/10.1002/pits.10164

Baker, B. L., Blacher, J., Crnic, K. A., & Edelbrock, C. (2002). Behavior
problems and parenting stress in families of three-year-old children with
References and without developmental delays. American Journal on Mental Retar-
dation, 107, 433 444. http://dx.doi.org/10.1352/0895-8017(2002)
References marked with an asterisk indicate studies included in the 1070433:BPAPSI2.0.CO;2
meta-analysis.
Baker, B. L., & Heller, T. L. (1996). Preschool children with externalizing
Achenbach, T. M. (2011). Commentary: Definitely more than measure- behaviors: Experience of fathers and mothers. Journal of Abnormal
ment error: But how should we understand and deal with informant Child Psychology, 24, 513532. http://dx.doi.org/10.1007/BF01441572

discrepancies? Journal of Clinical Child and Adolescent Psychology, 40, Baker, B. L., McIntyre, L. L., Blacher, J., Crnic, K., Edelbrock, C., &
80 86. http://dx.doi.org/10.1080/15374416.2011.533416 Low, C. (2003). Pre-school children with and without developmental
Achenbach, T. M., Krukowski, R. A., Dumenci, L., & Ivanova, M. Y. delay: Behaviour problems and parenting stress over time. Journal of
(2005). Assessment of adult psychopathology: Meta-analyses and im- Intellectual Disability Research, 47, 217230. http://dx.doi.org/10.1046/
plications of cross-informant correlations. Psychological Bulletin, 131, j.1365-2788.2003.00484.x
361382. http://dx.doi.org/10.1037/0033-2909.131.3.361 Baker, L. A., Jacobson, K. C., Raine, A., Lozano, D. I., & Bezdjian, S.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/ (2007). Genetic and environmental bases of childhood antisocial behav-
adolescent behavioral and emotional problems: Implications of cross- ior: A multi-informant twin study. Journal of Abnormal Psychology,
informant correlations for situational specificity. Psychological Bulletin, 116, 219 235. http://dx.doi.org/10.1037/0021-843X.116.2.219

101, 213232. http://dx.doi.org/10.1037/0033-2909.101.2.213 Baldwin, J. S., & Dadds, M. R. (2007). Reliability and validity of parent
Achenbach, T. M., & Rescorla, L. (2001). Manual for the ASEBA school- and child versions of the Multidimensional Anxiety Scale for Children in
age forms and profiles. Burlington, VT: University of Vermont Research community samples. Journal of the American Academy of Child and
Center for Children, Youth, and Families. Adolescent Psychiatry, 46, 252260. http://dx.doi.org/10.1097/01.chi
Achenbach, T. M. (2006). As others see us: Clinical and research impli- .0000246065.93200.a1

cations of cross-informant correlations for psychopathology. Current Banez, G. A., & Compas, B. E. (1990). Childrens and parents daily
Directions in Psychological Science, 15, 94 98. http://dx.doi.org/10.1111/ stressful events and psychological symptoms. Journal of Abnormal
j.0963-7214.2006.00414.x Child Psychology, 18, 591 605. http://dx.doi.org/10.1007/BF01342749
VALIDITY OF MULTI-INFORMANT ASSESSMENT 885

Barker, E. T., Bornstein, M. H., Putnick, D. L., Hendricks, C., & Suwal- Social Sciences, 3, 126. Retrieved from https://journals.uair.arizona
sky, J. T. D. (2007). Adolescent-mother agreement about adolescent .edu/index.php/jmmss/article/view/16110/16035
problem behaviors: Direction and predictors of disagreement. Journal of Bird, H. R., Gould, M. S., & Staghezza, B. (1992). Aggregating data from
Youth and Adolescence, 36, 950 962. http://dx.doi.org/10.1007/s10964- multiple informants in child psychiatry epidemiological research. Jour-
006-9164-0 nal of the American Academy of Child and Adolescent Psychiatry, 31,
78 85. http://dx.doi.org/10.1097/00004583-199201000-00012
Bartels, M., Boomsma, D. I., Hudziak, J. J., Rietveld, M. J. H., van
Beijsterveldt, T. C. E. M., & van den Oord, E. J. C. G. (2004). Bitsika, V., Sharpley, C. F., Sweeney, J. A., & McFarlane, J. R. (2014).
Disentangling genetic, environmental, and rater effects on internalizing HPA and SAM axis responses as correlates of self- vs. parental ratings
and externalizing problem behavior in 10-year-old twins. Twin Re- of anxiety in boys with an Autistic Disorder. Physiology & Behavior,
search, 7, 162175. http://dx.doi.org/10.1375/136905204323016140 127, 17. http://dx.doi.org/10.1016/j.physbeh.2013.12.011
Bartels, M., Boomsma, D. I., Hudziak, J. J., van Beijsterveldt, T. C., & van Blais, M. A., Hilsenroth, M. J., Castlebury, F., Fowler, J. C., & Baity,
den Oord, E. J. (2007). Twins and the study of rater (dis)agreement. M. R. (2001). Predicting DSM-IV cluster B personality disorder criteria
Psychological Methods, 12, 451 466. http://dx.doi.org/10.1037/1082- from MMPI-2 and Rorschach data: A test of incremental validity.
989X.12.4.451 Journal of Personality Assessment, 76, 150 168. http://dx.doi.org/10.1207/
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

S15327752JPA7601_9
Bartels, M., Hudziak, J. J., Boomsma, D. I., Rietveld, M. J. H., Van

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Beijsterveldt, T. C. E. M., & Van den Oord, E. J. C. G. (2003). A study Blakeley-Smith, A., Reaven, J., Ridge, K., & Hepburn, S. (2012). Parent
of parent ratings of internalizing and externalizing problem behavior in child agreement of anxiety symptoms in youth with autism spectrum
12-year-old twins. Journal of the American Academy of Child & Ado- disorders. Research in Autism Spectrum Disorders, 6, 707716. http://
lescent Psychiatry, 42, 13511359. http://dx.doi.org/10.1097/01.CHI dx.doi.org/10.1016/j.rasd.2011.07.020
.0000085755.71002.5d Bgels, S. M., Alden, L., Beidel, D. C., Clark, L. A., Pine, D. S., Stein,

Bass, J. K., Ayash, C., Betancourt, T. S., Haroz, E. E., Verdeli, H., M. B., & Voncken, M. (2010). Social anxiety disorder: Questions and
Neugebauer, R., . . . Bolton, P. A. (2014). Mental health problems of answers for the DSM-V. Depression and Anxiety, 27, 168 189. http://
displaced war-affected adolescents in northern Uganda: Patterns of dx.doi.org/10.1002/da.20670

agreement between self and caregiver assessment. Journal of Child and Bohnert, A. M., Kane, P., & Garber, J. (2008). Organized activity partic-
ipation and internalizing and externalizing symptoms: Reciprocal rela-
Family Studies, 23, 1000 1010. http://dx.doi.org/10.1007/s10826-013-
tions during adolescence. Journal of Youth and Adolescence, 37, 239
9755-9
250. http://dx.doi.org/10.1007/s10964-007-9195-1
Beidel, D. C., Rao, P. A., Scharfstein, L., Wong, N., & Alfano, C. A.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009).
(2010). Social skills and social phobia: An investigation of DSMIV
Introduction to meta-analysis. Chichester, UK: Wiley. http://dx.doi.org/
subtypes. Behaviour Research and Therapy, 48, 9921001. http://dx.doi
10.1002/9780470743386
.org/10.1016/j.brat.2010.06.005
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept
Beidel, D. C., Turner, S. M., Young, B. J., Ammerman, R. T., Sallee, F. R.,
of validity. Psychological Review, 111, 10611071. http://dx.doi.org/
& Crosby, L. (2007). Psychopathology of adolescent social phobia.
10.1037/0033-295X.111.4.1061
Journal of Psychopathology and Behavioral Assessment, 29, 4754.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou,
http://dx.doi.org/10.1007/s10862-006-9021-1
P. P., Irwig, L. M., . . . Lijmer, J. G. (2003). The STARD statement for
Bein, L. A., Petrik, M. L., Saunders, S. M., & Wojcik, J. V. (2014).
reporting studies of diagnostic accuracy: Explanation and elaboration.
Discrepancy between parents and children in reporting of distress and
Clinical Chemistry, 49, 718. http://dx.doi.org/10.1373/49.1.7
impairment: Association with critical symptoms. Clinical Child Psy-
Braaten, E. B., Biederman, J., DiMauro, A., Mick, E., Monuteaux, M. C.,
chology and Psychiatry. Advance online publication. http://dx.doi.org/
Muehl, K., & Faraone, S. V. (2001). Methodological complexities in the
10.1177/1359104514532185 diagnosis of major depression in youth: An analysis of mother and youth

Bendersky, M., Bennett, D., & Lewis, M. (2006). Aggression at age 5 as self-reports. Journal of Child and Adolescent Psychopharmacology, 11,
a function of prenatal exposure to cocaine, gender, and environmental 395 407. http://dx.doi.org/10.1089/104454601317261573
risk. Journal of Pediatric Psychology, 31, 71 84. http://dx.doi.org/ Brookman-Frazee, L., Haine, R. A., Gabayan, E. N., & Garland, A. F.
10.1093/jpepsy/jsj025 (2008). Predicting frequency of treatment visits in community-based

Benjamin, C. L., Puleo, C. M., & Kendall, P. C. (2011). Informant youth psychotherapy. Psychological Services, 5, 126 138. http://dx.doi
agreement in treatment gains for child anxiety. Child & Family Behavior .org/10.1037/1541-1559.5.2.126
Therapy, 33, 199 216. http://dx.doi.org/10.1080/07317107.2011
Brody, G. H., Stoneman, Z., Flor, D., McCrary, C., Hastings, L., &
.595987 Conyers, O. (1994). Financial resources, parent psychological function-

Bennett, D. S., Snooks, Q., Llera, S., Vogel, K., Conklin, D., & Varlotta, ing, parent co-caregiving, and early adolescent competence in rural
L. (2008). Monitoring and internalizing symptoms among youths with two-parent African-American families. Child Development, 65, 590
cystic fibrosis. Childrens Health Care, 37, 278 292. http://dx.doi.org/ 605. http://dx.doi.org/10.2307/113140310.2307/1131403
10.1080/02739610802437426
Brown, J. D., Wissow, L. S., Gadomski, A., Zachary, C., Bartlett, E., &

Berg-Nielsen, T. S., Solheim, E., Belsky, J., & Wichstrom, L. (2012). Horn, I. (2006). Parent and teacher mental health ratings of children
Preschoolers psychosocial problems: In the eyes of the beholder? Add- using primary-care services: Interrater agreement and implications for
ing teacher characteristics as determinants of discrepant parentteacher mental health screening. Ambulatory Pediatrics, 6, 347351. http://dx
reports. Child Psychiatry & Human Development, 43, 393 413. http:// .doi.org/10.1016/j.ambp.2006.09.004
dx.doi.org/10.1007/s10578-011-0271-0 Brown-Jacobsen, A. M., Wallace, D. P., & Whiteside, S. P. H. (2011).

Berkhout, L., Hoekman, J., & Goorhuis-Brouwer, S. M. (2012). Psycho- Multimethod, multi-informant agreement, and positive predictive value
social health of 4-year-old boys and girls, as observed by parents and in the identification of child anxiety disorders using the SCAS and
teachers. European Journal of Developmental Psychology, 9, 500 507. ADIS-C. Assessment, 18, 382392. http://dx.doi.org/10.1177/
http://dx.doi.org/10.1080/17405629.2012.662590 1073191110375792

Bickman, L., Wighton, L. G., Lambert, E. W., Karver, M. S., & Steding, Burt, S. A., McGue, M., Krueger, R. F., & Iacono, W. G. (2005). How are
L. (2012). Problems in using diagnosis in child and adolescent mental parent child conflict and childhood externalizing symptoms related over
health services research. Journal of Methods and Measurement in the time? Results from a genetically informative cross-lagged study. Devel-
886 DE LOS REYES ET AL.

opment and Psychopathology, 17, 145165. http://dx.doi.org/10.1017/ hyperactivity disorder. Journal of School Psychology, 28, 189 202.
S095457940505008X http://dx.doi.org/10.1016/0022-4405(90)90011-U

Buschgens, C. J. M., Aken, M. A. van Swinkels, S. H. N., Ormel, J., Cohen, N. J., Coyne, J., & Duvall, J. (1993). Adopted and biological
Verhulst, F. C., & Buitelaar, J. K. (2010). Externalizing behaviors in children in the clinic: Family, parental and child characteristics. Child
preadolescents: Familial risk to externalizing behaviors and perceived Psychology & Psychiatry & Allied Disciplines, 34, 545562. http://dx
parenting styles. European Child & Adolescent Psychiatry, 19, 567575. .doi.org/10.1111/j.1469-7610.1993.tb01035.x
http://dx.doi.org/10.1007/s00787-009-0086-8
Cole, D. A., Hoffman, K., Tram, J. M., & Maxwell, S. E. (2000).

Buschgens, C. J. M., Swinkels, S. H. N., Aken, M. A. van Ormel, J., Structural differences in parent and child reports of childrens symptoms
Verhulst, F. C., & Buitelaar, J. K. (2009). Externalizing behaviors in of depression and anxiety. Psychological Assessment, 12, 174 185.
preadolescents: Familial risk to externalizing behaviors, prenatal and http://dx.doi.org/10.1037/1040-3590.12.2.174
perinatal risks, and their interactions. European Child & Adolescent
Cole, D. A., & Martin, N. C. (2005). The longitudinal structure of the
Psychiatry, 18, 6574. http://dx.doi.org/10.1007/s00787-008-0704-x Childrens Depression Inventory: Testing a latent trait-state model.
Butler, A. C., Chapman, J. E., Forman, E. M., & Beck, A. T. (2006). The Psychological Assessment, 17, 144 155. http://dx.doi.org/10.1037/
empirical status of cognitive-behavioral therapy: A review of meta- 1040-3590.17.2.144
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

analyses. Clinical Psychology Review, 26, 1731. http://dx.doi.org/


Cole, D. A., Peeke, L. G., Martin, J. M., Truglio, R., & Seroczynski, A. D.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

10.1016/j.cpr.2005.07.003 (1998). A longitudinal look at the relation between depression and



Campbell, S. B., Pierce, E. W., Moore, G., Marakovitz, S., & Newby, K. anxiety in children and adolescents. Journal of Consulting and Clinical
(1996). Boys externalizing problems at elementary school age: Path- Psychology, 66, 451 460. http://dx.doi.org/10.1037/0022-006X.66.3
ways from early behavior problems, maternal control, and family stress. .451
Development and Psychopathology, 8, 701719. http://dx.doi.org/
Cole, D. A., Truglio, R., & Peeke, L. (1997). Relation between symptoms
10.1017/S0954579400007379 of anxiety and depression in children: A multitrait-multimethod-
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant multigroup assessment. Journal of Consulting and Clinical Psychology,
validation by the multitrait-multimethod matrix. Psychological Bulletin, 65, 110 119. http://dx.doi.org/10.1037/0022-006X.65.1.110
56, 81105. http://dx.doi.org/10.1037/h0046016
Collishaw, S., Goodman, R., Ford, T., Rabe-Hesketh, S., & Pickles, A.
Carey, W. B. (1998). Temperament and behavior problems in the class- (2009). How far are associations between child, family and community
room. School Psychology Review, 27, 522533. Retrieved from http://
factors and child psychopathology informant-specific and informant-
psycnet.apa.org/psycinfo/1999-00154-005
general? Journal of Child Psychology and Psychiatry, 50, 571580.
Carlson, G. A., & Blader, J. C. (2011). Diagnostic implications of infor-
http://dx.doi.org/10.1111/j.1469-7610.2008.02026.x
mant disagreement for manic symptoms. Journal of Child and Adoles-
Comer, J. S., & Kendall, P. C. (2004). A symptom-level examination of
cent Psychopharmacology, 21, 399 405. http://dx.doi.org/10.1089/cap
parent-child agreement in the diagnosis of anxious youths. Journal of the
.2011.0007
American Academy of Child and Adolescent Psychiatry, 43, 878 886.
Carlson, G. A., & Youngstrom, E. A. (2003). Clinical implications of
http://dx.doi.org/10.1097/01.chi.0000125092.35109.c5
pervasive manic symptoms in children. Biological Psychiatry, 53,
Connell, A. M., & Dishion, T. J. (2006). The contribution of peers to
1050 1058. http://dx.doi.org/10.1016/S0006-3223(03)00068-4
monthly variation in adolescent depressed mood: A short-term longitu-
Carlston, D. L., & Ogles, B. M. (2009). Age, gender, and ethnicity effects
dinal study with time-varying predictors. Development and Psychopa-
on parent child discrepancy using identical item measures. Journal of
thology, 18, 139 154. http://dx.doi.org/10.1017/S0954579406060081
Child and Family Studies, 18, 125135. http://dx.doi.org/10.1007/
Connolly, A., & Vance, A. (2010). Psychosocial factors associated with
s10826-008-9213-2
parent and teacher reports of aggression in children and adolescents with
Carter, A. S., Grigorenko, E. L., & Pauls, D. L. (1995). A Russian
adaption of the Child Behavior Checklist: Psychometric properties and attention deficit hyperactivity disorder. Australian and New Zealand
associations with child and maternal affective symptomatology and Journal of Psychiatry, 44, 667 675. http://dx.doi.org/10.3109/
family functioning. Journal of Abnormal Child Psychology, 23, 661 00048671003664697

684. http://dx.doi.org/10.1007/BF01447471 Connor, J., & Rueter, M. (2009). Predicting adolescent suicidality: Com-

Carter, R., Jaccard, J., Silverman, W. K., & Pina, A. A. (2009). Pubertal paring multiple informants and assessment techniques. Journal of Ado-
timing and its link to behavioral and emotional problems among at-risk lescence, 32, 619 631. http://dx.doi.org/10.1016/j.adolescence.2008.06
African American adolescent girls. Journal of Adolescence, 32, 467 .005

481. http://dx.doi.org/10.1016/j.adolescence.2008.07.005 Connors, L. L., Connolly, J., & Toplak, M. E. (2012). Self-reported

Caye, A., Machado, J. D., & Rohde, L. A. (2013). Evaluating parental inattention in early adolescence in a community sample. Journal of
disagreement in ADHD diagnosis: Can we rely on a single report from Attention Disorders, 16, 60 70. http://dx.doi.org/10.1177/
home? Journal of Attention Disorders. Advance online publication. 1087054710379734
http://dx.doi.org/10.1177/1087054713504134 Conrad, M., & Hammen, C. (1989). Role of maternal depression in
perceptions of child maladjustment. Journal of Consulting and Clinical
Choudhury, M. S., Pimentel, S. S., & Kendall, P. C. (2003). Childhood
anxiety disorders: Parent child (dis) agreement using a structured inter- Psychology, 57, 663 667. http://dx.doi.org/10.1037/0022-006X.57.5
view for the DSM-IV. Journal of the American Academy of Child & .663
Adolescent Psychiatry, 42, 957964. http://dx.doi.org/10.1097/01.CHI Crick, N. R., Werner, N. E., Casas, J. F., OBrien, K. M., Nelson, D. A.,
.0000046898.27264.A2 Grotpeter, J. K., & Markon, K. (1998). Childhood aggression and
Cicchetti, D. (1984). The emergence of developmental psychopathology. gender; A new look at an old problem. In D. Bernstein (Ed.), Nebraska
Child Development, 55, 17. http://dx.doi.org/10.2307/1129830 Symposium on Motivation (pp. 75141). Lincoln, NE: University of
Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple Nebraska Press.

regression/correlation analysis for the behavioral sciences (3rd ed.). Crockenberg, S., & Lourie, A. (1996). Parents conflict strategies with
Hillsdale, NJ: Erlbaum. children and childrens conflict strategies with peers. Merrill-Palmer

Cohen, M., Becker, M. G., & Campbell, R. (1990). Relationships among Quarterly, 42, 495518. Retrieved from http://www.jstor.org/stable/
four methods of assessment of children with attention deficit- 23087467
VALIDITY OF MULTI-INFORMANT ASSESSMENT 887

Croskerry, P. (2003). The importance of cognitive errors in diagnosis and De Los Reyes, A., Alfano, C. A., & Beidel, D. C. (2011). Are clinicians
strategies to minimize them. Academic Medicine, 78, 775780. http:// assessments of improvements in childrens functioning global? Jour-
dx.doi.org/10.1097/00001888-200308000-00003 nal of Clinical Child and Adolescent Psychology, 40, 281294. http://

Crowley, S. L., Worchel, F. F., & Ash, M. J. (1992). Self-report, peer- dx.doi.org/10.1080/15374416.2011.546043
report, and teacher-report measures of childhood depression: An analysis De Los Reyes, A., Augenstein, T. M., Aldao, A., Thomas, S. A., Daruwala,
by item. Journal of Personality Assessment, 59, 189 203. http://dx.doi S. E., Kline, K., & Regan, T. (2015). Implementing psychophysiology in
.org/10.1207/s15327752jpa5901_16 clinical assessments of adolescent social anxiety: Use of rater judgments
Cuijpers, P., & Dekker, J. (2005). Psychologische behandeling van de- based on graphical representations of psychophysiology. Journal of
pressie; een systematisch overzicht van meta-analysen. [Psychological Clinical Child and Adolescent Psychology, 44, 264 279. http://dx.doi
treatment for depression: A systematic review of meta-analyses]. Ned- .org/10.1080/15374416.2013.859080
erlands Tijdschrift voor Geneeskunde, 149, 18921897. Retrieved from De Los Reyes, A., Bunnell, B. E., & Beidel, D. C. (2013). Informant
http://hdl.handle.net/1871/17373 discrepancies in adult social anxiety disorder assessments: Links with
Curtis, W. J., & Cicchetti, D. (2007). Emotion and resilience: A contextual variations in observed behavior. Journal of Abnormal Psy-
multilevel investigation of hemispheric electroencephalogram asym- chology, 122, 376 386. http://dx.doi.org/10.1037/a0031150
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

metry and emotion regulation in maltreated and nonmaltreated chil- De Los Reyes, A., Ehrlich, K. B., Swan, A. J., Luo, T., Van Wie, M., &
This document is copyrighted by the American Psychological Association or one of its allied publishers.

dren. Development and Psychopathology, 19, 811 840. http://dx.doi Pabn, S. C. (2013). An experimental test of whether informants can
.org/10.1017/S0954579407000405 report about child and family behavior based on settings of behavioral
Dadds, M. R., Perrin, S., & Yule, W. (1998). Social desirability and expression. Journal of Child and Family Studies, 22, 177191. http://dx
self-reported anxiety in children: An analysis of the RCMAS Lie scale. .doi.org/10.1007/s10826-012-9567-3
Journal of Abnormal Child Psychology, 26, 311317. http://dx.doi.org/ De Los Reyes, A., Goodman, K. L., Kliewer, W., & Reid-Quiones, K.
10.1023/A:1022610702439 (2008). Whose depression relates to discrepancies? Testing relations

Daley, T. C. (2006). Perceptions and congruence of symptoms and com- between informant characteristics and informant discrepancies from
munication among second-generation Cambodian youth and parents: A both informants perspectives. Psychological Assessment, 20, 139 149.
matched-control design. Child Psychiatry and Human Development, 37, http://dx.doi.org/10.1037/1040-3590.20.2.139
39 53. http://dx.doi.org/10.1007/s10578-006-0018-5 De Los Reyes, A., Goodman, K. L., Kliewer, W., & Reid-Quiones, K.
Dawes, R. M. (1999). Two methods for studying the incremental validity (2010). The longitudinal consistency of mother-child reporting discrep-
of a Rorschach variable. Psychological Assessment, 11, 297302. http:// ancies of parental monitoring and their ability to predict child delinquent
dx.doi.org/10.1037/1040-3590.11.3.297 behaviors two years later. Journal of Youth and Adolescence, 39, 1417

Deater-Deckard, K., & Scarr, S. (1996). Parenting stress among dual- 1430. http://dx.doi.org/10.1007/s10964-009-9496-7

earner mothers and fathers: Are there gender differences? Journal of De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009).
Family Psychology, 10, 4559. http://dx.doi.org/10.1037/0893-3200.10 Linking informant discrepancies to observed variations in young chil-
.1.45 drens disruptive behavior. Journal of Abnormal Child Psychology, 37,

deKeyser, L., Svedin, C. G., Agnafors, S., Bladh, M., & Sydsjo, G. (2014). 637 652. http://dx.doi.org/10.1007/s10802-009-9307-3

Multi-informant reports of mental health in Swedish-born children of im- De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrep-
migrants and children born to non-immigrants the SESBiC-study. BMC ancies in clinical child research. Psychological Assessment, 16, 330
Pediatrics, 14: 95. http://dx.doi.org/10.1186/1471-2431-14-95 334. http://dx.doi.org/10.1037/1040-3590.16.3.330

de la Osa, N., Ezpelata, L., Granero, R., Olaya, B., & Domenech, J. M. De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the
(2011). Diagnostic value of the dominic interactive assessment with assessment of childhood psychopathology: A critical review, theoretical
children exposed to intimate partner violence. Psicothema, 23, 648 653. framework, and recommendations for further study. Psychological Bul-
Retrieved from http://www.unioviedo.es/reunido/index.php/PST/article/ letin, 131, 483509. http://dx.doi.org/10.1037/0033-2909.131.4.483
viewFile/9139/9003 De Los Reyes, A., & Kazdin, A. E. (2006). Conceptualizing changes in
De Los Reyes, A. (2011). Introduction to the special section: More than behavior in intervention research: The range of possible changes model.
measurement error: Discovering meaning behind informant discrepan- Psychological Review, 113, 554 583. http://dx.doi.org/10.1037/0033-
cies in clinical assessments of children and adolescents. Journal of 295X.113.3.554

Clinical Child and Adolescent Psychology, 40, 19. http://dx.doi.org/ De Los Reyes, A., & Kazdin, A. E. (2006). Informant discrepancies in
10.1080/15374416.2011.533405 assessing child dysfunction relate to dysfunction within mother-child
De Los Reyes, A. (2013). Strategic objectives for improving understanding interactions. Journal of Child and Family Studies, 15, 645 663. http://
of informant discrepancies in developmental psychopathology research. dx.doi.org/10.1007/s10826-006-9031-3
Development and Psychopathology, 25, 669 682. http://dx.doi.org/ De Los Reyes, A., Kundey, S. M. A., & Wang, M. (2011). The end of the
10.1017/S0954579413000096 primary outcome measure: A research agenda for constructing its re-
De Los Reyes, A., & Aldao, A. (2015). Introduction to the special issue: placement. Clinical Psychology Review, 31, 829 838. http://dx.doi.org/
Toward implementing physiological measures in clinical child and ad- 10.1016/j.cpr.2011.03.011
olescent assessments. Journal of Clinical Child and Adolescent Psychol- De Los Reyes, A., & Marsh, J. K. (2011). Patients contexts and their
ogy, 44, 221237. http://dx.doi.org/10.1080/15374416.2014.891227 effects on clinicians impressions of conduct disorder symptoms. Jour-

De Los Reyes, A., Aldao, A., Thomas, S. A., Daruwala, S. E., Swan, A. J., nal of Clinical Child and Adolescent Psychology, 40, 479 485. http://
Van Wie, M., . . . Lechner, W. V. (2012). Adolescent self-reports of dx.doi.org/10.1080/15374416.2011.563471
social anxiety: Can they disagree with objective psychophysiological De Los Reyes, A., Salas, S., Menzer, M. M., & Daruwala, S. E. (2013).
measures and still be valid? Journal of Psychopathology and Behavioral Criterion validity of interpreting scores from multi-informant statistical
Assessment, 34, 308 322. http://dx.doi.org/10.1007/s10862-012-9289-2 interactions as measures of informant discrepancies in psychological

De Los Reyes, A., Alfano, C. A., & Beidel, D. C. (2010). The relations assessments of children and adolescents. Psychological Assessment, 25,
among measurements of informant discrepancies within a multisite trial 509 519. http://dx.doi.org/10.1037/a0032081
of treatments for childhood social phobia. Journal of Abnormal Child De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. A.
Psychology, 38, 395 404. http://dx.doi.org/10.1007/s10802-009-9373-6 (2013). Principles underlying the use of multiple informants reports.
888 DE LOS REYES ET AL.

Annual Review of Clinical Psychology, 9, 123149. http://dx.doi.org/ Adolescent Psychology, 33, 813 818. http://dx.doi.org/10.1207/
10.1146/annurev-clinpsy-050212-185617 s15374424jccp3304_16

De Los Reyes, A., Youngstrom, E. A., Pabn, S. C., Youngstrom, J. K., Donenberg, G. R., Emerson, E., Bryant, F. B., Wilson, H., & Weber-
Feeny, N. C., & Findling, R. L. (2011a). Internal consistency and Shifrin, E. (2001). Understanding AIDS-risk behavior among adoles-
associated characteristics of informant discrepancies in clinic referred cents in psychiatric care: Links to psychopathology and peer relation-
youths age 11 to 17 years. Journal of Clinical Child and Adolescent ships. Journal of the American Academy of Child & Adolescent
Psychology, 40, 36 53. http://dx.doi.org/10.1080/15374416.2011 Psychiatry, 40, 642 653. http://dx.doi.org/10.1097/00004583-
.533402 200106000-00008

De Los Reyes, A., Youngstrom, E. A., Swan, A. J., Youngstrom, J. K., Douma, J. C. H., Dekker, M. C., Verhulst, F. C., & Koot, H. M. (2006).
Feeny, N. C., & Findling, R. L. (2011b). Informant discrepancies in Self-reports on mental health problems of youth with moderate to
clinical reports of youths and interviewers impressions of the reliability borderline intellectual disabilities. Journal of the American Academy of
of informants. Journal of Child and Adolescent Psychopharmacology, Child & Adolescent Psychiatry, 45, 1224 1231. http://dx.doi.org/
21, 417 424. http://dx.doi.org/10.1089/cap.2011.0011 10.1097/01.chi.0000233158.21925.95

Dempsey, A. G., Llorens, A., Brewton, C., Mulchandani, S., & Goin- Drabick, D. A. G., Bubier, J., Chen, D., Price, J., & Lanza, I. H. (2011).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Kochel, R. P. (2012). Emotional and behavioral adjustment in typically Source-specific oppositional defiant disorder among inner-city children:
This document is copyrighted by the American Psychological Association or one of its allied publishers.

developing siblings of children with autism spectrum disorders. Journal Prospective prediction and moderation. Journal of Clinical Child and
of Autism and Developmental Disorders, 42, 13931402. http://dx.doi Adolescent Psychology, 40, 2335. http://dx.doi.org/10.1080/15374416
.org/10.1007/s10803-011-1368-9 .2011.533401

Deng, S., Liu, X., & Roosa, M. W. (2004). Agreement between parent and Drabick, D. A. G., Gadow, K. D., & Loney, J. (2007). Source-specific
teacher reports on behavioral problems among Chinese children. Journal oppositional defiant disorder: Comorbidity and risk factors in referred
of Developmental and Behavioral Pediatrics, 25, 407 417. http://dx.doi elementary schoolboys. Journal of the American Academy of Child and
.org/10.1097/00004703-200412000-00004 Adolescent Psychiatry, 46, 92101. http://dx.doi.org/10.1097/01.chi

De Nijs, P. F. A., Ferdinand, R. F., de Bruin, E. I., Dekker, M. C. J., van .0000242245.00174.90
Duijn, C. M., & Verhulst, D. C. (2004). Attention-deficit/hyperactivity Drabick, D. A. G., Gadow, K. D., & Loney, J. (2008). Co-occurring ODD
and GAD symptom groups: Source-specific syndromes and cross-
disorder (ADHD): Parents judgment about school, teachers judgment
informant comorbidity. Journal of Clinical Child and Adolescent Psy-
about home. European Child & Adolescent Psychiatry, 13, 315320.
chology, 37, 314 326. http://dx.doi.org/10.1080/15374410801955862
http://dx.doi.org/10.1007/s00787-004-0405-z
Drabick, D. A. G., & Kendall, P. C. (2010). Developmental psychopathol-
De Pauw, S. S. W., Mervielde, I., De Clercq, B. J., De Fruyt, F.,
ogy and the diagnosis of mental health problems among youth. Clinical
Tremmery, S., & Deboutte, D. (2009). Personality symptoms and self-
Psychology: Science and Practice, 17, 272280. http://dx.doi.org/10.1111/
esteem as correlates of psychopathology in child psychiatric patients:
j.1468-2850.2010.01219.x
Evaluating multiple informant data. Child Psychiatry and Human De-
Du, Y., Kou, J., & Coghill, D. (2008). The validity, reliability and
velopment, 40, 499 515. http://dx.doi.org/10.1007/s10578-009-0140-2
normative scores of the parent, teacher and self report versions of the
Derks, E. M., Hudziak, J. J., Van Beijsterveldt, C. E., Dolan, C. V., &
Strengths and Difficulties Questionnaire in China. Child and Adolescent
Boomsma, D. I. (2006). Genetic analyses of maternal and teacher ratings
Psychiatry and Mental Health, 2:8. http://dx.doi.org/10.1186/1753-
on attention problems in 7-year-old Dutch twins. Behavior Genetics, 36,
2000-2-8
833 844. http://dx.doi.org/10.1007/s10519-006-9084-5

Dubi, K., & Schneider, S. (2009). The Picture Anxiety Test (PAT): A new
Diamond, G. M., Izzard, M. C., Kedar, T., Hutlzer, A., & Mell, H. (2005).
pictorial assessment of anxiety symptoms in young children. Journal of
Psychological symptoms and drug use severity among Israeli adoles-
Anxiety Disorders, 23, 1148 1157. http://dx.doi.org/10.1016/j.janxdis
cents presenting for outpatient drug abuse treatment. Journal of Adoles- .2009.07.020
cence, 28, 495505. http://dx.doi.org/10.1016/j.adolescence.2004.10 Duhig, A. M., Renk, K., Epstein, M. K., & Phares, V. (2000). Interparental
.005 agreement on internalizing, externalizing, and total behavior problems:
DiBartolo, P. M., Albano, A. M., Barlow, D. H., & Heimberg, R. G. A meta-analysis. Clinical Psychology: Science and Practice, 7, 435
(1998). Cross-informant agreement in the assessment of social phobia in 453. http://dx.doi.org/10.1093/clipsy.7.4.435
youth. Journal of Abnormal Child Psychology, 26, 213220. http://dx
Duke, N., Ireland, M., & Borowsky, I. W. (2005). Identifying psychoso-
.doi.org/10.1023/A:1022624318795 cial problems among youth: Factors associated with youth agreement on

DiBartolo, P. M., & Grills, A. E. (2006). Who is best at predicting a positive parent-completed PSC-17. Child: Care, Health and Develop-
childrens anxiety in response to a social evaluative task? A comparison ment, 31, 563573. http://dx.doi.org/10.1111/j.1365-2214.2005.00551.x
of child, parent, and teacher reports. Journal of Anxiety Disorders, 20,
Ebesutani, C., Bernstein, A., Martinez, J. I., Chorpita, B. F., & Weisz,
630 645. http://dx.doi.org/10.1016/j.janxdis.2005.06.003 J. R. (2011). The youth self report: Applicability and validity across

Dirks, M. A., Boyle, M. H., & Georgiades, K. (2011). Psychological younger and older youths. Journal of Clinical Child and Adolescent
symptoms in youth and later socioeconomic functioning: Do associa- Psychology, 40, 338 346. http://dx.doi.org/10.1080/15374416.2011
tions vary by informant? Journal of Clinical Child and Adolescent .546041
Psychology, 40, 10 22. http://dx.doi.org/10.1080/15374416.2011
Ebesutani, C., Okamura, K., Higa-McMillan, C., & Chorpita, B. F. (2011).
.533403 A psychometric analysis of the Positive and Negative Affect Schedule
Dirks, M. A., De Los Reyes, A., Briggs-Gowan, M. J., Cella, D., & for ChildrenParent Version in a school sample. Psychological Assess-
Wakschlag, L. S. (2012). Embracing not erasing contextual variability in ment, 23, 406 416. http://dx.doi.org/10.1037/a0022057
childrens behaviortheory and utility in the selection and use of meth-
Efstratopoulou, M., Janssen, R., & Simons, J. (2012). Agreement among
ods and informants in developmental psychopathology. Journal of Child physical educators, teachers and parents on childrens behaviors: A
Psychology and Psychiatry, 53, 558 574. http://dx.doi.org/10.1111/j multitraitmultimethod design approach. Research in Developmental
.1469-7610.2012.02537.x Disabilities, 33, 13431351. http://dx.doi.org/10.1016/j.ridd.2012.03

Doctoroff, G. L., & Arnold, D. H. (2004). Parent-rated externalizing .015

behavior in preschoolers: The predictive utility of structured interviews, Efstratopoulou, M., Simons, J., & Janssen, R. (2013). Concordance among
teacher reports, and classroom observations. Journal of Clinical Child & physical educators, teachers, and parents perceptions of attention
VALIDITY OF MULTI-INFORMANT ASSESSMENT 889

problems in children. Journal of Attention Disorders, 17, 437 443. try, 43, 452 460. http://dx.doi.org/10.1097/00004583-200404000-
http://dx.doi.org/10.1177/1087054711431698 00011

Ehrlich, K. B., Cassidy, J., & Dykas, M. J. (2011). Reporter discrepancies Forehand, R., Frame, C. L., Wierson, M., Armistead, L., & Kempton, T.
among parents, adolescents, and peers: Adolescent attachment and in- (1991). Assessment of incarcerated juvenile delinquents: Agreement
formant depressive symptoms as explanatory factors. Child Develop- across raters and approaches to psychopathology. Journal of Psychopa-
ment, 82, 999 1012. http://dx.doi.org/10.1111/j.1467-8624.2010 thology and Behavioral Assessment, 13, 1725. http://dx.doi.org/
.01530.x 10.1007/BF00960736

Eisenberg, N., Cumberland, A., Spinrad, T. L., Fabes, R. A., Shepard, Forman, E. M., & Patrick, T. (2005). Assessing childrens appraisals of
S. A., Reiser, M., . . . Guthrie, I. K. (2001). The relations of regulation security in the family system: The development of the Security in the
and emotionality to childrens externalizing and internalizing problem Family System (SIFS) scales. Journal of Child Psychology and Psychi-
behavior. Child Development, 72, 11121134. http://dx.doi.org/10.1111/ atry, 46, 900 916. http://dx.doi.org/10.1111/j.1469-7610.2004.00385.x
1467-8624.00337 Frank, S. J., Van Egeren, L. A., Fortier, J. L., & Chase, P. (2000).

Eisenberg, N., Valiente, C., Morris, A. S., Fabes, R. A., Cumberland, A., Structural, relative, and absolute agreement between parents and ado-
Reiser, M., . . . Losoya, S. (2003). Longitudinal relations among parental lescent inpatients reports of adolescent functional impairment. Journal
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

emotional expressivity, childrens regulation, and quality of socioemo- of Abnormal Child Psychology, 28, 395 402. http://dx.doi.org/10.1023/
This document is copyrighted by the American Psychological Association or one of its allied publishers.

tional functioning. Developmental Psychology, 39, 319. http://dx.doi A:1005125211187



.org/10.1037/0012-1649.39.1.3 Frick, P. J., Silverthorn, P., & Evans, C. (1994). Assessment of childhood

Eiser, C., Havermans, T., Pancer, M., & Eiser, J. R. (1992). Adjustment to anxiety using structured interviews: Patterns of agreement among infor-
chronic disease in relation to age and gender: Mothers and fathers mants and association with maternal anxiety. Psychological Assessment,
reports of their childrens behavior. Journal of Pediatric Psychology, 17, 6, 372379. http://dx.doi.org/10.1037/1040-3590.6.4.372

261275. http://dx.doi.org/10.1093/jpepsy/17.3.261 Frigerio, A., Cattaneo, C., Cataldo, M. G., Schiatti, A., Molteni, M., &

Epkins, C. C, & Meyers, A. W. (1994). Assessment of childhood depres- Battaglia, M. (2004). Behavioral and emotional problems among Italian
sion, anxiety, and aggression: Convergent and discriminant validity of children and adolescents aged 4 to 18 years as reported by parents and
self-, parent-, teacher-, and peer-report measures. Journal of teachers. European Journal of Psychological Assessment, 20, 124 133.
Personality Assessment, 62, 364 381. http://dx.doi.org/10.1207/ http://dx.doi.org/10.1027/1015-5759.20.2.124

s15327752jpa6202_16 Frosch, C. A., & Mangelsdorf, S. C. (2001). Marital behavior, parenting

Eremsoy, C. E., Karanci, A. N., & Berument, S. K. (2010). How do the behavior, and multiple reports of preschoolers behavior problems: Me-
predictors of conduct problems/hyperactivity and callous-unemotional diation or moderation? Developmental Psychology, 37, 502519. http://
(CU) traits in children differ according to mothers and teachers rat- dx.doi.org/10.1037/0012-1649.37.4.502

ings? Procedia - Social and Behavioral Sciences, 5, 471 476. http://dx Funderburk, B. W., Eyberg, S. M., Rich, B. A., & Behar, L. (2003).
.doi.org/10.1016/j.sbspro.2010.07.126 Further psychometric evaluation of the Eyberg and Behar rating scales

Escobar, M. J., Pereira, X., & Santelices, M. P. (2014). Behavior problems for parents and teachers of preschoolers. Early Education and Develop-
and attachment in adopted and non-adopted adolescents. Children and ment, 14, 67 82. http://dx.doi.org/10.1207/s15566935eed1401_5

Youth Services Review, 42, 59 66. http://dx.doi.org/10.1016/j Fung, J. J., & Lau, A. S. (2010). Factors associated with parent child
.childyouth.2014.04.001 (dis)agreement on child behavior and parenting problems in Chinese

Fagot, B. I. (1995). Classification of problem behaviors in young children: immigrant families. Journal of Clinical Child & Adolescent Psychology,
A comparison of four systems. Journal of Applied Developmental Psy- 39, 314 327. http://dx.doi.org/10.1080/15374411003691693

chology, 16, 95106. http://dx.doi.org/10.1016/0193-3973(95)90018-7 Gagnon-Oosterwaal, N., Cossette, L., Smolla, N., Pomerleau, A., Malcuit,

Feeney-Kettler, K. A., Kratochwill, T. R., & Kettler, R. J. (2011). Iden- G., Chicoine, J.-F., . . . Berthiaume, C. (2012). Pre-adoption adversity
tification of preschool children at risk for emotional and behavioral and self-reported behavior problems in 7 year-old international adoptees.
disorders: Development and validation of a universal screening system. Child Psychiatry & Human Development, 43, 648 660. http://dx.doi
Journal of School Psychology, 49, 197216. http://dx.doi.org/10.1016/j .org/10.1007/s10578-011-0279-5
.jsp.2011.02.001 Galione, J. N., & Oltmanns, T. F. (2013). Identifying personality pathology

Fellinger, J., Holzinger, D., Sattel, H., & Laucht, M. (2008). Mental health associated with major depressive episodes: Incremental validity of in-
and quality of life in deaf pupils. European Child & Adolescent Psychi- formant reports. Journal of Personality Assessment, 95, 625 632. http://
atry, 17, 414 423. http://dx.doi.org/10.1007/s00787-008-0683-y dx.doi.org/10.1080/00223891.2013.825624

Ferdinand, R. F., van der Ende, J., & Verhulst, F. C. (2004). Parent- Gao, W., Paterson, J., Carter, S., Iustini, L., & Sundborn, G. (2011).
adolescent disagreement regarding psychopathology in adolescents from Agreement and discordance of parents and teachers reports of behav-
the general population as a risk factor for adverse outcome. Journal of ioural problems among Pacific children living in New Zealand. Pacific
Abnormal Psychology, 113, 198 206. http://dx.doi.org/10.1037/0021- Health Dialog, 17, 6577. Retrieved from http://forum.nuama.org/
843X.113.2.198 attachments/article/220/132299%20PHD%20Journal.pdf#page65

Ferdinand, R. F., van der Ende, J., & Verhulst, F. C. (2006). Prognostic Garb, H. N. (1998). Studying the clinician: Judgment research and psy-
value of parentadolescent disagreement in a referred sample. European chological assessment. Washington, DC: American Psychological As-
Child & Adolescent Psychiatry, 15, 156 162. http://dx.doi.org/10.1007/ sociation.
s00787-005-0518-z Garb, H. N. (2003). Incremental validity and the assessment of psychopa-

Fisher, S. L., Bucholz, K. K., Reich, W., Fox, L., Kuperman, S., Kramer, thology in adults. Psychological Assessment, 15, 508 520. http://dx.doi
J., . . . Bierut, L. J. (2006). Teenagers are rightParents do not know .org/10.1037/1040-3590.15.4.508

much: An analysis of adolescentparent agreement on reports of ado- Garner, A. A., Mrug, S., Hodgens, B., & Patterson, C. (2013). Do
lescent substance use, abuse, and dependence. Alcoholism: Clinical and symptoms of sluggish cognitive tempo in children with ADHD symp-
Experimental Research, 30, 1699 1710. http://dx.doi.org/10.1111/j toms represent comorbid internalizing difficulties? Journal of Attention
.1530-0277.2006.00205.x Disorders, 17, 510 518. http://dx.doi.org/10.1177/1087054711431456

Foley, D., Rutter, M., Pickles, A., Angold, A., Maes, H., Silberg, J., & Garner, W. R., Hake, H. W., & Eriksen, C. W. (1956). Operationism and
Eaves, L. (2004). Informant disagreement for separation anxiety disor- the concept of perception. Psychological Review, 63, 149 159. http://
der. Journal of the American Academy of Child & Adolescent Psychia- dx.doi.org/10.1037/h0042992
890 DE LOS REYES ET AL.


Germn, M., Gonzales, N. A., & Dumka, L. (2009). Familism values as a children. Journal of Psychopathology and Behavioral Assessment, 26,
protective factor for Mexican-origin adolescents exposed to deviant 137146. http://dx.doi.org/10.1023/B:JOBA.0000013661.14995.59

peers. The Journal of Early Adolescence, 29, 16 42. http://dx.doi.org/ Grigorenko, E. L., Geiser, C., Slobodskaya, H. R., & Francis, D. J. (2010).
10.1177/0272431608324475 Cross-informant symptoms from CBCL, TRF, and YSR: Trait and

Ghassabian, A., Bongers-Schokking, J. J., de Rijke, Y. B., van Mil, N., method variance in a normative sample of Russian youths. Psycholog-
Jaddoe, V. W., de Muinck Keizer-Schrama, S. M., . . . Tiemeier, H. ical Assessment, 22, 893911. http://dx.doi.org/10.1037/a0020703
(2012). Maternal thyroid autoimmunity during pregnancy and the risk of Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and
attention deficit/hyperactivity problems in children: The Generation R the Anxiety Disorders Interview Schedule for parents and children.
study. Thyroid: Official Journal of the American Thyroid Association, Journal of the American Academy of Child and Adolescent Psychiatry,
22, 178 186. http://dx.doi.org/10.1089/thy.2011.0318 42, 30 40. http://dx.doi.org/10.1097/00004583-200301000-00008
Groth-Marnat, G. (2009). Handbook of psychological assessment (5th ed.).
Ghassabian, A., Bongers-Schokking, J. J., Henrichs, J., Jaddoe, V. W.,
Visser, T. J., Visser, W., . . . Tiemeier, H. (2011). Maternal thyroid Hoboken, NJ: Wiley.

function during pregnancy and behavioral problems in the offspring: The Gustafsson, A., Arnup, K., Broberg, A. G., Bodin, L., & Berggren, U.
generation R study. Pediatric Research, 69, 454 459. http://dx.doi.org/ (2010). Child dental fear as measured with the Dental Subscale of the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

10.1203/PDR.0b013e3182125b0c Childrens Fear Survey Schedule: The impact of referral status and type
This document is copyrighted by the American Psychological Association or one of its allied publishers.


Gizer, I. R., Waldman, I. D., Abramowitz, A., Barr, C., Feng, Y., Wigg, of informant (child versus parent). Community Dentistry and Oral Ep-
K. G., . . . Rowe, D. C. (2008). Relations between multi-informant idemiology, 38, 256 266. http://dx.doi.org/10.1111/j.1600-0528.2009
assessments of ADHD symptoms, DAT1, and DRD4. Journal of Ab- .00521.x

normal Psychology, 117, 869 880. http://dx.doi.org/10.1037/a0013297 Haine, R. A., Ayers, T. S., Sandler, I. N., Wolchik, S. A., & Weyer, J. L.

Gomez, R. (2007). Australian parent and teacher ratings of the DSM-IV (2003). Locus of control and self-esteem as stress-moderators or stress-
ADHD symptoms: Differential symptom functioning, and parent- mediators in parentally bereaved children. Death Studies, 27, 619 640.
teacher agreement and differences. Journal of Attention Disorders, 11, http://dx.doi.org/10.1080/07481180302894

1727. http://dx.doi.org/10.1177/1087054706295665 Handwerk, M. L., Larzelere, R. E., Soper, S. H., & Friman, P. C. (1999).

Gomez, R., Burns, G. L., Walsh, J. A., & de Moura, M. A. (2003). A Parent and child discrepancies in reporting severity of problem behav-
iors in three out-of-home settings. Psychological Assessment, 11, 14 23.
multitrait-multisource confirmatory factor analytic approach to the con-
http://dx.doi.org/10.1037/1040-3590.11.1.14
struct validity of ADHD rating scales. Psychological Assessment, 15,
Hanssen-Bauer, K., Langsrud, O., Kvernmo, S., & Heyerdahl, S. (2010).
316. http://dx.doi.org/10.1037/1040-3590.15.1.3
Clinician-rated mental health in outpatient child and adolescent mental
Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use
health services: Associations with parent, teacher and adolescent ratings.
broader internalising and externalising subscales instead of the hypoth-
Child and Adolescent Psychiatry and Mental Health, 4:29. http://dx.doi
esised five subscales on the strengths and difficulties questionnaire
.org/10.1186/1753-2000-4-29
(SDQ): Data from British parents, teachers and children. Journal of
Harding, H. G., Morelen, D., Thomassin, K., Bradbury, L., & Shaffer, A.
Abnormal Child Psychology, 38, 1179 1191. http://dx.doi.org/10.1007/
(2013). Exposure to maternal- and paternal-perpetrated intimate partner
s10802-010-9434-x
violence, emotion regulation, and child outcomes. Journal of Family
Goodman, K. L., De Los Reyes, A., & Bradshaw, C. P. (2010). Under-
Violence, 28, 6372. http://dx.doi.org/10.1007/s10896-012-9487-4
standing and using informants reporting discrepancies of youth victim-
Hartley, A. G., Zakriski, A. L., & Wright, J. C. (2011). Probing the depths
ization: A conceptual model and recommendations for research. Clinical
of informant discrepancies: Contextual influences on divergence and
Child and Family Psychology Review, 13, 366 383. http://dx.doi.org/
convergence. Journal of Clinical Child and Adolescent Psychology, 40,
10.1007/s10567-010-0076-x

54 66. http://dx.doi.org/10.1080/15374416.2011.533404
Goodman, R. (2001). Psychometric properties of the strengths and diffi-
Hartos, J. L., & Power, T. G. (2000). Association between mother and
culties questionnaire. Journal of the American Academy of Child & adolescent reports for assessing relations between parentadolescent
Adolescent Psychiatry, 40, 13371345. http://dx.doi.org/10.1097/ communication and adolescent adjustment. Journal of Youth and Ado-
00004583-200111000-00015 lescence, 29, 441 450. http://dx.doi.org/10.1023/A:1005158425861
Goodman, S. H., & Gotlib, I. H. (1999). Risk for psychopathology in the
Harvey, E. A., Fischer, C., Weieneth, J. L., Hurwitz, S. D., & Sayer, A. G.
children of depressed mothers: A developmental model for understand- (2013). Predictors of discrepancies between informants ratings of
ing mechanisms of transmission. Psychological Review, 106, 458 490. preschool-aged childrens behavior: An examination of ethnicity, child
http://dx.doi.org/10.1037/0033-295X.106.3.458 characteristics, and family functioning. Early Childhood Research

Gordis, E. B., Granger, D. A., Susman, E. J., & Trickett, P. K. (2006). Quarterly, 28, 668 682. http://dx.doi.org/10.1016/j.ecresq.2013.05.002
Asymmetry between salivary cortisol and -amylase reactivity to stress: Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist
Relation to aggressive behavior in adolescents. Psychoneuroendocrinol- (dis)agreement on target problems in outpatient therapy: The therapists
ogy, 31, 976 987. http://dx.doi.org/10.1016/j.psyneuen.2006.05.010 dilemma and its implications. Journal of Consulting and Clinical Psy-

Gordis, E. B., Margolin, G., & John, R. S. (2001). Parents hostility in chology, 71, 6270. http://dx.doi.org/10.1037/0022-006X.71.1.62
dyadic marital and triadic family settings and childrens behavior prob- Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical
lems. Journal of Consulting and Clinical Psychology, 69, 727734. assessment measures. Psychological Assessment, 15, 456 466. http://
http://dx.doi.org/10.1037/0022-006X.69.4.727 dx.doi.org/10.1037/1040-3590.15.4.456

Greenbaum, R. L., Stevens, S. A., Nash, K., Koren, G., & Rovet, J. Hemmelgarn, A. L., Glisson, C., & Sharp, S. R. (2003). The validity of
(2009). Social cognitive and emotion processing abilities of children the Shortform Assessment for Children (SAC). Research on Social
with fetal alcohol spectrum disorders: A comparison with attention Work Practice, 13, 510 530. http://dx.doi.org/10.1177/
deficit hyperactivity disorder. Alcoholism: Clinical and Experimental 1049731503013004006

Research, 33, 1656 1670. http://dx.doi.org/10.1111/j.1530-0277.2009 Hendriks, A. H. C., de Moor, J. M. H., Oud, J. H. L., Franken, W. M., &
.01003.x Savelberg, M. M. H. W. (2001). Behaviour problems of young motor

Grietens, H., Onghena, P., Prinzie, P., Gadeyne, E., Van Assche, C., disabled children at home and in the therapeutic toddler class. European
Ghesquire, P., & Hellinckx, W. (2004). Comparison of mothers, fa- Journal of Special Needs Education, 16, 1528. http://dx.doi.org/
thers, and teachers reports on problem behavior in 5- to 6-year-old 10.1080/08856250150501770
VALIDITY OF MULTI-INFORMANT ASSESSMENT 891

Henggeler, S. W., Watson, S. M., & Whelan, J. P. (1990). Peer relations Psychological Assessment, 17. http://dx.doi.org/10.1037/1040-3590.17
of hearing-impaired adolescents. Journal of Pediatric Psychology, 15, .3.251
721731. http://dx.doi.org/10.1093/jpepsy/15.6.721 Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psycho-

Henrich, C. C., Kuperminc, G. P., Sack, A., Blatt, S. J., & Leadbeater, logical testing and assessment: Conceptual, methodological, and statis-
B. J. (2000). Characteristics and homogeneity of early adolescent friend- tical issues. Psychological Assessment, 15, 446 455. http://dx.doi.org/
ship groups: A comparison of male and female clique and nonclique 10.1037/1040-3590.15.4.446
members. Applied Developmental Science, 4, 1526. http://dx.doi.org/
Hwang, J., & Lee, B. R. (2013). Response agreement rates between child
10.1207/S1532480XADS0401_2 welfareinvolved youth and their caregivers. Journal of Social Service

Henry, B., Moffitt, T. E., Caspi, A., Langley, J., & Silva, P. A. (1994). On Research, 39, 218 232. http://dx.doi.org/10.1080/01488376.2012
the remembrance of things past: A longitudinal evaluation of the .746768
retrospective method. Psychological Assessment, 6, 92101. http://dx
Jaffee, S. R., Caspi, A., Moffitt, T. E., Polo-Tomas, M., Price, T. S., &
.doi.org/10.1037/1040-3590.6.2.92 Taylor, A. (2004). The limits of child effects: Evidence for genetically

Henry, D. B., & Metropolitan Area Child Study Research Group US. mediated child effects on corporal punishment but not on physical
(2006). Associations between peer nominations, teacher ratings, self- maltreatment. Developmental Psychology, 40, 10471058. http://dx.doi
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

reports, and observations of malicious and disruptive behavior. Assess- .org/10.1037/0012-1649.40.6.1047


This document is copyrighted by the American Psychological Association or one of its allied publishers.

ment, 13, 241252. http://dx.doi.org/10.1177/1073191106287668


Jaffee, S. R., Moffitt, T. E., Caspi, A., Taylor, A., & Arseneault, L. (2002).

Higa, C. K., Daleiden, E. L., & Chorpita, B. F. (2002). Psychometric Influence of adult domestic violence on childrens internalizing and
properties and clinical utility of the school refusal assessment scale in a externalizing problems: An environmentally informative twin study.
multiethnic sample. Journal of Psychopathology and Behavioral Assess- Journal of the American Academy of Child & Adolescent Psychiatry, 41,
ment, 24, 247258. http://dx.doi.org/10.1023/A:1020727016113 10951103. http://dx.doi.org/10.1097/00004583-200209000-00010

Higa, C. K., Fernandez, S. N., Nakamura, B. J., Chorpita, B. F., & Daleiden,
Janssens, J. M. A. M., De Bruyn, E. E. J., Manders, W. A., & Scholte,
E. L. (2006). Parental assessment of childhood social phobia: Psychometric R. H. J. (2005). The multitrait-multimethod approach in family assess-
properties of the Social Phobia and Anxiety Inventory for ChildrenParent ment: Mutual parent-child relationships assessed by questionnaires and
Report. Journal of Clinical Child & Adolescent Psychology, 35, 590 597. observations. European Journal of Psychological Assessment, 21, 232
http://dx.doi.org/10.1207/s15374424jccp3504_11
239. http://dx.doi.org/10.1027/1015-5759.21.4.232
Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003).
Jaser, S. S., Langrock, A. M., Keller, G., Merchant, M. J., Benson, M. A.,
Measuring inconsistency in meta-analyses. British Medical Journal,
Reeslund, K., . . . Compas, B. E. (2005). Coping with the stress of
327, 557560. http://dx.doi.org/10.1136/bmj.327.7414.557
parental depression II: Adolescent and parent reports of coping and
Hogue, A., Dauber, S., Lichvar, E., & Spiewag, G. (2014). Adolescent and
adjustment. Journal of Clinical Child and Adolescent Psychology, 34,
caregiver reports of ADHD symptoms among inner-city youth agree-
193205. http://dx.doi.org/10.1207/s15374424jccp3401_18
ment, perceived need for treatment, and behavioral correlates. Journal of
Jaycox, L. H., & Repetti, R. L. (1993). Conflict in families and the
Attention Disorders, 18, 212225. http://dx.doi.org/10.1177/
psychological adjustment of preadolescent children. Journal of Family
1087054712443160
Psychology, 7, 344 355. http://dx.doi.org/10.1037/0893-3200.7.3.344
Hollocks, M. J., Papadopoulos, A. S., Khondoker, M., & Simonoff, E.
Jensen-Doss, A., & Weisz, J. R. (2008). Diagnostic agreement predicts
(2014). Differences in HPA-axis and heart rate responsiveness to psy-
treatment process and outcomes in youth mental health clinics. Journal
chosocial stress in children with autism spectrum disorders with and
of Consulting and Clinical Psychology, 76, 711722. http://dx.doi.org/
without co-morbid anxiety. Psychoneuroendocrinology, 46, 32 45.
10.1037/0022-006X.76.5.711
http://dx.doi.org/10.1016/j.psyneuen.2014.04.004
Joffe, B. I., Van Lieshout, R. J., Duncan, L., & Boyle, M. H. (2014).
Hommersen, P., Murray, C., Ohan, J. L., & Johnston, C. (2006). Oppo-
sitional defiant disorder rating scale preliminary evidence of reliability Suicidal ideation and behavior in adolescents aged 1216 years: A
and validity. Journal of Emotional and Behavioral Disorders, 14, 118 17-year follow-up. Suicide and Life-Threatening Behavior. Advance
125. http://dx.doi.org/10.1177/10634266060140020201 online publication. http://dx.doi.org/10.1111/sltb.12077

Honomichi, R. D., & Donnellan, M. B. (2012). Dimensions of tempera- Johnston, C., & Murray, C. (2003). Incremental validity in the psycholog-
ment in preschoolers predict risk taking and externalizing behaviors in ical assessment of children and adolescents. Psychological Assessment,
adolescents. Social Psychological and Personality Science, 3, 14 22. 15, 496 507. http://dx.doi.org/10.1037/1040-3590.15.4.496

http://dx.doi.org/10.1177/1948550611407344 Jouriles, E. M., Norwood, W. D., McDonald, R., Vincent, J. P., &

Huberty, T. J., Austin, J. K., Harezlak, J., Dunn, D. W., & Ambrosius, Mahoney, A. (1996). Physical violence and other forms of marital
W. T. (2000). Informant agreement in behavior ratings for children with aggression: Links with childrens behavior problems. Journal of Family
epilepsy. Epilepsy & Behavior, 1, 427 435. http://dx.doi.org/10.1006/ Psychology, 10, 223234. http://dx.doi.org/10.1037/0893-3200.10.2.223

ebeh.2000.0119 Jouriles, E. N., Rosenfield, D., McDonald, R., & Mueller, V. (2014).
Child involvement in interparental conflict and child adjustment prob-
Hughes, E. K., & Gullone, E. (2010). Discrepancies between adolescent,
mother, and father reports of adolescent internalizing symptom levels lems: A longitudinal study of violent families. Journal of Abnormal
and their association with parent symptoms. Journal of Clinical Psy- Child Psychology, 42, 693704. http://dx.doi.org/10.1007/s10802-013-
chology, 66, 978 995. http://dx.doi.org/10.1002/jclp.20695 9821-1
Hunsley, J. (2003). Introduction to the special section on incremental Jouriles, E. N., & Thompson, S. M. (1993). Effects of mood on mothers
validity and utility in clinical assessment. Psychological Assessment, 15, evaluations of childrens behavior. Journal of Family Psychology, 6,
443 445. http://dx.doi.org/10.1037/1040-3590.15.4.443 300 307. http://dx.doi.org/10.1037/0893-3200.6.3.300

Hunsley, J., & Lee, C. M. (2010). Introduction to clinical psychology. Jung, E., Raikes, H. H., & Chazan-Cohen, R. (2013). Maternal depressive
Hoboken, NJ: Wiley. symptoms and behavior problems in preschool children from low-
Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual income families: Comparison of reports from mothers and teachers.
Review of Clinical Psychology, 3, 29 51. http://dx.doi.org/10.1146/ Journal of Child and Family Studies, 22, 757768. http://dx.doi.org/
annurev.clinpsy.3.022806.091419 10.1007/s10826-012-9630-0

Hunsley, J., & Mash, E. J. (Eds.). (2005). Developing guidelines for the Karustis, J. L., Power, T. J., Rescorla, L. A., Eiraldi, R. B., & Gallagher,
evidence-based assessment (EBA) of adult disorders [Special section]. P. R. (2000). Anxiety and depression in children with ADHD: Unique
892 DE LOS REYES ET AL.

associations with academic and social functioning. Journal of Attention Kramer, T. L., Phillips, S. D., Hargis, M. B., Miller, T. L., Burns, B. J., &
Disorders, 4, 133149. http://dx.doi.org/10.1177/108705470000400301 Robbins, J. M. (2004). Disagreement between parent and adolescent

Karver, M. S. (2006). Determinants of multiple informant agreement on reports of functional impairment. Journal of Child Psychology and
child and adolescent behavior. Journal of Abnormal Child Psychology, Psychiatry, 45, 248 259. http://dx.doi.org/10.1111/j.1469-7610.2004
34, 251262. http://dx.doi.org/10.1007/s10802-005-9015-6 .00217.x

Kassam-Adams, N., Garca-Espaa, J. F., Miller, V. A., & Winston, F. Kugler, B. B., Burke, N. L., Bloom, M., Truax, T. V., Kaercher, L. B., &
(2006). Parent-child agreement regarding childrens acute stress: The Storch, E. A. (2013). Caregiverteacher agreement on emotional and
role of parent acute stress reactions. Journal of the American Academy behavioral problems in traumatized youth in residential treatment. Res-
of Child & Adolescent Psychiatry, 45, 14851493. http://dx.doi.org/ idential Treatment for Children & Youth, 30, 151167. http://dx.doi.org/
10.1097/01.chi.0000237703.97518.12 10.1080/0886571X.2013.813343

Kazak, A. E., Christakis, D., Alderfer, M., & Coiro, M. J. (1994). Young Kuijpers, R. C. W. M., Otten, R., Krol, N. P. C. M., Vermulst, A. A., &
adolescent cancer survivors and their parents: Adjustment, learning Engels, R. C. M. E. (2013). The reliability and validity of the Dominic
problems, and gender. Journal of Family Psychology, 8, 74 84. http:// interactive: A computerized child report instrument for mental health
dx.doi.org/10.1037/0893-3200.8.1.74 problems. Child & Youth Care Forum, 42, 3552. http://dx.doi.org/
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Kazdin, A. E., & Kagan, J. (1994). Models of dysfunction in developmen- 10.1007/s10566-012-9185-7


This document is copyrighted by the American Psychological Association or one of its allied publishers.


tal psychopathology. Clinical Psychology: Science and Practice, 1, Kumpulainen, K., Rasanen, E., Heuttonen, L., Moilanen, I., Piha, J.,
3552. http://dx.doi.org/10.1111/j.1468-2850.1994.tb00005.x Puura, K., . . . Almqvist, F. (1999). Childrens behavioral/emotional

Kearney, C. A., & Silverman, W. K. (1993). Measuring the function of problems: A comparison of parents and teachers reports for elementary
school refusal behavior: The School Refusal Assessment scale. Journal school-aged children. European Child and Adolescent Psychiatry,
of Clinical Child Psychology, 22, 8596. http://dx.doi.org/10.1207/ 8(Supplement), S41S47. http://dx.doi.org/10.1007/BF03159641
s15374424jccp2201_9 Kwon, K., Kim, E. M., & Sheridan, S. M. (2012). A contextual approach

Kemper, T. S., Gerhardstein, R., Repper, K. K., & Kistner, J. A. (2003). to social skills assessment in the peer group: Who is the best judge?
Mother child agreement on reports of internalizing symptoms among School Psychology Quarterly, 27, 121133. http://dx.doi.org/10.1037/
children referred for evaluation of ADHD. Journal of Psychopathology a0028696

and Behavioral Assessment, 25, 239 250. http://dx.doi.org/10.1023/A: Ladd, G. W., & Ettekal, I. (2013). Peer-related loneliness across early to
1025847012201 late adolescence: Normative trends, intra-individual trajectories, and

Kendall, P. C., Puliafico, A. C., Barmish, A. J., Choudhury, M. S., Henin, links with depressive symptoms. Journal of Adolescence, 36, 1269
A., & Treadwell, K. S. (2007). Assessing anxiety with the Child Behav- 1282. http://dx.doi.org/10.1016/j.adolescence.2013.05.004
ior Checklist and the Teacher Report Form. Journal of Anxiety Disor- Ladd, G. W., & Kochenderfer-Ladd, B. (2002). Identifying victims of peer
ders, 21, 1004 1015. http://dx.doi.org/10.1016/j.janxdis.2006.10.012 aggression from early to middle childhood: Analysis of cross-informant

Keown, L. J. (2012). Predictors of boys ADHD symptoms from early to data for concordance, estimation of relational adjustment, prevalence of
middle childhood: The role of father-child and mother-child interactions. victimization, and characteristics of identified victims. Psychological
Journal of Abnormal Child Psychology, 40, 569 581. http://dx.doi.org/ Assessment, 14, 74 96. http://dx.doi.org/10.1037/1040-3590.14.1.74

10.1007/s10802-011-9586-3 Lagattuta, K. H., Sayfan, L., & Bamford, C. (2012). Do you know how I

King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the feel? Parents underestimate worry and overestimate optimism compared
validity of the behavioral and emotional screening system in a rural to child self-report. Journal of Experimental Child Psychology, 113,
elementary school validity of the BESS. Journal of Psychoeducational 211232. http://dx.doi.org/10.1016/j.jecp.2012.04.001

Assessment, 30, 527538. http://dx.doi.org/10.1177/0734282912440673 Lahey, B. B., Applegate, B., Waldman, I. D., Loft, J. D., Hankin, B. L.,

Klaus, N. M., Mobilio, A., & King, C. A. (2009). Parentadolescent & Rick, J. (2004). The structure of child and adolescent psychopathol-
agreement concerning adolescents suicidal thoughts and behaviors. ogy: Generating new hypotheses. Journal of Abnormal Psychology, 113,
Journal of Clinical Child & Adolescent Psychology, 38, 245255. http:// 358 385. http://dx.doi.org/10.1037/0021-843X.113.3.358

dx.doi.org/10.1080/15374410802698412 Laird, R. D., & Weems, C. F. (2011). The equivalence of regression
Klein, D. N., Dougherty, L. R., & Olino, T. M. (2005). Toward guidelines models using difference scores and models using separate scores for
for evidence-based assessment of depression in children and adolescents. each informant: Implications for the study of informant discrepancies.
Journal of Clinical Child and Adolescent Psychology, 34, 412 432. Psychological Assessment, 23, 388 397. http://dx.doi.org/10.1037/
http://dx.doi.org/10.1207/s15374424jccp3403_3 a0021926

Knox, M., King, C., Hanna, G. L., Logan, D., & Ghaziuddin, N. (2000). Langberg, J. M., Epstein, J. N., Simon, J. O., Loren, R. E. A., Arnold,
Aggressive behavior in clinically depressed adolescents. Journal of the L. E., Hechtman, L., . . . Wigal, T. (2010). Parent agreement on ratings
American Academy of Child & Adolescent Psychiatry, 39, 611 618. of childrens attention deficit/hyperactivity disorder and broadband ex-
http://dx.doi.org/10.1097/00004583-200005000-00015 ternalizing behaviors. Journal of Emotional and Behavioral Disorders,

Kraatz Keiley, M., Bates, J. E., Dodge, K. A., & Pettit, G. S. (2000). A 18, 4150. http://dx.doi.org/10.1177/1063426608330792

cross-domain growth analysis: Externalizing and internalizing behaviors Langeveld, J. H., Gundersen, K. K., & Svartdal, F. (2012). Social com-
during 8 years of childhood. Journal of Abnormal Child Psychology, 28, petence as a mediating factor in reduction of behavioral problems.
161179. http://dx.doi.org/10.1023/A:1005122814723 Scandinavian Journal of Educational Research, 56, 381399. http://dx
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., .doi.org/10.1080/00313831.2011.594614
& Kupfer, D. J. (2003). A new approach to integrating data from Lapouse, R., & Monk, M. A. (1958). An epidemiologic study of behavior
multiple informants in psychiatric assessment and research: Mixing and characteristics in children. American Journal of Public Health, 48,
matching contexts and perspectives. The American Journal of Psychia- 1134 1144. http://dx.doi.org/10.2105/AJPH.48.9.1134

try, 160, 1566 1577. http://dx.doi.org/10.1176/appi.ajp.160.9.1566 Lauth, B., Arnkelsson, G. B., Magnusson, P., Skarpheoinsson, G. A.,

Krain, A. L., & Kendall, P. C. (2000). The role of parental Ferrari, P., & Petursson, H. (2010). Parentyouth agreement on symp-
emotional distress in parent report of child anxiety. Journal of Clin- toms and diagnosis: Assessment with a diagnostic interview in an
ical Child Psychology, 29, 328 335. http://dx.doi.org/10.1207/ adolescent inpatient clinical population. Journal of Physiology-Paris,
S15374424JCCP2903_4 104, 315322. http://dx.doi.org/10.1016/j.jphysparis.2010.09.004
VALIDITY OF MULTI-INFORMANT ASSESSMENT 893

Lavigne, J. V., Dahl, K. P., Gouze, K. R., LeBailly, S. A., & Hopkins, J. tility, parent-child hostility, and childrens adjustment. Journal of Fam-
(2014). Multi-domain predictors of oppositional defiant disorder symp- ily Psychology, 19, 394 403. http://dx.doi.org/10.1037/0893-3200.19.3
toms in preschool children: Cross-informant differences. Child Psychi- .394

atry & Human Development. Advance online publication. http://dx.doi Luby, J. L., Belden, A., Sullivan, J., & Spitznagel, E. (2007). Preschool-
.org/10.1007/s10578-014-0472-4 ers contribution to their diagnosis of depression and anxiety: Uses and

Lee, H., & Schepp, K. G. (2011). Multiple informants in assessing stress limitations of young child self-report of symptoms. Child Psychiatry and
and symptoms in adolescents with schizophrenia. Archives of Psychiat- Human Development, 38, 321338. http://dx.doi.org/10.1007/s10578-
ric Nursing, 25, 120 128. http://dx.doi.org/10.1016/j.apnu.2010.06.004 007-0063-8

Lee, S. W., Elliott, J., & Barbour, J. D. (1994). A comparison of cross- Luthar, S. S., Cicchetti, D., & Becker, B. (2000). The construct of resil-
informant behavior ratings in school-based diagnosis. Behavioral Dis- ience: A critical evaluation and guidelines for future work. Child De-
orders, 19, 8797. Retrieved from http://www.jstor.org/stable/23887490 velopment, 71, 543562. http://dx.doi.org/10.1111/1467-8624.00164

Leeuwen, K. V., Meerschaert, T., Bosmans, G., De Medts, L., & Braet, C. Lyneham, H. J., Street, A. K., Abbott, M. J., & Rapee, R. M. (2008).
(2006). The Strengths and Difficulties Questionnaire in a community Psychometric properties of the School Anxiety ScaleTeacher report
sample of young children in Flanders. European Journal of Psycholog- (SAS-TR). Journal of Anxiety Disorders, 22, 292300. http://dx.doi.org/
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

ical Assessment, 22, 189 197. http://dx.doi.org/10.1027/1015-5759.22 10.1016/j.janxdis.2007.02.001


This document is copyrighted by the American Psychological Association or one of its allied publishers.

.3.189
Mahone, E. M., Zabel, T. A., Levey, E., Verda, M., & Kinsman, S. (2002).

Lehti, V., Niemel, S., Heinze, M., Sillanmki, L., Helenius, H., Piha, J., Parent and self-report ratings of executive function in adolescents with
. . . Sourander, A. (2012). Childhood predictors of becoming a teenage myelomeningocele and hydrocephalus. Child Neuropsychology, 8, 258
mother among Finnish girls. Acta Obstetricia et Gynecologica Scandi- 270. http://dx.doi.org/10.1076/chin.8.4.258.13510
navica, 91, 1319 1325. http://dx.doi.org/10.1111/j.1600-0412.2012
Maikovich-Fong, A. K., & Jaffee, S. R. (2010). Sex differences in child-
.01517.x hood sexual abuse characteristics and victims emotional and behavioral

Lengua, L. J. (2003). Associations among emotionality, self-regulation, problems: Findings from a national sample of youth. Child Abuse &
adjustment problems, and positive adjustment in middle childhood. Neglect, 34, 429 437. http://dx.doi.org/10.1016/j.chiabu.2009.10.006
Journal of Applied Developmental Psychology, 24, 595 618. http://dx
Mann, B. J., & MacKenzie, E. P. (1996). Pathways among marital
.doi.org/10.1016/j.appdev.2003.08.002
functioning, parental behaviors, and child behavior problems in school-
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological,
age boys. Journal of Clinical Child Psychology, 25, 183191. http://dx
educational, and behavioral treatment. Confirmation from meta-analysis.
.doi.org/10.1207/s15374424jccp2502_7
American Psychologist, 48, 11811209. http://dx.doi.org/10.1037/0003-
Markon, K. E., Chmielewski, M., & Miller, C. J. (2011). The reliability and
066X.48.12.1181
validity of discrete and continuous measures of psychopathology: A
Little, S. A., & Garber, J. (2005). The role of social stressors and
quantitative review. Psychological Bulletin, 137, 856 879. http://dx.doi
interpersonal orientation in explaining the longitudinal relation between
.org/10.1037/a0023678
externalizing and depressive symptoms. Journal of Abnormal Psychol-
Marsh, J. K., De Los Reyes, A., & Wallerstein, A. (2014). The influence
ogy, 114, 432 443. http://dx.doi.org/10.1037/0021-843X.114.3.432
of contextual information on lay judgments of childhood mental health
Liu, J., Cheng, H., & Leung, P. W. L. (2011). The application of the
concerns. Psychological Assessment, 26, 1268 1280. http://dx.doi.org/
Preschool Child Behavior Checklist and the CaregiverTeacher Report
10.1037/pas0000012
Form to mainland Chinese children: Syndrome structure, gender differ-
Martin, J. L., Ford, C. B., Dyer-Friedman, J., Tang, J., & Huffman, L. C.
ences, country effects, and inter-informant agreement. Journal of Ab-
(2004). Patterns of agreement between parent and child ratings of
normal Child Psychology, 39, 251264. http://dx.doi.org/10.1007/
emotional and behavioral problems in an outpatient clinical setting:
s10802-010-9452-8

Liu, X., Sun, Z., Neiderhiser, J. M., Uchiyama, M., Okawa, M., & Rogan, When children endorse more problems. Journal of Developmental and
W. (2001). Behavioral and emotional problems in Chinese adolescents: Behavioral Pediatrics, 25, 150 155. http://dx.doi.org/10.1097/
Parent and teacher reports. Journal of the American Academy of Child & 00004703-200406000-00002

Adolescent Psychiatry, 40, 828 836. http://dx.doi.org/10.1097/ Martin-Storey, A., Serbin, L. A., Stack, D. M., & Schwartzman, A. E.
00004583-200107000-00018 (2009). The behaviour style observation system for young children
Loeber, R., Green, S. M., & Lahey, B. B. (1990). Mental health predicts teacher-reported externalizing behaviour in middle childhood.
professionals perception of the utility of children, mothers, and Infant and Child Development, 18, 337350. http://dx.doi.org/10.1002/
teachers as informants of childhood psychopathology. Journal of icd.601

Clinical Child Psychology, 19, 136 143. http://dx.doi.org/10.1207/ Mascendaro, P. M., Herman, K. C., & Webster-Stratton, C. (2012). Parent
s15374424jccp1902_5 discrepancies in ratings of young childrens co-occurring internalizing
Loeber, R., Green, S. M., Lahey, B. B., & Stouthamer-Loeber, M. (1989). symptoms. School Psychology Quarterly, 27, 134 143. http://dx.doi
Optimal informants on childhood disruptive behaviors. Development .org/10.1037/a0029320
and Psychopathology, 1, 317337. http://dx.doi.org/10.1017/ Mash, E. J., & Hunsley, J. (Eds.). (2005). Evidence-based assessment of
S095457940000050X child and adolescent disorders: Issues and challenges [Special section].
Journal of Clinical Child and Adolescent Psychology. http://dx.doi.org/
Loomans, E. M., van der Stelt, O., van Eijsden, M., Gemke, R. J.,
Vrijkotte, T., & den Bergh, B. R. (2011). Antenatal maternal anxiety is 10.1207/s15374424jccp3403_1

associated with problem behaviour at age five. Early Human Develop- Mattison, R. E., Carlson, G. A., Cantwell, D. P., & Asarnow, J. R. (2007).
ment, 87, 565570. http://dx.doi.org/10.1016/j.earlhumdev.2011.04.014 Teacher and parent ratings of children with depressive disorders. Journal
of Emotional and Behavioral Disorders, 15, 184 192. http://dx.doi.org/
Lopata, C., Toomey, J. A., Fox, J. D., Volker, M. A., Chow, S. Y.,
Thomeer, M. L., . . . Smerbeck, A. M. (2010). Anxiety and depression 10.1177/10634266070150030501

in children with HFASDs: Symptom levels and source differences. McConaughy, S. H., Stanger, C., & Achenbach, T. M. (1992). Three-year
Journal of Abnormal Child Psychology, 38, 765776. http://dx.doi.org/ course of behavioral/emotional problems in a national sample of 4- to
10.1007/s10802-010-9406-1 16-year-olds: I. Agreement among informants. Journal of the American

Low, S. M., & Stocker, C. (2005). Family functioning and childrens Academy of Child & Adolescent Psychiatry, 31, 932940. http://dx.doi
adjustment: Associations among parents depressed mood, marital hos- .org/10.1097/00004583-199209000-00023
894 DE LOS REYES ET AL.


McLaughlin, K. A., Alves, S., & Sheridan, M. A. (2014). Vagal regulation Mulvaney, M. K., Mebert, C. J., & Flint, J. (2007). Parental affect and
and internalizing psychopathology among adolescents exposed to child- childrearing beliefs uniquely predict mothers and fathers ratings of
hood adversity. Developmental Psychobiology, 56, 1036 1051. http:// childrens behavior problems. Journal of Applied Developmental Psy-
dx.doi.org/10.1002/dev.21187 chology, 28, 445 457. http://dx.doi.org/10.1016/j.appdev.2007.06.001

McMahon, R. J., & Frick, P. J. (2005). Evidence-based assessment of Munkvold, L., Lundervold, A., Lie, S. A., & Manger, T. (2009). Should
conduct problems in children and adolescents. Journal of Clinical Child there be separate parent and teacher-based categories of ODD? Evidence
and Adolescent Psychology, 34, 477505. http://dx.doi.org/10.1207/ from a general population. Journal of Child Psychology and Psychiatry,
s15374424jccp3403_6 50, 1264 1272. http://dx.doi.org/10.1111/j.1469-7610.2009.02091.x

Meesters, C., Muris, P., Ghys, A., Reumerman, T., & Rooijmans, M. Muris, P., & Maas, A. (2004). Strengths and difficulties as correlates of
(2003). The Childrens Somatization Inventory: Further evidence for its attachment style in institutionalized and non-institutionalized children
reliability and validity in a pediatric and a community sample of Dutch with below-average intellectual abilities. Child Psychiatry and Human
children and adolescents. Journal of Pediatric Psychology, 28, 413 422. Development, 34, 317328. http://dx.doi.org/10.1023/B:CHUD
http://dx.doi.org/10.1093/jpepsy/jsg031 .0000020682.55697.4f

Mellor, D. (2004). Furthering the use of the Strengths and Difficulties Muris, P., & Meesters, C. (2003). The validity of attention deficit hyper-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Questionnaire: Reliability with younger child respondents. Psychologi- activity and hyperkinetic disorder symptom domains in nonclinical
This document is copyrighted by the American Psychological Association or one of its allied publishers.

cal Assessment, 16, 396 401. http://dx.doi.org/10.1037/1040-3590.16.4 Dutch children. Journal of Clinical Child & Adolescent Psychology, 32,
.396 460 466. http://dx.doi.org/10.1207/S15374424JCCP3203_14

Mellor, D., Wong, J., & Xu, X. (2011). Interparent agreement on the Muris, P., Meesters, C., & Rompelberg, L. (2007). Attention control in
strengths and difficulties questionnaire: A Chinese study. Journal of middle childhood: Relations to psychopathological symptoms and threat
Clinical Child & Adolescent Psychology, 40, 890 896. http://dx.doi perception distortions. Behaviour Research and Therapy, 45, 9971010.
.org/10.1080/15374416.2011.614580 http://dx.doi.org/10.1016/j.brat.2006.07.010

Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, Muris, P., Meesters, C., & Timmermans, A. (2013). Some youths have a
R. R., . . . Reed, G. M. (2001). Psychological testing and psychological gloomy side: Correlates of the dark triad personality traits in non-clinical
assessment. A review of evidence and issues. American Psychologist, adolescents. Child Psychiatry & Human Development, 44, 658 665.
56, 128 165. http://dx.doi.org/10.1037/0003-066X.56.2.128 http://dx.doi.org/10.1007/s10578-013-0359-9

Mikami, A. Y., Lee, S. S., Hinshaw, S. P., & Mullin, B. C. (2008). Murray, D. W., Kollins, S. H., Hardy, K. K., Abikoff, H. B., Swanson,
Relationships between social information processing and aggression J. M., Cunningham, C., . . . Chuang, S. Z. (2007). Parent versus teacher
among adolescent girls with and without ADHD. Journal of Youth and ratings of attention-deficit/hyperactivity disorder symptoms in the Pre-
Adolescence, 37, 761771. http://dx.doi.org/10.1007/s10964-007- schoolers with Attention-Deficit/Hyperactivity Disorder Treatment
9237-8 Study (PATS). Journal of Child and Adolescent Psychopharmacology,

Mikami, A. Y., & Pfiffner, L. J. (2008). Sibling relationships among 17, 605 619. http://dx.doi.org/10.1089/cap.2007.0060

children with ADHD. Journal of Attention Disorders, 11, 482 492. Najman, J. M., Williams, G. M., Nikles, J., Spence, S., Bor, W.,
http://dx.doi.org/10.1177/1087054706295670 OCallaghan, M., . . . Andersen, M. J. (2000). Mothers mental illness
Miller, J. D., Pilkonis, P. A., & Clifton, A. (2005). Self- and other-reports and child behavior problems: Cause effect association or observation
of traits from the five-factor model: Relations to personality disorder. bias? Journal of the American Academy of Child & Adolescent Psychi-
Journal of Personality Disorders, 19, 400 419. http://dx.doi.org/10.1521/ atry, 39, 592 602. http://dx.doi.org/10.1097/00004583-200005000-
pedi.2005.19.4.400 00013
Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of National Institute of Mental Health. (2008). The National Institute of
personality: Reconceptualizing situations, dispositions, dynamics, and Mental Health strategic plan (NIH Publication No. 08 6368). Retrieved
invariance in personality structure. Psychological Review, 102, 246 from http://www.nimh.nih.gov/about/strategic-planning-reports/index
268. http://dx.doi.org/10.1037/0033-295X.102.2.246 .shtml

Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn, J. H., & Halperin, Natsuaki, M. N., Klimes-Dougan, B., Ge, X., Shirtcliff, E. A., Hastings,
J. M. (2000). Parentteacher concordance for DSM-IV attention-deficit/ P. D., & Zahn-Waxler, C. (2009). Early pubertal maturation and inter-
hyperactivity disorder in a clinic-referred sample. Journal of the Amer- nalizing problems in adolescence: Sex differences in the role of
ican Academy of Child & Adolescent Psychiatry, 39, 308 313. http:// cortisol reactivity to interpersonal stress. Journal of Clinical Child and
dx.doi.org/10.1097/00004583-200003000-00012 Adolescent Psychology, 38, 513524. http://dx.doi.org/10.1080/
Mller, A. P., & Jennions, M. D. (2002). How much variance can be 15374410902976320

explained by ecologists and evolutionary biologists? Oecologia, 132, Nauta, M. H., Scholing, A., Rapee, R. M., Abbott, M., Spence, S. H., &
492500. http://dx.doi.org/10.1007/s00442-002-0952-2 Waters, A. (2004). A parent-report measure of childrens anxiety: Psy-

Montgomery, E. (2008). Self- and parent assessment of mental health: chometric properties and comparison with child-report in a clinic and
Disagreement on externalizing and internalizing behaviour in young normal sample. Behaviour Research and Therapy, 42, 813 839. http://
refugees from the Middle East. Clinical Child Psychology and Psychi- dx.doi.org/10.1016/S0005-7967(03)00200-6
atry, 13, 49 63. http://dx.doi.org/10.1177/1359104507086341 Neal, J. W., Cappella, E., Wagner, C., & Atkins, M. S. (2011). Seeing eye

Morrow, M. T., Hubbard, J. A., Rubin, R. M., & McAuliffe, M. D. (2008). to eye: Predicting teacher-student agreement on classroom social net-
The relation between childhood aggression and depressive symptoms: works. Social Development, 20, 376 393. http://dx.doi.org/10.1111/j
The unique and joint mediating roles of peer rejection and peer victim- .1467-9507.2010.00582.x

ization. Merrill-Palmer Quarterly, 54, 316 340. http://dx.doi.org/ New, M., Razzino, B., Lewin, A., Schlumpf, K., & Joseph, J. (2002).
10.1353/mpq.0.0000 Mental health service use in a community head start population. Ar-

Moss, E., Smolla, N., Cyr, C., Dubois-Comtois, K., Mazzarello, T., & chives of Pediatrics and Adolescent Medicine, 156, 721727. http://dx
Berthiaume, C. (2006). Attachment and behavior problems in middle .doi.org/10.1001/archpedi.156.7.721

childhood as reported by adult and child informants. Development Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent
and Psychopathology, 18, 425 444. http://dx.doi.org/10.1017/ validity of the Devereux Student Strengths Assessment. School Psychol-
S0954579406060238 ogy Quarterly, 24, 48 59. http://dx.doi.org/10.1037/a0015147
VALIDITY OF MULTI-INFORMANT ASSESSMENT 895

Niditch, L. A., & Varela, R. E. (2011). Mother child disagreement in Perry-Parrish, C., Waasdorp, T. E., & Bradshaw, C. P. (2012). Peer
reports of child anxiety: Effects of child age and maternal anxiety. nominations of emotional expressivity among urban children: Social and
Journal of Anxiety Disorders, 25, 450 455. http://dx.doi.org/10.1016/j psychological correlates. Social Development, 21, 88 108. http://dx.doi
.janxdis.2010.11.009 .org/10.1111/j.1467-9507.2011.00615.x
Nimon, K. (2010). Regression commonality analysis: Demonstration of an Peterson, R. A. (2001). On the use of college students in social science
SPSS solution. Multiple Linear Regression Viewpoints, 36, 10 17. Re- research: Insights from a second-order meta-analysis. Journal of Con-
trieved from http://mlrv.ua.edu/2010/vol36_1/nimon_proof_2.pdf sumer Research, 28, 450 461. http://dx.doi.org/10.1086/323732

OBrien, J., Margolin, G., & John, R. S. (1995). Relation among Peterson, R. L., Kirkwood, M. W., Taylor, H. G., Stancin, T., Brown,
marital conflict, child coping, and child adjustment. Journal of Clinical T. M., & Wade, S. L. (2013). Adolescents internalizing problems
Child Psychology, 24, 346 361. http://dx.doi.org/10.1207/ following traumatic brain injury are related to parents psychiatric symp-
s15374424jccp2403_12 toms. Journal of Head Trauma Rehabilitation, 28. http://dx.doi.org/
Odgers, C. L., Caspi, A., Bates, C. J., Sampson, R. J., & Moffitt, T. E. 10.1097/HTR.0b013e318263f5ba

(2012). Systematic social observation of childrens neighborhoods using Petot, D., Rescorla, L., & Petot, J.-M. (2011). Agreement between parent-
Google Street View: A reliable and cost-effective method. Journal of and self-reports of Algerian adolescents behavioral and emotional prob-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Child Psychology and Psychiatry, 53, 1009 1017. http://dx.doi.org/ lems. Journal of Adolescence, 34, 977986. http://dx.doi.org/10.1016/j
This document is copyrighted by the American Psychological Association or one of its allied publishers.

10.1111/j.1469-7610.2012.02565.x .adolescence.2010.11.011

Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Phares, V., & Compas, B. E. (1990). Adolescents subjective distress over
Sanford, M., & Lipman, E. L. (1996). Integrating assessment data from their emotional/behavioral problems. Journal of Consulting and Clinical
multiple informants. Journal of the American Academy of Child and Psychology, 58, 596 603. http://dx.doi.org/10.1037/0022-006X.58.5
Adolescent Psychiatry, 35, 1078 1085. http://dx.doi.org/10.1097/ .596

00004583-199608000-00019 Phares, V., Compas, B. E., & Howell, D. C. (1989). Perspectives on child

Oldehinkel, A. J., Hartman, C. A., De Winter, A. F., Veenstra, R., & behavior problems: Comparisons of childrens self-reports with parent
Ormel, J. (2004). Temperament profiles associated with internalizing and teacher reports. Psychological Assessment, 1, 68 71. http://dx.doi
and externalizing problems in preadolescence. Development and .org/10.1037/1040-3590.1.1.68
Psychopathology, 16, 421 440. http://dx.doi.org/10.1017/ Phares, V., & Danforth, J. S. (1994). Adolescents, parents, and teachers
S0954579404044591 distress over adolescents behavior. Journal of Abnormal Child Psychol-

Oldehinkel, A. J., Veenstra, R., Ormel, J., de Winter, A. F., & Verhulst, ogy, 22, 721732. http://dx.doi.org/10.1007/BF02171998

F. C. (2006). Temperament, parenting, and depressive symptoms in a Phillips, B. M., Lonigan, C. J., Driscoll, K., & Hooe, E. S. (2002). Positive
population sample of preadolescents. Journal of Child Psychology and and negative affectivity in children: A multitrait-multimethod investiga-
Psychiatry, 47, 684 695. http://dx.doi.org/10.1111/j.1469-7610.2005 tion. Journal of Clinical Child and Adolescent Psychology, 31, 465 479.
.01535.x http://dx.doi.org/10.1207/153744202320802142

OLeary-Barrett, M., Topper, L., Al-Khudhairy, N., Pihl, R. O., Phillips, P. L., Greenson, J. N., Collett, B. R., & Gimpel, G. A. (2002).
Castellanos-Ryan, N., Mackie, C. J., & Conrod, P. J. (2013). Two-year Assessing ADHD symptoms in preschool children: Use of the ADHD
impact of personality-targeted, teacher-delivered interventions on youth symptoms rating scale. Early Education and Development, 13, 283299.
internalizing and externalizing problems: A cluster-randomized trial. http://dx.doi.org/10.1207/s15566935eed1303_3
Journal of the American Academy of Child & Adolescent Psychiatry, 52, Piacentini, J. C., Cohen, P., & Cohen, J. (1992). Combining discrepant
911920. http://dx.doi.org/10.1016/j.jaac.2013.05.020 diagnostic information from multiple sources: Are complex algorithms
Oltmanns, T. F., & Turkheimer, E. (2009). Person perception and person- better than simple ones? Journal of Abnormal Child Psychology, 20,
ality pathology. Current Directions in Psychological Science, 18, 3236. 51 63. http://dx.doi.org/10.1007/BF00927116

http://dx.doi.org/10.1111/j.1467-8721.2009.01601.x Pierrehumbert, B., Bader, M., Thevoz, S., Kinal, A., & Halfon, O. (2006).
Owens, J., & Hoza, B. (2003). Diagnostic utility of DSMIVTR symptoms Hyperactivity and attention problems in a Swiss sample of school-aged
in the prediction of DSMIVTR ADHD subtypes and ODD. Journal children: Effects of school achievement, child gender, and informants.
of Attention Disorders, 7, 1127. http://dx.doi.org/10.1177/ Journal of Attention Disorders, 10, 6576. http://dx.doi.org/10.1177/
108705470300700102 1087054705286050

Pastorelli, C., Barbaranelli, C., Cermak, I., Rozsa, S., & Caprara, G. V. Power, T. J., Andrews, T. J., Eiraldi, R. B., Doherty, B. J., Ikeda, M. J.,
(1997). Measuring emotional instability, prosocial behavior and aggres- DuPaul, G. J., & Landau, S. (1998). Evaluating attention deficit hyper-
sion in pre-adolescents: A cross-national study. Personality and Indi- activity disorder using multiple informants: The incremental utility of
vidual Differences, 23, 691703. http://dx.doi.org/10.1016/S0191- combining teacher with parent reports. Psychological Assessment, 10,
8869(97)00056-1 250 260. http://dx.doi.org/10.1037/1040-3590.10.3.250

Pearson, D. A., Aman, M. G., Arnold, L. E., Lane, D. M., Loveland, K. A., Puura, K., Almqvist, F., Tamminen, T., Piha, J., Rasanen, E., Kumpulai-
Santos, C. W., . . . Cleveland, L. A. (2012). High concordance of parent nen, K., . . . Koivisto, A.-M. (1998). Psychiatric disturbances among
and teacher attention-deficit/hyperactivity disorder ratings in medicated pre-pubertal children in Southern Finland. Social Psychiatry and Psy-
and unmedicated children with autism spectrum disorders. Journal of chiatric Epidemiology, 33, 310 318. http://dx.doi.org/10.1007/
Child and Adolescent Psychopharmacology, 22, 284 291. http://dx.doi s001270050060

.org/10.1089/cap.2011.0067 Randazzo, K. V., Landsverk, J., & Ganger, W. (2003). Three informants
Pelham, W. E., Jr., Fabiano, G. A., & Massetti, G. M. (2005). Evidence- reports of child behavior: Parents, teachers, and foster parents. Journal
based assessment of attention deficit hyperactivity disorder in children of the American Academy of Child & Adolescent Psychiatry, 42, 1343
and adolescents. Journal of Clinical Child and Adolescent Psychology, 1350. http://dx.doi.org/10.1097/01.chi.0000085753.71002.da

34, 449 476. http://dx.doi.org/10.1207/s15374424jccp3403_5 Rapee, R. M., Barrett, P. M., Dadds, M. R., & Evans, L. (1994). Reli-

Penney, S. R., & Skilling, T. A. (2012). Moderators of informant agree- ability of the DSMIIIR childhood anxiety disorders using structured
ment in the assessment of adolescent psychopathology: Extension to a interview: Interrater and parent-child agreement. Journal of the Ameri-
forensic sample. Psychological Assessment, 24, 386 401. http://dx.doi can Academy of Child and Adolescent Psychiatry, 33, 984 992. http://
.org/10.1037/a0025693 dx.doi.org/10.1097/00004583-199409000-00008
896 DE LOS REYES ET AL.


Raviv, A., Erel, O., Fox, N. A., Leavitt, L. A., Raviv, A., Dar, I., . . . Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach
Greenbaum, C. W. (2001). Individual measurement of exposure to to the measurement of change. Psychological Bulletin, 92, 726 748.
everyday violence among elementary schoolchildren across various set- http://dx.doi.org/10.1037/0033-2909.92.3.726
tings. Journal of Community Psychology, 29, 117140. http://dx.doi.org/ Rogosa, D. R., & Willett, J. B. (1983). Demonstrating the reliability of the
10.1002/1520-6629(200103)29:2117::AID-JCOP10093.0.CO;2-2 difference score in the measurement of change. Journal of Educational
Ready, R. E., Watson, D., & Clark, L. A. (2002). Psychiatric patient- Measurement, 20, 335343. http://dx.doi.org/10.1111/j.1745-3984.1983
and informant-reported personality: Predicting concurrent and future .tb00211.x

behavior. Assessment, 9, 361372. http://dx.doi.org/10.1177/ Rosenbaum, M., & Ronen, T. (1997). Parents and childrens appraisals
1073191102238157 of each others anxiety while facing a common threat. Journal of

Redmond, S. M., & Rich, M. L. (2002). Stability of behavioral ratings of Clinical Child Psychology, 26, 4352. http://dx.doi.org/10.1207/
children with SLI. Journal of Speech, Language, and Hearing Research, s15374424jccp2601_5

45, 190 201. http://dx.doi.org/10.1044/1092-4388(2002/014) Rosenberg, L. A., Brown, J., & Singer, H. S. (1994). Self-reporting of
behavior problems in patients with tic disorders. Psychological Reports,
Reeve, E. A., Bernstein, G. A., & Christenson, G. A. (1992). Clinical
74, 653 654. http://dx.doi.org/10.2466/pr0.1994.74.2.653
characteristics and psychiatric comorbidity in children with trichotillo-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Rosenthal, J. A., & Curiel, H. F. (2006). Modeling behavioral problems of


mania. Journal of the American Academy of Child & Adolescent Psy-
This document is copyrighted by the American Psychological Association or one of its allied publishers.

children in the child welfare system: Caregiver, youth, and teacher


chiatry, 31, 132138. http://dx.doi.org/10.1097/00004583-199201000-
perceptions. Children and Youth Services Review, 28, 13911408. http://
00020
dx.doi.org/10.1016/j.childyouth.2006.02.007
Reidler, E. B., & Swenson, L. P. (2012). Discrepancies between youth and
Roskam, I., & Stievenart, M. (2014). Is there a common pathway to
mothers perceptions of their mother child relationship quality and
maladjustment for internationally adopted and non-adopted adolescents?
self-disclosure: Implications for youth- and mother-reported youth ad- Journal of Applied Developmental Psychology, 35, 215222. http://dx
justment. Journal of Youth and Adolescence, 41, 11511167. http://dx .doi.org/10.1016/j.appdev.2013.12.005
.doi.org/10.1007/s10964-012-9773-8
Rosnati, R., Montirosso, R., & Barni, D. (2008). Behavioral and emotional
Renk, K., & Phares, V. (2004). Cross-informant ratings of social compe- problems among Italian international adoptees and non-adopted chil-
tence in children and adolescents. Clinical Psychology Review, 24, dren: Fathers and mothers reports. Journal of Family Psychology, 22,
239 254. http://dx.doi.org/10.1016/j.cpr.2004.01.004 541549. http://dx.doi.org/10.1037/0893-3200.22.3.541

Rescorla, L. A., Achenbach, T. M., Ivanova, M. Y., Bilenberg, N., Roth, A., & Fonagy, P. (2005). What works for whom?: A critical review
Bjarnadottir, G., Denner, S., . . . Verhulst, F. C. (2012). Behavioral/ of psychotherapy research (2nd ed.). New York, NY: Guilford Press.
emotional problems of preschoolers: Caregiver/teacher reports from 15
Rothbaum, F., Rosen, K. S., Pott, M., & Beatty, M. (1995). Early parent-
societies. Journal of Emotional and Behavioral Disorders, 20, 68 81. child relationships and later problem behavior: A longitudinal study.
http://dx.doi.org/10.1177/1063426611434158 Merrill-Palmer Quarterly, 41, 133151. Retrieved from http://www

Rescorla, L. A., Bochicchio, L., Achenbach, T. M., Ivanova, M. Y., .jstor.org/stable/23090528

Almqvist, F., Begovac, I., . . . Verhulst, F. C. (2014). Parent-teacher Rubio-Stipec, M., Canino, G. J., Shrout, P., Dulcan, M., Freeman, D., &
agreement on childrens problems in 21 societies. Journal of Clinical Bravo, M. (1994). Psychometric properties of parents and children as
Child and Adolescent Psychology, 43, 627 642. http://dx.doi.org/ informants in child psychiatry epidemiology with the Spanish Diagnos-
10.1080/15374416.2014.900719 tic Interview Schedule for Children (DISC-2). Journal of Abnormal
Child Psychology, 22, 703720. http://dx.doi.org/10.1007/BF02171997
Rescorla, L. A., Ginzburg, S., Achenbach, T. M., Ivanova, M. Y.,

Almqvist, F., Begovac, I., . . . Verhulst, F. C. (2013). Cross-informant Ruchkin, V., Koposov, R., Vermeiren, R., & Schwab-Stone, M. (2012).
agreement between parent-reported and adolescent self-reported prob- The Strength and Difficulties Questionnaire: Russian validation of the
lems in 25 societies. Journal of Clinical Child and Adolescent Psychol- teacher version and comparison of teacher and student reports. Journal
ogy, 42, 262273. http://dx.doi.org/10.1080/15374416.2012.717870 of Adolescence, 35, 8796. http://dx.doi.org/10.1016/j.adolescence.2011

Rettew, D. C., van Oort, F. V. A., Verhulst, F. C., Buitelaar, J. K., Ormel, .06.003
J., Hartman, C., . . . Hudziak, J. J. (2011). When parent and teacher Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richard-
ratings dont agree: The Tracking Adolescents Individual Lives Survey son, W. S. (1996). Evidence based medicine: What it is and what it isnt.
British Medical Journal, 312, 7172. http://dx.doi.org/10.1136/bmj.312
(TRAILS). Journal of Child and Adolescent Psychopharmacology, 21,
.7023.71
389 397. http://dx.doi.org/10.1089/cap.2010.0153
Salbach-Andrae, H., Klinkowski, N., Lenz, K., & Lehmkuhl, U. (2009).
Reuterskild, L., Ost, L.-G., & Ollendick, T. (2008). Exploring child and
Agreement between youth-reported and parent-reported psychopathol-
parent factors in the diagnostic agreement on the Anxiety Disorders
ogy in a referred sample. European Child & Adolescent Psychiatry, 18,
Interview Schedule. Journal of Psychopathology and Behavioral Assess-
136 143. http://dx.doi.org/10.1007/s00787-008-0710-z
ment, 30, 279 290. http://dx.doi.org/10.1007/s10862-008-9081-5
Salbach-Andrae, H., Klinkowski, N., Lenz, K., Pfeiffer, E., Lehmkuhl, U.,
Richters, J. E. (1992). Depressed mothers as informants about their chil-
& Ehrlich, S. (2008). Correspondence between self-reported and parent-
dren: A critical review of the evidence for distortion. Psychological reported psychopathology in adolescents with eating disorders. Psycho-
Bulletin, 112, 485 499. http://dx.doi.org/10.1037/0033-2909.112.3.485 pathology, 41, 307312. http://dx.doi.org/10.1159/000146068

Rockhill, C. M., Russo, J. E., McCauley, E., Katon, W. J., Richardson,
Salbach-Andrae, H., Lenz, K., & Lehmkuhl, U. (2009). Patterns of agree-
L. P., & Lozano, P. (2007). Agreement between parents and children ment among parent, teacher and youth ratings in a referred sample.
regarding anxiety and depression diagnoses in children with asthma. European Psychiatry, 24, 345351. http://dx.doi.org/10.1016/j.eurpsy
Journal of Nervous & Mental Disease, 195, 897904. http://dx.doi.org/ .2008.07.008
10.1097/NMD.0b013e318159289c Sanislow, C. A., Pine, D. S., Quinn, K. J., Kozak, M. J., Garvey, M. A.,

Rodriguez, E. M., Nichols, S. R., Javdani, S., Emerson, E., & Donenberg, Heinssen, R. K., . . . Cuthbert, B. N. (2010). Developing constructs for
G. R. (2013). Economic hardship, parent positive communication and psychopathology research: Research domain criteria. Journal of Abnor-
mental health in urban adolescents seeking outpatient psychiatric care. mal Psychology, 119, 631 639. http://dx.doi.org/10.1037/a0020909

Journal of Child and Family Studies. Advance online publication. http:// Satake, H., Yoshida, K., Yamashita, H., Kinukawa, N., & Takagishi, T.
dx.doi.org/10.1007/s10826-013-9872-5 (2003). Agreement between parents and teachers on behavioral/
VALIDITY OF MULTI-INFORMANT ASSESSMENT 897

emotional problems in Japanese school children using the Child Behav- rica. AIDS and Behavior, 18, 1174 1185. http://dx.doi.org/10.1007/
ior Checklist. Child Psychiatry and Human Development, 34, 111126. s10461-014-0739-6

http://dx.doi.org/10.1023/A:1027342007736 Shemmassian, S. K., & Lee, S. S. (2014). Comparative validity of

Saudino, K. J., Ronald, A., & Plomin, R. (2005). The etiology of behavior DSM-IV and alternative empirically derived approaches for the assess-
problems in 7-year-old twins: Substantial genetic influence and negli- ment of ADHD. Journal of Attention Disorders. Advance online publi-
gible shared environmental influence for parent ratings and ratings by cation. http://dx.doi.org/10.1177/1087054714522511
same and different teachers. Journal of Abnormal Child Psychology, 33, Silver, C. H. (2014). Sources of data about childrens executive function-
113130. http://dx.doi.org/10.1007/s10802-005-0939-7 ing: Review and commentary. Child Neuropsychology, 20, 113. http://

Saunders, C. D., Voelker, S. L., & Shore, D. L. (2002). Comparison of dx.doi.org/10.1080/09297049.2012.727793
mother and father report on the personality inventory for children in a Silverman, W. K., & Ollendick, T. H. (2005). Evidence-based assessment
non referred Canadian sample. Canadian Journal of School Psychology, of anxiety and its disorders in children and adolescents. Journal of
17, 8595. http://dx.doi.org/10.1177/082957350201700107 Clinical Child and Adolescent Psychology, 34, 380 411. http://dx.doi

Sawyer, M. G., Baghurst, P., & Clark, J. (1992). Differences between .org/10.1207/s15374424jccp3403_2
reports from children, parents and teachers: Implications for epidemio- Silverman, W. K., & Rabian, B. (1995). Testretest reliability of the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

logical studies. Australian and New Zealand Journal of Psychiatry, 26, DSMIIIR childhood anxiety disorders symptoms using the Anxiety
This document is copyrighted by the American Psychological Association or one of its allied publishers.

652 660. http://dx.doi.org/10.3109/00048679209072102 Disorders Interview Schedule for children. Journal of Anxiety Disorders,

Sawyer, M. G., Baghurst, P., & Mathias, J. (1992). Differences between 9, 139 150. http://dx.doi.org/10.1016/0887-6185(94)00032-8

informants reports describing emotional and behavioural problems in Simmel, C., Lee, I., Kim, S., & Miles, J. (2014). Multiple assessors of
community and clinic-referred children: A research note. Child Psychol- child welfare youths mental-health functioning: Comparing perceptions
ogy & Psychiatry & Allied Disciplines, 33, 441 449. http://dx.doi.org/ of adolescents, caregivers and teachers. Child & Family Social Work, 19,
10.1111/j.1469-7610.1992.tb00878.x 343354. http://dx.doi.org/10.1111/cfs.12024

Schenck, C. E., Braver, S. L., Wolchik, S. A., Saenz, D., Cookston, J. T., Smetana, J. G. (2008). Its 10 oclock: Do you know where your children
& Fabricius, W. V. (2009). Relations between mattering to step- and are? Recent advances in understanding parental monitoring and ado-
non-residential fathers and adolescent mental health. Fathering, 7, 70 lescents information management. Child Development Perspectives, 2,
90. http://dx.doi.org/10.3149/fth.0701.70 19 25. http://dx.doi.org/10.1111/j.1750-8606.2008.00036.x
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological Smetana, J. G., Campione-Barr, N., & Metzger, A. (2006). Adolescent
research: Lessons from 26 research scenarios. Psychological Methods, 1, development in interpersonal and societal contexts. Annual Review of
199 223. http://dx.doi.org/10.1037/1082-989X.1.2.199 Psychology, 57, 255284. http://dx.doi.org/10.1146/annurev.psych.57

Schroeder, J. F., Hood, M. M., & Hughes, H. M. (2010). Inter-parent .102904.190124

agreement on the syndrome scales of the Child Behavior Checklist Smider, N. A., Essex, M. J., Kalin, N. H., Buss, K. A., Klein, M. H.,
(CBCL): Correspondence and discrepancies. Journal of Child and Fam- Davidson, R. J., & Goldsmith, H. H. (2002). Salivary cortisol as a
ily Studies, 19, 646 653. http://dx.doi.org/10.1007/s10826-010-9352-0 predictor of socioemotional adjustment during kindergarten: A prospec-
Schwarz, N. (1999). Self-reports: How the questions shape the answers. tive study. Child Development, 73, 7592. http://dx.doi.org/10.1111/
American Psychologist, 54, 93105. http://dx.doi.org/10.1037/0003- 1467-8624.00393
066X.54.2.93 Smith, B. H., Pelham, W. E., Jr., Gnagy, E., Molina, B., & Evans, S.

Seiffge-Krenke, I. (2000). Causal links between stressful events, coping (2000). The reliability, validity, and unique contributions of self-report
style, and adolescent symptomatology. Journal of Adolescence, 23, by adolescents receiving treatment for attention-deficit/hyperactivity
675 691. http://dx.doi.org/10.1006/jado.2000.0352 disorder. Journal of Consulting and Clinical Psychology, 68, 489 499.

Selbo-Bruns, A., Nakamura, B. J., Higa-McMillan, C. K., Kentaro, H., & http://dx.doi.org/10.1037/0022-006X.68.3.489
Chorpita, B. F. (2012). A multi-informant multi-measure approach to the Smith, G. T., Fischer, S., & Fister, S. M. (2003). Incremental validity
two factor model of emotion with youth. Journal of Psychopathology principles in test construction. Psychological Assessment, 15, 467 477.
and Behavioral Assessment, 34, 520 530. http://dx.doi.org/10.1007/ http://dx.doi.org/10.1037/1040-3590.15.4.467

s10862-012-9298-1 Smith, M. A., & Jenkins, J. M. (1991). The effects of marital disharmony

Self-Brown, S., LeBlanc, M., & Kelley, M. L. (2004). Effects of violence on prepubertal children. Journal of Abnormal Child Psychology, 19,
exposure and daily stressors on psychological outcomes in urban ado- 625 644. http://dx.doi.org/10.1007/BF00918903

lescents. Journal of Traumatic Stress, 17, 519 527. http://dx.doi.org/ Smith Carter, J., Garber, J., Jeffrey, A., & Cole, D. A. (2006). Modeling
10.1007/s10960-004-5801-0 relations between hassles and internalizing and externalizing symptoms

Sentse, M., & Laird, R. D. (2010). Parent child relationships and dyadic in adolescents: A four-year prospective study. Journal of Abnormal
friendship experiences as predictors of behavior problems in early ado- Psychology, 115, 428 442. http://dx.doi.org/10.1037/0021-843X.115.3
lescence. Journal of Clinical Child and Adolescent Psychology, 39, .428

873 884. http://dx.doi.org/10.1080/15374416.2010.517160 Snyder, H. T., Bilboul, M. J., & Pope, A. W. (2005). Psychosocial

Sentse, M., Lindenberg, S., Omvlee, A., Ormel, J., & Veenstra, R. (2010). adjustment in adolescents with craniofacial anomalies: A comparison of
Rejection and acceptance across contexts: Parents and peers as risks and parent and self-reports. The Cleft Palate-Craniofacial Journal, 42, 548
buffers for early adolescent psychopathology. The TRAILS study. Jour- 555. http://dx.doi.org/10.1597/04-078R.1

nal of Abnormal Child Psychology, 38, 119 130. http://dx.doi.org/ Sollie, H., Larsson, B., & Morch, W. T. (2013). Comparison of mother,
10.1007/s10802-009-9351-z father, and teacher reports of ADHD core symptoms in a sample of child

Sharp, C., Mosko, O., Chang, B., & Ha, C. (2011). The cross-informant psychiatric outpatients. Journal of Attention Disorders, 17, 699 710.
concordance and concurrent validity of the Borderline Personality Fea- http://dx.doi.org/10.1177/1087054711436010

tures Scale for Children in a community sample of boys. Clinical Child Soper, A. C., Wolchik, S. A., Sandler, I. N., Tein, J.-Y., & Lustig, J. L.
Psychology and Psychiatry, 16, 335349. http://dx.doi.org/10.1177/ (2007). Predictors of threat to self-appraisals for children from divorced
1359104510366279 homes. Journal of Divorce & Remarriage, 48, 127. http://dx.doi.org/

Sharp, C., Venta, A., Marais, L., Skinner, D., Lenka, M., & Serekoane, J. 10.1300/J087v48n01_01

(2014). First evaluation of a population-based screen to detect Sorlie, M.-A., Hagen, K. A., & Ogden, T. (2008). Social competence and
emotional-behavior disorders in orphaned children in Sub-Saharan Af- antisocial behavior: Continuity and distinctiveness across early adoles-
898 DE LOS REYES ET AL.

cence. Journal of Research on Adolescence, 18, 121144. http://dx.doi search, Practice, and Policy, 2, 159 168. http://dx.doi.org/10.1037/
.org/10.1111/j.1532-7795.2008.00553.x a0019156

Sourander, A., Fossum, S., Rnning, J. A., Elonheimo, H., Ristkari, T., Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2011).
Kumpulainen, K., . . . Almqvist, F. (2012). What is the long-term Evidence-based medicine: How to practice and teach EBM (4th ed.).
outcome of boys who steal at age eight? Findings from the Finnish New York, NY: Churchill Livingstone.

nationwide From A Boy To A Man birth cohort study. Social Psychi- Strickland, J., Hopkins, J., & Keenan, K. (2012). Mother-teacher agree-
atry and Psychiatric Epidemiology, 47, 13911400. http://dx.doi.org/ ment on preschoolers symptoms of ODD and CD: Does context matter?
10.1007/s00127-011-0455-8 Journal of Abnormal Child Psychology, 40, 933943. http://dx.doi.org/
10.1007/s10802-012-9622-y
Southam-Gerow, M. A., Flannery-Schroeder, E. C., & Kendall, P. C.

(2003). A psychometric evaluation of the parent report form of the Strijker, J., van Oijen, S., & Knot-Dickscheit, J. (2011). Assessment of
State-Trait Anxiety Inventory for ChildrenTrait Version. Journal of problem behaviour by foster parents and their foster children. Child &
Anxiety Disorders, 17, 427 446. http://dx.doi.org/10.1016/S0887- Family Social Work, 16, 93100. http://dx.doi.org/10.1111/j.1365-2206
6185(02)00223-2 .2010.00717.x

Spear, L. P. (2000). The adolescent brain and age-related behavioral Stuewig, J., Tangney, J. P., Heigel, C., Harty, L., & McCloskey, L.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

manifestations. Neuroscience and Biobehavioral Reviews, 24, 417 463. (2010). Shaming, blaming, and maiming: Functional links among the
This document is copyrighted by the American Psychological Association or one of its allied publishers.

http://dx.doi.org/10.1016/S0149-7634(00)00014-2 moral emotions, externalization of blame, and aggression. Journal of



Spiker, D., Kraemer, H. C., Constantine, N. A., & Bryant, D. (1992). Research in Personality, 44, 91102. http://dx.doi.org/10.1016/j.jrp
Reliability and validity of behavior problem checklists as measures of .2009.12.005
stable traits in low birth weight, premature preschoolers. Child Devel- Swets, J. A. (1992). The science of choosing the right decision threshold in
opment, 63, 14811496. http://dx.doi.org/10.2307/1131570 high-stakes diagnostics. American Psychologist, 47, 522532. http://dx

Stanger, C., & Lewis, M. (1993). Agreement among parents, teachers, and .doi.org/10.1037/0003-066X.47.4.522
children on internalizing and externalizing behavior problems. Journal Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science
of Clinical Child Psychology, 22, 107115. http://dx.doi.org/10.1207/ can improve diagnostic decisions. Psychological Science in the Public
s15374424jccp2201_11 Interest, 1, 126. http://dx.doi.org/10.1111/1529-1006.001

Tackett, J. L., Herzhoff, K., Reardon, K. W., Smack, A. J., & Kushner,
Steele, R. G., Forehand, R., & Armistead, L. (1997). The role of family
S. C. (2013). The relevance of informant discrepancies for the assess-
processes and coping strategies in the relationship between parental
ment of adolescent personality pathology. Clinical Psychology: Science
chronic illness and childhood internalizing problems. Journal of Abnor-
and Practice, 20, 378 392. http://dx.doi.org/10.1111/cpsp.12048
mal Child Psychology, 25, 8394. http://dx.doi.org/10.1023/A:
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid,
1025771210350
R. F. (2011). What forty years of research says about the impact of
Sterling, L., Renno, P., Storch, E. A., Ehrenreich-May, J., Lewin, A. B.,
technology on learning: A second-order meta-analysis and validation
Arnold, E., . . . Wood, J. (2014). Validity of the Revised Childrens
study. Review of Educational Research, 81, 4 28. http://dx.doi.org/
Anxiety and Depression Scale for youth with autism spectrum disorders.
10.3102/0034654310393361
Autism. Advance online publication. http://dx.doi.org/10.1177/
Termine, C., Luoni, C., Selvini, C., Bandera, V., Balottin, U., Eddy,
1362361313510066
C. M., & Cavanna, A. E. (2014). Mother-child agreement on behavioral
Sternberg, K. J., Lamb, M. E., Greenbaum, C., Cicchetti, D., Dawud, S.,
ratings in Tourette syndrome: A controlled study. Journal of Child
Cortes, R. M., . . . Lorey, F. (1993). Effects of domestic violence on
Neurology, 29, 79 83. http://dx.doi.org/10.1177/0883073812470003
childrens behavior problems and depression. Developmental Psychol-
Thomas, S. A., Aldao, A., & De Los Reyes, A. (2012). Implementing
ogy, 29, 44 52. http://dx.doi.org/10.1037/0012-1649.29.1.44

clinically feasible psychophysiological measures in evidence-based as-
Sternberg, K. J., Lamb, M. E., Guterman, E., & Abbott, C. B. (2006). sessments of adolescent social anxiety. Professional Psychology: Re-
Effects of early and later family violence on childrens behavior prob- search and Practice, 43, 510 519. http://dx.doi.org/10.1037/a0029183
lems and depression: A longitudinal, multi-informant perspective. Child
Thuppal, M., Carlson, G. A., Sprafkin, J., & Gadow, K. D. (2002).
Abuse & Neglect, 30, 283306. http://dx.doi.org/10.1016/j.chiabu.2005 Correspondence between adolescent report, parent report, and teacher
.10.008 report of manic symptoms. Journal of Child and Adolescent Psychop-

Stice, E., & Barrera, M., Jr. (1995). A longitudinal examination of the harmacology, 12, 2735. http://dx.doi.org/10.1089/10445460252943542
reciprocal relations between perceived parenting and adolescents sub-
Thurber, S., & Osborn, R. A. (1993). Comparisons of parent and adoles-
stance use and externalizing behaviors. Developmental Psychology, 31, cent perspectives on deviance. The Journal of Genetic Psychology:
322334. http://dx.doi.org/10.1037/0012-1649.31.2.322 Research and Theory on Human Development, 154, 2532. http://dx.doi

Stocker, C. M., Richmond, M. K., Low, S. M., Alexander, E. K., & Elias, .org/10.1080/00221325.1993.9914718
N. M. (2003). Marital conflict and childrens adjustment: Parental hos-
Tinoco, V. M., Lagares, I. J., Moreno, C., Tessier, N. G., & Schneider,
tility and childrens interpretations as mediators. Social Development, B. H. (2009). The nature and basis of teachers concerns about Spanish
12, 149 161. http://dx.doi.org/10.1111/1467-9507.00226 secondary-school students. School Psychology International, 30, 585

Stocker, C. M., Richmond, M. K., Rhoades, G. K., & Kiang, L. (2007). 602. http://dx.doi.org/10.1177/0143034309107078
Family emotional processes and adolescents adjustment. Social Devel-
Trzesniewski, K. H., Moffitt, T. E., Caspi, A., Taylor, A., & Maughan, B.
opment, 16, 310 325. http://dx.doi.org/10.1111/j.1467-9507.2007 (2006). Revisiting the association between reading achievement and
.00386.x antisocial behavior: New evidence of an environmental explanation from

Stone, L. L., van Daal, C., van der Maten, M., Engels, R. C. M. E., a twin study. Child Development, 77, 72 88. http://dx.doi.org/10.1111/
Janssens, J. M. A. M., & Otten, R. (2014). The Berkeley Puppet j.1467-8624.2006.00857.x
Interview: A screening instrument for measuring psychopathology in
Tu, K. M., & Erath, S. A. (2013). Social discomfort in preadolescence:
young children. Child & Youth Care Forum, 43, 211225. http://dx.doi Predictors of discrepancies between preadolescents and their parents and
.org/10.1007/s10566-013-9235-9 teachers. Child Psychiatry & Human Development, 44, 201216. http://

Stover, C. S., Hahn, H., Im, J. J. Y., & Berkowitz, S. (2010). Agreement dx.doi.org/10.1007/s10578-012-0319-9
of parent and child reports of trauma exposure and symptoms in the early Tulbure, B. T., Szentagotai, A., Dobrean, A., & David, D. (2012). Evidence
aftermath of a traumatic event. Psychological Trauma: Theory, Re- based clinical assessment of child and adolescent social phobia: A
VALIDITY OF MULTI-INFORMANT ASSESSMENT 899

critical review of rating scales. Child Psychiatry and Human Develop- Vierhaus, M., & Lohaus, A. (2008). Children and parents as informants of
ment, 43, 795 820. http://dx.doi.org/10.1007/s10578-012-0297-y emotional and behavioural problems predicting female and male ado-

Turchik, J. A., Karpenko, V., & Ogles, B. M. (2007). Further evidence of lescent risk behaviour: A longitudinal cross-informant study. Journal of
the utility and validity of a measure of outcome for children and Youth and Adolescence, 37, 211224. http://dx.doi.org/10.1007/s10964-
adolescents. Journal of Emotional and Behavioral Disorders, 15, 119 007-9193-3
128. http://dx.doi.org/10.1177/10634266070150020201
Villab, M., Gere, M., Torgersen, S., March, J. S., & Kendall, P. C.

van der Ende, J., & Verhulst, F. C. (2005). Informant, gender and age (2012). Diagnostic efficiency of the child and parent versions of the
differences in ratings of adolescent problem behaviour. European Child Multidimensional Anxiety Scale for Children. Journal of Clinical Child
& Adolescent Psychiatry, 14, 117126. http://dx.doi.org/10.1007/ and Adolescent Psychology, 41, 75 85. http://dx.doi.org/10.1080/
s00787-005-0438-y 15374416.2012.632350

van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2012). Agreement of Visser, A., Huizinga, G. A., Hoekstra, H. J., Van Der Graaf, W. T. A.,
informants on emotional and behavioral problems from childhood to Klip, E. C., Pras, E., & Hoekstra-Weebers, J. E. H. M. (2005). Emotional
adulthood. Psychological Assessment, 24, 293300. http://dx.doi.org/ and behavioural functioning of children of a parent diagnosed with
10.1037/a0025500 cancer: A cross-informant perspective. Psycho-Oncology, 14, 746 758.

This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

van der Oord, S., Prins, P. J., Oosterlaan, J., & Emmelkamp, P. M. (2006). http://dx.doi.org/10.1002/pon.902
This document is copyrighted by the American Psychological Association or one of its allied publishers.

The association between parenting stress, depressed mood and informant


Vitiello, B., Malone, R., Buschle, P. R., Delaney, M. A., & Behar, D.
agreement in ADHD and ODD. Behaviour Research and Therapy, 44, (1990). Reliability of DSM-III diagnoses of hospitalized children. Hos-
15851595. http://dx.doi.org/10.1016/j.brat.2005.11.011 pital and Community Psychiatry, 41, 63 67. http://dx.doi.org/10.1176/

van der Sluis, C. M., van der Bruggen, C. O., Brechman-Toussaint, M. L., ps.41.1.63
Thissen, M. A., & Bgels, S. M. (2012). Parent-directed cognitive Wakschlag, L. S., Tolan, P. H., & Leventhal, B. L. (2010). Research
behavioral therapy for young anxious children: A pilot study. Behavior review: Aint misbehavin: Towards a developmentally-specified no-
Therapy, 43, 583592. http://dx.doi.org/10.1016/j.beth.2011.10.002 sology for preschool disruptive behavior. Journal of Child Psychology

van der Valk, J. C., van den Oord, E. J. C. G., Verhulst, F. C., & and Psychiatry, 51, 322. http://dx.doi.org/10.1111/j.1469-7610.2009
Boomsma, D. I. (2001). Using parental ratings to study the etiology of .02184.x
3-year-old twins problem behaviors: Different views or rater bias?
Walker, K. C., & Bracken, B. A. (1996). Inter-parent agreement on four
Journal of Child Psychology and Psychiatry, 42, 921931. http://dx.doi
preschool behavior rating scales: Effects of parent and child gender.
.org/10.1111/1469-7610.00788

Psychology in the Schools, 33, 273281. http://dx.doi.org/10.1002/
van der Valk, J. C., van den Oord, E. J. C. G., Verhulst, F. C., &
(SICI)1520-6807(199610)33:4273::AID-PITS23.0.CO;2-O
Boomsma, D. I. (2003). Using shared and unique parental views to study
Wang, J., Liu, L., Wu, H., Yang, X., Wang, Y., & Wang, L. (2014).
the etiology of 7-year-old twins internalizing and externalizing prob-
Agreement between parents and adolescents on emotional and behav-
lems. Behavior Genetics, 33, 409 420. http://dx.doi.org/10.1023/A:
ioral problems and its associated factors among Chinese school adoles-
1025369525924
cents: A cross-sectional study. BMC Psychiatry, 14:114. http://dx.doi
van Dulmen, M. H. M., & Egeland, B. (2011). Analyzing multiple
.org/10.1186/1471-244X-14-114
informant data on child and adolescent behavior problems: Predictive
Waschbusch, D. A., & Elgar, F. J. (2007). Development and validation of
validity and comparison of aggregation procedure. International Journal
the Conduct Disorder Rating Scale. Assessment, 14, 6574. http://dx.doi
of Behavioral Development, 35, 84 92. http://dx.doi.org/10.1177/
.org/10.1177/1073191106289908
0165025410392112
Waschbusch, D. A., & Willoughby, M. T. (2008). Parent and teacher
van Hoeke, E., Hoebeke, P., Braet, C., & Walle, J. V. (2004). An
ratings on the IOWA Conners Rating Scale. Journal of Psychopathology
assessment of internalizing problems in children with enuresis. The
Journal of Urology, 171, 2580 2583. http://dx.doi.org/10.1097/01.ju and Behavioral Assessment, 30, 180 192. http://dx.doi.org/10.1007/
.0000110521.20103.14 s10862-007-9064-y


Veenstra, R., Lindenberg, S., Oldehinkel, A. J., De Winter, A. F., Ver- Watts, S. E., & Weems, C. F. (2006). Associations among selective
hulst, F. C., & Ormel, J. (2008). Prosocial and antisocial behavior in attention, memory bias, cognitive errors and symptoms of anxiety in
preadolescence: Teachers and parents perceptions of the behavior of youth. Journal of Abnormal Child Psychology, 34, 841 852. http://dx
girls and boys. International Journal of Behavioral Development, 32, .doi.org/10.1007/s10802-006-9066-3
243251. http://dx.doi.org/10.1177/0165025408089274 Weissman, M. M., Wickramaratne, P., Warner, V., John, K., Prusoff, B. A.,

Veenstra, R., Oldehinkel, A. J., De Winter, A. F., Lindenberg, S., & Merikangas, K. R., & Gammon, G. D. (1987). Assessing psychiatric
Ormel, J. (2006). Temperament, environment, and antisocial behavior in disorders in children. Discrepancies between mothers and childrens
a population sample of preadolescent boys and girls. International reports. Archives of General Psychiatry, 44, 747753. http://dx.doi.org/
Journal of Behavioral Development, 30, 422 432. http://dx.doi.org/ 10.1001/archpsyc.1987.01800200075011
10.1177/0165025406071490 Weisz, J. R., Doss, A. J., & Hawley, K. M. (2005). Youth psychotherapy
outcome research: A review and critique of the evidence base. Annual
Verhulst, F. C., & Akkerhuis, G. W. (1989). Agreement between parents
and teachers ratings of behavioral/emotional problems of children aged Review of Psychology, 56, 337363. http://dx.doi.org/10.1146/annurev
4-12. Journal of Child Psychology & Psychiatry & Allied Disciplines, .psych.55.090902.141449

30, 123136. http://dx.doi.org/10.1111/j.1469-7610.1989.tb00772.x Weisz, J. R., Southam-Gerow, M. A., & McCarty, C. A. (2001). Control-
related beliefs and depressive symptoms in clinic-referred children and
Verhulst, F. C., & van der Ende, J. (1991). Assessment of child psycho-
pathology: Relationships between different methods, different infor- adolescents: Developmental differences and model specificity. Journal
mants and clinical judgment of severity. Acta Psychiatrica Scandi- of Abnormal Psychology, 110, 97109. http://dx.doi.org/10.1037/0021-
navica, 84, 155159. http://dx.doi.org/10.1111/j.1600-0447.1991 843X.110.1.97

.tb03120.x Weitkamp, K., Romer, G., Rosenthal, S., Wiegand-Grefe, S., & Daniels,

Verhulst, F. C., & van der Ende, J. (1992). Agreement between parents J. (2010). German Screen for Child Anxiety Related Emotional Disor-
reports and adolescents self-reports of problem behavior. Journal of ders (SCARED): Reliability, validity, and cross-informant agreement in
Child Psychology & Psychiatry & Allied Disciplines, 33, 10111023. a clinical sample. Child and Adolescent Psychiatry and Mental Health,
http://dx.doi.org/10.1111/j.1469-7610.1992.tb00922.x 4:19. http://dx.doi.org/10.1186/1753-2000-4-19
900 DE LOS REYES ET AL.


Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: Two Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns
simple measures. Journal of Personality and Social Psychology, 84, and correlates of agreement between parent, teacher, and male adoles-
608 618. http://dx.doi.org/10.1037/0022-3514.84.3.608 cent ratings of externalizing and internalizing problems. Journal of
Westen, D., & Rosenthal, R. (2005). Improving construct validity: Cron- Consulting and Clinical Psychology, 68, 1038 1050. http://dx.doi.org/
bach, Meehl, and Neuraths ship. Psychological Assessment, 17, 409 10.1037/0022-006X.68.6.1038
412. http://dx.doi.org/10.1037/1040-3590.17.4.409 Youngstrom, E. A. (2013). Future directions in psychological assessment:
Combining evidence-based medicine innovations with psychologys his-
Westerman, M. A., & Schonholtz, J. (1993). Marital adjustment, joint
parental support in a triadic problem-solving task, and child behavior torical strengths to enhance utility. Journal of Clinical Child and Ado-
problems. Journal of Clinical Child Psychology, 22, 97106. http://dx lescent Psychology, 42, 139 159. http://dx.doi.org/10.1080/15374416
.doi.org/10.1207/s15374424jccp2201_10 .2012.736358

Whiteside, S. P. (2009). Adapting the Sheehan Disability Scale to assess Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2003). Who are
child and parent impairment related to childhood anxiety disorders. the comorbid adolescents? Agreement between psychiatric diagnosis,
Journal of Clinical Child & Adolescent Psychology, 38, 721730. http:// youth, parent, and teacher report. Journal of Abnormal Child Psychol-
dx.doi.org/10.1080/15374410903103551 ogy, 31, 231245. http://dx.doi.org/10.1023/A:1023244512119
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Whiting, P. F., Rutjes, A. W., Westwood, M. E., Mallett, S., Deeks, J. J., Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2004b). Effects of
adolescent manic symptoms on agreement between youth, parent, and
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Reitsma, J. B., . . . Bossuyt, P. M.. (2011). QUADAS-2: A revised tool


for the quality assessment of diagnostic accuracy studies. Annals of teacher ratings of behavior problems. Journal of Affective Disorders, 82,
Internal Medicine, 155, 529 536. http://dx.doi.org/10.7326/0003-4819- S5S16. http://dx.doi.org/10.1016/j.jad.2004.05.016
155-8-201110180-00009 Youngstrom, E. A., Findling, R. L., Calabrese, J. R., Gracious, B. L.,

Will, M. N., & Wilson, B. J. (2014). A longitudinal analysis of parent and Demeter, C., . . . Price, M. (2004a). Comparing the diagnostic accuracy
of six potential screening instruments for bipolar disorder in youths aged
teacher ratings of problem behavior in boys with and without develop-
5 to 17 years. Journal of the American Academy of Child and Adolescent
mental delays. Journal of Intellectual Disabilities, 18, 176 187. http://
Psychiatry, 43, 847 858. http://dx.doi.org/10.1097/01.chi.0000125091
dx.doi.org/10.1177/1744629514528828
.35109.1e
Winsler, A., & Wallace, G. L. (2002). Behavior problems and social skills
Youngstrom, E., Izard, C., & Ackerman, B. (1999). Dysphoria-related bias
in preschool children: Parent-teacher agreement and relations with class-
in maternal ratings of children. Journal of Consulting and Clinical
room observations. Early Education and Development, 13, 4158. http://
Psychology, 67, 905916. http://dx.doi.org/10.1037/0022-006X.67.6
dx.doi.org/10.1207/s15566935eed1301_3
.905
Wong, H. Y., Jenvey, V., & Lill, A. (2012). Childrens self-reports and
Youngstrom, E. A., Kahana, S. Y., Starr, M., & Schwartz, C. (2004,
parents reports of internalising and externalising problems in Chinese
November). Emotion states and bias in clinical ratings: Causal but
and Anglo-Celtic children in Australia. Australian Journal of Psychol-
complicated. In A. De Los Reyes & L. M. Saavedra (Chairs), Informant
ogy, 64, 155163. http://dx.doi.org/10.1111/j.1742-9536.2011.00038.x discrepancies in clinical child research: Processes of informant discrep-

Wren, F. J., Bridge, J. A., & Birmaher, B. (2004). Screening for childhood ancies and distortions and impact on clinical decision-making. Sympo-
anxiety symptoms in primary care: Integrating child and parent reports. sium conducted at the annual meeting of the Association for the Ad-
Journal of the American Academy of Child & Adolescent Psychiatry, 43, vancement of Behavior Therapy, New Orleans.
1364 1371. http://dx.doi.org/10.1097/01.chi.0000138350.60487.d3
Zahn-Waxler, C., Schmitz, S., Fulker, D., Robinson, J., & Emde, R.

Xu, X., Mellor, D., & Wong, J. (2011). Childhood cruelty to animals: Do (1996). Behavior problems in five-year-old monozygotic and dizygotic
mothers and fathers reports agree? Anthrozoos: A Multidisciplinary twins: Genetic and environmental influences, patterns of regulation, and
Journal of the Interactions of People & Animals, 24, 263271. http:// internalization of control. Development and Psychopathology, 8, 103
dx.doi.org/10.2752/175303711X13045914865105 122. http://dx.doi.org/10.1017/S0954579400006994
Yates, B. T., & Taub, J. (2003). Assessing the costs, benefits, cost-
Zhou, Q., Hofer, C., Eisenberg, N., Reiser, M., Spinrad, T. L., & Fabes,
effectiveness, and cost-benefit of psychological assessment: We should, R. A. (2007). The developmental trajectories of attention focusing,
we can, and heres how. Psychological Assessment, 15, 478 495. http:// attentional and behavioral persistence, and externalizing problems dur-
dx.doi.org/10.1037/1040-3590.15.4.478 ing school-age years. Developmental Psychology, 43, 369 385. http://
Yeh, M., & Weisz, J. R. (2001). Why are we here at the clinic? Parent-child dx.doi.org/10.1037/0012-1649.43.2.369
(dis)agreement on referral problems at outpatient treatment entry. Jour-
Zhou, Q., Lengua, L. J., & Wang, Y. (2009). The relations of temperament
nal of Consulting and Clinical Psychology, 69, 1018 1025. http://dx.doi reactivity and effortful control to childrens adjustment problems in
.org/10.1037/0022-006X.69.6.1018 China and the United States. Developmental Psychology, 45, 724 739.
Youngstrom, E. (2008). Evidence-based strategies for the assessment of http://dx.doi.org/10.1037/a0013776
developmental psychopathology: Measuring prediction, prescription,
and process. In W. E. Craighead & D. J. Milkowitz (Eds.), Psychopa- Received March 14, 2013
thology. History, diagnosis, and empirical foundations (pp. 34 77). Revision received November 1, 2014
Hoboken, NJ: Wiley. Accepted November 7, 2014

Vous aimerez peut-être aussi