2011; 33: 820–827

Development and analysis of D-RECT,

an instrument measuring residents’
learning climate
KLARKE BOOR1, CEES VAN DER VLEUTEN2, PIM TEUNISSEN2,3, ALBERT SCHERPBIER2
& FEDDE SCHEELE1,3
1
St Lucas Andreas Hospital, The Netherlands, 2Maastricht University, The Netherlands, 3VU University Medical Centre,
The Netherlands

Abstract
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14

Background: Measurement of learning climates can serve as an indicator of a department’s educational functioning.
Aim: This article describes the development and psychometric qualities of an instrument to measure learning climates in
postgraduate specialist training: the Dutch Residency Educational Climate Test (D-RECT).
Method: A preliminary questionnaire was evaluated in a modified Delphi procedure. Simultaneously, all residents in the
Netherlands were invited to fill out the preliminary questionnaire. We used exploratory factor analysis to analyze the outcomes and
construct the definitive D-RECT. Confirmatory factor analysis tested the questionnaire’s goodness of fit. Generalizability studies
tested the number of residents needed for a reliable outcome.
Results: In two rounds, the Delphi panel reached consensus. In addition, 1278 residents representing 26 specialties completed the
questionnaire. The Delphi panel’s input in combination with the exploratory factor analysis of 600 completed surveys led to the
definitive D-RECT, consisting of 50 items and 11 subscales (e.g., feedback, supervision, patient handover and professional relations
For personal use only.

between attendings). Confirmatory factor analyses of the remaining surveys confirmed the construct. The results showed that a
feasible number of residents is needed for a reliable outcome.
Conclusion: D-RECT appears to be a valid, reliable and feasible tool to measure the quality of clinical learning climates.

Background Practice points

Governments, regulators as well as medical training boards
expect, and progressively inspect, a high quality of residency
training (Iverson 1998; Genn 2001; Afrin et al. 2006). One way
to evaluate the quality of training programs is to evaluate
learning climates. Learning climates inform us about the
contexts residents participate in. It is a construct that relates
to multiple facets of residents’ training. It reflects the way
people in departments approach learning and it incorporates
shared perceptions of these people of themes like atmosphere,
supervision, and the status of learning. Learning climates are
constructed through interactions of learners and other
healthcare workers and are influenced by organizational
arrangements and artifacts (Boor 2009). Measurement of
learning climates can serve as a broad indicator of a
department’s educational functioning because of the con-
struct’s versatility. This article describes the development and
testing of an instrument to measure learning climates in
graduate medical education (GME), the Dutch Residency
Educational Climate Test (D-RECT). As can be expected from
the description above, evaluation of learning climates is an
arduous endeavor. We point out two problems.
First, a learning climate is a theoretical construct that cannot
be measured immediately. What can be measured immediately
. D-RECT is a valid and reliable instrument to measure
learning climates in GME.
. The 11 subscales give departments specific feedback as
to what should be maintained and what could be
improved
. D-RECT highlights important issues such as supervision
and feedback, but also covers less obvious themes such
as ‘‘patient handover’’ and ‘‘professional relations
between attendings.’’
are residents’ daily experiences reflecting an underlying
construct, that of a learning climate. To apprehend those
daily experiences, we performed a qualitative study, reported
elsewhere (Boor 2009), among 40 residents at different levels
of training in various specialties to explore which events can
‘‘make or break’’ learning climates. Residents explained that in
an optimal learning climate they experienced that their daily
work was a continuation of their training (as opposed to
working for service imperatives only). So, for instance, in an
optimal learning climate residents were able to make choices
in the content of their work in relation to their personal
learning needs. Residents also stressed the important role of

Correspondence: K. Boor, Department of Medical Education, St Lucas Andreas Hospital, Amsterdam, The Netherlands. Tel: 31 64 797 6494;
fax: þ31 20 5108791; email: klarkeboor@gmail.com

820 ISSN 0142–159X print/ISSN 1466–187X online/11/100820–8 ß 2011 Informa UK Ltd.

DOI: 10.3109/0142159X.2010.541533

interaction with attendings, peers, nurses, and other healthcare
personnel. In summary, an optimal learning climate was
characterized by integration of work and training and tailored
to individual residents’ needs. Residents’ input from this study
served as basis for the themes reflecting a good learning
climate.
Second, it is difficult to construct an instrument with sound
psychometric properties. Both within the field of organiza-
tional psychology (Bartram et al. 1993; Ashkanasy et al. 2000)
and the medical educational field (Roff et al. 1997; Kanashiro
et al. 2006), many instruments have been developed that tap
into (learning) climates. Still, the development and psychome-
trical testing of most instruments can be improved. For
instance, Dundee ready education environment measure
(DREEM; Roff et al. 1997) and postgraduate hospital educa-
tional environment measure (PHEEM; Roff et al. 2005), two
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14
widely used instruments to measure learning environments,
lack a clearly described theoretical base and their underlying
factor structure is disputed. For instance, PHEEM has been
described as possessing three different subscales in three
different publications (Roff et al. 2005; Boor et al. 2007;
Schonrock-Adema et al. 2008). Such a controversial subscale
structure hinders the possibilities for specific feedback to
departments.
Aims
For personal use only.
This article describes our attempt to overcome the above-
described obstacles. We developed D-RECT guided by the
results from our qualitative research. We searched for validity
evidence on test content and internal structure (American
Education Research Association and American Psychological
Association 1999). We constructed subscales and tested them
using input from a Delphi panel (obtaining validity evidence
based on test content) and an exploratory factor analysis; the
definitive questionnaire was confirmed with a confirmatory
factor analysis (obtaining validity evidence based on internal
structure). Moreover, this article studied D-RECTs reliability:
using generalizability theory, we estimated the number of
participants needed to get a reproducible outcome, i.e., how
many residents must fill out a questionnaire to get a reliable
(reproducible) result?
Methods
Setting
In the Netherlands, medical students obtain a basic medical
degree after 6 years of undergraduate medical training. This
entitles them to apply for a place in a training program in one
of the 27 specialties. Depending on the specialty, training lasts
from 4 to 6 years. In this article, we use the term resident to
refer to a junior doctor who is undertaking specialty training in
GME. Specialist training programs consist of rotations in a
university hospital and an affiliated general hospital. In every
hospital department where training is offered, one specialist is
the ‘‘specialty tutor’’ and formally responsible for residents’
education and training as well as (bi-) annual assessments.
Development and analysis of D-RECT
Ethical considerations
This study was exempt from Institutional Board Review under
Dutch law. However, we made sure that no possible harm
could come to our participants. In the invitation to the Delphi
panelists and in the letter inviting the residents to take part in
the questionnaire study, we explicitly stated that participation
was voluntary and full anonymity was guaranteed.
Using multiple sources and different methods, we aimed to
develop a questionnaire with reproducible subscales and offer
validity and reliability evidence. Figure 1 shows an overview of
the different analysis steps.
Development of D-RECT
In an earlier study, we found that an optimal learning climate
was characterized by the integration of work and training and
tailored to residents’ needs (Boor 2009). We used input from
this qualitative study in designing the first version of the
learning climate questionnaire, which consisted of 83 items
and 21 subscales (e.g., feedback, collaboration between peers,
patient handover, workload, or teamwork). These 83 items
and 21 subscales were discussed in an expert group of two
medical education experts (CvdV and AS), two medical
doctors (PT and KB), and one specialty tutor (FS).
Additionally, four residents, two specialty tutors, and three
medical educationalists (all of whom were not in the research
team) checked the items for face validity and made sugges-
tions for removal or rewording. Duplicate or unclear items
were removed. This resulted in a 75-item preliminary D-RECT.
Every item invited agreement on a five-point Likert scale
(1 ¼ totally disagree and 5 ¼ totally agree); we also included a
not applicable option.
We submitted the 75 items of the preliminary D-RECT to a
Delphi panel. Simultaneously, residents were asked to fill out
the preliminary D-RECT and we performed several analyses on
the pool of completed questionnaires.
Delphi procedure
A Delphi procedure is aimed at achieving consensus among
experts in a systematic manner (Fink et al. 1984; Jones &
Hunter 1995). In multiple consultation rounds, experts indicate
their (dis)agreement with statements or concepts. After the first
round, the experts can change their own rating in light of the
summarized (anonymous) ratings of the other panel members
(Jones & Hunter 1995). In a modified Delphi procedure, the
statements or items are not generated by the expert group
but – as in this study – carefully selected based on earlier
research (Rowe et al. 1991; Jones & Hunter 1995; Boor 2009).
In April and May 2008, we invited 10 medical education-
alists, 10 policymakers, 10 residents, and 10 specialists (all
specialty tutors) for our Delphi panel; the latter two groups
represented different specialties as well as university and non-
university hospitals. The 40 panelists were chosen for their
involvement in postgraduate specialist training. They were
offered a monetary incentive for participating in the complete
Delphi procedure. Experts received the preliminary D-RECT
and rated every item’s relevance in relation to postgraduate
821
 K. Boor et al.

Qualitative input (40 residents)

Preliminary instrument: 83 items

Discussion (5 educationalists,
6 residents, 3 attendings)

Preliminary instrument: 75 items

Delphi panel (38 experts) Filled out preliminary instruments

(2 rounds) by 1278 residents (1251 usable)

Exploratory factor analysis

Preliminary instrument: 45 items
(600 at random residents)
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14

Interpretation of outcomes with

help of factor loadings and Delphi
panel results

Definitive instrument: 50 items

Calculation of number of Check of instrument’s goodness-

residents needed for reliable of-fit with a confirmatory factor
outcome (G-analysis) analysis (other 651 residents)
For personal use only.

Figure 1. Flowchart of different analysis steps.

learning climate on a five-point scale (1¼ not relevant and
5¼ highly relevant). For the analysis, the ratings were
dichotomized (1, 2, and 3 were interpreted as not relevant; 4
and 5 as relevant). In the absence of undisputed guidelines to
decide when consensus is reached (Holey et al. 2007), we
decided that agreement among 80% of participants would lead
to inclusion or exclusion of an item and that consensus was
considered to have been reached when 90% of the items that
were included or excluded in one round remained unchanged
in the subsequent round.
(which presupposes no correlations between the factors) and
Oblimin rotation (which presupposes some correlations
between the factors). Because the ‘‘Component Correlation
Matrix’’ indicated correlations between the factors, from then
on we only used Oblimin rotations (Field 2005). Items with
weak factor loadings were eliminated and internal consistency
of the factors was determined by calculating Cronbach’s alpha.
Our theoretical framework (Boor 2009) and the outcomes
of the Delphi procedure guided our decisions on the inclusion
of ambiguous items. This resulted in a multi-factorial model:
the definitive D-RECT.

Questionnaire mailing
In May 2008, a letter was sent to all residents in the
Netherlands, asking them to complete the web-based prelim-
inary D-RECT and answer some demographic questions.
D-RECT was administered in Dutch. Psychiatric residents
were not included for logistical reasons. Some of the respon-
dents could win an incentive provided they completed the
questionnaire. No reminders were sent. We compared the
response group with respect to sex and specialty to the entire
population of residents using the chi-squared test ( p 5 0.05
was considered significant).
Testing the definitive D-RECT
Confirmatory factor analysis
Using structural equation modeling, we tested the goodness-
of-fit of the multi-factorial model on the other half of the
returned questionnaires. The comparative fit index (CFI), the
root mean square error of approximation (RMSEA), and
the relative chi-square (CMIN/DF) were used as indices of
the goodness-of-fit (McDonald & Ho 2002).

Exploratory factor analysis
Factor analysis is used to identify clusters of related variables.
We randomly selected half of all returned questionnaires for
exploratory factor analysis using both Varimax rotation
822
Reliability analysis
We used the same data (the second half of the returned
questionnaires) in order to determine reliability. A variance
component analysis was performed to measure the contribu-
tions of all relevant components (in this case, residents,

departments, items, and their interactions) to the variance in an
outcome measure (Crossley et al. 2002). We performed
generalizability analysis for the mean total score and each
separate subscale, to estimate the number of residents needed
to obtain reliable test scores.
We treated the total number of items as fixed. The number
of residents within a single department and the number of
departments were allowed to vary. Following variance com-
ponent estimation, we estimated the standard error of
measurement (SEM) for a single department. The SEM can
be interpreted on the original scoring scale (in this case 1–5)
and we decided to accept a maximum ‘‘noise level’’ of 1.0
on the scale. We therefore used a SEM 5 0.26
(1.96  0.26  2 ¼ 1.0) as the smallest admissible value for a
95% confidence interval interpretation.
To use D-RECT across a group of departments, we
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14
estimated the RMSE, which can be interpreted in the same
way as the SEM but now at group level.
We used Amos structural equation modeling software for
the confirmatory factor analysis, URGENOVA software to
analyze generalizability, and SPSS for all the other analyses.
Results
Creating the definitive D-RECT
For personal use only.
Delphi procedure. Two respondents did not complete the
Delphi procedure: one specialist was unable to take part
because of health problems and one resident failed to respond
in time; the other 38 respondents completed the full Delphi
procedure. In the first round, one of the researchers (KB) sent
the preliminary D-RECT to all participants asking them to rate
every item’s relevance for their learning climate. If desired,
they could add qualitative remarks to explain their ratings. The
completed questionnaires were returned to KB, who summa-
rized the ratings and remarks. In the second round, the
preliminary D-RECT was sent to all participants, with the item
mean scores and standard deviations of all items in the first
round as well as (anonymous) summaries of the qualitative
remarks. The participants were also asked to (dis)agree to
some slight adjustments to seven items, which were simplified
to improve clarity. The panel again completed the question-
naire and returned it to KB, who analyzed the new ratings and
remarks. All participants approved the seven adjusted items.
Moreover, consensus was reached in this round: 91% of the
items included and excluded in the first and second rounds
were identical. The Delphi procedure led to elimination of 30
items, leaving a 45-item questionnaire. Items were left out
because the panel members found them too extreme or
irrelevant to the learning climate (for instance, the item
‘‘attendings know my name’’ or the whole subcategory on
‘‘workload’’ with, for instance, an item related to possibilities to
work part-time).
Questionnaire mailing. We invited 4835 residents to com-
plete the preliminary D-RECT; 1276 residents in 26 specialties
returned the questionnaire, 53.1% were female. The response
group was comparable to the total population of
residents (http://knmg.artsennet.nl/opleidingenregistratie/
Development and analysis of D-RECT
jaarverslagenopleidingregistratie-1.htm, annual report 2007)
with respect to sex and the 26 specialties, except for internal
medicine residents ( p ¼ 0.01), obstetric-gynecological resi-
dents ( p 5 0.01), and rehabilitation medicine residents
( p ¼ 0.01). Twenty-five questionnaires that had more than 25
unanswered items were excluded from the analysis. 591
respondents checked ‘‘not applicable’’ on some items, but the
response rate per item was never lower than 94%. The values
of these items were replaced for the psychometric analysis
using two-way imputation, a method that corrects both for
person effects and item effects (Sijtsma & van der Ark 2003).
Exploratory factor analysis. We first analyzed 600 randomly
chosen questionnaires using an exploratory factor analysis. For
a sound factor analysis a number of 5 subjects per item is the
minimum, so this was a large enough sample for our analysis
(Streiner 1994). We eliminated 14 items with weak factor
loadings. Another 11 items were removed following the advice
of the Delphi panel. Fifteen items were included in the D-RECT
because residents in earlier qualitative research highlighted its
specific importance and factor loadings were high, although
these items were not accepted by the Delphi panel (Table 1).
Five subscales were excluded completely, and five subscales
merged with other, related, subscales. This led to the definitive
50-item D-RECT with 11 subscales. Cronbach’s alpha var-
ied between 0.64 and 0.85 for the different factors (Tables 1
and 2).
Testing the definitive D-RECT
Confirmatory factor analysis
We tested the definitive 50-item D-RECT by analyzing the
remaining 651 questionnaires using confirmatory factor anal-
ysis. The goodness-of-fit indices were: CFI, 0.89; RMSEA, 0.04;
and CMIN/DF, 2.9. Overall, the indices showed a good fit.
Reliability analysis
We examined the generalizability of the 50 items of the
definitive D-RECT. In order to obtain reliable outcomes based
on the overall score for one department, three residents had to
fill out D-RECT. Eleven residents were needed for reliable
outcomes for every subscale in one department (Table 3). For
reliable outcomes for groups of departments, two residents
from three departments were sufficient to obtain a reproduc-
ible total score. For a reliable outcome for every subscale,
four residents from six different departments were required
(Table 3).
Conclusions
Principal findings
Different approaches guided the development and validation
of D-RECT, a questionnaire to evaluate the learning climate in
GME. The theoretical foundation of the questionnaire was a
model of the clinical learning climate based on earlier
empirical findings (Boor 2009). A modified Delphi procedure
823
 K. Boor et al.

Table 1. D-RECT questionnaire.

Items N Mean SD  Delphi

Subscale: Supervision 0.64
1 The guidelines clearly outline when to request input from a supervisor 600 3.77 1.06 A
2 The amount of supervision I receive is appropriate for my level of 600 4.09 0.91 A
experience
3 It is clear which attending supervises me 600 4.26 0.95 A
Subscale: Coaching and assessment 0.80
4 I am asked on a regular basis to provide a rationale for my management 600 3.88 0.88 A
decisions and actions
5 My attendings coach me on how to communicate with difficult patients 600 3.42 1.03 NA
6 My attendings take the initiative to explain their actions 600 3.46 0.98 NA
7 My attendings take the initiative to evaluate my performance 600 3.08 1.06 NA
8 My attendings take the initiative to evaluate difficult situations I have been 600 3.01 1.04 A
involved in
9 My attendings evaluate whether my performance in patient care is 600 3.40 1.09 A
commensurate with my level of training
10 My attendings occasionally observe me taking a history 600 2.85 1.22 A
11 My attendings assess not only my medical expertise but also other skills 600 3.67 1.12 A
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14

such as teamwork, organization or professional behavior

Subscale: Feedback 0.75
12 My attendings give regular feedback on my strengths and weaknesses 600 3.37 1.14 A
13 Observation forms (i.e., Mini-CEX) are used to structure feedback 600 3.81 1.19 NA
14 Observation forms (i.e., Mini-CEX) are used periodically to monitor my 600 3.42 1.27 NA
progress
Subscale: Teamwork 0.69
15 Attendings, nursing staff, other allied health professionals and residents 600 3.82 1.00 NA
work together as a team
16 Nursing staff and other allied health professionals make a positive 600 3.66 1.09 NA
contribution to my training
17 Nursing staff and other allied health professionals are willing to reflect 600 3.76 1.02 NA
with me on the delivery of patient care
18 Teamwork is an integral part of my training 600 3.00 1.04 A
For personal use only.

Subscale: Peer collaboration 0.76

19 Residents work well together 600 4.48 0.79 A
20 Residents, as a group, make sure the day’s work gets done 600 4.14 1.04 NA
21 Within our group of residents it is easy to find someone to cover or 600 4.26 0.91 NA
exchange a call
Subscale: Professional relations between attendings 0.77
22 Continuity of care is not affected by differences of opinion between 600 3.76 1.12 A
attendings
23 Differences of opinion between attendings about patient management 600 3.50 1.14 A
are discussed in such a manner that is instructive to others present
24 Differences of opinion are not such that they have a negative impact on 600 3.62 1.16 A
the work climate
Subscale: Work is adapted to residents’ competence 0.66
25 The work I am doing is commensurate with my level of experience 600 4.12 0.84 A
26 The work I am doing suits my learning objectives at this stage of my 600 3.97 0.92 A
training
27 It is possible to do follow up with patients 600 3.96 1.08 A
28 There is enough time in the schedule for me to learn new skills 600 3.24 1.08 A
Subscale: Attendings’ role 0.85
29 My attendings take time to explain things when asked for advice 600 4.12 0.80 A
30 My attendings are happy to discuss patient care 600 4.16 0.82 A
31 There is (are) NO attending physician(s) who have a negative impact on 600 3.34 1.27 NA
the educational climate
32 My attendings treat me as an individual 600 3.47 1.18 NA
33 My attendings treat me with respect 600 4.46 0.87 A
34 My attendings are all in their own way positive role models 600 3.48 1.04 NA
35 When I need a attending, I can always contact one 600 4.55 0.69 A
36 When I need to consult a attending, they are readily available 600 4.52 0.72 A
Subscale: Formal education 0.75
37 Residents are generally able to attend scheduled educational activities 600 3.60 1.06 A
38 Educational activities take place as scheduled 600 3.89 0.98 A
39 Attendings contribute actively to the delivery of high-quality formal 600 3.63 1.14 A
education
40 Formal education and training activities are appropriate to my needs 600 3.61 1.06 NA
Subscale: Role of the specialty tutor 600 0.78
41 The specialty tutor monitors the progress of my training 600 4.49 0.84 A
42 The specialty tutor provides guidance to other attendings when needed 600 3.46 1.26 A
43 The specialty tutor is actively involved in improving the quality of 600 4.12 1.06 A
education and training
44 In this rotation evaluations are useful discussions about my performance 600 3.65 1.11 A
45 My plans for the future are part of the discussion 600 3.61 1.16 NA

(continued )
824
 Development and analysis of D-RECT

Table 1. Continued.

Items N Mean SD  Delphi

46 During evaluations, input from several attendings is considered 600 3.83 1.17 A
Subscale: Patient sign out 0.75
47 When there is criticism of a management plan I have developed in 600 3.74 1.13 A
consultation with my attending physician, I know the attending
physician will back me up
48 Sign out takes place in a safe climate 600 4.02 1.04 A
49 Sign out is used as a teaching opportunity 600 3.96 1.06 A
50 Attendings encourage residents to join in the discussion during sign out 600 3.69 1.15 A

Notes: A professional translator rendered the original Dutch questionnaire into English. This version was checked by a British medical specialist for clarity. A native
speaker translated the questionnaire back into a Dutch version; this version was comparable to the original version.
N, number of respondents; , Cronbach’s alpha; and Delphi, item accepted (A) or not accepted (NA) by Delphi panel.

test scores. The analyses resulted in extensive tables per

Table 2. Total variance explained per subscale. subscale (all available from first author), which showed that for
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14

one department, a minimum number of 11 residents is needed

Rotation sums of for a reliable outcome, although most subscales can be judged
Subscale squared loadings reliably using eight residents’ input. For groups of depart-
Supervision 3, 29 ments, four residents from six departments are necessary for
Coaching and assessment 5, 22 reliable test scores.
Feedback 4, 10
Team work 4, 53 D-RECT touches on important issues such as ‘‘supervision’’,
Peer collaboration 4, 42 ‘‘coaching and assessment,’’ and ‘‘feedback’’. In addition,
Professional relations between attendings 5, 52 D-RECT includes less obvious themes such as ‘‘professional
Work is adapted to resident’s competence 5, 05
Attendings’ role 6, 21 relations between attendings’’ and ‘‘patient handover’’ (which
Formal education 5, 29 relates to the moment when two groups of doctors exchange
For personal use only.

Role of the specialty tutor 5, 65 patient data because one shift is ending and the other shift is
Patient sign out 5, 23
beginning). These issues were brought to the fore by residents
in earlier qualitative research and confirmed by the Delphi
panel. They are hardly described in relation to learning
Table 3. Generalizability analysis of D-RECT total scores and climates (Sanfey et al. 2008) and can offer new insights why
subscales. some climates ‘‘work’’ while others do not.

SEM RMSE
n (residents)/ Strengths and weaknesses
n (residents) n (departments)
It is a strength of this study that different strategies were used
Total 3 2/3 in developing the questionnaire. Other studies reporting on
Supervision 7 4/4 the development and/or validation of learning climate ques-
Coaching and assessment 6 3/3
tionnaires did not describe their literature review or analytical
Feedback 11 4/6
Team work 8 4/4 methods and used Cronbach’s alpha as the sole indicator of
Peer collaboration 7 4/4 questionnaire stability (Roff et al. 1997; Cassar 2004; Roff et al.
Professional relations 9 4/6
2005). Some studies included an exploratory factor analysis
between attendings
Work is adapted to 7 4/3 (Bligh & Slade 1996; Pololi & Price 2000) but, to our
residents’ competence knowledge, no earlier study combined theoretical input, a
Attendings’ role 5 4/3
Formal education 7 4/6
Delphi procedure, exploratory and confirmatory analyses, and
Role of the specialty tutor 7 4/4 generalizability studies in developing and validating an
Patient sign out 8 4/5 instrument. Another strength of this study is that the data for
the psychometric analyses were obtained from residents in 26
Notes: SEM, standard error of measurement; number of residents needed to
get a reliable result for one department, RMSE ¼ root mean square error;
different specialties, at different levels of training, and from 76
number of residents needed to get a reliable outcome for a group of different hospitals. This strengthens the comprehensive appli-
departments (for instance, three residents from three departments are cability of D-RECT. Furthermore, the number of residents
needed to get a reliable outcome for the subscale ‘‘coaching and
assessment’’).
needed for a reliable outcome for one department lies
between 3 and 11, but most subscales can be judged reliably
by eight residents. This supports the feasibility of the instru-
among experts determined the final inclusion and exclusion of ment. As for groups of departments, even fewer residents (four
items and extensive psychometric analyses revealed a multi- from four different departments) would yield a reliable
factorial questionnaire. Generalizability theory was used to impression of most subscales. Other studies have shown
determine the number of respondents necessary for reliable similar generalizability outcomes or required (much) larger
825
 K. Boor et al.
numbers of participants for a reliable outcome (Bierer et al.
2004; van der Hem-Stokroos et al. 2005; Boor et al. 2007).
There are some caveats to take into account. First, the 26%
response rate can be a source of potential bias. This could
have been caused by the possible sensitive nature of the
questionnaire and the lack of funding for sending a reminder –
a strategy proven to be effective to increase response rates
(Edwards et al. 2009). However, the goal of this part of the
study was to test D-RECTs psychometric properties: for this
goal, the number of respondents is sufficiently high (Streiner
1994; Field 2005). Second, we conducted our study within the
context of GME in the Netherlands. Whether D-RECT is also
valid outside the Netherlands warrants further investigations.
Finally, D-RECT has not been tested in its final form;
administering the 50-item questionnaire instead of the prelim-
inary 75-item version could possibly lead to slightly different
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14
outcomes.
Implications and future research
Our study resulted in a multi-factorial questionnaire that can
yield reliable results that can be used to give specific feedback
to departments on their local learning climate. Based on the
questionnaire outcome, a (group of) department(s) can be
informed which elements (i.e., subscales) scored highly and
should be continued and which factors need to be improved in
For personal use only.
order to offer a better learning climate for residents. In
addition, D-RECT offers benchmarking opportunities.
Departments who devote time and effort to creating an
optimal learning climate can substantiate their qualities, while
departments in need of help can be made aware of their
performance in relation to that of other departments. This is
also relevant in light of governmental and public calls for
transparency and monitoring of the quality of postgraduate
medical education (Genn 2001; Afrin et al. 2006).
From a research point of view, it would be interesting to
examine D-RECT’s consequential validity to strengthen the
instrument’s evidence base (American Education Research
Association and American Psychological Association 1999).
D-RECT is intended to improve the educational climate. Does
it succeed? Are there unintended positive or adverse effects?
Another line of investigation would be to validate D-RECT for
use in international settings. Finally, D-RECT may enable us to
study the relationship between a good learning climate and
patient and resident outcomes. Are high D-RECT scores related
to the delivery of better patient care? Do they result in better
specialists? Research could also examine whether some
subscales in particular are associated with the delivery of
better patient care or better doctors.
Acknowledgments
The authors thank the participants of the Delphi panel and all
residents who completed D-RECT. We also thank Ron
Hoogenboom and Henk van Berkel for their invaluable
assistance in analyzing the data. In addition, we thank
Mereke Gorsira for her support in editing this article’s
English and Tim Dornan, for commenting on D-RECT’s
proper usage of English terms. The CBOG (an institute
826
involved in residents’ training) dispatched the questionnaire
and offered administrative support in analyzing the filled out
surveys; they had no influence on the content or analysis of
the data.
Declaration of interest: The authors report no conflicts of
interest. The authors alone are responsible for the content and
writing of the article.
Notes on contributors
KLARKE BOOR is a resident and obtained a PhD in medical education.
CEES VAN DER VLEUTEN is a professor of education.
PIM TEUNISSEN is a resident and obtained a PhD in medical education.
ALBERT SCHERPBIER is a professor of education.
FEDDE SCHEELE is a gynecologist and is a professor of education.
References
Afrin LB, Arana GW, Medio FJ, Ybarra AF, Clarke Jr HS. 2006. Improving
oversight of the graduate medical education enterprise: One institu-
tion’s strategies and tools. Acad Med 81(5):419–425.
American Education Research Association and American Psychological
Association 1999. Standards for Educational and Psychological Testing.
Washington DC: American Education Research Association.
Ashkanasy NM, Wilderom CPM, Peterson MF. 2000. Introduction. In:
Ashkanasy NM, Wilderom CPM, Peterson MF, editors. Handbook of
organizational culture and climate. Thousand Oaks CA: Sage
Publications. pp 1–18.
Bartram D, Foster F, Lindley PA, Brown AJ, Nixon S. 1993. The Learning
Climate Questionnaire. Employment Service and Newland Park
Associates Ltd.
Bierer SB, Fishleder AJ, Dannefer E, Farrow N, Hull AL. 2004. Psychometric
properties of an instrument designed to measure the educational
quality of graduate training programs. Eval Health Prof 27(4):410–424.
Bligh J, Slade P. 1996. A questionnaire examining learning in general
practice. Med Educ 30(1): 65–70.
Boor K. 2009. The clinical learning climate. Amsterdam: VU Medical
Center.
Boor K, Scheele F, van der Vleuten CP, Scherpbier AJ, Teunissen PW,
Sijtsma K. 2007. Psychometric properties of an instrument to measure
the clinical learning environment. Med Educ 41(1):92–99.
Cassar K. 2004. Development of an instrument to measure the surgical
operating theatre learning environment as perceived by basic surgical
trainees. Med Teach 26(3):260–264.
Crossley J, Davies H, Humphris G, Jolly B. 2002. Generalisability: A key to
unlock professional assessment. Med Educ 36(10):972–978.
Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, Cooper R,
Felix LM, Pratap S. 2009. Methods to increase response to postal and
electronic questionnaires. Cochrane database of systematic review.
3:MR000008.
Field A. 2005. Exploratory factor analysis. In: Discovering statistics using
SPSS. 2nd ed. London: Sage. pp 619–680.
Fink A, Kosecoff J, Chassin M, Brook RH. 1984. Consensus
methods: Characteristics and guidelines for use. Am J Publ Health
74(9):979–983.
Genn JM. 2001. AMEE medical education guide no. 23 ( part 2): Curriculum,
environment, climate, quality and change in medical education – a
unifying perspective. Med Teach 23(5):445–454.
Holey EA, Feeley JL, Dixon J, Whittaker VJ. 2007. An exploration of the use
of simple statistics to measure consensus and stability in Delphi studies.
BMC Med Res Meth 7(52):doi:10.1186/1471-2288-7-52.
Iverson DJ. 1998. Meritocracy in graduate medical education? Some
suggestions for creating a report card. Acad Med 73(12):1223–1225.
Jones J, Hunter D. 1995. Consensus methods for medical and health
services research. Br Med J 311(7001):376–380.

Kanashiro J, McAleer S, Roff S. 2006. Assessing the educational environ-
ment in the operating room: A measure of resident perception at one
Canadian institution. Surgery 139(2):150–158.
McDonald RP, Ho MH. 2002. Principles and practice in reporting structural
equation analyses. Psychol Methods 7(1): 64–82.
Pololi L, Price J. 2000. Validation and use of an instrument to measure the
learning environment as perceived by medical students. Teach Learn
Med. 12(4):201–207.
Roff S, McAleer S, Harden RM, Al-Qahtani M, Ahmed AU, Deza H,
Groenen G, Primparyon P. 1997. Development and validation of the
Dundee Ready Education Environment Measure (DREEM). Med
Teach 19:295–299.
Roff S, McAleer S, Skinner A. 2005. Development and validation of an
instrument to measure the postgraduate clinical learning and teaching
educational environment for hospital-based junior doctors in the UK.
Med Teach 27(4): 326–331.
Med Teach Downloaded from informahealthcare.com by The University of Manchester on 11/11/14
For personal use only.
Development and analysis of D-RECT
Rowe G, Wright G, Bolger F. 1991. Delphi. A reevaluation of research and
theory. Technol Forecast Soc Change 39:235–251.
Sanfey H, Stiles B, Hedrick T, Sawyer RG. 2008. Morning
report: Combining education with patient handover. Surgeon
6(2):94–100.
Schonrock-Adema J, Heijne-Penninga M, Hell EA, Cohen-Schotanus J.
2008. Necessary steps in factor analysis: Enhancing validation studies of
educational instruments. The PHEEM applied to clerks as an example.
Med Teach. 31:1–7.
Sijtsma K, van der Ark LA. 2003. Investigation and treatment of missing item
scores in test and questionnaire data. Multivariate Behav Res
38:505–528.
Streiner DL. 1994. Figuring out factors: The use and misuse of factor
analysis. Can J Psychiat 39(3):135–140.
van der Hem-Stokroos HH, van der Vleuten CP, Daelmans HE, Haarman
HJ, Scherpbier AJ. 2005. Reliability of the clinical teaching effectiveness
instrument. Med Educ 39(9):904–910.
827