Animal Behaviour 90 (2014) 273e280

Contents lists available at ScienceDirect

Animal Behaviour
journal homepage: www.elsevier.com/locate/anbehav

Commentary

Observer bias in animal behaviour research: can we believe what

we score, if we score what we believe?
F. A. M. Tuyttens a, b, *, S. de Graaf a, J. L. T. Heerkens a, L. Jacobs a, E. Nalon a, b, S. Ott b, c,
L. Stadig a, E. Van Laer a, B. Ampe a
a
Animal Sciences Unit, Institute for Agricultural and Fisheries Research (ILVO), Melle, Belgium
b
Faculty of Veterinary Medicine, Ghent University, Merelbeke, Belgium
c
Faculty of Bioscience Engineering, Katholieke Universiteit Leuven, Belgium

a r t i c l e i n f o
Most observers in behaviour studies are aware of relevant information about the animals being observed.
Article history: We investigated whether observer expectations influence subjective scoring methods during a class
Received 28 October 2013 practicum. Veterinary students were trained in recording negative and positive interactions between
Initial acceptance 19 November 2013 pigs, in scoring the degree of panting in cattle and in applying qualitative behaviour assessment (QBA)
Final acceptance 14 January 2014 using a fixed set of terms for assessing hens’ behaviour. The students applied these methods in three
Available online 12 March 2014 trials in which they were shown duplicated video recordings of the same animals: the original and a
MS. number: 13-00901 slightly modified version (to prevent recognition at second viewing). When scoring the duplicated re-
cordings they were told either correct or false information about the conditions in which the animals had
Keywords: been filmed. The false information reflected plausible study scenarios in ethology and was used to create
animal welfare
expectations about the outcome. As in reality the students scored the identical behaviour twice, the
behaviour scoring
difference in the scores for the original and modified recordings reflects expectation bias due to
cognitive bias
confirmation bias
providing different contextual information. In all trials there was evidence of expectation bias: students
double-blind experiment scored the ratio of positive to negative interactions higher when told that the observed pigs had been
expectation bias selected for high social breeding value, they scored cattle panting higher when told that the ambient
information bias temperature was 5
C higher than in reality, and in the QBA they indicated more positive and fewer
observer bias negative emotions when told that the hens were from an organic instead of a conventional farm. The
panting score magnitude of the bias in the QBA trial was related to the opinion of the students about hen welfare in
qualitative behaviour assessment organic versus conventional farms. Although veterinary students may not be representative of practising
ethologists, these findings do indicate that observer bias could influence subjective scores of animal
behaviour and welfare.
Ó 2014 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.

Scientific research that relies on observation and interpretation
by the investigator has long been confronted with a well-known
and fundamental problem: humans cannot be assumed always to
process information objectively and accurately. Natural selection
has shaped the human sensory processing system to promote
behaviour that enhances the spread of our genes, not necessarily to
provide us with a complete and correct picture of reality. There is
ample evidence that human perception can be selective and biased,
and that a healthy human brain often makes incorrect associations
and deductions (Braeckman & Boudry, 2011). For example, psy-
chologists have long recognized that people are prone to expecta-
tion bias, which refers to the psychological sway towards one
* Correspondence: F. Tuyttens, Animal Sciences Unit, Institute for Agricultural
and Fisheries Research (ILVO), Scheldeweg 68, 9090 Melle, Belgium.
E-mail address: frank.tuyttens@ilvo.vlaanderen.be (F. A. M. Tuyttens).
opinion versus another as a result of possessing information
extraneous to the task at hand (Page, Taylor, & Blenking, 2012).
Information that confirms one’s beliefs or hypotheses is often fav-
oured (Nickerson, 1998; Wason, 1960). Investigators and research
staff are also susceptible to these pitfalls of the human brain. Often,
they carry out experiments while they are predisposed by strong
expectations about the outcome and deep-rooted assumptions
about what is and what is not possible. These expectations may
lead to conscious or unconscious biases in observation and
recording of data.
The risk of these types of observer bias can be reduced by
ensuring that the person collecting the data is unaware of which
treatment each subject has received until after the experiment.
Such blind trials are widely considered as the best study design
to minimize observer bias and are often required to attain reg-
ularity approval for new drugs, dietary supplements and medical

http://dx.doi.org/10.1016/j.anbehav.2014.02.007
0003-3472/Ó 2014 The Association for the Study of Animal Behaviour. Published by Elsevier Ltd. All rights reserved.
274 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280
devices (Kaptchuk, 2001; Miller & Stewart, 2011). Meta-analyses
have convincingly shown that randomized trials have substan-
tially larger treatment effects if the assessor is not blinded
(Hróbjartsson et al., 2012, 2013; Schulz, Chalmers, Hayes, &
Altman, 1995). Nevertheless, the use of nonblinded assessors in
nonpharmacological trials remains common (Hróbjartsson et al.,
2013; Schulz et al., 1995), particularly in animal behaviour
studies. Burghardt et al. (2012) reviewed several hundred articles
published in five leading animal behaviour journals during the
last five decades. They found that, despite the numerous and
widely used texts on research methods in animal behaviour that
advocate researchers should minimize potential observer bias in
their studies, only 6.3% of the empirical research articles re-
ported that at least one component of the research was con-
ducted blind. This percentage was much higher in two more
human-focused comparison journals that publish research
based on similar behavioural observations and coding strategies
(25% for Behavioral Neuroscience and 47.5% for Infancy). We
checked the 2012 volumes of Animal Behaviour and Applied An-
imal Behaviour Science; only 15.3% (37/242) and 9.9% (13/131) of
the papers for which we judged it relevant reported that they
were conducted blind.
Marsh and Hanlon (2004) remarked that while the potential
for observer biases in animal behaviour research has often been
discussed, there have been few quantitative analyses of the
kinds of biases that may affect behavioural data. We cannot find
any scientific reason to justify the limited attention being paid to
minimizing observer bias in animal behaviour science. Observer
bias is particularly likely when the investigator has strong pre-
conceptions or a vested interest in the outcome, when the un-
derlying data are ambiguous and when the scoring method is
subjective (Hoyt & Kerns, 1999; Page et al., 2012; Risinger, Saks,
Thompson, & Rosenthal, 2002; Rosenthal, 1966, Schulz,
Chalmers, & Altman, 2002). In our opinion, all three predispos-
ing factors are commonly present in animal behaviour research.
In this study, we investigated the potential of observer bias in
animal behaviour studies. Veterinary medicine students un-
knowingly participated in an experiment during a class practicum.
The students were first given a brief demonstration of several
subjective (in the sense that they rely on an individual’s percep-
tion and judgement, and can therefore be influenced by experi-
ence or personal views, cf. Meagher, 2009) animal-based scoring
methods commonly used in ethology. They then received a short
training session on using these methods. Last, they applied the
methods to score video clips of farm animals that they were led to
believe had been subjected to different conditions. In reality,
however, they scored the same video clips twice, with the second
clip being slightly modified to trick the students into thinking that
the videos were of different animals subjected to the mock con-
ditions. The false information was specifically chosen to create
expectations among the students about the outcome of the ob-
servations. The research objective was to calculate the magnitude
of observer bias as the difference between the first and second
scoring of the video clips. In addition to this research objective, the
practicum was also designed to meet two educational goals: to
learn and practise a limited but diverse set of scoring methods
used in ethological research, and to raise the students’ awareness
about observer bias by allowing them to experience how prior
expectations affected their scoring. Goldstein, Hopkins, and Strube
(1994) argued that a classroom demonstration in which the stu-
dents personally experience the powerful effects of previous ex-
pectations on perception should improve their learning and
memory about observer bias. We do not include that aspect of this
study in the current paper and report on the research objective
only.
METHODS
Subjects and Overall Experimental Set-up
The trials were conducted using third-year veterinary medicine
students at Ghent University during the practicum of the Ethology,
Ethics and Animal Welfare course. The students (N ¼ 157) were
informed that the aim of the practicum was to learn some tech-
niques and methods commonly used in ethological research: (trial
1) identifying negative and positive social interactions in pigs; (trial
2) scoring panting as an indicator of heat stress in cattle; (trial 3)
qualitative behaviour assessment (QBA) of laying hens. For logistic
reasons, the training and trials were based on photographs and
video recordings rather than live observations. These video re-
cordings were obtained from recent or ongoing research projects
approved by the ILVO ethical committee for experiments on ani-
mals. All methods concerned animal-based measures that were
subjective in that they required some interpretation and judgement
by the students when giving scores or values to their observations
of the animals. The students were not informed about the research
aim of the practicum, namely to investigate the extent to which
they were affected by misleading background information when
applying the three scoring methods they had learned.
After the students had been informed about the content and
organization of the practicum, they were randomly split into two
groups of approximately equal size. For the training and the three
trials group A (N ¼ 80) remained in the auditorium, whereas group
B (N ¼ 77) moved to another nearby auditorium. Each trial was
chaired by a different designated pair of researchers and lasted 30e
40 min followed by a 10 min break. The order of the sessions
differed to enable the designated researchers to repeat their trial in
both places with both groups. After the scoring sessions, the two
groups were reunited for a plenary session during which they were
informed about the hidden research aim of the practicum and the
reason why we investigated this subject (i.e. to raise awareness
about the pitfalls of their own senses and brain when applying
subjective animal-based measures for assessing animal behaviour
and welfare).
Trial 1: Negative and Positive Social Interactions
The students were taught how to recognize negative and posi-
tive social interactions in fattening pigs (using video recordings of
various interactions) and were briefly informed about the relevance
of these interactions for animal welfare and farm management.
They were taught an ethogram that included four types of negative
interactions (head butt, belly nosing, ear or tail biting, biting other
body parts) and three types of positive interactions (play, sniffing/
nosing the body of a pen mate, nose-to-nose contact). During a
video-based training session, the students then practised counting
social interactions, differentiating between positive and negative
ones.
Subsequently, for the trial proper, the students were instructed
to tally each negative and positive social interaction in a pen of six
fattening pigs during two 5 min video clips. Both video clips were
the same, but the second clip was shown in a mirror image, its
brightness was slightly adjusted and fictional pen numbers and
dates were shown. The goal of showing the same clip twice (once in
the original version and once in the slightly modified version) was
to mislead the students into believing that it concerned two
different groups of pigs from an experiment on social breeding
value and housed in an opposite pen of the same pig stable. The
theory of social breeding value, in which pigs with a high social
breeding value have a positive effect on the growth of their pen
mates (Bergsma, Kanis, Knol, & Bijma, 2008), was explained. The
 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280 275

students were falsely informed that one of the video clips was of a
control group and the other of a group selected for high social
breeding value, and that the aim of the trial was to test whether the
ratio of positive to negative social interactions differed between the
two clips. Group A was told that the original clip was of the high
breeding value group and the subsequent modified clip was the
control group; the opposite was told to group B. The true research
aim, however, was to test whether telling the students that one of
the clips was of high social breeding value pigs affected the number
of negative and positive interactions they recorded.
Trial 2: Panting Score
The students were informed about thermoregulation and about
the signs and consequences of thermal stress in cattle. The panting
score was introduced as a measure of heat stress based on behav-
ioural signs including the rate and depth of respiration and the
amount of drooling (Gaughan, Mader, Holt, & Lisle, 2008; Mader,
Davis, & Brown-Brandl, 2006). They were told how to score pant-
ing using a tagged visual analogue scale labelled with descriptors of
an increasing degree of panting (Fig. 1). Such tagged visual
analogue scales do not limit the precision and sensitivity with
which observers can distinguish different degrees of severity of the
condition concerned. They have been shown to have equal or su-
perior interobserver reliability in comparison to ordinal scales for
scoring lameness (Nalon et al., in press; Tuyttens, Sprenger, Van
Nuffel, Maertens, & Van Dongen, 2009). During the training
phase, the students were shown six video clips of cattle with
varying degrees of panting.
Subsequently, the students were instructed to score 24 video
clips of cattle using the tagged visual analogue scale. During this
trial they could use a printed hand-out of the scale with the
different descriptors (Fig. 1). A vertical coloured bar at the right-
hand side of each video clip, as well as a video stamp in the up-
per right-hand corner, indicated ambient temperature at the time
of recording. In half of the video clips, however, not the true tem-
perature but a 5
C higher temperature was indicated. Seven video
clips were shown with the correct temperature to group A and with
the elevated temperature to group B, and seven other video clips
were shown with the correct temperature to group B and with the
elevated temperature to group A. In addition, five video clips were
shown twice, once with the correct temperature and once with the
elevated temperature, to both groups. To reduce the risk of students
recognizing the video clip when viewing it for the second time, one
clip was shown as a mirror image and was slightly modified (by
zooming in or out and by adjusting the brightness or image
contrast). We tested whether the students recorded a higher
panting score when an elevated ambient temperature was falsely
indicated on the clips.
Trial 3: Qualitative Behaviour Assessment
QBA was introduced to the students as an integrative method-
ology that characterizes animal behaviour as a dynamic, expressive
body language (Wemelsfelder 2007; Wemelsfelder, Hunter,
Lawrence, & Mendl, 2001). It was explained that it was originally
developed as a free-choice profiling in which the assessors can
generate their own descriptors for describing the expressive quality
of how animals behave and how they interact with each other and
the environment. For the purpose of the practicum, however, we
worked with a fixed list of descriptors as in the Welfare Quality
(2009) protocol in which QBA is used as an animal welfare mea-
sure at group level. After the hens were observed (from video for
the sake of this practical) their behavioural expression was scored
on a visual analogue scale for each of the 19 fixed qualitative de-
scriptors. For each term (such as ‘calm’, ‘content’ and ‘frustrated’;
see Table 1 for all terms) we used a 100 mm visual analogue scale
defined by the left-hand point, which represented ‘minimum’ (i.e.
the expressive quality indicated by the term is entirely absent in
any of the animals seen) and the right-hand point which repre-
sented ‘maximum’ (i.e. the expressive quality is dominant across all
observed animals).
For the trial proper we used a 5 min video recording of laying
hens in a conventional commercial aviary. This 5 min video was
split into two clips (clip A and clip B). One of these video clips (clip
B) was mirrored and slightly modified (made brighter) to give the
impression that it was filmed on a different farm from clip A. The
students were told that one of the video clips (clip B for group A and
clip A for group B) was from an organic laying hen farm. Prior to the
QBA trial, the main differences between organic and conventional
egg production were briefly explained. To test their pretest attitude
towards organic farming, students were instructed to indicate on a
100 mm visual analogue scale whether, in their opinion, hen wel-
fare was much worse (left side of the scale), equal (midpoint) or
much better (right side of the scale) on organic versus conventional
farms. Subsequently, to explain what they would see in the video
clips, the students were shown a demonstration video of hens from
a different farm but with a similar housing system. The layout of the

No panting, Slight panting, Fast As for 2 but Open mouth As for 3 but with Open mouth with As for 4 but
normal mouth closed, panting, occasional and the tongue out tongue fully head held
respiration, no drool, easy drool open mouth excessive slightly and extended for down, cattle
difficult to to see chest present, panting, drooling, occasionally prolonged periods “breathe”
see chest movement. no open tongue not neck fully extended with excessive from the
movement. mouth. extended. extended, for short drooling, neck flank,
head held periods. extended and drooling may
up. head up. cease.

PS 0 1 2 3 4 5

0 10 20 30 40 50 60 70 80 90 100 mm

Figure 1. The 100 mm tagged visual analogue scale labelled with descriptors used for scoring panting in cattle during trial 2.
276 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280
aviary housing system (litter area, elevated platforms with nest-
boxes, feeding chains and drinker lines) was explained. After this
explanation, the students then watched and scored video clip A first
and then clip B. We tested whether the QBA terms were scored
differently if the students had been told that the hens were filmed
on an organic farm and whether this difference was related to their
opinion about hen welfare on organic as compared to conventional
farms.
Statistical Analysis
The results of all trials were analysed using a mixed regression
model with a random effect for student to correct for repeated
measures. In each model, treatment (prior knowledge: e.g. control
12 12
Number of scored behaviours
(a)
10 *** 10
8 8
6 6
4 4
2 2
0 0
Positive social behaviour
Figure 2. The mean number  SE of (a) positive and (b) negative behaviours scored during the 5 min video clip of the control group and the video clip where the students were told
that the animals were selected for high social breeding value (SBVþ). ***P < 0.001.
or high breeding value group, true or elevated temperature indi-
cation, conventional or organic production system), group (A or B)
and video clip (original or modified) and all interactions were
introduced as fixed effects. Interactions and fixed factors (except
main effect of treatment and video clip) were removed from the
final model if the estimated effect was not significant. The model
assumptions were evaluated using graphical procedures (normality
of the residuals was evaluated with a histogram and QQ-plot and
the homogeneity of variance was evaluated on a plot of the re-
siduals versus fitted values).
For the panting score trial, the data set was divided into two
parts: (1) the 14 video clips that were shown only once in the same
group combined with the first video clip from those shown twice to
both groups (five video clips); and (2) the five video clips that were
shown twice to both groups. The unique video clip number or true
temperature was also introduced in the model. For the 19 video
clips of the first part, we also tested whether the difference be-
tween the scores with correct and elevated temperature indication
was associated with the corresponding standard deviation of the
scores (as an indication of the variation between observers when
scoring the same clip). A large standard deviation could indicate
that these clips were ambiguous and therefore hard to score
consistently by the students.
For the QBA trial, the QBA terms were handled using principal
component analysis (PCA), with no rotation, using the princomp
function in R 2.15.1 (The R Foundation for Statistical Computing,
Vienna, Austria, http://www.r-project.org). The first two PCA
components and the individual QBA terms were analysed using a
mixed regression model.
All tests were two tailed at a significance level of 5% and all
calculations were performed using the lme function from the nlme
package in R 2.15.1 for Windows.
RESULTS
Trial 1: Negative and Positive Social Interactions
A higher number of positive (ANOVA: F1,153 ¼ 29.6, P < 0.001)
and fewer negative (ANOVA: F1,153 ¼ 75.7, P < 0.001) social in-
teractions were counted when the students were informed that the
pigs had been selected for high social breeding value as compared
to the control pigs (Fig. 2). Modifying the video clips itself did not
affect the recorded number of positive (ANOVA: F1,153 ¼ 1.6,
P ¼ 0.212) or negative interactions (ANOVA: F1,153 ¼ 0.6, P ¼ 0.437).
Also, no differences between the two groups of students were
found (linear regression: F1,153 ¼ 1.0, P ¼ 0.309 and F1,153 ¼ 0.00,
P ¼ 0.978).
***
(b)
Control
SBV+
Negative social behaviour
Trial 2: Panting Score
For the 19 cattle recordings that were shown to each group only
once, the students that had been led to believe that ambient tem-
perature was 5
C higher than in reality recorded a higher panting
score on average than the students that had been shown the real
temperature (ANOVA: F1,2772 ¼ 8.5, P ¼ 0.004; Fig. 3). The magni-
tude of this difference was small, however (3.1% at 20
C). The di-
rection of the difference and its magnitude were not consistent for
the various video clips (Fig. 3). The average panting score increased
with the true ambient temperature at the time of video recording
(linear regression: F1,2772 ¼ 984.0, P < 0.001), but did not differ
between the two groups of students (linear regression,:
F1,153 ¼ 0.12, P ¼ 0.729). There was a nonsignificant trend for the
difference between the panting score allocated to the clips with the
indication of an elevated temperature versus that for the real
temperature to decrease with increasing true ambient temperature
(ANOVA: F1,2772 ¼ 3.0, P ¼ 0.081). This difference was also not
related to the standard deviation of the scores (as an indicator of
the amount of variation between observers when scoring the same
clips; ANOVA: F1,17 ¼ 0.7, P ¼ 0.41).
For the five cattle recordings that had been shown twice to each
group (once with the correct temperature and once with an
elevated temperature of 5
C), the panting score was 5.0% higher (at
20
C) on average for the indication of an elevated temperature
versus the real temperature (ANOVA: F1,1384 ¼ 54.6, P < 0.001;
Fig. 4). The average panting score increased with the true ambient
temperature at the time of video recording (linear regression:
F1,619 ¼ 1070.3, P < 0.001), but did not differ between the two stu-
dent groups (linear regression: F1,153 ¼ 1.1, P ¼ 0.287). The differ-
ence between the panting score allocated to the clips with the
indication of an elevated temperature versus the real temperature
 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280 277

100
Real temperature
90
Elevated temperature
80

70
Panting score

60

50 *

40 **
*
30
**
20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 Video no.
19 20 20 20 20 23 23 23 23 23 32 32 32 34 Real temperature (°C)

Figure 3. The mean þ SE panting score (0e100 visual analogue score) given to the 14 video clips of cattle shown to one group of students with the correct ambient temperature
indication (real temperature) versus the other group of students with an elevated temperature indication (elevated temperature). The real ambient temperature at the time of
filming is indicated at the bottom of the figure. *P < 0.05; **P < 0.01.

100 recorded on an organic farm they gave, or tended to give, a lower

Real temperature score to all negative descriptors and a higher score to all positive
90 descriptors with the exception of the descriptor ‘active’ (effect of
Elevated temperature system in Table 1). The size of these effects increased along with a
80 * more positive opinion about hen welfare in organic versus con-
ventional systems (opinion*system interactions in Table 1). The
70
scores for most terms also differed between video clips A and B
(effect of video in Table 1). This was surprising, because both video
Panting score

60
clips were recorded in the same aviary at almost exactly the same
50 time and without any clear disturbance of the flock during
** recording.
40 The PCA was carried out to elucidate the relationship between
the 19 descriptors and revealed two major components. The first
30 *** component explained 42.9%, and the second component 12.8%, of
20
Table 1
***
10 Estimates and significance levels of linear mixed-effects models on each of the 19
fixed descriptors used in the qualitative behaviour assessment trial
0 Descriptor Opinion System Video Opinion* System*
15 16 17 18 19 Video no. System Video

20 23 29 32 32 Real temperature (°C) Active 0.1 3.6 9.3** 0.1 1.4

Relaxed 0.3* 48.4*** 2.0 0.8*** 12.8*
Figure 4. The mean þ SE panting score (0e100 visual analogue score) given to the five Fearful 0.3* 27.6** 2.9 0.5*** 0.7
video clips of cattle shown twice to all students: once with the correct ambient Agitated 0.1 26.6* 2.4 0.5** 8.2
temperature indication (real temperature) and once slightly modified and with an Confident 0.3* 37.7*** 12.0*** 0.7*** 9.0(*)
elevated temperature indication (elevated temperature). The real ambient temperature Depressed 0.3* 32.4*** 2.2 0.5*** 8.2
at the time of filming is indicated at the bottom of the figure. *P < 0.5; **P < 0.01; Calm 0.2 25.6* 8.3* 0.5** 1.5
***P < 0.001. Content 0.3** 41.5*** 9.5** 0.7*** 14.0**
Tense 0.3* 58.1*** 7.4* 0.9*** 17.5**
Unsure 0.3* 33.2** 7.6* 0.6*** 7.8
Energetic 0.3* 19.3(*) 12.0*** 0.4** 1.1
decreased with increasing true ambient temperature (linear Frustrated 0.3* 57.8*** 7.8* 0.9*** 18.0**
regression: F1,767 ¼ 22.6, P < 0.001). Bored 0.3* 45.4*** 10.4** 0.8*** 10.6(*)
Friendly 0.4** 32.7*** 1.5 0.6*** 1.1
Positively occupied 0.3* 39.0*** 6.3* 0.8*** 7.1
Trial 3: Qualitative Behaviour Assessment Scared 0.2(*) 17.4(*) 2.4 0.4** 3.0
Nervous 0.1 51.6*** 9.4* 0.9*** 12.6*
The students had a rather positive opinion about laying hen Happy 0.4*** 55.2*** 7.4* 0.9*** 6.6
Distressed 0.2 63.2*** 13.1*** 1.0*** 20.1**
welfare on organic versus conventional egg production units, as
indicated by their average score being above the neutral point of Variables included in the models are self-reported opinion about hen welfare in
the scale (50 mm). The opinion of female students was more pos- organic versus conventional production systems (Opinion), whether the students
were deceived into believing that the video was recorded in an organic versus
itive than that of male students (females: 66 mm (95%CI: conventional farm (System), video clip A or B (Video), and interactions between
63.3;67.8); males: 58 mm (95%CI: 54.8;62.11); ANOVA: F1,153 ¼ 10.6, Opinion and System and between System and Video.
P ¼ 0.001). When the students were told that the hens were video (*)P < 0.1; *P < 0.5; **P < 0.01; ***P < 0.001.
278
2 teractions was increased and the number of negative interactions
Standardized PC2 (12.8% explained variance)
1 (trial 2); and the QBA scoring indicated more positive and fewer
Depr Bored
essed
Unsure
0 Frustratedared
Sc arful
Fe
es s e d
Distr enses
T u ed
rvo at
Ne Agit
Activ
–1
En
erg
e
–2
–2 –1 0 1
Standardized PC1 (42.9% explained variance)
Figure 5. A plot of scores showing the relation between principal component 1 and
principal component 2 and the loadings of the 19 fixed descriptors used in the qual-
itative behaviour assessment session. Diamonds indicate the scores for students who
were correctly informed that the video was recorded in a conventional aviary. Rect-
angles indicate the scores for students who were told that the video was recorded in an
organic aviary system. The black (conventional) and grey (organic) ellipses are the
normal data ellipses which contain 70% of the data of the two groups.
the total variance. Other components explained only a small part of
the variance and are not considered further for analysis. The
loadings (magnitude and direction) of the different descriptors to
both components are shown in Fig. 5. In general, the first compo-
nent seems to describe the affective states of the hens (the higher
the score, the more positive the affective state), whereas the second
component seems to describe the level and type of activity (the
lower the score, the more active and energetic the hens). The stu-
dents’ score for component 1 was higher if they had been told that
the video was recorded on an organic farm instead of a conven-
tional farm (ANOVA: F1,148 ¼ 54, P < 0.001; Fig. 5). The magnitude
of this effect was positively correlated with the students’ opinion
about hen welfare in organic versus conventional farms (ANOVA:
opinion*system interaction: F1,148 ¼ 75.9, P < 0.001). Deceiving the
students into believing that the hens were filmed on an organic
farm did not affect component 2 (ANOVA: F1,148 ¼ 0.4, P ¼ 0.504).
DISCUSSION
This study has demonstrated that observer bias is likely to be a
more important problem in applied animal behaviour science than
is often realized. Observer bias was evaluated in the three student
trials by manipulating the contextual information about the ani-
mals they observed and scored. The false contextual information
was intended to simulate a variety of plausible, realistic and
contemporary study scenarios that an animal behaviour researcher
may encounter. In addition, these scenarios were deliberately
chosen to evoke expectations about the likely scoring outcomes.
We were specifically interested in evaluating the extent to which
such a priori expectations influence the interpretation and scoring
of animal behaviour. Although in reality the students unknowingly
scored the same (group of) animals twice, their beliefs influenced
F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280
their scoring in all three trials. The number of positive social in-
decreased when the assessors were told that the pigs had been
selected for high social breeding value (trial 1); the panting score
allocated to cattle was higher when the observers were led to
believe that the ambient temperature was higher than in reality
negative emotions when the observers were told that the hens
lm were kept in an organic instead of a conventional farm. Thus,
Ca misleading background information affected scores in all three
trials, showing direct evidence of expectation bias by the assessors.
Relaxed Because we gathered additional pretest data quantifying observers’
Frie
ndl views on animal welfare on organic farms, the QBA trial was able to
y
HaCon provide the most convincing evidence that the bias was related to
Co P pp ten
nf os y t expectations of the student observers.
id O
en cc Although students are often involved in collecting data in ani-
t
mal behaviour, an obvious question in the present study is to what
extent the documented observer biases of undergraduate veteri-
e
nary science students are similar to the biases of practising ethol-
tic
ogists. Probably in most research the data recorders will be better
trained as observers and therefore have more experience than the
students used in our trials. Although we are not aware of studies
2 that have shown that observer training successfully reduces
expectation bias, the possibility exists that the observed biases
would be less prevalent in practising scientists than in our sample
of university students. Undergraduate students may be especially
prone to see, or even pretend to see, what they are told to expect.
On the other hand, undergraduate students may not be as strongly
personally committed to particular views of reality as experienced
scientists who might have firm expectations and large investments
in particular outcomes of their research (Marsh & Hanlon, 2004).
Indeed, research hypotheses are often also personal prophesies
which most researchers like to see come true in order to strengthen
confidence in their scientific insights (Lehner, 1979, p. 129). This
could make experienced researchers particularly subject to pre-
diction or confirmatory bias. Moreover, misleading the students by
providing them with false contextual information about the treat-
ments may not evoke equally strong expectations as in real situa-
tions. For example, the effect of different thermal conditions when
scoring the degree of panting in cattle is likely to be stronger when
the observer is personally exposed to these conditions during live
scoring in the field instead of being given this information when
scoring from video.
Our three trials with farm animals all required some degree of
interpretation and subjective judgement when recording or scoring
the behavioural observations from video. As a consequence, these
trials are not representative for trials with clearly objective mea-
surement scale outcomes which leave little room for bias. Examples
of such measures could be the measurement of body temperature
using a thermometer, the concentration of cortisol in plasma as
analysed in the laboratory or the automated recording of activity
level using pedometers. Although some would like science to focus
exclusively on empirical facts that can be observed and quantified
objectively, in reality many recording methods and scales used in
animal behaviour science are subjective to some extent. Gradations
or categories in the outcome of behavioural studies as well as
clinical inspections are rarely cut-and-dried or easy to define in a
precise way. Recording methods often require subjective judge-
ments by the assessor to categorize behaviours (e.g. differentiating
between play and aggression in ethograms), to interpret the
outcome of behavioural interactions (e.g. indicating the winners
and losers of agonistic interactions for calculating social hierarchy),
to characterize the personality of an animal, or to quantify the
severity of a physical or clinical condition (e.g. locomotion scoring
for assessing the severity of lameness). Meagher (2009) has even
 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280
argued that such subjective observer ratings may offer not only
practical (e.g. money-saving) but also scientific advantages (e.g. for
integrating multimodal information across time and situations, and
for constructs that are otherwise difficult to assess such as pain).
Moreover, she claims that such ratings can be both reliable and
valid conditional upon careful experimental design to minimize
observer bias.
Textbooks on behavioural recording methods therefore high-
light the need to reduce observer bias by properly training asses-
sors, by testing inter- and intraobserver reliability, and by randomly
assigning observers to different groups or treatments (Dawkins,
2007; Lehner, 1979; Martin & Bateson, 1993). Another common
recommendation is to use precise and unambiguous operational
definitions of behavioural categories (Lerman et al., 2010). Very
explicit criteria for each value on a scale make the ratings less
dependent on personal experience or other factors that will make
the judgement vary greatly between individuals (Meagher, 2009).
Of the three scoring scales that were used in our study, the panting
score was the most precisely defined. Although the indication of an
elevated temperature did significantly increase the students’
average panting score, the absolute size of this effect was small. The
specific descriptors on the scale may have forced the observers to
check for the appropriate signs by closely observing the animals.
This may have reduced any expectation bias effect. With regard to
QBA it should be noted that asking observers to score one video clip
in isolation as was done in this study is not representative of its
methods. QBA trials normally involve comparative assessment of
20 video clips or more, within a specific investigative context
presented by the experimenter, which allows for a more carefully
weighted assessment of individual clips in relation to each other.
Although expectation bias may still be present in observers, this
design offers greater opportunity for managing the context in
which clips are assessed.
Such recommendations and actions to reduce interobserver
variation may not suffice, however, to prevent all possible sources
of observer bias. For example, Hróbjartsson et al. (2013) reviewed
16 randomized clinical trials with subjective outcome assessment
by both blinded and nonblinded assessors. The trial with the largest
degree of observer bias actually had a very good interobserver
agreement. A high inter- and intraobserver agreement is by no
means a guarantee of absence of expectation bias, as observers may
have been biased in the same direction because they hold similar
expectations about the outcome of the study. Also, in our panting
score trial we found no indication that expectation bias was lowest
(i.e. small difference between the scores with the correct and
elevated temperature indication) for those video clips that were
scored similarly between students (as indicated by a low standard
deviation).
Observer blinding, that is, withholding information about the
differences between the animals or the experimental treatments
from the observer, is one of the most effective and powerful solu-
tions to avoid the conscious or unconscious suppositions of the
assessor influencing the scoring outcomes. For example, a ‘blinded’
QBA study that compared pigs treated with the anxiolytic drug
Azaperone with nontreated pigs in an elevated plus-maze found
that observers, although unaware of the treatment, could clearly
distinguish between the expressions of treated and untreated pigs,
indicating that when observer bias is managed adequately,
methods such as QBA can produce valuable results (Rutherford,
Donald, Lawrence, & Wemelsfelder, 2012). Our findings thus call
for the more widespread use of blind experiments in animal
behaviour research whenever possible and for placing greater
credence in results when observers were blinded as compared to
not blinded. Withholding information from researchers and their
data collection staff becomes an increasingly important part of
279
preparing well-designed studies as the collection and recording of
data become more subjective (Schulz et al., 2002). Although such
blind trial designs are likely to increase the complexity and cost of
research, the technology now available to record behaviour can in
theory make this possible in most experimental trials (Boutron
et al., 2007; Day & Altman, 2000).
Nevertheless, we do acknowledge that blinding the assessors
can be difficult to achieve in certain research set-ups. Blinding
observers may be hindered where treatment effects are easy to see
or guess, or in less experimental research settings such as obser-
vational studies. An example in place is the on-farm monitoring of
animal welfare. Farm animal welfare monitoring and auditing
protocols increasingly apply behavioural observations and ratings
similar to those used in the present study. There is growing
consensus that such protocols ought to use animal-based welfare
measures where possible (Anonymous, 2012; Blokhuis, Veissier,
Miele, & Jones, 2010; Main, Whay, Leeb, & Webster, 2007). These
measures are believed to be more directly linked to the true welfare
status of the animals than resource-based measures that describe
the housing and management conditions in which the animals are
kept (Roe, Buller, & Bull, 2011; Webster, Main, & Whay, 2004), but
may also involve a greater level of subjectivity. Animal welfare
assessment schemes such as the Welfare Quality (2009) protocols
not only have to be valid and reliable, they also ought to be feasible
and cost-effective if a wide uptake by the food industry is to be
achieved. The latter requirement favours assessments based on
observer ratings that are relatively nonintrusive, inexpensive and
can be used to integrate multimodal information across time and
situations at the expense of tests that are perhaps more accurate
but that are expensive, time consuming or dependent on laboratory
or technical equipment (Meagher, 2009). With manpower being
the main cost of such animal-based monitoring schemes, it is likely
that farms are visited and assessed by only a single auditor.
Consequently, interobserver reliability has been rightly considered
in the selection of the welfare indicators and the training of audi-
tors (Dalmau et al., 2010; Gibbons, Vasseur, Rushen, & de Pasillé,
2012; Phytian et al., 2013). In addition to these precautions, our
findings point to the need to pay more attention to how the ex-
pectations and predispositions of the auditors may bias their re-
cordings when they assess the welfare of the animals on the farms
they visit.
Acknowledgments
We thank C. Moons, T. Decroos, J. Vander Linden and T. Martens
for their help with the student trials; M. Levenson for English
revision; D. Lapage for the data entry; and S. Buijs, S. De Campe-
neere and F. Wemelsfelder for commenting on the manuscript. We
thank the veterinary students of Ghent University who participated
in the trial.
References
Anonymous. (2012). EFSA recommends use of animal-based measures when
assessing welfare. Veterinary Record, 170, 112.
Bergsma, R., Kanis, E., Knol, E. F., & Bijma, P. (2008). The contribution of social effects
to heritable variation in finishing traits of domestic pigs (Sus scrofa). Genetics,
178, 1559e1570.
Blokhuis, H. J., Veissier, I., Miele, M., & Jones, B. (2010). The Welfare QualityÒ project
and beyond: safeguarding farm animal well-being. Acta Agriculturae Scandi-
navica Section A: Animal Science, 60, 129e140.
Boutron, I., Guittet, L., Estellat, C., Mohar, D., Hróbjartsson, A., & Ravaud, P. (2007).
Reporting methods of blinding in randomized trials assessing non-
pharmacological treatments: a systematic review. PLoS Med, 4, e61.
Braeckman, J., & Boudry, M. (2011). De ongelovige thomas heeft een punt: Een han-
dleiding voor kritisch denken. (The reasonableness of doubting Thomas: An invi-
tation to critical thinking). Antwerp, Belgium: Houtekiet.
280 F. A. M. Tuyttens et al. / Animal Behaviour 90 (2014) 273e280
Burghardt, G. M., Bartmess-LeVasseur, J. N., Browning, S. A., Morrsion, K. E.,
Stec, C. L., Zachau, C. E., et al. (2012). Perspectives: minimizing observer bias in
behavioral studies: a review and recommendations. Ethology, 118, 511e517.
Dalmau, A., Geverink, N. A., Van Nuffel, A., Van Reenen, K., Hautekiet, V.,
Vermeulen, K., et al. (2010). Repeatability of lameness, fear and slipping scores
to assess animal welfare upon arrival in pig slaughterhouses. Animal, 4, 804e
809.
Dawkins, M. S. (2007). Observing animal behaviour: Design and analysis of quanti-
tative data. Oxford, UK: Oxford University Press.
Day, S. J., & Altman, D. G. (2000). Statistics notes: blinding in clinical trials and other
studies. British Medical Journal, 321, 504.
Gaughan, J. B., Mader, T. L., Holt, S. M., & Lisle, A. (2008). A new heat load index for
feedlot cattle. Journal of Animal Science, 86, 226e234.
Gibbons, J., Vasseur, E., Rushen, J., & de Pasillé, A. M. (2012). A training programme
to ensure high repeatability of injury scoring of dairy cows. Animal Welfare, 21,
379e388.
Goldstein, M. D., Hopkins, J. R., & Strube, M. J. (1994). ‘The eye of the beholder’: a
classroom demonstration of observer bias. Teaching of Psychology, 21, 154e157.
Hoyt, W. T., & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer
ratings: a meta-analysis. Psychological Methods, 4, 403e424.
Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I.,
et al. (2012). Observer bias in randomised clinical trials with binary outcomes: a
systematic review of trails with both blinded and non-blinded outcome as-
sessors. British Medical Journal, 344. e1119.
Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I.,
et al. (2013). Observer bias in randomized clinical trials with measurement
scale outcomes: a systematic review of trails with blinded and non-blinded
assessors. Canadian Medical Association Journal, 185, E201eE211.
Kaptchuk, T. J. (2001). The double-blind, randomized, placebo-controlled trial: gold
standard or golden calf? Journal of Clinical Epidemiology, 54, 541e549.
Lehner, P. N. (1979). Handbook of ethological methods. New York: Garland STPM
Press.
Lerman, D. C., Tetreault, A., Hovanetz, A., Bellaci, E., Miller, J., Karp, H., et al. (2010).
Applying signal-detection theory to the study of observer accuracy and bias in
behavioural assessment. Journal of Applied Behaviour Analysis, 43, 195e213.
Mader, T. L., Davis, M. S., & Brown-Brandl, T. (2006). Environmental factors influ-
encing heat stress in feedlot cattle. Journal of Animal Science, 84, 712e719.
Main, D. C. J., Whay, H. R., Leeb, C., & Webster, A. J. F. (2007). Formal animal-based
welfare assessment in UK certification schemes. Animal Welfare, 16, 233e236.
Marsh, D. M., & Hanlon, T. J. (2004). Observer gender and observer bias in animal
behaviour research: experimental tests with red-backed salamanders. Animal
Behaviour, 68, 1425e1433.
Martin, P., & Bateson, P. (1993). Measuring behaviour: An introductory guide (2nd
ed.). Cambridge, UK: Cambridge University Press.
Meagher, R. K. (2009). Observer ratings: validity and value as a tool for animal
welfare research. Applied Animal Behaviour Science, 119, 1e14.
Miller, L. E., & Stewart, M. E. (2011). The blind leading the blind: use and misuse
of blinding in randomized controlled trials. Contemporary Clinical Trials, 32,
240e243.
Nalon, E., Maes, D., Van Dongen, S., van Riet, M. M. J., Janssens, G. P. J., Millet, S.,
et al. (2014). Comparison of the inter- and intra-observer repeatability of three
gait-scoring scales for sows. Animal (in press). http://dx.doi.org/10.1017/
S1751731113002462.
Nickerson, R. S. (1998). Confirmation bias: an ubiquitous phenomenon in many
guises. Review of General Psychology, 2, 175e220.
Page, M., Taylor, J., & Blenking, M. (2012). Context effects and observer bias: im-
plications for forensic odontology. Journal of Forensic Sciences, 57, 108e112.
Phytian, C. J., Toft, N., Cripps, P. J., Michalaopolou, E., Winter, A. C., Jones, P. H., et al.
(2013). Inter-observer agreement, diagnostic sensitivity and specificity of
animal-based indicators of young lamb welfare. Animal, 7, 1182e1190.
Risinger, D. M., Saks, M. J., Thompson, W. C., & Rosenthal, R. (2002). The Daubert/
Kumho implications of observer effects in forensic science: hidden problems of
expectation and suggestion. California Law Review, 90, 1e56.
Roe, E., Buller, H., & Bull, J. (2011). The performance of farm animal assessment.
Animal Welfare, 20, 69e78.
Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Apple-
ton-Century-Crofts.
Rutherford, K. M. D., Donald, R. D., Lawrence, A. B., & Wemelsfelder, F. (2012).
Qualitative behavioural assessment of emotionality in pigs. Applied Animal
Behaviour Science, 139, 218e224.
Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The landscape and lexicon of
blinding in randomized trials. Annals of Internal Medicine, 136, 254e259.
Schulz, K. F., Chalmers, I., Hayes, R. J., & Altman, D. G. (1995). Empirical evidence of
bias: dimensions of methodological quality associated with estimates of
treatment effects in controlled trials. Journal of the American Medical Associa-
tion, 273, 408e412.
Schulz, K. F., & Grimes, D. A. (2002). Blinding in randomised trials: hiding who got
what. Lancet, 359, 696e700.
Tuyttens, F. A. M., Sprenger, M., Van Nuffel, A., Maertens, W., & Van Dongen, S.
(2009). Reliability of categorical versus continuous scoring of welfare in-
dicators: lameness in cows as a case study. Animal Welfare, 18, 399e405.
Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task.
Quarterly Journal of Experimental Psychology, 12, 129e140.
Webster, A. J. F., Main, D. C. J., & Whay, H. R. (2004). Welfare assessment: indices
from clinical observations. Animal Welfare, 13, S93eS98.
Welfare Quality. (2009). Welfare QualityÒ assessment protocol for poultry (broilers,
laying hens). Lelystad, Netherlands: Welfare QualityÒ Consortium.
Wemelsfelder, F. (2007). How animals communicate quality of life: the qualitative
assessment of animal behavior. Animal Welfare, Supplement, 16, 25e31.
Wemelsfelder, F., Hunter, E. A., Lawrence, A. B., & Mendl, M. T. (2001). Assessing
the ‘whole-animal’: a free-choice-profiling approach. Animal Behaviour, 62,
209e220.