Research Methodology

1.1 Classifying Empirical Research

In any research activity, the researchers aim is to find out something about something.
This can be done in different ways; one method, widely used in HCI, is empirical
research. Empirical research is characterised by the fact that knowledge or theory that
is derived from it is arrived at as a result of observations or experiments. (Robson,
2002) writes that empirical research ‘involves a systematic investigation of an
experience which should be both sceptical and ethical.’. (Cresswell, 2005) identifies
the separate processes that make up empirical research to be
Identification of a research problem
Review of the existing literature
Specification of a purpose
Collection of Data
Analysis and Interpretation of data
Reporting on and evaluating data
Commonly a research problem is formulated in a very general way and then the
researcher identifies different purposes that dictate how the subsequent data gathering
will take place and how it will be analysed. In some instances, the researcher has a
theory that he wishes to test out; in other cases the researcher has observed a
phenomenon that he wishes to develop a theory about. These two approaches are
sometimes referred to as normative and non-normative or deductive and inductive. It
is common to match these approaches to two research styles, these being quantitative
research and qualitative research (Cohen and Manion, 1994).

1.1.1 Quantitative Research

The general process in quantitative research is to test a theory by relating independent

variables to dependent variables in a controlled setting. Surveys and experiments are
commonly used for quantitative studies

1.1.2 Qualitative Research

It is often said (wrongly) that qualitative research is any research that does not include
numbers! This is not a good definition as it is the approach to the data gathering that
is different in the two styles of research rather than the outputs. Qualitative research
aims to explore, discover, understand or describe phenomena that have already been
identified but are not well understood. The tools that are used for qualitative research
include observations and interviews and the methodological tool is interpretation. In
this sort of research, theories are often ‘grounded’ in data and ethnographic and
narrative methods are used to assist in the interpretation and understanding of social
interactions and phenomena.
1.1.3 Research Methods
The tools that are used in research have generally been developed to fit into one of the
two styles of research outlined in the previous section. The following figure (adapted
from (Cresswell, 2005)) (Figure 1) shows the relationships between research styles
and methods.

Figure 1 - Research Designs

1.1.3.1 Experimental Research

Sometimes referred to as intervention studies or group comparison studies; these are
research activities in which the impact of an activity is measured by offering it to one
group, whilst withholding it from another. They are used to establish cause and effect
and the emphasis is on controlling the variables in such a way that the cause of an
effect can be attributed to one condition. The validity of this sort of research depends
on the random assignment of individuals to the groups that are being compared, the
amount of control that is exerted over extraneous factors, how the treatment
conditions are manipulated, what is measured as outcomes, and how groups compare.
Experimental research is generally carried out in one of two ways; between – subjects
and within – subjects.
In between - subjects’ designs, the subjects are randomly assigned to one of two or
more groups and they receive different conditions of the experimental variable. At
the end of the experiment, the two groups are tested to determine the effect of the
conditions. This sort of design is often accompanied by a pre test to increase the
validity of the findings.
In a within - subjects design, there is only one group formed and each member of this
group is treated in the same way. There are different ways of carrying out within-
subjects investigations: one common method (time series) is where the group is
subjected to a stream of interventions over a length of time, and these interventions
are interspersed by pre and post tests or behavioural observations. Aligned to this is
the classic repeated measures design in which all the group participate in a set of
different experimental treatments and in this way, the group becomes its’ own control
group. The major problem with this approach is the effect of learning whereby
individuals carrying out two different tasks carry over skills, habits and knowledge
into the second task, thus ‘muddling’ the effects. It is not possible to ‘unlearn’ and so
this effect is commonly limited this by balancing the order of the treatments on
individuals by using a Latin square approach. This is applied in the following way; if
six subjects are each being presented with two different interfaces (A and B), then
three subjects will look at A then B, and the other three will look at B then A.
1.1.3.2 Correlational Research
In correlational design, two or more sets of data are compared to see if they vary
consistently. It is sometimes used to identify which variables might predict certain
results but is more commonly used in HCI research to determine to what extent two
sets of data vary. In this form, correlational research is sometimes referred to as
relational design (Cohen et al., 1994) or explanatory design (Fraenkel and Wallen,
2000). To carry out a correlational study, it is necessary to collect at least two sets of
data for each participant and these are compared using statistical tests to determine the
strength of association.
1.1.3.3 Survey Research
Surveys are used to describe attitudes, opinions, behaviours or characteristics of a
group. They are typically administered in one of two ways, either at a moment in
time over a cross section, or over a length of time with the same population. This
latter method is often used to find changes of opinion or to identify trends. In cross
sectional research, the intention is sometimes to describe current practice or to
evaluate a program or activity in which the participants have been involved. Surveys
have been used in empirical research with both adults and children for many years
with (Bogdan and Biklen, 1998) reporting that surveys were carried out with children
as early as the 1890s.
There are essentially two instruments for survey research; these are the questionnaire
and the interview. In a questionnaire it is the participant that records the data, in an
interview it is the researcher that records the data. Interviews can be carried out in
one to one settings or can be in a group setting, often referred to as a focus group
(Krueger, 1994).
1.1.3.4 Grounded Theory Research
Grounded theory research is a qualitative procedure that is used to generate a theory
to explain a process. It is commonly used to study complex processes, such as how
students learn to write (Neff, 1998), but has also been used to explain actions or
interactions by and with groups of people (Cresswell, 2005), this makes it an
attractive method for HCI research with several authors using grounded theory based
approaches. The key concept in grounded theory is that data is collected to assist in
the development of a theory. It is common to zigzag in and out of data collection and
data analysis, often with the same individuals, and it is the researcher that determines
when the data collection should stop; i.e. when the theory is complete. Data that is
collected is coded into categories and one such category is then selected to be the core
category and relationships between this and other categories are determined. Once a
theory has been developed, it is generally validated, by comparing it with other
sources.
1.1.3.5 Ethnographic Research
‘Ethnographic designs are procedures for describing, analyzing, and interpreting a
culture- sharing group’s shared patterns of behaviour, beliefs, and language that
develop over time’ (Cresswell, 2005), page 436. Ethnography developed from
cultural anthropology where researchers attempted to investigate primitive cultures by
becoming immersed in their way of life (Mead, 1928). There are three approaches;
realist ethnography, where the study is of the culture – sharing group and how it
works (as carried out by Mead), critical ethnography with the emphasis on identifying
the critical issue in a group with some identified problem, and case studies where a
process, an event, or an individual, is studied in depth (Cresswell, 2005). The first
two of these methods are generally confined to sociology and related disciplines, but
the case study can be used in multiple areas, including HCI.
1.1.3.6 Narrative Research
Narrative research focuses on the lives and experiences of individuals and the findings
from narrative research are generally written up as stories. It involves the researcher
in the collecting of stories, the retelling of these stories, collaboration with the
storyteller about what has been recorded and then the production of a narrative report.

1.1.4 Doing Research With Children

There is considerable literature on carrying out effective research with children.
Much of this focuses on practical issues such as the appearance of the research
environment or the clothing of the researcher but there are some interesting debates
about the role of the children in the research process (which can be likened to the role
of the children in the design process as outlined in Chapter 4). (Christensen and
James, 1999) comment that children can be involved in research at four levels:
• Children as Objects
• Children as Subjects
• Children as Social Actors (Participants in Society)
• Children as Participants (in doing research)
Children who are being objects are being acted upon and it is the change in the child
that is being observed or recorded. This is not usual in HCI research but does
sometimes take place when educational technologies are being evaluated. In the
second level of involvement, children as subjects are doing something with
something, this is common in HCI, for instance when a child is interacting with an
interface and the interface is being logged. Children as social actors recognises that
the child will have an effect on a research activity, as he may not be prevented from
changing the experimental set up. In the last category, children as participants
assumes the child is embedded in the research activity in much the same way that
design partners are embedded in design activities (Druin, Bederson, Boltman, Miura
et al., 1999). This level of participation of subjects in research is not widely found in
HCI.
1.1.5 Experimental Validity
The validity of the findings from an experiment can be threatened in a number of
ways (Cook and Campbell, 1979). When an experiment is carried out and an effect is
observed, the effect has to be examined to be sure that it has internal validity. There
are two main sources of error that cause poor internal validity and these are described
below.
Statistical errors or the use of inappropriate statistical procedures can result in an
effect being reported which is not true. Statistical validity is threatened by low
statistical power within an experiment (that being a high chance of false negatives),
this is generally a result of the experimenter or investigator making errors (generally
unintentional) in the gathering or in the analysis of data, these errors include
employing selective analysis of data, unreliable measuring, fishing for results by
carrying out multiple significance test, and by fudging results or applying bias in error
checking of statistical results (Barber, 1976).
Sometimes in an experiment there is an effect but the effect may not have been caused
by the treatment. Any change in the experimental set up constitutes a threat to
internal validity, for instance, the subjects may learn something between two tests
which may impact on the results from the second test (maturation), they may
experience an event that changes how they perceive a second test (history), a pre-test
itself may change how the subjects respond, the researcher may become better at
recording data and may apply different standards to earlier and later instances of a
single test, subjects may drop put of experiments, groups may be poorly selected and
the attributes of the experimenter may impact on the experimental results (Friedman,
1967). Other threats include hypothesis guessing (where the subjects try to guess
what the researcher is trying to find out), and uneven treatment (where the researcher
doesn’t treat each subject in exactly the same way).
Construct validity refers to the possibility that in the design of the experiment, the
operations that are carried out do not adequately represent the normal use. This is
threatened by the use of poorly designed data sets.
The external validity of an experiment is concerned with the extent to which the
results can be generalised to other people, other places, and other time periods (Bracht
and Glass, 1968). This is threatened by the selection of subjects for the experiments,
who may not be representative of the population, by the settings in which the
experiments took place, which may not be typical, and by the tasks that were carried
out, including the timing of the tasks and the diversity of the tasks.

1.2 Research Design

1.2.1 A Possible Approach
As HCI is a multi disciplinary subject, it has inherited research methods from many
different sources. Two of the most influential sources are from design and science.
Scientific research is generally carried out using empirical studies, as described in the
preceding section, but in design disciplines, research is more often based on the
development of prototype products, their evaluation, and their refinement. The
marriage of these two approaches can be difficult to formalise but work by (Mackay
and Fayard, 1997) has brought the two together in a framework that is illustrated in
Figure 2.
 Figure 2 - Integration of Science and Design (Mackay et al., 1997)

In this framework, the researcher is able to complement research findings from theory
and observations whilst developing a prototype application.

1.3 Research Practice

The involvement in the research of young children means that there are several ethical
issues that have to be addressed. These issues are outlined here.

1.3.1 Access and Privacy

To protect the privacy of the children, no information should be asked for that would
identify their religion, sexuality, parental income bracket, or ethnicity. The names of
children should be removed from the data. If children are photographed or videoed
permission must be sought and obtained from children and their parents and the
intended use of the data needs to be made specific.

1.3.2 Being Inclusive

Children should all be given a chance to participate in an experiment. Children with
special needs should normally be included, and if necessary their results discounted..

1.3.3 Informed Consent

Diener et al., (1978) describe informed consent as ‘the procedures in which the
individuals choose whether to participate in an investigation after being informed of
facts that would be likely to influence their decisions.’ This definition identifies four
elements that need to be present for informed consent. These being;
♦ Competence to make decisions
♦ Voluntary participation
♦ Full information about the research activity
♦ Comprehension of the processes.
There is a risk with children that they will not ‘know what they are letting themselves
in for’. In their text on research with children, (Fine and Sandstrom, 1988) observe
that ‘children should be told as much as possible, even if some of them cannot
understand the full explanation. Their age should not diminish their rights.’
One ‘risky research method is the use of ‘Wizard of Oz’ experiments that, by their
nature, involve some deception (Dahlback, Jonsson and Ahrenberg, 1993). This ‘first
order’ deception can be justified in some cases where the results cannot be gained by
any other means and where no harm is done (Aronson, Ellsworth, Carlsmith and
Gonzalez, 1990).

1.3.4 Dissemination
Consider how you can gain support for your work by sharing some of your findings
with the children and the teachers.
Aronson, E., Ellsworth, P. C., Carlsmith, J. M. and Gonzalez, M. H. (1990) Methods
of Research in Social Psychology, Mc Graw-Hill, New York.
Barber, T. X. (1976) Pitfalls in human research, Pergamon, Oxford.
Bogdan, R. C. and Biklen, S. K. (1998) Qualitative research for education: An
introduction to theory and methods, Allyn and Bacon, Boston.
Bracht, G. H. and Glass, G. V. (1968) The external validity of experiments, American
Educational Research Journal, 5, 437 - 474.
Christensen, P. and James, A. (Eds.) (1999) Research with Children: Perspectives
and Practices, RoutledgeFalmer.
Cohen, L. and Manion, L. (1994) Research Methods in Education, Routledge,
London.
Cook, T. D. and Campbell, D. T. (1979) Quasi - experimentation: design and
analysis issues for field settings, Rand McNally, Chicago.
Cresswell, J. W. (2005) Educational Research, Pearson, New Jersey.
Dahlback, N., Jonsson, A. and Ahrenberg, L. (1993) Wizard of Oz Studies - Why and
How, Knowledge based systems, 6, (4) 258 - 266.
Diener, E. and Crandall, R. (1978) Ethics in Social and Behavioural Research,
University of Chicago Press, Chicago.
Druin, A., Bederson, B., Boltman, A., Miura, A., Knotts-Callaghan, D. and Platt, M.
(1999) Children as our technology design partners in The Design of Children's
technology(Ed,Druin, A.) Morgan Kaufmann, San Francisco, CA, pp. 51 - 72.
Fine, G. A. and Sandstrom, K. L. (1988) Knowing Children: Participant Observation
with Minors, Sage Publications, Beverly Hills, CA.
Fraenkel, J. R. and Wallen, N. E. (2000) How to design and evaluate research in
education, McGraw-Hill, Boston.
Friedman, N. (1967) The social nature of psychological research, Basic Books, New
York.
Krueger, R. A. (1994) Focus groups: A practical guide for applied research, Sage,
Thousand Oaks, CA.
Mackay, W. E. and Fayard, A.-L. (1997) HCI, Natural Science and Design: A
Framework for Triangulation Across Disciplines DIS ''97 Amsterdam ACM
Press 223 - 234
Mead, M. (1928) Coming of age in Samoa: A psychological study of primitive youth
for Western civilization, Morrow, New York.
Neff, J. M. (1998) From a distance: Teaching writing on interactive television,
Research in the teaching of English, 33, 146 - 157.
Robson, C. (2002) Real World Research, Blackwell, Oxford.