Research Analysis & Critique

Research Analysis & Critique

ETEC 500 – 66A
University of British Columbia

Submitted by: Kevin Monkman

August 10, 2018

1
Research Analysis & Critique

Introduction

The study I have chosen to analyze and critique is titled “The Effects of Adult Learning on Self-

Efficacy”, and was published in the London Review of Education in 2005, by Cathie Hammond

and Leon Feinstein.

Analysis

Purpose

The purpose of the study is, as Hammond and Feinstein state, to “investigate the links between

participation in adult learning and self-efficacy, particularly for the subgroup of adults who had

low levels of achievement at school” (Hammond and Feinstein, 2005, pg. 265). The researchers

hypothesize that “adults with relatively high levels of self-efficacy are more likely to engage in

learning, and also that the experience of adult learning raises levels of self-efficacy” (Hammond

& Feinstein, 2005, pg. 266). The study focuses on a subgroup of women with what Feinstein

and Hammond classify as “poor school attainment” (Hammond and Feinstein, pg. 266, 2005).

Three corollary research questions addressed are as follows: “Is participation in adult learning

related to transformed or sustained efficacy? Are relationships found for men and women with

poor and good school achievement? What are the causal pathways that underlie any

relationships found?” (Hammond and Feinstein, 2005, pg. 269). The knowledge gap is related

to a lack of research regarding which processes might underlie an association between adult

learning and self-efficacy, and whether participation in learning contributes to transformed self-

efficacy (Hammond and Feinstein, 2005). The study conducts a qualitative and quantitative

analysis, synthesizing findings to address the knowledge gap.

Research Analysis & Critique

Prior Research and Compelling Ideas

A notable prior study was conducted by Schuller et al., 2002, 2004. A compelling idea emerging

from Schuller et al. 2002, 2004 is that adult learners show increased confidence, and even

though confidence differs from self-efficacy, there are many wider benefits that flow from

greater confidence, and several of these could be described as the manifestations of increased

self-efficacy (as cited in Hammond and Feinstein, 2005). Ultimately, the implications are that

increased self-efficacy can benefit adult learners in a wide variety of ways. Another notable

study was conducted by Bandura 1997. An important idea from this study is that self-efficacy

plays important roles in protection from the development of anxiety disorders, depression and

dependency on drugs, as well as in their treatment; in the management of pain, athletic

performance, the enactment and fulfillment of occupational roles, creative productivity and

effective social relationships (as cited in Hammond and Feinstein, 2005).

Constructs

The most significant constructs examined in the study are sustained and transformed self-

efficacy, as well as participation in adult education. Sustained self-efficacy refers to self-efficacy

that stays the same between measurements taken at 33 and then at 43. Transformed self-

efficacy refers to changes in self-efficacy during this period. These self-efficacy constructs are

operationally defined by a binary value of 1 or 0, with 1 representing high self-efficacy and 0

representing low self-efficacy. The researchers create this operational definition (binary

variable) by having cohort members select from three binary positive/negative statement pairs.

Those who chose all three positive statements are assigned a value of 1, indicating high self-

efficacy. Those who chose at least one negative statement were assigned a value of 0. These
Research Analysis & Critique

constructs were further analyzed and operationally defined as an odds ratio in Table 2.

Participation in Adult Education was operationally defined as whether cohort members took

any classes between ages 33 and 43. Another construct in the study is school attainment,

which researchers operationally define through a binary variable based on whether the

member attained an O level by age 16; they are given a value of 1 and are considered to have

had school attainment if they have achieved an O level by 16, and are given a value of 0 if they

haven’t.

Research Type

The study can be defined as theory based, as it draws on past theoretical research regarding

self-efficacy and school attainment, and creates a hypothesis out of this. The opening line of

the study is “we use quantitative and qualitative methods to investigate…” which indicates a

clear mixed methods approach (Hammond and Feinstein, 2005, pg. 265). Researchers use a

quantitative approach to identify an underlying association between dependent and

independent variables, and a qualitative approach to further explore the association on an in-

depth macro level. The study does not use an experimental design, instead focusing on a non-

intervention design that examines a potential causal relationship between school attainment

and self-efficacy. The study is considered causal-comparative as it compares two groups (those

with school attainment and those without) in an important dimension (presence of self-

efficacy). The researchers also examine qualitative and quantitative data from a longitudinal

study conducted previously by the National Child Development Study (NCDS).

Research Analysis & Critique

Variables

The study’s dependent variables are sustained or transformed self-efficacy. The independent

variable in the study is participation in adult learning, represented as a binary variable of

whether cohort member took any taught-courses between the ages of 33 and 42 between 1991

and 2000 (Hammond and Feinstein, 2005). An attribute variable, male versus female, was also

used to determine whether the relationship between participation in adult education may be

different for men and for women (Hammond and Feinstein, 2005).

Research Design

The study’s primary research design is non-intervention causal-comparative. It analyzes data

gathered from a previous longitudinal study, and compares two groups (those with school

attainment and those without) across an important dimension (presence of self-efficacy).

Control Procedures

Researchers attempt to control for social, family, cognitive and biological factors through

several methods (Hammond and Feinstein, 2005). First, “the childhood controls include

measures derived from an instrument devised by Rutter et al. (1970) to measure social and

emotional adjustment based on a parent’s perceptions of the child. Adjustment at school is

assessed using the Bristol Social Adjustment Guide (BSAG)” (Hammond and Feinstein, 2005, pg.

272). Second, researchers “control for socio-economic status based on occupation and the

Registrar-General’s schema, in five categories with manual semi-skilled and unskilled combined

(Hammond and Feinstein, 2005, pg. 272). Third, researchers control for age self-efficacy as
Research Analysis & Critique

by creating dummy variables that describe sixteen groups of individuals depending on these

criteria (Hammond and Feinstein, 2005). Unfortunately, control procedures for the original

longitudinal study by the NCDS are not provided.

Sampling

The sample of research participants varies throughout the study. The quantitative data is

derived from a longitudinal study of surveys conducted between 1958 and 2000, of all UK

children born between March 3rd and March 9th, 1958. Hammond and Feinstein focus on

participants at ages 7, 11, 16, 23, 33 and 42. The first data collected by the NCDS study

featured 17, 415 participants, and the final sweep of data in 1999 featured 11, 419 participants.

For the qualitative portion of the study, the researchers sampled 15 women with poor school

attainment (no O level equivalents at age 16) from the original NCDS study (Hammond and

Feinstein, 2005). These women were all aged 46 when they were interviewed in 2004.

Reliability and Validity

The fact that the binary odds ratios used to measure the constructs were estimates, validity is

questionable. Researchers somewhat address the reliability of the qualitative research by

pointing out that the participant interviewee cross-referenced the interview data with the

quantitative survey data, and found no contradictions in results. Another potential reliability

concern is the fact that for the qualitative portion, there are only three sets of binary questions,

which may not be enough to adequately measure self-efficacy. Though the researchers do not

provide specific reliability indicators, the cross referencing does lend itself to reliability.

Ultimately, however, the researchers do not present clear evidence of either validity or

reliability.
Research Analysis & Critique

Alternative Hypothesis

The researchers suggest the alternative hypothesis that it is possible that life experience

(health, family/social background/socio-economic status, etc.) could explain both a

participant’s self-efficacy measure, as well as his or her school attainment measure. They seek

to eliminate this confounding bias by enacting a series of statistical controls (as detailed in

section 7 above), but ultimately admit that their controls cannot eliminate all potential bias,

and as such does not completely explain away this alternate hypothesis. They also conducted

their analyses for men and women separately because they believed that participation and self-

efficacy would be different for men and women (Hammond and Feinstein, 2005).

Data Analysis and Major Findings

For the quantitative section of the study, Hammond and Feinstein used estimate odds ratios to

analyze the data, and logit regression analysis (to deal with the binary variables). The major

findings for the quantitative portion of the study suggest that those who took courses during

the periods measured were far more likely to experience transformed self-efficacy. However,

there is no evidence to suggest that sustained self-efficacy is more likely for those who took

courses during the periods measured. For the qualitative portion of the study, interviewers

collected data, completed a 2000 word reflection on the responses, and then used analytic and

inductive software NVivo 1.2 to analyze the response data. Several major findings emerged

from this portion of the study. First, an adult’s perception of achievement in adult education

can increase self-efficacy. Second, as adults increase their self-efficacy, there is less resistance

to participation in adult learning. Third, those who find success in adult education may find
Research Analysis & Critique

more challenging jobs and have higher self-efficacy. Finally, challenging backgrounds and life-

circumstances can be barriers to shaping impacts of adult learning on self-efficacy.

Conclusions

Hammond and Feinstein found that adults, particularly women with low school attainment, and

at specific periods of their lives, may engage in adult learning and see increased-self efficacy.

They identify a positive feedback loop in which self-efficacy increases, along with participation

in adult-learning, which in turn increases self-confidence and participation in more challenging

employment opportunities. However, they recognize the complexity of life-processes and as

such do not suggest adult learning as being directly causal of self-efficacy and increased self-

confidence.

Critique

In their study, Hammond and Feinstein seek to investigate potential links between participation

in adult learning and self-efficacy. They believe that adult learning can contribute to self-

increased efficacy, and hypothesize that “adults with relatively high levels of self-efficacy are

more likely to engage in learning, and also that the experience of adult learning raises levels of

self-efficacy” (Hammond & Feinstein, 2005, pg. 266). The study is situated in the context of

previous studies that suggest increased self-efficacy can protect adults from depression and

anxiety, two factors that can also contribute to low school attainment or a lack of participation

in adult learning. As such, the study examines the role of adult education as a potential aid for

the reduction of these processes.

Research Analysis & Critique

One major strength of the study is its methodology. That is, a mixed methods approach allows

the researchers to supplement a quantitative data analysis with an in-depth qualitative study.

Further, the quantitative portion has the benefit of accessing data from a previous large scale

longitudinal national study. Though the quantitative data was collected from a previous study,

the researcher do a commendable job incorporating the data into their study through the

creation of a summary ratio formula, thereby making the data relevant and useful to their

examination. They further complete a logit regression analysis to compensate for the

shortcomings of the estimation ratio. Another strength of the study is in the sample size of the

quantitative study, which collects data from nearly 15,000 participants.

Though Hammond and Feinstein do their best to utilize this data to address their hypothesis,

several weaknesses are apparent in the study. One such weakness is the inability of the

researchers to adequately control for confounding bias and reverse causality. This is a

consequence of the chosen methodology of the study. That is, because the quantitative

portion of the study relies on previously collected data from the NCDS, the researchers are

unable to account for several factors. Most notable, is their inability to access information

about the timing of changes in self-efficacy of the participant cohort. Hammond and Feinstein

note several times in the study that their inability to control for certain factors, mostly biases,

makes them cautious about identifying causal relationships from the quantitative data alone.

Their inability to do so, suggests a potential inability of them to generalize from the findings.

Even though the qualitative portion of the study is beneficial when synthesized with the

qualitative data, it too has its weaknesses. Though the researchers suggest that the subjectivity
Research Analysis & Critique

of both the participants and the interviewers is a potential weakness of the study, this

subjectivity is a hallmark of qualitative research and allows for personal, macro level insights

and should be viewed as a strength. Another potential threat to validity is the fact that the

odds ratios used to measure the constructs of self-efficacy are estimates, and as such construct

validity is questionable. However, Hammond and Feinstein argue that the findings are still

relevant “if interpreted appropriately.” (Hammond and Feinstein, 2005).

Ultimately, though this study does uncover some interesting findings, it does not do enough to

account for biases. Though it does result in findings that are general in nature, it is not easily

generalizable. A study of this nature is dependent on longitudinal data, and although the

researchers access this data via the NCDS, they did not collect the data themselves. As such,

they are not able to control for the necessary variables that allow their findings to go much

further beyond the previous research they reference in their work. Though I find the study’s

major findings particularly interesting given my subjective professional context, I am not

convinced that this study fills the knowledge gap or controls for bias enough to justify its

publication.
Research Analysis & Critique

References

Hammond, C., & Feinstein, L. (2005). The effects of adult learning on self-efficacy. London

Review of Education, 3(3), 265-287. 10.1080/14748460500372754

Bandura, A. (1997) Self-efficacy: the exercise of control (New York, W. H. Freeman & Company).

Schuller, T., Brassett-Grundy, A., Green, A., Hammond, C. & Preston, J. (2002) Learning,

continuity and change in adult life, wider benefits of learning research report no. 3

(London, Centre for Research on the Wider Benefits of Learning).

Schuller, T., Preston, J., Hammond, C., Brassett-Grundy, A. & Bynner, J. (2004) The benefits of

learning: the impacts of formal and informal education on social capital, health and

family life (London, Routledge Farmer).

Research Analysis & Critique