Running head: CRITICAL ANALYSIS ARTICLE 1

Team Project 2 - Critical Analysis:

Designing Authentic Learning Activities to Train Service Teachers About Teaching Online

Team 1: Joshua Eisenberg, Damian Perez, Aaron Tisdale, Andrew McCarty, Lynn Mann

California State University, Monterey Bay

April 12, 2018

IST520: Learning Theories

Dr. Tourtellotte and Dr. Fischer

CRITICAL ANALYSIS ARTICLE 2

Team Project 2 - Critical Analysis Article

The authors sought to examine the use of authentic learning in pre-service teaching

programs as a way to better prepare pre-service teachers for modern classrooms. They posited

that using authentic learning would help better prepare teachers for real-world problem solving,

and because online learning is a growing trend, teachers should be prepared to teach in an online

environment. The “authentic learning” experience in this course focused on having teachers

create online resources for their classroom. The discussion for this study focuses mostly on the

qualitative data. There is some quantitative analysis of survey results (averaged Likert scales)

presented. However, there are no numerical comparisons or identified correlations. Through

incorporating the authentic learning activities as a teaching intervention, the aim was to improve

course design.

The researchers defined the research questions clearly focusing on three main ideas:

• How was Herrington framework used to integrate authentic activities into the design of

an online course for pre-service teachers?

• What were the pre-service teachers' perceptions of the online authentic learning

experiences?

• How did technology facilitate the authentic online learning process?

The researchers expressed a positive bias in the wording of the hypotheses and no

commentary on the correlation of the design of the experiment. There is little explanation of

particular questions, especially considering the questions do not test the efficacy of the program.

Instead they assess user experience/reaction.

CRITICAL ANALYSIS ARTICLE 3

Research Methods

The Procedure

The procedure of the study aligns with “design-based research” and describes design-

based research methodology as one which “involves constant reflection and revision.” The study

is primarily qualitative, as it has a degree of flexibility built in and relies on observations over

controlled, quantitative data.

The format of the study:

• Pre-survey to assess participant’s comfort level with technology

• Fifteen week online class that met three times in person.

• Given two weeks of instruction on a new technology

• Given three weeks to create a product.

• Exiting survey

Sampling

The caliber of participant seemed to be specific for the study and not random.

Participants of the study were a small collection of students who were enrolled in a teacher

education program. Participants were millennials who might be open to online learning and

instruction, and all were familiar with technology. The size of the sample was 48. Two ‘macro

cycles’ of subjects consecutively participated in the study. Between the macro cycles

adjustments were made to the study. It should be noted no control group was established to have

a baseline by which to compare results against.

CRITICAL ANALYSIS ARTICLE 4

Measurement Validity

Self-reporting can be inconsistent. Without baseline or observable and quantifiable

outcomes, emotions and perceptions can make results inconsistent. The pre-survey gauged an

arbitrary scale of comfort level with technology. The post-survey did not revisit or reference

these questions to show a change in perception. The evaluators used student blogs and the

progress of their Weebly websites to gauge change. The percent of change needed to determine if

this was a viable outcome was never stated and the data was anecdotal.

Research Procedures

This was not a controlled experiment, so it seems doubtful that replication could occur.

The instructional materials were not presented and there was no control group. The article

mentioned how the professor that taught the second cycle implemented suggested modifications

of the professor that taught the first cycle. There is absolutely no indication of what those

modifications were. Macro cycles were also mentioned without details. Basic procedures were

described, but an attempted reproduction of this study could result in very different instructional

methods and therefore different results.

Research Results

Statistical techniques

The research questions focused on user experience - not performance. Empirical data

was not the focus of the study and therefore the data received was appropriate for the questions

asked. The authors calculated averages for quantitative questions in the surveys and reported

standard deviations for those questions. It makes sense to calculate averages from Likert scales

and report standard deviations, but since there is no control or baseline to compare those
CRITICAL ANALYSIS ARTICLE 5

numbers to, there is no clear interpretation. Overall, the statistics were very limited, providing

only means and standard deviations, with no comparisons.

Variables of study

Pre-class survey questions differed from post questions, thus changes in perception of

online technology proficiency cannot be determined. Each research question was not addressed

in a manner that could be measurable with standard deviation; the results are addressed in a

purely descriptive way. Only the reactions (as scored on a Likert scale) were quantitative.

Hypotheses

The researchers’ conclusions stated that the subjects had overall positive perceptions of

the authentic learning process, and showed evidence of a comfort level in using online web

building applications. The study suggested that students were highly motivated to engage based

on perceived real-world relevance. This hypothesis comes from qualitative survey responses, but

does not include specifics of those responses and does not compare their motivation to engage in

any other activity. The conclusions show sweeping general statements that the subjects had

overall good perceptions of the authentic learning process.

Discussion of Results

Supportive Data

There is no comparison data that show how this study’s use of authentic learning

provides a quantitative or qualitative improvement over the typical courses. The only “data”

were the expression of a few positive assertions on surveys (which had no baseline, rubric or

control).
CRITICAL ANALYSIS ARTICLE 6

Researchers reasonable findings

The study does not give access to the qualitative results. Lack of access to results and

lack of analysis makes it impossible to support the validity of this study. The researchers state,

“The study primarily relied on self-report data, some objective dimensions…, such as

performance, were not measured and evaluated” (Luo, Murray, Cropton, 2017). Within these

parameters the “findings show that the pre-service teachers perceived the online learning

authentic learning activities to be a good introduction to learning how to structure an online

learning site, a technical skill in which they would consider using in the future.”

Summary

This topic warrants further study. This study raises questions for the academic

community, but presents no actionable data. The findings of this research leave much to be

desired. The small and selective approach of the research population is not representative of the

researchers’ claim: “Teacher training programs have consistently been challenged by their

incompetency in preparing pre-service teachers for the ever-changing reality of modern

classrooms” (Luo, Murray, Cropton, 2017). Pre-service teachers do not have the experience to

effectively analyze the value of Classroom 2.0 technologies in applying authentic learning

experiences. Therefore, the sample size should have included experienced teachers. A control

group should have been present that uses traditional instruction to compare results from user

experiences. It would also be beneficial to have a task to evaluate performance, rather than just

user experience.
CRITICAL ANALYSIS ARTICLE 7

References

Luo, T., Murray, A., & Crompton, H. (2017). Designing Authentic Learning Activities to

Train Pre-Service Teachers About Teaching Online. The International Review of

Research in Open and Distributed Learning,18(7).

doi:10.19173/irrodl.v18i7.3037