Article

Journal of Educational Computing

Enhancing and Research

2019, Vol. 57(2) 360–384
! The Author(s) 2018
Modeling Teachers’ Article reuse guidelines:
sagepub.com/journals-permissions
Design Beliefs DOI: 10.1177/0735633117752453
journals.sagepub.com/home/jec
and Efficacy of
Technological
Pedagogical Content
Knowledge for 21st
Century Quality
Learning

Ching Sing Chai1, Joyce Hwee Ling Koh2, and

Yiong Hwee Teo2

Abstract
This study proposed a new conceptualization of technological pedagogical content
knowledge (TPACK) that focuses exclusively on the intersection of technology, peda-
gogy, and content specifically for selected dimensions of 21st century learning. In
addition, teachers’ design beliefs were investigated with the teachers’ TPACK. Given
the conceptualization, a new instrument was designed and validated. An associated
intervention program to enhance the preservice teachers’ TPACK was designed and
the pre- and post-course surveys were conducted. To unpack the relationships
between teachers’ design beliefs and their TPACK, structural equation models
were constructed and validated. The findings indicate that the instrument possesses
good construct, discriminant and convergence validity, and reliabilities. The interven-
tion enhanced the teachers’ TPACK efficacies and their design beliefs significantly,

1
The Chinese University of Hong Kong, Shatin, Hong Kong
2
National Institute of Education, Nanyang Technological University, Nanyang Walk, Singapore
Corresponding Author:
Ching Sing Chai, Faculty of Education, Department of Curriculum & Instruction, The Chinese University
of Hong Kong, Shatin, N. T., Hong Kong.
Email: CSChai@cuhk.edu.hk
Chai et al. 361

and the structural equation models indicate that the teachers’ design beliefs are
significant predictors of the teachers’ TPACK. The implications of this study suggest
that TPACK may be conceived differently and this may promote new intervention
programs to foster preservice teachers’ TPACK and design beliefs.

Keywords
technological pedagogical content knowledge, design thinking, teachers’ beliefs,
structural equation modeling, quality learning, preservice teachers, scale
development

Introduction
As technology continues to enhance the affordances of the 21st century class-
room, teaching and learning practices that are founded on the epistemological
assumptions and pedagogical research of the last century are continuously being
disrupted (Collins & Halverson, 2010; Geisinger, 2016). The status of teachers
and textbooks, which once were the authoritative sources of knowledge are now
being challenged. Multiple sources of high-quality knowledge, and even peda-
gogically strong delivery of subject matter, are readily available in the Internet.
Many YouTube teaching videos are created by well-respected educators.
In other words, the advantages of traditional schooling can be and is increas-
ingly being replicated and propagated through information and communication
technologies (ICTs). Consequently, classroom pedagogical practices and
teachers’ pedagogical competencies need to be transformed beyond excellent
content delivery. Emerging literature on teacher education is pointing toward
teachers’ ability to create technological, pedagogical, and content knowledge
(TPACK; Koh, Chai, Wong & Hong, 2015; Mishra & Koehler, 2006;
Valtonen et al., 2017) for 21st century classrooms.
As with pedagogical content knowledge (PCK), TPACK has emerged to be a
powerful notion to unpack the knowledge and skills teachers need to design
lessons for 21st century classrooms (Harris, Phillips, Koehler & Rosenberg,
2017). Nonetheless, current research indicates that the emergence of TPACK
has not transformed the state of technology integration in classrooms (Heitink,
Voogt, Fisser, Verplanken, & van Braak, 2017; Pringle, Dawson, & Ritzhaupt,
2015; Tondeur, Aesaert, et al., 2017). More research on developing teachers’
TPACK specifically to promote 21st century competencies through various
pathways is needed (Koehler, Mishra, Kereluik, Shin, & Graham, 2014;
Valtonen et al., 2017). On the other hand, quantitative TPACK research has
focused more on validating the seven-factor model (see Mishra & Koehler, 2006)
and using the validated models to assess teachers’ growth in terms of their effi-
cacy before and after ICT courses (e.g., see Chai & Koh, 2017). While several
studies have been able to validate the seven-factor model quantitatively (Chai,
362 Journal of Educational Computing Research 57(2)

Koh, & Tsai, 2016), recent research has challenged the model with different ways
with which TPACK could be conceived of (e.g., see Jang & Tsai, 2013; Lee &
Tsai, 2010; Yeh, Hsu, Wu, Hwang, & Lin, 2014).
Given the aforementioned background, this study attempts to conceptualize a
new representation of TPACK created specifically for 21st century learning,
which is labeled as TPACK-21st century quality learning (TPACK-21CQL).
TPACK-21CQL deals directly with the intersection among technology, peda-
gogy, and content of the seven-factor model of TPACK without considering the
intermediate factors such as technological pedagogical knowledge (TPK), PCK,
or technological content knowledge (TCK) or the elementary factors such
as technological knowledge, pedagogical knowledge, or content knowledge. In
other words, instead of conceptualizing TPACK as constituted by various sub-
factors, it contributes to current TPACK research by exploring directly the
intersection of TPACK and hypothesizes that the central TPACK factor can
be multidimensional when it addresses selected dimensions of 21st century learn-
ing. TPACK-21CQL may open up new perspectives on how TPACK should be
conceptualized, measured, and fostered. In addition, teachers’ beliefs have been
recognized as an area that needs to be researched in conjunction with teachers’
TPACK as these constructs are closely intertwined in influencing teachers’
instructional decision-making (Tondeur, van Braak, Ertmer, & Ottenbreit-
Leftwich, 2017). In particular, teachers’ design beliefs have been identified as a
multidimensional construct that is associated with teachers’ TPACK, and tea-
chers’ design beliefs may predict the teachers’ TPACK (Chai & Koh, 2017).
Given the new conceptualization of TPACK, this study aims to (a) create and
validate an instrument that represents TPACK-21CQL, (b) investigate how the
preservice teachers’ beliefs and TPACK-21QL efficacies change throughout an
intervention program designed around TPACK-21QL, and (c) examine the
structural equation model (SEM) of design beliefs and TPACK-21QL before
and after the intervention to further understand the effects of the intervention.

Literature Review
The following section first identifies the significant dimensions of 21st century
learning and its relation to TPACK. It then argues for a new representation of
TPACK-21CQL. Subsequently, professional devlopment models for TPACK
are reviewed along with research about relevant aspects of teachers’ beliefs.
The purpose is to identify important design heruistics that could promote
TPACK-21CQL and foster teachers’ beliefs toward 21st century learning.

Dimensions of 21st Century Learning for TPACK-21CQL

Driven by the advances in technologies, multiple frameworks of 21st century
learning has been formulated (see Dede, 2010; Geisinger, 2016; Voogt &
Chai et al. 363

Roblin, 2012). Examples of well-known international frameworks include the

Partnership for 21st century skills (P21CS, 2009), Organisation for Economic
Co-operation and Development (OECD) 21st century skills and competences for
new millennium learners (Ananiadou & Claro, 2009), and the Intel, Microsoft,
and Cisco sponsored assessment and teaching of 21st century skills (Binkley
et al., 2012; see also http://www.atc21s.org/). These international frameworks
are from three different continents, which are America, Europe, and Australia,
respectively. While variation exists among the frameworks, the common under-
lying competencies are learners’ ability to leverage on ICT to communicate and
collaborate with others. The ACT21S classified ICT as tools for working in the
21st century workplace while the P21CS framework further divides the ICT
competencies as information, media, and technological skills. Communication
and collaboration are the desired ways of working for the knowledge society,
which is intertwined with critical and creative thinking (P21CS, 2009).
The OECD framework, however, took the approach of categories of skills
and organized the 21st century competencies as cogntive skills, intrapersonal
skills, interpersonal skills, and technical skills (Ananiadou & Claro, 2009).
Cognitive skills involve solving emerging real-world problems which are com-
plex in nature through different modes of thinking. Intrapersonal skills involve
self-management such that one could adapt and perform self-regulated learning
in a dynamically changing world.
Drawing from the OECD framework and also the meaningful learning frame-
work which has been portrayed as being compatible with the 21st century learn-
ing framework (Howland, Jonassen, & Marra, 2014), this study identifies
authentic learning (AUL), collaborative learning (COL), reflective learning
(RL), and active and cosntructive learning (ACTL) as the four key dimensions
of 21st century quality learning that preservice teachers need to be able to foster
in the classroom, supported by technologies to facilitate content mastery.
AUL emphasizes getting students to resolve real-world complex problems.
COL involves students co-construction of content knowledge through in-depth
negotiation. RL engages students in planning, monitoring, reviewing, and
reflecting their learning processes while ACTL requires students to construct
digital artifacts with appropriate ICT tools as mediators of learning. The frame-
work is contextualized to the current development of Singapore education and
its emphasis on 21st century learning and it is named as 21st century quality
learning (Chee & Chai, 2017).
While teachers are called to transform pedagogical practices to create 21st
century learning in their classroom, it is obvious that specific research into how
to equip teachers for such transformations is needed. Generally, the notion of
21st century learning is discussed conceptually and methods of assessing 21st
century learning has just begun to emerge (Care, Scoular, & Griffin, 2016;
Geisinger, 2016). Voogt, Erstad, Dede, and Mishra’s (2013) review of inter-
national studies indicates that while there is consensus about the major
364 Journal of Educational Computing Research 57(2)

constituents of 21st century skills, the enactment of 21st century learning in

classrooms is lacking, owing to the absence of integration between the 21st
century skills with curriculum and assessment. Kereluik, Mishra, Fahnoe, and
Terry (2013) highlighted the need for teachers to develop TPACK for 21st cen-
tury learning. The TPACK framework articluates the complex forms of know-
ledge and knowing that teachers need to create lessons that promote technology-
based learning. While many TPACK studies are based on the premise of engen-
dering 21st century learning (e.g., Angeli & Valanides, 2009; Mishra & Koehler,
2006), specific efforts that are directed toward promoting specific 21st century
skills may need further research (Valtonen et al., 2017), especially given that the
current state of technology integration generally supports traditional learning
(Pringle et al., 2015; Tondeur, Aesaert, et al., 2017).

Current Conceptualizations of TPACK and Associated

Quantitaive Surveys
Since the inception of TPACK around 2005, its representations have been con-
tested (see Angeli & Valanides, 2009; Koehler et al., 2014). Nonetheless,
the most widely cited framework is the seven-factor model articulated by
Mishra and Koehler (2006). Schmidt et al. (2009) created the first instrument
to measure TPACK based on the seven-factor representation. Subsequently,
many researchers have built on their efforts to test, refine, or create and further
contextualize the TPACK efficacy questionnaire (see Chai et al., 2016; Koehler
et al., 2014). For example, Lee and Tsai (2010) attempted to contextualize
TPACK for web-based learning and they were able to identify some relevant
factors. Earlier research has also questioned whether it is possible to identify the
seven-factor model (e.g., see Archambault & Barnett, 2010). Recent reviews have
provided substantial evidences that it is possible to identify the seven factors
with adequate validity and reliability (Chai et al., 2016; Drummond & Sweeney,
2017; Harris et al., 2017). The seven-factor model has also guided professional
development activities to build teachers’ TPACK, mainly drawing on the tea-
chers’ TK, PCK, and TPK to be synthesized and transformed into ICT-based
lesson design (Chai & Koh, 2017; Koehler, Mishra & Yahya, 2007).
It is obvious that the short academic development history of the seven-factor
model illustrates that conceptual models guide measurement, research, the
design of intervention, and subsequently the development of teachers’ compe-
tencies. Researchers who are sensitive to the implications of the conceptual
model have thus challenged the model from both the theoretical and empirical
fronts (e.g., Archambault & Barnett, 2010; Brantley-Dias & Ertmer, 2013). The
seven-factor model has also been criticized as adopting a more integrative view
rather than a transformative view of TPACK (Angeli & Valanides, 2009).
In other words, there are other possible representations of TPACK that could
guide teachers’ development.
Chai et al. 365

Through Delphi technique, Yeh et al. (2014) presented an eight-factor model

based on experts’ inputs about TPACK from the perspective of educators’ prac-
tical experiences with integrating technology. Examples of the eight factors
include using ICT to assess students and using ICT-integrated strategies for
teaching. Closer examination of the factors reveals that they can be considered
as expanded forms of TPK. Yeh et al.’s (2014) study implies that TPK can be
further broken down to subfactors. However, as the authors argue that their
representation of TPK is solely drawn from practicing science educators, the
eight factors may be taken as representing technological pedagogical science
knowledge.
Recent reviews have also suggested that future TPACK research may move
into measuring specific combinations of technology, pedagogy, and content
areas (e.g., Chai et al., 2016; Jang & Tsai, 2013; Koehler et al., 2014; Lee &
Tsai, 2010). To focus teachers’ development of TPACK specifically for selected
dimensions of 21st century learning, this study proposed and validated a survey
that focused on TPACK-21CLQ as explained earlier.

Models to Facilitate Preservice Teachers’ TPACK Development

Several models based on different contexts of teacher education have been
researched and shown to be effective for raising preservice teachers’ compe-
tencies for creating TPACK in the form of ICT-based lesson designs. Most
notable among them is Angeli and Valanides’s (2009) work on technology
mapping. Technology mapping begins with the identification of difficult
topics that ICT could better represent, followed by the creation of techno-
logical representations of the content that could facilitate student-centric learn-
ing. Appropriate teaching and learning strategies are then chosen to engage
students in learning the content. The effectiveness of technology mapping is
verified by assessing preservice teachers’ lesson design. The study indicates that
it is important to support preservice teachers with an overall instructional
model.
Building on the technology mapping processes, Kramarski and Michalsky
(2010) articulated a set of 16 metacognitive prompts to help preservice teachers
in different phases of designing ICT-integrated lessons. The underlying reason
for supporting the preservice teachers with the metacognitive prompts was based
on the complexities of design tasks where multiple sources of knowledge have to
be activated and transformed to design a coherent and effective lesson.
Kramarski and Michalsky (2010) tested the effectiveness of their model with
TPACK comprehension tests and evaluated the preservice teachers’ lesson
design. Apparently, their study indicates that preservice teachers who were sup-
ported by the prompts outperformed those who did not have the prompts.
An important lesson that can be derived from their study is that the complex
366 Journal of Educational Computing Research 57(2)

and knowledge intensive design processes need to be scaffolded especially for

preservice teachers. One limitation of this model could be that it was imple-
mented with a 56-hour timeframe which many teacher education institutes
may not be able to afford. There may be a need to find ways to support the
preservice teachers in a just-in-time manner.
To date, there are criticisms about the TPACK framework as it is complex
and may not help teachers in their day-to-day practice (Brantley-Dias & Ertmer,
2013; Dobozy & Campbell, 2016). In addition, Angeli, Valanides and
Christodoulou (2016) have advocated that meaningful TPACK could not be
achieved through the integrative approach. These authors have argued for the
transformative approach that requires the teachers to synthesize isolated know-
ledge of technology, pedagogy, and content in a manner that result in new
teaching and learning practices (see Angeli & Valnides, 2009). Thus, to be well
versed in utilizing the framework, extensive help to challenge teachers’ assump-
tion about what constitutes good technology integration and to help them trans-
form isolated knowledge is needed (Koh, Chai, & Lim, 2017). For preservice
teachers who may lack experience, the preservice ICT course design needs to
help them to interweave multiple source of knowledge and tutors facilitation is
essential.
Integrating the aforementioned models, Chai and Koh (2017) created the
Scaffolded TPACK Lesson Design Model (STLDM) as an overall design pro-
cess for ICT integrated lesson. Based on the theory of epistemic framing (Koh,
Chai, Wong, et al., 2015; see also Rosenberg, Hammer, & Phelan, 2006) that
emphasizes helping learners to view the task at hand as knowledge construction,
the STLDM synthesizes important metacognitive prompts from past research as
a means to activate the relevant knowledge of the various subfactors of TPACK
(see Figure 2) to facilitate the preservice teachers’ design. While previous
research indicated that the STLDM (Chai & Koh, 2017) improves the preservice
teachers’ self-efficacy for designing lessons and fosters relevant design beliefs,
more instructional support can be further integrated. These supports could be
instructor modeling of design thinking to avoid early fixation (Razzouk & Shute,
2012), experiential learning among the preservice teachers to foster deep under-
standing, and reflection and collaboration among the preservice teachers
(Heitink et al., 2017; Tondeur et al., 2012).
Past effort in promoting teachers’ TPACK has surfaced some important les-
sons. First, learning design through authentic design experience, that is, learning
by design, is a tried-and-tested approach. Second, framing the design with rele-
vant knowledge and considerations helps preservice teachers to draw upon rele-
vant knowledge needed for good lesson design. Third, multiple forms of
cognitive prompts can support the preservice teachers to consider the myriad
contextual and pedagogical knowledge. The actualization of these learning from
past research is elaborated in our revised STLDM (see Methods section).
Chai et al. 367

The revised STLDM may contribute to promote educators’ effort in enhancing

preservice teachers TPACK.

Teachers’ Design Beliefs

The importance of teachers’ beliefs in shaping teachers’ instructional decision has
been emphasized in the context of engaging teachers to design lesson (Heitink
et al., 2017; Tondeur, van Braak, et al., 2017). Teachers possess many different
forms of beliefs (e.g., beliefs about technology, students, subject matter, etc.) and
these beliefs could be highly individualized and implicit. Nonetheless, these
beliefs shape teachers’ design instruction. In relation to TPACK, empirical
research on teachers’ beliefs and how it is associated with their TPACK has
drawn attention but with relatively few studies being published. Boschman,
McKenny, and Voogt’s (2015) qualitative content analysis indicates that early
literacy teachers’ existing beliefs and attitudes form the basis of their technology
use in classrooms. Quantitative research, however, was largely confined to
teachers’ design beliefs (Chai & Koh, 2017; Koh, Chai, Hong, & Tsai, 2015;
Koh et al., 2017). The teachers’ design beliefs reported in these studies include
their beliefs about new culture of learning, beliefs about themselves as designers,
their design disposition (i.e., their comfort level with the inherently ambiguous
design situation), and their self-efficacy for design thinking. The beliefs
toward new culture of learning assess if teachers are agreeable with emerging
participatory culture of learning that employs ICT as a collaborative, curative,
constructive, and inquiry tool (Thomas & Brown, 2011).
Past research (Chai & Koh, 2017; Koh, Chai, Hong, et al., 2015) indicates
that the teachers’ design beliefs predict their TPACK when TPACK is treated as
a single factor where all other TPACK subfactors intersect. In addition, tea-
chers’ design thinking efficacy may mediate other form of teacher design beliefs
such as design disposition. This study hypothesizes that the predictive relation-
ship should be generally supported as a second-order SEM was supported in the
previous study (see Chai & Koh, 2017). As the conceptualization of TPACK
evolves, it seems necessary for research in teachers’ beliefs to coevolve given the
importance of teachers’ beliefs in influencing instructional decisions. In particu-
lar, research on how the SEM may change before and after the preservice
teachers have been engaged in creating TPACK is apparently still lacking.

Methods
Based on the foregoing arguments that a new conceptualization of TPACK-
21CQL could open up new ways to measure and foster preservice teachers’
competencies and beliefs about 21st century learning, and it may shape the
relationships between the teachers’ design beliefs and their TPACK-21CQL,
the following research questions were formulated to guide this study.
368 Journal of Educational Computing Research 57(2)

Research Questions
1. Does the TPACK-21CQL with teachers’ design beliefs questionnaire possess
adequate psychometric properties to be regarded as a reliable and valid
instrument?
2. Does the revised STLDM enhance Singapore preservice teachers’ TPACK-
21CQL efficacies and their design beliefs significantly?
3. Can the hypotheses of teachers’ design beliefs predicting TPACK-21CQL
efficacies be supported in the SEMs for both the pre- and postcourse data?

Participants
A total of 564 preservice teachers who enrolled for a 1-year teacher certification
program participated in this study, accounting for 73.2% of the overall preser-
vice teachers enrolled for the year of 2016. These preservice teachers were train-
ing for a wide range of subjects including languages, science, mathematics,
humanities, and social studies. Of them, 295 (52.3%) are from primary school
teacher training while the rest are from secondary or high school teacher train-
ing. There are 403 female teachers (71.5%). The mean age for the teachers is 26.2
years (SD ¼ 5.71). Participation in the survey is voluntary with informed consent
as per the procedures approved by the university’s institutional review board.
As a result of the emphasis placed on ICT in education in Singapore that
begins in elementary school, the participants are generally well versed with basic
technological skills which include the abilities to use Microsoft Office packages,
surfing the Internet, and using Web 2.0 applications to collaborate. These soft-
ware were used during the course for a variety of learning activities and their
assignments. In addition, it should be noted that all participants passed the
course which required them to design technology-enhanced lessons that meets
the course requirements.

Instrument
The TPACK-21CQL survey is a newly constructed TPACK questionnaire with
five items each for the four dimensions of 21st century (AUL, COL, RL, and
ACTL) mentioned earlier. The initial pool of items was generated by the authors
based on our understanding of TPACK and 21st century learning. The items
were subjected to three university professors’ review to establish face validity.
The items for teachers’ design beliefs were adopted from previous research (Chai
& Koh, 2017), which has been established as valid and reliable. The factors
include beliefs about new culture of learning (BNCL, seven items), teachers as
designer (TAD; five items), design disposition (DD, six items), and efficacy in
design thinking (DT, five items). All items are scored on a 7-point Likert scale
(1 ¼ strongly disagree to 7 ¼ strongly agree).
Chai et al. 369

Data Collection
There were two points of data collection. The precourse survey was conducted
during the first lesson before any teaching commenced. The postcourse survey
was conducted 12 weeks later during the last session of the course. All data were
collected through an online survey and the preservice teachers spent around
15 minutes completing the survey.

Data Analysis
After the responses were matched for the pre and postsurveys, the precourse
data were analyzed to establish construct validity through exploratory factor
analysis using principal component analysis and the alpha reliabilities were
computed. This was followed by further analysis of average variance extracted
(AVE) and construct reliability to support convergent validity. The correlations
between the factors were also computed along discriminant validity obtained
through the square root of AVEs. Subsequently, confirmatory factor analysis
was performed on the postcourse data to provide further evidence about the
construct validity. To examine the effects of the course on the preservice tea-
chers’ efficacy of designing 21CQL and the changes in their design beliefs, paired
sample t-tests and analysis of Cohen’s d were conducted. The SEM with hypoth-
eses based on previous research (Chai & Koh, 2017; Koh, Chai, Hong, et al.,
2015) were constructed (see Figure 1) and tested.

Figure 1. Structural equation model of teachers’ design beliefs predicting 21CQL.

370 Journal of Educational Computing Research 57(2)

Description of the Intervention

The mandatory course, ‘‘ICT for meaningful learning,’’ is a 24-hour course
conducted over 12 weeks for all preservice teachers in Singapore. Several
design principles undergird the course. First, learning by design through
actual designing. To equip the preservice teachers with the necessary TPACK
design skills for TPACK-21CQL, there were two design assignments, supported
by a lesson design template. The first design assignment requires the preservice
teachers to choose a topic pertaining to a cyberwellness issue (e.g., cyber bully-
ing) and design a lesson to facilitate students in learning how to be safe and
responsible users of ICT after they have been taught the basic skills of designing
lessons with the support of the revised STLDM (see the following section). The
skills taught include how to formulate lesson objectives, design ICT-based lesson
activities, prepare learning resources, and evaluate students’ understanding. The
preservice teachers worked in groups of four to five, drawing on collaborative
design to reduce the preservice teachers’ cognitive load of learning the design
process. The second assigment is an individual assignment for a topic based on
the subject they were training to teach. This assignment was intended to further
strengthen the preservice teachers’ design skills. In both assignments, the pre-
service teachers were supported by coaching from the tutors through consult-
ation within and outside class sessions.
Second, framing lesson design with relevant technological, pedagogical, and
content knowledge through cognitive prompts. The revised STLDM as depicted
in Figure 2 was provided to guide the preservice teachers in creating ICT-
integrated lessons for 21CQL. As shown in Figure 2, multiple cognitive prompts
denoting relevant TPACK domains were formulated and spread across the two
stages of design. In Stage 1, the cognitive prompts help the preservice teachers to
activate relevant knowledge resources and gather information as the basis of
diagnosing and deciding appropriate learning objectives for the lesson. In Stage
2, the cognitive prompts push the preservice teachers to consider good practices
for using specifics of technologies and to create lesson activities that incorporate
21CQL. The revised STLDM has incorporated Kirschner’s (2015) suggestions
on building the teaching professionals which focuses on teachers as designer.
Third, building preservice teachers’ TPACK through reflection of learning
experiences. To build the preservice teachers’ TPACK, multiple forms of tech-
nology-supported lessons were introduced weekly. For instance, to build the
preservice teachers’ knowledge about 21CQL, the preservice teachers were pro-
vided with website resources and a set of Powerpoint or Google slides with
guiding questions. Preservice teachers were tasked to co-construct knowledge
for one dimension of 21CQL and subsequently perform peer teaching. After the
peer teaching, the tutors invited the preservice teachers to consider the strengths
and weaknesses of peer teaching as they have experienced, the possible technol-
ogies, resources, grouping instruction, and so forth, needed to support peer
Chai et al. 371

Figure 2. The revised scaffolded TPACK lesson design model (R-STLDM).

teaching; possible problems and solutions with the pedagogical approach and
what is needed to improve it. In other words, reflecting on the learning experi-
ences created by the teacher, educators’ design of TPACK with the aims of
improving the lesson was one of the means to help the preservice teachers to
unpack lesson design.
Fourth, tutor modeling of TPACK co-creation. In many sessions, the pre-
service teachers were tasked to tinker with ICT tools for the content areas, learn
the technical skills, and build knowledge on how to use the tool meaningfully for
relevant content-based topics. The tutors modeled how they investigate the
372 Journal of Educational Computing Research 57(2)

affordances and pedagogical considerations needed to use the ICT tools in the
classroom in the first few lessons. The tutors’ way of pedagogical sense making
was gradually transferred to the preservice teachers to build their ability to make
sense of emerging ICT tools. For instance, the tutors may task the preservice
teachers to build a ICT-based concept map using Cmap (see https://cmap.ihmc.
us/) for a given topic relevant to their teaching needs. This is followed by a
knowledge construction question ‘‘How can an ICT-based concept map
be used to facilitate 21st century quality learning in your subject matter?’’ The
preservice teachers contribute ideas about suitable topics and tasks that the tool
can be used for, the pedagogical provision such as providing guiding questions
and linking words between the concepts to help students, video resources show-
ing advanced features of the tool, possible problems and solutions when using
the tool, and how collaboration and reflection should be structured.

Findings
Research Question 1 assesses the psychometric properties of the survey to estab-
lish its validity and reliability. The exploratory factor analysis yielded eight
factors with Eigen value above 1 as it was designed to measure. The total vari-
ance explained was 73.0%. Nine items with factor loading lower than .50 were
removed. The overall alpha reliability of the remaining items was .94. Table 1
provides further information about means, standard deviations, factor loadings,
alpha reliabilities, AVE, and CR.
In addition to the psychometric information provided in Table 1, Table 2
provides evidence for the discriminant validity. In addition, confirmatory factor
analysis (CFA) of the postcourse survey indicates good model fit with a range of
indices (chi-square ¼ 1192.26, degree of freedom ¼ 532, 2/df ¼ 2.24, Root Mean
Square Error of Approximation (RMSEA) ¼ 0.047, Goodness of Fit Index
(GFI) ¼ 0.89, Confirmatory Fit Index (CFI) ¼ 0.96; see Hair, Black, Babin, &
Anderson, 2010). Other fit indexes include Normed Fit Index (NFI) ¼ 0.93;
Non-Normed Fit Index (NNFI) ¼ 0.96; Incremental fit index (IFI) ¼ 0.96,
Root Mean Square Residual (RMSR) ¼ 0.025, and Adjusted Goodness of Fit
Index (AGFI) ¼ 0.87. Overall, there are sufficient evidences to support the con-
struct, convergent, and divergent validity and the reliability of the TPACK-
21CQL with teachers’ design beliefs survey.
Research Question 2 is concerned with the effects of the course design sup-
ported by the revised STLDM on the preservice teachers’ TPACK21CQL effi-
cacies and their design beliefs. Table 3 shows that other than the factor teachers
as designer (TAD), all factors measured were significantly enhanced pre- and
postcourse. Analyses of the effect sizes indicate that, in general, the course effects
on TPACK-21CQL dimensions (RL, AUL, and COL) and the teachers’ DT
were large (d > 0.8; see Table 3). The effects on ACTL are medium while the
Chai et al. 373

Table 1. Exploratory Factor Analysis With Descriptive Statistics, Factor Loadings, Alpha
Reliabilities (), Average Variance Extracted (AVE), and Composite Reliabilities (CR),
N ¼ 564.

Factor
No Items loadings

Reflective Learning with ICT (RL); Mean ¼ 4.73, SD ¼ 1.16,  ¼ .93, AVE ¼ 0.57, CR ¼ 0.87
RL2 I can use technology-based tools (e.g., online KWL, concept maps) .81
to prompt students in monitoring their progress in under-
standing the content
RL4 I can create self-directed learning activities of the content know- .78
ledge with appropriate ICT tools (e.g., Webquest, Flipped
lessons)
RL3 I can guide students in diagnosing their knowledge gaps with .75
various forms of online feedback systems (quizzes, analytics
etc.).
RL5 I can help students to review the strength and weaknesses of their .75
ICT-supported learning for the subject I teach.
RL1 I can use technology (e.g., blogs. e-portfolio) to support students’ .69
self-reflection about their learning strategies for the subject
matter.
Authentic Learning with ICT (AUL), Mean ¼ 4.61, SD ¼ 1.05,  ¼ .88, AVE ¼ 0.49, CR ¼ 0.82
AUL2 I can engage students in learning the subject matter using the ICT .73
tools that subject matter experts use
AUL4 I can use technologies to scaffold students’ in solving complex .73
problems arising from the topics that I teach.
AUL5 I can arouse students’ interest in solving real-world problems .73
using subject related software.
AUL1 I am able to create real-world problem scenarios for my teaching .66
subject using online web site creators.
AUL3 I am competent in searching for online video resources to initiate .63
real-world problem solving related to the subject matter.
Collaborative Learning with ICT (COL) Mean ¼ 4.49, SD ¼ 1.02,  ¼ .91,
AVE ¼ 0.53, CR ¼ 0.82
COL4 I am competent in prompting students to talk deeply about the .73
content knowledge in online platforms.
COL1 I can formulate in-depth discussion topics about the content .73
knowledge for students’ online discussion.
COL3 I can engage students in substantial peer critiquing work through .73
collaborative software.
COL2 I can facilitate students’ co-construction of subject matter repre- .72
sentations when they are working in small groups around a
computer.
(continued)
374 Journal of Educational Computing Research 57(2)

Table 1. Continued

Factor
No Items loadings

Active Constructive Learning with ICT (ACTL) Mean ¼ 4.92, SD ¼ 0.99,  ¼ .84,
AVE ¼ 0.56, CR ¼ 0.79
ACTL1 I am able to use technology to stimulate my students’ higher order .81
thinking about the subject matter.
ACTL2 I can engage students in constructing deep understanding about .80
the subject matter with various forms of technology (e.g.,
Google site, concept maps etc.)
ACTL3 I am competent in helping my students to critically synthesize .63
information from various web-based resources for content
learning.
Beliefs of New Culture of Learning (BNCL) Mean ¼ 5.80, SD ¼ 0.76,  ¼ .88,
AVE ¼ 0.59, CR ¼ 0.88
BNCL5 Today’s learners should be able to remix relevant resources to .81
publish their ideas.
BNCL3 Remeshing digital resources responsibly is a good way to learn. .79
BNCL1 It is important nowadays for students to be able to participate in .76
online learning communities.
BNCL7 Managing personal online learning resources is a desirable skill. .74
BNCL4 Producing creative digital works is a meaningful task. .74
Design Disposition (DD), Mean ¼ 5.35, SD ¼ 0.85,  ¼ .84, AVE ¼ 0.54, CR ¼ 0.86
DD4 I am comfortable to deviate from established practices. .780
DD5 I am comfortable with occasional failures from trying out new .78
approaches for teaching.
DD3 I am comfortable to explore conflicting ideas .76
DD6 I am constantly seeking to turn constraints into opportunities. .70
DD1 I am comfortable with the presence of uncertainty. .66
Design Thinking Efficacy (DT), Mean ¼ 4.27, SD ¼ 1.12,  ¼ .91, AVE ¼ 0.57, CR ¼ 0.84
DT3 I am competent in detecting possible implementation difficulties in .78
the ICT-based lesson plan.
DT2 I am able to choose the most feasible ICT-based lesson design for .78
my students’ learning.
DT1 I can easily generate a few ICT-based lesson ideas. .74
DT4 I am confident that my design will optimize my students’ learning. .73
Teachers as designer (TAD), Mean ¼ 6.08, SD ¼ 0.72,  ¼ .91, AVE ¼ 0.64, CR ¼ 0.87
TAD3 Teachers should be empowered to design lessons .85
TAD2 My students’ learning experience is an outcome of my lesson .83
design.
TAD1 It is my responsibility to master the skills of designing lessons. .80
TAD4 Working like designer is part of the teacher’s duty. .70
Chai et al. 375

Table 2. Correlations and Discriminant Validity of the Factors Based on Precourse Survey.

Components RL AUL COL ACTL BNCL DD DT TAD

RL (0.75)
AUL .65 (.70)
COL .68 .64 (.73)
ACTL .56 .58 .56 (.75)
BNCL .30 .32 .30 .31 (.77)
DD .32 .31 .38 .34 .48 (.73)
DT .66 .63 .63 .51 .28 .32 (.75)
TAD .21 .18 .19 .23 .49 .44 .20 (.80)
Note. All correlations are significant at .01, figures in parenthesis are the square root of AVE.

Table 3. Paired Sample t-Test and Effect Sizes (N ¼ 564).

Pre-study survey Post-study survey

Measured
factors M SD M SD t-test Cohen’s d

BNCL 5.80 0.76 5.89 0.74 2.42* 0.10

DT 4.27 1.12 5.61 0.75 28.61*** 1.20
DD 5.35 0.85 5.42 0.80 5.37*** 0.23
TAD 6.08 0.72 6.12 0.71 1.22 0.05
ACTL 4.92 0.99 5.61 0.73 15.02*** 0.63
COL 4.49 1.02 5.54 0.75 24.02*** 1.01
AUL 4.61 1.05 5.66 0.74 23.40*** 0.98
RL 4.73 1.16 5.74 0.75 20.44*** 0.86
*p < 0.05, ***p < .001.

effects on two of the teachers’ design beliefs (DD and BNCL) were small.
Overall, the course effects should be regarded as positive though some improve-
ment may be needed.
The SEM tested 19 hypotheses with regard to whether the teachers’ design
beliefs could predict directly or indirectly their TPACK-21CQL. Table 4 reports
the findings for both the pre- and postcourse surveys. Figure 3 depicts the sup-
ported hypotheses graphically with standardized estimates. The SEM fit indices
show that both models based on pre- and postcourse data can be accepted
(precourse: chi-square ¼ 1447.80, degree of freedom ¼ 538, 2/df ¼ 2.69,
RMSEA ¼ 0.055, TLI ¼ 0.92, CFI ¼ 0.93; postcourse: chi-square ¼ 1335.72,
376 Journal of Educational Computing Research 57(2)

Table 4. Supported or Unsupported Hypotheses of Structural Equation Models.

Standardized Standard Critical Supported

estimate error ratio Yes/No

Hypotheses Pre Post Pre Post Pre Post Pre Post

H1 BNCL!DT 0.29 0.19 0.09 0.05 3.12** 3.50*** Yes Yes

H2 BNCL!RL 0.13 0.14 0.06 0.04 2.23* 3.41*** Yes Yes
H3 BNCL!AUL 0.24 0.33 0.07 0.04 3.49*** 7.58*** Yes Yes
H4 BNCL!COL 0.10 0.16 0.06 0.04 1.83 3.89*** No Yes
H5 BNCL!ACTL 0.21 0.30 0.07 0.05 3.05** 6.52*** Yes Yes
H6 DD!DT 0.43 0.37 0.08 0.04 5.43*** 8.35*** Yes Yes
H7 DD!RL 0.05 0.10 0.05 0.03 1.04 2.95** No Yes
H8 DD!AUL 0.09 0.09 0.06 0.03 1.46 2.67** No Yes
H9 DD!COL 0.20 0.22 0.05 0.04 4.02 6.11*** No Yes
H10 DD!ACTL 0.10 0.00 0.06 0.04 1.66 0.05 No No
H11 TAD!DT 0.10 0.29 0.10 0.05 1.04 5.62*** No Yes
H12 TAD!RL 0.03 0.21 0.06 0.04 0.45 5.44*** No Yes
H13 TAD!AUL 0.11 0.03 0.07 0.04 1.55 0.77 No No
H14 TAD!COL 0.12 0.06 0.06 0.04 2.05* 1.45 No No
H15 TAD!ACTL 0.01 0.04 0.07 0.04 0.08 1.01 No No
H16 DT!RL 0.63 0.57 0.04 0.04 15.44*** 13.31*** Yes Yes
H17 DT!AUL 0.69 0.53 0.05 0.05 14.34*** 11.74*** Yes Yes
H18 DT!COL 0.52 0.57 0.04 0.04 14.31*** 13.03*** Yes Yes
H19 DT!ACTL 0.43 0.50 0.04 0.05 11.16*** 10.88*** Yes Yes
*p < .05. **p < .01. ***p < .001.

Figure 3. SEM of teachers’ design beliefs predicting 21CQL for pre- and post-course
surveys.
Chai et al. 377

degree of freedom ¼ 538, 2/df ¼ 2.48, RMSEA ¼ 0.051, TLI ¼ 0.95, CFI ¼ 0.95;
see Hair et al., 2010). For the precourse survey, 9 out of 19 hypotheses were
supported. For the postcourse survey, 15 out of 19 hypotheses were supported.
The comparison reveals that before the intervention, only BNCL and DT were
significant predictors of the factors of TPACK-21CQL. DD only has indirect
effect through DT on the TPACK-21CQL factors while TAD has no effect.
After the intervention, the teachers’ design beliefs were more able to predict
the preservice teachers’ TPACK-21CQL either directly or indirectly through
DT. The notable changes in predictive relationship are for the factor of DD
follow by TAD. This indicates that the course may have increased the strength
of the teachers’ design beliefs as positive predictors. Overall, BNCL and DT
have emerged as important positive predictors of the teachers’ TPACK-21CQL
for both pre- and postcourse data.

Discussion
This section is organized according to the findings. First, this study proposed a
new conceptualization of TPACK-21CQL, created a new survey to operation-
alize the concept, and provided adequate evidences that the new survey is valid
and reliable. It may contribute to research in 21st century learning, specifically in
the area of teacher education for the 21st century. Measurement for 21st century
learning is in its nascent stage (Care et al., 2016; Geisinger, 2016). The survey can
enrich the field with more instruments to assess teachers’ TPACK, especially for
21st century learning. Valtonen et al. (2017) attempted to create an instrument to
measure 21st century TPACK. However, their instrument adopted the seven-
factor model and it did not deal directly with the intersection of technology,
pedagogy, and content, that is, they did not have a factor that represents the
intersection of technology, pedagogy, and content. They have also suggested that
TPACK for 21st century learning needs to consider reflective learning. Tondeur,
Aesaert, et al. (2017) also recognize the needs to create new instruments for 21st
century learning with ICT but their effort was focused on ICT competencies.
Building on past research that propose alternatives of the seven-factor TPACK
model (e.g., Jang & Tsai, 2013; Yeh et al., 2014), this study factorized the inter-
section of technology, pedagogy, and content with dimensions of 21st century.
The essence of TPACK lies in teachers’ pedagogical reasoning of the intersection
among TK, PK, and CK (Mishra & Koehler, 2006). This instrument focuses on
the intersection and thus is a potentially useful instrument for teacher educators
who are interested in promoting teachers’ 21st century TPACK. We believe that
the four factors could be used independently for teacher education programs
which may focus on one or more dimensions of 21st century learning.
Nonetheless, our current instrument did not include other important 21st cen-
tury skills such as critical and creative thinking. Future research may consider
expanding the instrument to include other subscales.
378 Journal of Educational Computing Research 57(2)

The second research question dealt with the effects of the course design based
on TPACK-21CQL that we have implemented. Building on the lessons learnt
from past research (Angeli & Valanides, 2009; Kramarski & Michalsky, 2010;
Mishra & Koehler, 2006), the course design was undergirded by learning by
design and supported through cognitive prompts, reflective experiential learning,
tutor coaching and modeling, and collaborative design. The paired-sample t-test
indicates that the preservice teachers’ gain in self-efficacies for design thinking
and TPACK-21CQL are significant with good effect sizes. This study thus pro-
vides evidence that fostering preservice teachers’ TPACK may include the design
principles mentioned. The intervention may contribute to Kirschner’s (2015)
vision of professional teachers who are adept at designing instruction.
Kirschner has pointed out that studies about teacher as designer tends to
adopt the case study methodology which involves small samples. This research
is a large-scale study that promotes teachers as designer with possible design
principles for teacher educators to consider. A limitation of this study is that the
evidence is confined to teachers’ self-report. Future study needs to include more
evidence such as scoring the lesson plans with rubrics (see e.g., Harris,
Grandgenett, & Hofer, 2010). In addition, the effect sizes of the change in the
teachers’ design beliefs are mostly small. Further effort in fostering the teachers’
design beliefs is needed.
The findings with regard to teachers’ design beliefs in this study indicate that
preservice teachers’ beliefs toward TAD and BNCL were high even before the
course commenced (TAD ¼ 6.08, BNCL ¼ 5.80). Preservice teachers in this
study were generally in agreement that teachers should be designers and teachers
should design lesson that promote new culture of learning. Nonetheless,
these teachers did not possess strong beliefs in dealing with design as indicated
by the precourse mean scores for DT (M ¼ 4.27) and DD (M ¼ 5.35). In general,
these teachers have positive beliefs about design that could facilitate their learn-
ing in the course. The precourse findings also show that all factors of the
teachers’ design beliefs are significantly related to their TPACK-21CQL (see
Table 2).
The pre- and postcourse SEMs indicate that before the intervention, only
BNCL and DT were significant predictors for the TPACK-21CQL. After the
course, other factors of teachers’ design beliefs (DD and TAD) also gained
importance as predictors. In particular, closer direct relationships between
design dispositions and TPACK-21CQL were established. The teachers’ design
disposition indicates their comfort level to deal with ambiguous design situ-
ations. The results seem to imply that while the changes in design disposition
were significant though small, the intervention has improved their capacity to
deal with design situations sufficiently such that they have become more adept
and comfortable with the demands of creating TPACK-21CQL. We would
argue that it is important to raise teachers’ comfort level in dealing with the
emerging volatile and unpredictable technological pedagogical contexts of
Chai et al. 379

today’s classroom and equip them with the disposition to rise above the chal-
lenges through design thinking.
There are a number of hypotheses that were not supported during the pre-
course survey. The unsupported hypotheses were mainly from DD and TAD
(see Figure 3). While factors can be established and are positively correlated to
all other factors surveyed, they are not predictors of the teachers DT or the
factors of TPACK-21CQL when the preservice teachers are untrained. This may
indicate that the SEM may not be representative of how DD and TAD are
structurally associated to the mediating factor DT and the subsequent factors.
Both factors are relatively new in research and hypothesized in accordance to
past research (Koh, Chai, Hong, et al., 2015). An alternative model could be that
they predict BNCL, which in turn predict DT and subsequent factors. This is
however beyond the scope of this study and could be tested in future research.
The SEM based on the postcourse data also did not support several hypoth-
eses, namely DD as predictor for ACTL and TAD as predictor to AUL, COL,
and ACTL. Further correlation analysis indicates stronger correlations for the
factors concerned after the course (generally > 0.4, see Table 2). Nonetheless, it
seems clear that TAD effects are mediated mostly through DT. Again, this
may indicate that there may be better models beyond the one examined in this
study.
In summary, this study provides further evidences that support past research
about the importance of teachers’ beliefs in influencing teachers’ lesson designs
(Heitink et al., 2017; Tondeur, van Braak, et al., 2017). The SEM we obtained
for the postcourse survey is congruent with past research (Chai & Koh, 2017;
Koh, Chai, Hong, et al., 2015). These findings imply that teacher educators may
need to attend to teachers’ design beliefs. Together with the paired sample t-tests
on the teachers’ design beliefs, it seems that more effort in fostering the teachers’
design beliefs could be important. Previous research by Tee and Lee (2011)
indicates that the experience of designing TPACK shifted the teachers’ attribu-
tion of problems with ICT integration from external environment factor to
personal design ability. Thus, one possible way forward could be to structure
reflective questions for the preservice teachers to explicitly discuss their beliefs
about design and how their beliefs are connected to their design acts.
One limitation to our study with regard to teachers’ design beliefs and their
TPACK is that our sample comprised only preservice teachers. Practicing tea-
chers may have very different views about the beliefs we have measured due to
their embodied understanding of the teaching profession. It would be interesting
and valuable to replicate the study with practicing teachers and to compare how
preservice and practicing teachers differ in terms of their beliefs and their
responses toward the intervention. Practicing teachers may be less positive
about the conception of ‘‘teachers as designer’’ and their beliefs toward new
culture of learning. In addition, the revised STLDM may not have large effects
380 Journal of Educational Computing Research 57(2)

on their TPACK-21CQL as it may be oversimplified to capture the nuances in

teachers’ actual design considerations. Another limitation is that we did not test
the effects of the intervention for the different subject areas that the preservice
teachers were training to teach. As content specializations may pose different
challenges to the implementation of 21st century learning, further research is
definitely needed. Finally, as this research is quantitative in nature, future
research may consider interviewing and observing the preservice teachers to
provide richer and deeper understanding of the preservice teachers’ TPACK.

Implications and Conclusion

There are several rise-above implications that this study points to. First, to
promote 21st century learning which is most likely undergirded by technol-
ogy-enhanced learning, educators may need to create and validate specific
instruments to investigate the relations among the psychologically and peda-
gogically relevant factors and the effectiveness of instruction. Second, designing
comprehensive courses to address the challenges of 21st century technology-
enhanced learning requires teacher educators to address teachers’ design beliefs.
This is likely to require collective and coordinated efforts among teacher edu-
cators, especially for those who are specializing in different subject areas. It
seems that sustained effort in learning by design and reflecting the assumptions
of design is equally important for teacher educators and teachers. As pointed out
by Angeli et al. (2016), there are still many gaps that need attention for teacher
and teacher educators to create transformative TPACK. To build the knowledge
base needed to transform current educational practices and associated beliefs, it
is unlikely that the process can happen in isolation. It is more likely that beliefs
and practices are dialectically intertwined and thus need to be addressed
concurrently.
Raising preservice teachers’ competencies for the 21st century classroom is an
important mission that teacher education institutes need to address. This study
documents our effort in furthering the collective endeavor of teacher educators
in transforming today’s classroom. We have attempted to change how TPACK
can be theorized to influence the design of our ICT course into one that fosters
preservice teachers’ design thinking. Nonetheless, design is open-ended, context-
ually bounded, and subjected to dynamic challenges. More research on how
TPACK can be conceptualized and enacted is needed to enrich this nascent
field of research.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research,
authorship, and/or publication of this article.
Chai et al. 381

Funding
The authors received no financial support for the research, authorship, and/or publication
of this article.

References
Ananiadou, K., & Claro, M. (2009). 21st century skills and competences for new millen-
nium learners in OECD countries (OECD Education Working Papers, No. 41). OECD
Publishing. Retrieved from http://www.oecd.org/officialdocuments/publicdisplaydo-
cumentpdf/?cote¼edu/wkp(2009)20&doclanguage¼en
Angeli, C., & Valanides, N. (2009). Epistemological and methodological issues for the
conceptualization, development, and assessment of ICT-TPCK: Advances in techno-
logical pedagogical content knowledge (TPCK). Computers & Education, 52(1),
154–168.
Angeli, C., Valanides, N., & Christodoulou, A. (2016). Theoretical considerations of
technological, pedagogical content knowledge. In M. Herring, M. Koehler &
P. Mishra (Eds.), TPACK handbook V2.0: TPACK research and approaches
(pp. 11–30). New York, NY: Routledge.
Archambault, L. M., & Barnett, J. H. (2010). Revisiting technological pedagogical con-
tent knowledge: Exploring the TPACK framework. Computers & Education, 55(4),
1656–1662. doi:10.1016/j.compedu.2010.07.009
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble,
M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw & E. Care
(Eds.), Assessment and teaching of 21st century skills. London, NY: Springer.
Boschman, F., McKenney, S., & Voogt, J. (2015). Exploring teachers’ use of TPACK in
design talk: The collaborative design of technology-rich early literacy activities.
Computers & education, 82, 250–262.
Brantley-Dias, L., & Ertmer, P. A. (2013). Goldilocks and TPACK: Is the construct ‘Just
right?’ Journal of Research on Technology in Education, 46(2), 103–128.
Care, E., Scoular, C., & Griffin, P. (2016). Assessment of collaborative problem solving in
education environments. Applied Measurement in Education, 29(4), 250–264.
Chai, C. S., Koh, J. H. L., & Tsai, C.-C. (2016). A review of quantitative measures of
technological pedagogical content knowledge (TPACK). In H. Mary, K. Matthew &
M. Punya (Eds.), TPACK handbook V2.0: TPACK research and approaches
(pp. 87–106). New York, NY: Routledge.
Chai, C. S., & Koh, J. H. L. (2017). Changing teachers’ TPACK and design beliefs
through the Scaffolded TPACK lesson design model (STLDM). Learning: Research
and Practice, 3(2), 114–129.
Chee, Y. S., & Chai, C. S. (2017). Quality 21st century learning: A framework for 21st
century education. In C. S. Chai, H. L. J. Koh & Y. H. Teo (Eds.), Technology-
enhanced 21st century learning (pp. 1–14). Singapore: Pearson.
Collins, A., & Halverson, R. (2010). The second educational revolution: Rethinking
education in the age of technology. Journal of Computer Assisted Learning, 26(1),
18–27.
Dede, C. (2010). Comparing frameworks for 21st century skills. In J. Bellanca &
R. Brandt (Eds.), 21st century skills: Rethinking how students learn (pp. 51–75).
Bloomington, IN: Solution Tree Press.
382 Journal of Educational Computing Research 57(2)

Dobozy, E., & Campbell, C. (2016). The complementary nature of learning design and
TPACK. In J. Dalziel (Ed.), Learning design: Conceptualizing a framework for teaching
and learning online (pp. 96–116). New York, NY: Routledge.
Drummond, A., & Sweeney, T. (2017). Can an objective measure of technological peda-
gogical content knowledge (TPACK) supplement existing TPACK measures? British
Journal of Educational Technology, 48(4), 928–939.
Geisinger, K. F. (2016). 21st century skills: What are they and how do we assess them?
Applied Measurement in Education, 29(4), 245–249.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data
analysis (7th ed.). New York, NY: Prentice Hall.
Harris, J., Grandgenett, N., & Hofer, M. (2010). Testing a TPACK-based technology
integration assessment rubric. In C. Crawford, D. A. Willis, R. Carlsen, I. Gibson, K.
McFerrin, J. Price, & R. Weber (Eds.), Proceedings of the society for information
technology & teacher education international conference 2010 (pp. 3833–3840).
Chesapeake, VA: AACE.
Harris, J., Phillips, M., Koehler, M., & Rosenberg, J. (2017). TPCK/TPACK research
and development: Past, present, and future directions. Australasian Journal of
Educational Technology, 33(3), i–viii. doi:10.14742/ajet.3907
Heitink, M., Voogt, J., Fisser, P., Verplanken, L., & van Braak, J. (2017). Eliciting
teachers’ technological pedagogical knowledge. Australasian Journal of Educational
Technology, 33(3), 96–109. doi:10.14742/ajet.3505
Howland, J. L., Jonassen, D. H., & Marra, R. M. (2014). Meaningful learning with tech-
nology (4th ed.). Essex, England: Pearson.
Jang, S. J., & Tsai, M. F. (2013). Exploring the TPACK of Taiwanese secondary school
science teachers using a new contextualized TPACK model. Australasian Journal of
Educational Technology, 29(4), 566–580.
Kereluik, K., Mishra, P., Fahnoe, C., & Terry, L. (2013). What knowledge is of most
worth: Teacher knowledge for 21st century learning. Journal of Digital Learning in
Teacher Education, 29(4), 127–140.
Kirschner, P. A. (2015). Do we need teachers as designers of technology enhanced
learning? Instructional Science, 43(2), 309–322.
Koehler, M. J., Mishra, P., & Yahya, K. (2007). Tracing the development of teacher
knowledge in a design seminar: Integrating content, pedagogy and technology.
Computers & Education, 49(3), 740–762.
Koehler, M. J., Mishra, P., Kereluik, K., Shin, T. S., & Graham, C. R. (2014). The
technological pedagogical content knowledge framework. In J. M. Spector, M.
D. Merill, J. Elen & M. J. Bishop (Eds.), Handbook of research on educational com-
munications and technology (pp. 101–111). New York, NY: Springer.
Koh, J. H. L., Chai, C. S., Wong, B., & Hong, H. Y. (2015). Technological pedagogical
content knowledge (TPACK) and design thinking: A framework to support ICT
lesson design for 21st century learning. The Asia Pacific Education Researcher,
24(3), 535–543.
Koh, J. H. L., Chai, C. S., Hong, H. Y., & Tsai, C. C. (2015). A survey to examine
teachers’ perceptions of design dispositions, lesson design practices and their relation-
ships with technological pedagogical content knowledge (TPACK). Asia Pacific
Journal of Teacher Education, 43(5), 378–391.
Chai et al. 383

Koh, J. H. L., Chai, C. S., & Lim, W. Y. (2017). Teacher professional development for
TPACK-21CL: Effects on teacher ICT integration and student outcomes. Journal of
Educational Computing Research, 55(2), 172–196.
Kramarski, B., & Michalsky, T. (2010). Preparing preservice teachers for self-regulated
learning in the context of technological pedagogical content knowledge. Learning and
Instruction, 20(5), 434–447.
Lee, M. H., & Tsai, C. C. (2010). Exploring teachers’ perceived self efficacy and techno-
logical pedagogical content knowledge with respect to educational use of the World
Wide Web. Instructional Science, 38(1), 1–21.
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge:
A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054.
Partnership for 21st Century Skills. (2009). Framework for 21st century learning.
Retrieved from http://www.p21.org/documents/P21_Framework.pdf
Pringle, R. M., Dawson, K., & Ritzhaupt, A. D. (2015). Integrating science and technol-
ogy: Using technological pedagogical content knowledge as a framework to study the
practices of science teachers. Journal of Science Education and Technology, 24(5),
648–662.
Razzouk, R., & Shute, V. (2012). What is design thinking and why is it important? Review
of Educational Research, 82(3), 330–348.
Rosenberg, S., Hammer, D., & Phelan, J. (2006). Multiple epistemological coherences in
an eighth-grade discussion of the rock cycle. The Journal of the Learning Sciences,
15(2), 261–292.
Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S.
(2009). Technological pedagogical content knowledge (TPACK) the development and
validation of an assessment instrument for preservice teachers. Journal of Research on
Technology in Education, 42(2), 123–149.
Tee, M. Y., & Lee, S. S. (2011). From socialisation to internalisation: Cultivating techno-
logical pedagogical content knowledge through problem-based learning. Australasian
Journal of Educational Technology, 27(1), 89–104.
Thomas, D., & Brown, J. S. (2011). A new culture of learning: Cultivating the imagination
for a world of constant change. Lexington, KY: CreateSpace.
Tondeur, J., van Braak, J., Sang, G., Voogt, J., Fisser, P., & Ottenbreit-Leftwich, A.
(2012). Preparing pre-service teachers to integrate technology in education: A synthesis
of qualitative evidence. Computers & Education, 59, 134–144. doi:10.1016/
j.compedu.2011.10.009
Tondeur, J., Aesaert, K., Pynoo, B., Braak, J., Fraeyman, N., & Erstad, O. (2017).
Developing a validated instrument to measure preservice teachers’ ICT competencies:
Meeting the demands of the 21st century. British Journal of Educational Technology,
48(2), 462–472.
Tondeur, J., van Braak, J., Ertmer, P. A., & Ottenbreit-Leftwich, A. (2017).
Understanding the relationship between teachers’ pedagogical beliefs and technology
use in education: a systematic review of qualitative evidence. Educational Technology
Research and Development, 65(3), 555–575.
Valtonen, T., Sointu, W., Kukkonen, J., Kontkanen, S., Lambert, M., & Mäkitalo-Siegl,
K. (2017). TPACK updated to measure pre-service teachers’ twenty-first century skills.
Australasian Journal of Educational Technology, 33(3), 15–31. doi:10.14742/ajet.3518
384 Journal of Educational Computing Research 57(2)

Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks

for 21st century competences: Implications for national curriculum policies. Journal of
Curriculum Studies, 44, 299–321. doi:10.1080/00220272.2012.668938
Voogt, J., Erstad, O., Dede, C., & Mishra, P. (2013). Challenges to learning and schooling
in the digital networked world of the 21st century. Journal of Computer Assisted
Learning, 29(5), 403–413.
Yeh, Y. F., Hsu, Y. S., Wu, H. K., Hwang, F. K., & Lin, T. C. (2014). Developing and
validating technological pedagogical content knowledge-practical (TPACK-practical)
through the Delphi survey technique. British Journal of Educational Technology, 45(4),
707–722.

Author Biographies
Ching Sing Chai is a professor from the Chinese University of Hong Kong. His
research interest is in teacher education, specifically technological pedagogical
content knowledge, teacher’s beliefs, and design thinking.

Joyce Hwee Ling Koh is an associate professor with the Learning Sciences and
Technologies Academic Group at the National Institute of Education, Nanyang
Technological University, Singapore. She teaches preservice and graduate
courses in the areas of ICT integration and instructional design, respectively.
She has published numerous peer-reviewed articles in the area of TPACK and
design thinking in SSCI-listed journals and has served as the Principal
Investigator for two research projects focusing on design thinking and ICT
integration in the context of 21st century learning.

Yiong Hwee Teo is a lecturer with the Learning Sciences and Technologies
Academic Group at the National Institute of Education, Nanyang
Technological University, Singapore. He teaches preservice and graduate
courses in the areas of ICT integration, instructional design, and classroom
management. He was part of the team in the Ministry of Education that planned
and implemented Singapore’s first three ICT Masterplans for schools. His
research focus is in asynchronous online discussion and blended learning.