A Review of Introductory Programming Research 2003–2017

Andrew Luxton-Reilly∗ Simon† Ibrahim Albluwi

University of Auckland University of Newcastle Princeton University
New Zealand Australia United States of America
andrew@cs.auckland.ac.nz simon@newcastle.edu.au isma@cs.princeton.edu

Brett A. Becker Michail Giannakos Amruth N. Kumar

University College Dublin Norwegian University of Science and Ramapo College of New Jersey
Ireland Technology United States of America
brett.becker@ucd.ie Norway amruth@ramapo.edu
michailg@ntnu.no

Linda Ott James Paterson Michael James Scott

Michigan Technological University Glasgow Caledonian University Falmouth University
United States of America United Kingdom United Kingdom
linda@mtu.edu james.paterson@gcu.ac.uk michael.scott@falmouth.ac.uk

Judy Sheard Claudia Szabo

Monash University University of Adelaide
Australia Australia
judy.sheard@monash.edu claudia.szabo@adelaide.edu.au

ABSTRACT The working group will conduct a systematic literature review

A broad review of research on the teaching and learning of program-
ming was conducted by Robins et al. in 2003. Since this work there
have been several reviews of research concerned with the teach-
ing and learning of programming, in particular introductory pro-
gramming. However, these reviews have focused on highly specific
aspects, such as student misconceptions, teaching approaches, pro-
gram comprehension, potentially seminal papers, research methods
applied, automated feedback for exercises, competency-enhancing
games, and program visualisation. While these aspects encompass
a wide range of issues, they do not cover the full scope of research
into novice programming. Some notable areas that have not been
reviewed are assessment, academic integrity and novice student
attitudes to programming.
There does not appear to have been a comprehensive review of
research into introductory programming since that of Robins et
al. It is therefore timely to conduct and present such a review in
order to gain an understanding of the research focuses, to highlight
advances in knowledge since 2003, and to indicate possible future
directions for research.
∗ co-leader
† co-leader
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
© 2018 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-5707-4/18/07. . . $15.00
https://doi.org/10.1145/3197091.3205841
based on the guidelines proposed by Kitchenham et al. This research
project is well suited to an ITiCSE working group as the synthesis
and discussion of the literature will benefit from input from a variety
of researchers drawn from different backgrounds and countries.
CCS CONCEPTS
• Social and professional topics → Computing education;
KEYWORDS
ITiCSE working group; CS1; introductory programming; novice
programming; systematic literature review
ACM Reference Format:
Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail
Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James
Scott, Judy Sheard, and Claudia Szabo. 2018. A Review of Introductory
Programming Research 2003–2017. In Proceedings of 23rd Annual ACM
Conference on Innovation and Technology in Computer Science Education
(ITiCSE’18). ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/
3197091.3205841
1 BACKGROUND
The review by Robins et al. [7] in 2003 provided a substantive
overview of novice programming research prior to that year. The
first paragraph of the review sets the general tone by stating:
Learning to program is hard ... Novice programmers
suffer from a wide range of difficulties and deficits. Pro-
gramming courses are generally regarded as difficult,
and often have the highest dropout rates. [7, p137]

342
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus Luxton-Reilly, Simon, Albluwi, Becker, Giannakos, Kumar, Ott, Paterson, Scott, Sheard, Szabo
However, more recent studies have suggested that the situation is
not as dire as previously suggested. Studies indicate that dropout
rates among CS students are not unusually high [1, 13], and it has
been suggested that the difficulties faced by novices may be a con-
sequence of unrealistic expectations rather than intrinsic subject
complexity [4]. We believe that much has changed in the CS Edu-
cation community over the last 15 years, and a systematic review
of the novice programming literature would be of great benefit.
Although there have been several reviews of research involv-
ing novice programmers since 2003, those reviews have focused on
highly specific aspects, such as student misconceptions [6], teaching
approaches [12], program comprehension [8], potentially seminal
papers [5], research methods applied [9], automated feedback for ex-
ercises [2], competency-enhancing games [11], and program visuali-
sation [10]. There does not appear to have been an all-encompassing
review of research into introductory programming since that of
Robins et al. [7].
While the aspects listed above are wide-ranging, they clearly
do not cover the full scope of research into novice programming.
Picking just three further areas almost arbitrarily, we are not aware
of a review of research on novice programming assessment, aca-
demic integrity in novice programming, or novice student attitudes
to programming. It is therefore timely to conduct and present a
review of the area that is as broad-ranging as possible.
However, in this review, we make a clear distinction between
research involving introductory computing and research involving
introductory programming. Many aspects of introductory comput-
ing do not deal explicitly with programming. For example, the lit-
erature of computing education reports much research into aspects
of K-12 (school) education, and into computational thinking. This
review will address such research only where it clearly intersects
with introductory programming.
2 METHOD
The working group will conduct a systematic literature review
based on the guidelines proposed by Kitchenham et al. [3]. System-
atic reviews follow a highly structured process that involves:
(1) Specifying research questions
(2) Developing and reviewing research protocols
(3) Conducting searches of databases
(4) Selecting primary studies by applying inclusion and exclu-
sion criteria
(5) Filtering the studies by evaluating their quality
(6) Extracting data using standardized data extraction form
(7) Synthesising the results
(8) Writing the review report
It is possible that a purely systematic approach will prove un-
suitable for the project. A typical systematic review is relatively
narrow in scope, making it possible to define search terms that
343
will generate a workable number of papers. The deliberately broad
scope of this review is likely to generate a list of papers that is very
large, perhaps too large to give each paper the consideration that it
deserves. At the same time, it is possible that despite the group’s
best efforts to define the search terms, the search will miss papers
that are already known to members of the group. Therefore the
group accepts that it might need to deviate from the strict protocol
with which it will start, either to reduce or to supplement the list
of papers generated. If that does happen, the group will also report
on what it needed to change, and why.
REFERENCES
[1] Jens Bennedsen and Michael E. Caspersen. 2007. Failure Rates in Introductory
Programming. SIGCSE Bull. 39, 2 (June 2007), 32–36. https://doi.org/10.1145/
1272848.1272879
[2] Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2016. Towards a System-
atic Review of Automated Feedback Generation for Programming Exercises.
In Proceedings of the 2016 ACM Conference on Innovation and Technology in
Computer Science Education (ITiCSE ’16). ACM, New York, NY, USA, 41–46.
https://doi.org/10.1145/2899415.2899422
[3] Barbara Kitchenham, O Pearl Brereton, David Budgen, Mark Turner, John Bailey,
and Stephen Linkman. 2009. Systematic literature reviews in software engineer-
ing – A systematic literature review. Information and Software Technology 51, 1
(2009), 7–15. https://doi.org/10.1016/j.infsof.2008.09.009
[4] Andrew Luxton-Reilly. 2016. Learning to Program is Easy. In Proceedings of the
2016 ACM Conference on Innovation and Technology in Computer Science Education
(ITiCSE ’16). ACM, New York, NY, USA, 284–289. https://doi.org/10.1145/2899415.
2899432
[5] Arnold Pears, Stephen Seidman, Lauri Malmi, Linda Mannila, Elizabeth Adams,
Jens Bennedsen, Marie Devlin, and James Paterson. 2007. A Survey of Literature
on the Teaching of Introductory Programming. In Working Group Reports on
ITiCSE on Innovation and Technology in Computer Science Education (ITiCSE-WGR
’07). ACM, New York, NY, USA, 204–223. https://doi.org/10.1145/1345443.1345441
[6] Yizhou Qian and James Lehman. 2017. Students’ Misconceptions and Other
Difficulties in Introductory Programming: A Literature Review. ACM Trans.
Comput. Educ. 18, 1, Article 1 (Oct. 2017), 24 pages. https://doi.org/10.1145/
3077618
[7] Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and
Teaching Programming: A Review and Discussion. Computer Science Education
13, 2 (2003), 137–172. https://doi.org/10.1076/csed.13.2.137.14200
[8] Ivonne Schröter, Jacob Krüger, Janet Siegmund, and Thomas Leich. 2017. Compre-
hending Studies on Program Comprehension. In Proceedings of the 25th Interna-
tional Conference on Program Comprehension (ICPC ’17). IEEE Press, Piscataway,
NJ, USA, 308–311. https://doi.org/10.1109/ICPC.2017.9
[9] Judy Sheard, Simon, Margaret Hamilton, and Jan Lönnberg. 2009. Analysis of
Research into the Teaching and Learning of Programming. In Proceedings of the
Fifth International Workshop on Computing Education Research Workshop (ICER
’09). ACM, New York, NY, USA, 93–104. https://doi.org/10.1145/1584322.1584334
[10] Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A Review of Generic Program
Visualization Systems for Introductory Programming Education. Trans. Comput.
Educ. 13, 4, Article 15 (Nov. 2013), 64 pages. https://doi.org/10.1145/2490822
[11] A. Vahldick, A. J. Mendes, and M. J. Marcelino. 2014. A review of games designed
to improve introductory computer programming competencies. In 2014 IEEE
Frontiers in Education Conference (FIE) Proceedings. 1–7. https://doi.org/10.1109/
FIE.2014.7044114
[12] Arto Vihavainen, Jonne Airaksinen, and Christopher Watson. 2014. A System-
atic Review of Approaches for Teaching Introductory Programming and Their
Influence on Success. In Proceedings of the Tenth Annual Conference on Interna-
tional Computing Education Research (ICER ’14). ACM, New York, NY, USA, 19–26.
https://doi.org/10.1145/2632320.2632349
[13] Christopher Watson and Frederick W.B. Li. 2014. Failure Rates in Introductory
Programming Revisited. In Proceedings of the 2014 Conference on Innovation &
Technology in Computer Science Education (ITiCSE ’14). ACM, New York, NY, USA,
39–44. https://doi.org/10.1145/2591708.2591749