Programming Assistance Tools

Computers & Education 82 (2015) 162e178
Contents lists available at ScienceDirect
Computers & Education

journal homepage: www.elsevier.com/locate/compedu
Programming assistance tools to support the learning of IT

programming in South African secondary schools
Calitz 2
Melisa Koorsse*, Charmain Cilliers 1, Andre
Nelson Mandela Metropolitan University, P.O. Box 77000, Port Elizabeth, 6031, South Africa
a r t i c l e i n f o
a b s t r a c t
Article history:
Received 25 August 2014
Received in revised form
17 November 2014
Accepted 26 November 2014
Available online 3 December 2014
Novice programmers struggle to understand introductory programming concepts and this difculty,
associated with learning to program, contributes mainly to the lack of interest in the eld of Computer
Science at tertiary level. Programming assistance tools have been used to assist novice programmers
extensively at education institutions. A programming assistance tool (PAT) is a software program that can
be used by novice programmers to learn how to program and/or improve their understanding of programming concepts.
This research proposes that novice programmers, specically Information Technology (IT) scholars in
South African secondary schools, could be supported by PATs. The main objective of this research was to
determine whether the use of a PAT impacted IT scholars' understanding of programming concepts and
their motivation towards programming in general. Criteria for the selection of PATs were derived from
the programming difculties identied in literature and from surveys to IT teachers and scholars. The
selection criteria were grouped into programming knowledge and programming skills categories.
Existing PATs were evaluated using the selection criteria and three PATs, namely, RoboMind, Scratch and
B#, were selected for evaluation in this research study. A convenience sample of schools participated in
the study. The three PATs provided different approaches while being able to support the Delphi programming language used in schools that participated in the study.
The ndings of this research indicated that, although scholars perceived the PATs to be useful in the
explanation of certain of the programming concepts, there was no conclusive evidence that IT scholars
who used a PAT had a signicantly better understanding of programming concepts and motivation towards programming than scholars who did not use a PAT. Participant feedback was used to identify the
strengths and shortcomings of the three PATs and to provide recommendations for the development of
future PATs specically designed to support IT scholars.
2014 Elsevier Ltd. All rights reserved.
Keywords:
Programming and programming languages
Interactive learning environments
Secondary education
Evaluation of CAL systems
1. Introduction
Information and Communication Technology (ICT) related professions have been identied as top professions due to the increase in the
number of companies developing web, mobile and cloud computing applications (Strieber, 2011). Despite a great demand for ICT qualications, countries around the world, including the U.S. (Hug, Guenther, & Wenk, 2013; Wilson, Sudol, Stephenson, & Stehlik, 2010), European countries (Porta, Maillet, & Gil, 2010) and South Africa (Alexander & Twinomurinzi, 2012; Calitz, 2010), have experienced a decline in
the number of enrolments in Computer Science (CS) programs at tertiary level education institutions. This decline in enrolments has
resulted in a decrease in the number of software developers available to industry. Globally countries have identied the importance of ICT
professions for technological and economic growth of the country (Wilson et al., 2010). The importance of promoting ICT related skills and
knowledge at school level has also been recognised (Wilson et al., 2010).
* Corresponding author. Tel.: 27 41 504 3925.

E-mail addresses: Melisa.Koorsse@nmmu.ac.za (M. Koorsse), Charmain.Cilliers@nmmu.ac.za (C. Cilliers), Andre.Calitz@nmmu.ac.za (A. Calitz).
1
Tel.: 27 41 504 2639.
2
Tel.: 27 41 504 2235.
http://dx.doi.org/10.1016/j.compedu.2014.11.020
0360-1315/ 2014 Elsevier Ltd. All rights reserved.
M. Koorsse et al. / Computers & Education 82 (2015) 162e178
163
In South Africa, ICT related skills and knowledge including software development are included in the subject matter of the Information
Technology (IT) subject in South African secondary schools for Grade 10e12 scholars. IT in South African secondary schools is perceived as
being a difcult subject (Havenga & Mentz, 2009). This reputation impacts on scholars' decisions to take IT at school. Many scholars that do
attempt the subject change to another, easier subject before Grade 12 (the nal year) or remain in the subject class but lack the motivation
and interest in the programming content to achieve their full potential. The lack of interest and motivation in the subject and negative
impression created amongst scholars seems to affect negatively the number of scholars who decide to pursue further education in a
computing discipline.
IT scholars are faced with the same difculties of learning to program as that experienced by novice programmers in general (Havenga &
Mentz, 2009). These difculties are aggravated by the secondary school learning environment (Koorsse, Calitz & Cilliers, 2010). Short
(approximately 45 min), daily lessons make it difcult both to teach and practise complex programming concepts and skills during limited
school hours. There is consequently inadequate time available for IT teachers to address the specic programming difculties of individual IT
scholars as well as accurately assess scholar understanding of concepts during class time.
IT scholars further lack appropriate support in the form of resources to supplement the prescribed IT subject textbook. Programming
assistance tools (PATs) are one such supporting resource (Pears et al., 2007). PATs are designed specically for novice programmers in order
to enhance comprehension of algorithms and computer programs, assist with code debugging and/or assess programming knowledge and
skills (Kelleher & Pausch, 2005). PATs support self-study (Al-Imamy, Alizadeh, & Nour, 2006) and promote interaction (Baldwin & Kuljis,
2000). Research studies have evaluated the use of a PAT as part of the teaching environment such as Scratch (Malan & Leitner, 2007)
and a web-based tool (Al-Imamy et al. 2006). However, this study aims to evaluate the impact of PATs provided as part of self-regulated
learning. It is also not clear from literature if there are specic PATs that could be used to promote the understanding programming concepts and skills that are required specically for IT scholars. None of the PATs identied in literature have been evaluated in terms of the
programming knowledge and skills supported or the PATs can be used to support the learning of IT programming in South African (SA)
secondary schools.
This research study was important in identifying the programming knowledge and skills required by novice programmers in general
(Section 2), and specically IT scholars in SA schools (Section 2.2). This resulted in a list of selection criteria that indicated the programming
skills and knowledge that should be supported by a PAT (Section 2.2). Literature identies many PATs that have been developed for use by
novice programming subject (Section 3) with different programming environments. However, using the selection criteria, this research
study was able to select PATs that met the requirements of IT programmers (Section 3). The research study then evaluated whether or not
PATs could have an impact on IT scholar understanding of programming concepts and thus whether or not PATs could support the teaching
and learning of IT programming in SA schools (Section 6). Three PATs were selected for evaluation using the selection criteria identied. The
three PATs were RoboMind (Section 3.2.1), Scratch (Section 3.2.2) and B# (Section 3.2.3). The following research questions were addressed
by the study to determine the impact of these PATs:
RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?
RQ2: What impact do the PATs have on IT scholar perceived difculty of programming?
RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specic programming concepts?
The analysis of the results related to the research questions provided feedback on the impact, if any, of the different PATs on the understanding of programming concepts from a scholar's perspective and in terms of formal assessment (Section 5). Certain of the programming concepts such as loops were better supported by PATs and supported scholar understanding of these concepts (Section 5).
However, overall there was no evidence to support that scholars using the PATs had a signicantly better understanding of programming
concepts than scholars who did not. Qualitative feedback indicated the importance of the PAT interface, in particular that the interface
should not be too simple. The results and ndings of the research study provide further criteria and design guidelines for PATs to be
developed to support the learning of IT programming in SA schools, and programming in general (Section 6).
2. PAT selection criteria
A dependence relationship exists between the different types of knowledge required by a novice programmer (syntax, computers,
programming principles and programming concepts) and code comprehension and generation (Fig. 1). Novice programmers need to understand that code generation involves three sequential steps:
1. Studying a given problem statement or set of requirements and deciding on the best programming strategy to use. This skill requires an
understanding of the programming principles that can be applied as well as knowledge of how a computer executes code in order to
formulate a solution.
2. Producing an algorithm to solve the problem. The algorithm will often be formulated using pseudo code. Knowledge of programming
principles and concepts is required in order to select appropriate data types and structures or apply concepts such as looping or selection, for example. Knowledge of computer execution of code would assist programmers to efciently and effectively order the
sequence of algorithm instructions.
3. Translating the algorithm into the code of the programming language being used requires knowledge of the programming language syntax.
The program code is tested and changed until the program meets the original set of requirements and thus solves the problem.
The development of programming strategies in terms of applying programming knowledge correctly when formulating a solution to a
programming problem is important. Different knowledge and skills that are important in the programming process (Fig. 1). A particular PAT
may not address all these areas, instead supporting only a few areas. Surveys were administered to IT scholars and teachers to identify the
deciencies in the skill and knowledge areas required by a programmer (Section 2.2). By identifying skill and knowledge areas that need
attention for novice programming (Section 2.1) and particularly in IT schools (Section 2.2), selection criteria for PATs were formulated
164
Fig. 1. Knowledge and skills required by a programmer.
Table 1
Criteria for selecting PATs identied from introductory programming literature.
Category
Criteria item
Programming knowledge
Assist with the understanding of code execution

Assist with the learning of programming language syntax
Assists with developing knowledge of programming principles and concepts
Programming environment suitable for novice programmers
Assists with the application of programming knowledge (programming strategies)
Provides simple error message to assist with debugging
Promotes problem solving
Develops code comprehension
Programming skills
(Table 2) so that appropriate PATs could be identied and selected for use by IT scholars (Section 3). These PATs should then be able to
address the deciencies identied.
2.1. Novice programming knowledge and skills required
Specic programming knowledge and programming skills are required by novice programmers in order to program successfully. Aspects
which inuence programming ability are the application of knowledge and strategies and the relationship between skills such as code
comprehension and code generation (de Raadt, 2008; Robins, Rountree, & Rountree, 2003). Programming knowledge and strategies are
distinct but related (de Raadt, 2008). Programming knowledge relates to the body of information describing three areas:
1. Programming concepts and principles (for example: how a for-loop works or the purpose of a variable),
2. Knowledge of computers (for example: how an event is generated) and
3. Programming language knowledge or syntax.
If a programmer is not aware of concepts such as loops, data structures or objects, it would be difcult for the programmer to plan a
suitable solution. Knowledge of the syntax is required to implement a solution in a particular programming language (Pears et al., 2007).
Table 2
PAT criteria derived from literature (Lit) and IT teacher (TF) and scholar feedback (LF).
Category
Criteria for PAT
Lit
TF
Assists with the learning of programming language syntax (Delphi in this research study)
Constructivist to promote self-study
Assists with the application of programming knowledge
Assists with the understanding of code execution
Promotes problem solving and planning
Provides simple error messages to assist with debugging
Feedback to guide solution creation
Feedback regarding errors
C
C
C
C
C
C
C
C
C
C
C
C
Programming skills
C
C
C
C
LF
C
C
C
C
C
165
Regardless of how well the solution is designed, if the syntax is incorrect, the program will not run successfully. Knowledge in all three of
these areas is required for a novice programmer to be successful in programming.
Programming strategy refers to the way in which programming knowledge is applied to solve a particular problem (de Raadt, 2008;
Robins et al., 2003). Programming strategies are required for the rst step of code generation (Fig. 1). Novice programmers need assistance to develop both their programming knowledge and programming strategies. However, teachers generally spend more time teaching
programming language knowledge than programming strategy (Al-Imamy et al., 2006; Pears et al., 2007). Novice programmers often
combine steps 1 and 2 of the code generation process (Fig. 1) as they attempt to solve the algorithm in a particular programming language
(Garner, 2007; Rongas, Kaarna, & Kalvianen, 2004). Lack of planning or lack of algorithm design leads to a programmer having difculty in
solving a given problem (Shuhidan, Hamilton, & D'Souza, 2009).
Novice programmers also have an inability to transfer their knowledge of problem solutions to similar problems because they lack the
ability to comprehend program code and have a fragile knowledge of programming principles. Their knowledge is not deep enough to
understand how the principles can be applied to new or larger problems that are more complex. Related studies have indicated that novice
rvinen, 2005; Lister et al., 2004; Robins
programmers lack problem solving ability (Heines & Schedlbauer, 2007; Lahtinen, Ala-Mutka, & Ja
et al., 2003). However, novice programmers may be capable of problem solving but unable to express solutions in a programming language
or in a manner that a computer would understand. This is supported by the nding that novice programmers have a non-viable mental
model of programming concepts which results in misconceptions and difculties when trying to solve programming problems (Ma,
Ferguson, Roper, Ross, & Wood, 2008).
In addition to programming knowledge, novice programmers also require the skill to use a programming language and work in a
programming development environment. The programming language and environment used when novice programmers are learning to
program can also impact their understanding of programming concepts. Certain programming languages are too complex to explain or use
lling, 1999). A professional programming development
to teach some of the programming concepts taught to novice programmers (Ko
environment may overwhelm the students by presenting them with functionality and interfaces not needed by novice programmers
lling, 1999; Levy, Ben-Ari, & Uronen, 2001; Pendergast, 2006). Furthermore, the compiler messages provided by professional pro(Ko
gramming development environments are directed at professional programmers (Rongas et al., 2004).
Program compilers ensure that programs are syntactically and semantically correct prior to execution. If an error is found in the code, a
compiler message is displayed to the programmer describing the error and providing the location of the error in the code. Most compiler
messages are too complicated and low-level and do not explain sufciently well to a novice programmer which syntactical or semantic
errors have occurred. Novice programmers thus struggle to develop their skill to debug code.
Vague error messages such as the identication of incompatible data types do not provide programmers with information on how the
error can be corrected. In professional programming development environments, novice programmers struggle to understand compiler
error messages intended for professional programmers with a better understanding of the programming environment and more programming expertise.
Acquiring the programming knowledge and skills identied above should be supported by PATs provided to novice programmers.
Programming knowledge criteria would assist IT learner understanding of programming concepts and principles and programming skills
criteria would assist IT learner ability to perform programming related skills such as tracing code execution. PATs should assist scholars to
develop programming knowledge in computer execution of programming code, programming language syntax and programming principles and concepts (Table 1). Addressing these areas together with methods of improving code comprehension should implicitly assist users
to improve problem solving ability. PATs should also support the development of programming strategies in terms of applying programming
knowledge.
PATs selected for use with scholars should promote self-study (Table 1) for the scholar to improve knowledge in spite of the difculties
related to the teaching approach, programming language and development environment. PATs appropriate for use by IT scholars should
promote debugging by identifying errors in code and provide simple error messages that novice programmers can understand (Table 1).
Table 1 lists PAT selection criteria to address general novice programming difculties. A survey to IT teachers and scholars reected the
perceptions of the difculties faced by IT scholars learning to program in South African secondary schools. The survey ndings were used to
conrm the criteria for PAT selection from literature and included criteria that could specically be used to select PATs to support the
learning of IT programming.
2.2. PAT selection criteria to address IT programming difculties
Surveys were administered to participating IT teachers and Grade 11 IT scholars at four South African secondary schools in order to
identify the selection criteria for PATs to address programming difculties experienced by IT scholars. The four schools participating were a
sample of convenience and only IT teachers at these schools were included in the study. IT scholars participating in this study were taught
the Delphi programming language. Both questionnaire instruments were piloted to establish reliability and validity before administering to
participants. A combination of quantitative and qualitative data analysis techniques were used.
The IT teacher questionnaire was designed to gather information regarding the methods used by teachers to teach programming
concepts to their scholars and the teacher's perspective on difculties experienced by scholars. The aim of the questionnaire to participating
IT scholars was to identify factors contributing to the difculty of IT programming from an IT scholar perspective.
Feedback from IT teacher and scholar surveys conrmed the ndings from literature and identied additional criteria to be applied when
selecting an appropriate PAT to support the learning of IT programming. The feedback from the surveys to IT teachers resulted in a
comprehensive set of criteria as follows:
Promotes self-study in a constructivist way: IT teachers identied that the class lessons were short resulting in less time in class to
practise programming with the assistance of the IT teacher. Scholars should thus be able to use a PAT without the assistance of the
teacher to develop their knowledge and explore different programming concepts. This nding is conrmed by related studies (Areias &
Mendes, 2007).
166
Develops knowledge of programming principles and concepts: Certain programming principles and concepts such as objects, datastructures and planning algorithms were identied as difcult for IT scholars to understand. PATs that provide an explanation of
programming concepts and principles can improve scholar understanding of concepts during self-study.
Assists with the application of programming knowledge: It was found that scholars struggle to apply programming knowledge and the PAT
should provide support in the application of programming knowledge to different problems and exercises. An example would be
support in the form of scaffolding that assists the scholar to plan a coded or non-coded solution, as identied in related work (Wood,
Bruner, & Ross, 1976). When the scaffolding is removed, the scholar should be able to perform the task independently, as identied in
work by Kunkle (2010).
Assists with learning syntax knowledge: IT teachers indicated that struggling scholars have difculty with programming syntax.
Improving syntax knowledge is important for novice programmers to produce a successful code solution. In this research study, PATs
were required to assist IT scholars with the learning of the Delphi programming language syntax, particularly as scholars may not have
access to Delphi beyond the classroom.
Provides feedback regarding errors: PATs could benet IT scholars who struggle to identify syntax errors, by providing feedback regarding
errors in the code or solution. Error feedback would assist scholars in the development of debugging skills.
Provides simple error messages: If feedback is provided, simple error messages should be used that are easy to understand and help to
identify the error in order to improve an IT scholar's debugging skills.
Promotes problem solving and planning: IT teachers identied problem solving and planning of solutions as skills that IT scholars have
difculty understanding and applying. The PAT should help to promote these skills.
Provides feedback to guide solution creation: IT scholars need assistance to improve their ability to create algorithms. The PAT should
assist scholars to convert a planned solution to a code solution that can be executed.
PAT selection criteria were also formulated from IT scholar survey feedback:
Assists with understanding of code execution: Assistance with code execution could allow scholars to learn how to trace a program and
evaluate the state of variables at different points during the execution of the code. These criteria would assist scholars to improve
debugging skills as well as algorithm and code comprehension.
Provides simple error messages: Results indicated that scholars' understanding of compiler error messages rated lower than their use of
compiler error messages. Simple error messages would allow scholars who use error messages to understand the error during
debugging. Debugging is a skill scholars identied as difcult.
Develops code comprehension: Improving a scholar's ability to read and understand code could help scholars to evaluate their own code
solutions and assist with debugging. Code comprehension would also assist scholar understanding of common algorithms and how to
adapt algorithms to different problems by transferring knowledge from solutions to similar problems.
Promotes problem solving and planning: Results indicated that scholars do not create solutions independent of code when planning a
solution. The PAT should thus assist scholars to develop a program solution using diagrams or pseudo-code, including the provision of
feedback during code execution. PATs should guide scholars to develop a suitable solution to the problem and to identify any problems
in the logic of the program solution.
Feedback to guide solution creation: The purpose of this criterion is to assist scholars to convert a pseudo-code program solution into a
code solution implemented in a programming language, thus improving their ability to create code algorithms.
A comprehensive set of criteria (Table 2) incorporating characteristics evident in related studies as well as feedback from IT teachers and
scholars was formulated. The selection criteria were then used to evaluate and select suitable PATs specically for use in this study, where
participating IT scholars in South African secondary schools would independently use and evaluate one of the selected PATs.
3. Programming Assistance Tools (PATs)
PATs are specically designed for use by novice programmers to support their understanding of programming knowledge and assist with
the development of programming skills. The programming environment provided by a PAT is simpler to use than professional programming
environments, even supporting the learning of programming using visualisation techniques. The visualisation of programming concepts
supports novice programmers' understanding of abstract concepts, using interactive microworlds to make programming more interesting
and applicable (Pears et al., 2007).
An advantage of using a PAT is that it can assist a novice programmer to develop an understanding of programming concepts as well
as provide automated assessment of programming tasks and correction of simple errors (Rongas et al., 2004). Other advantages are that
PATs can promote interactivity as well as support self-study, providing novice programmers with different pedagogical learning
methods.
PATs can employ visualisation techniques to enhance algorithm comprehension, improve debugging ability and guide exploration to
understand concepts (Pears et al., 2007). The structure or execution of code can be visualised or a programming concept can be animated.
There are PATs that have environments that allow novice programmers to construct algorithms and visualisations graphically, using
owcharts or drag-and-drop techniques (Kelleher & Pausch, 2005) and thus prevent novice users from making syntax errors. Structured
editing, where elements are specied from a menu when typing the program, can also be used (Guzdial, 2004) where placeholders indicate
to users where additional code should be specied, such as variable names or values. Structured editing assists users by providing valid
constructs, thus allowing users to develop an executable solution, although logical errors may exist.
Microworlds can be used to make the understanding of programming more concrete as users can view the result of programming solutions as actions by characters or objects in a visual and/or animated environment (Kelleher & Pausch, 2005). General-purpose, professional
programming languages can be abstract and difcult to understand. Microworld environments motivate users to want to learn as the
environment is perceived as having a fun element (Kelleher & Pausch, 2005).
167
PATs use animation and/or visualisation in different ways. A solution can be created by typing code while the resulting program can be
animated, such as a robot performing actions in a 3D world. Alternatively, graphical objects can be used to create the program solution even
though the running program may not have any animations nor make use of a graphical user interface.
A PAT that uses visualisation techniques may not assist novice programmers. In some cases novice programmers that use a PAT do not
perform better than those not using it (Levy et al., 2001). The visualisation techniques used by a PAT may not be effective when the scholars
are not able to accurately and successfully map programming concepts to elements in the animation. PATs may also use animations that
require supplementary explanations from the teacher in order to be successful. However, scholars using a PAT with animations report
increased satisfaction and motivation compared to scholars who do not use a PAT (Levy et al., 2001).
The majority of PATs evaluated for selection in this research study were identied from a study of related work. The PAT selection criteria
were used to evaluate 10 PATs, from which three were identied for evaluation in this research study (Section 3.1.1). The programming
concepts and criteria which each of the three selected PATs address are identied (Section 3.1.2) and the three PATs are discussed (Section 3.2).
3.1. Selection and evaluation of PATs
The research study had to identify PATs suitable for use by IT programming scholars in South African secondary schools (Section 3.1.1). In
particular, three PATs had to be identied for evaluation at three participating schools as part of the experimental procedure (Section 4).
Since there is a possibility that PATs may not satisfy all of the derived selection criteria (Section 2.2), those PATs that were selected had to
satisfy the majority of the selection criteria (Section 3.1.2).
3.1.1. PATs for IT
The PATs selected for review and evaluation were identied based on availability. PATs identied had to be freely available for download
in order to make the PAT available to IT scholars. The PATs reviewed were a representative sample of tools developed to support novice
programmers.
PATs were evaluated using the derived selection criteria (Section 2.2). Only PATs that met most of the criteria (Table 3) were further
evaluated before selecting the nal three PATs for evaluation in the schools. The following PATs were evaluated and omitted as they did not
address the majority of the selection criteria: MatrixPro, KarelRobot, NoteTab, NotePad2, Phrogram, JHave, SmallTalk, EToys and GameMaker.
PATs selected for inclusion in the evaluation phase (Table 3) were: RoboMind, BlueJ, Greenfoot, Scratch, B#, Jeliot, Ville, PlanAni, Alice and
jGRASP.
Scratch and Alice allow users to create programs by dragging and dropping visual blocks. The blocks can only be joined in specic ways
thus only logical errors can occur. Error messages and feedback for errors are thus not applicable (n/a) for these two PATs (Table 3). Similarly,
PlanAni only allows users to run programs that have already been created and are running properly. The purpose of PlanAni is to animate a
program for users to understand the use of data structures as well as how looping, for example, works.
The ve highest ranking PATs (Koorsse, 2012), which were evaluated using the selection criteria were: B#, PlanAni, Scratch, Alice and
RoboMind. However, for this particular study only three PATs were required as three schools agreed to participate.
3.1.2. Match to PAT criteria
The three PATs selected as the intervention tools in the study, based on the evaluation of the PATs (Section 3.1.1) were RoboMind (Section
3.2.1), Scratch (Section 3.2.2) and B# (Section 3.2.3). B# is a PAT developed at the Nelson Mandela Metropolitan University (NMMU).
Scratch was recommended by the South African Department of Education to be used in the new IT subject curriculum implemented from
2012 for Grade 10 (Department of Basic Education, 2011). Scratch was thus selected as one of the three PATs. Alice and Scratch are similar
programming environments e allowing users to create a program by joining blocks to build a script that executes an animated program. In
order to evaluate different PATs, Alice was not included.
PlanAni satises many of the selection criteria and supports the Pascal programming language on which the Delphi programming
language is based. However, PlanAni does not allow users to create their own programs and only six example programs are provided for
users to work with. PlanAni was thus not included for this study.
The original RoboMind had a number of shortcomings in terms of the desired features for PATs. However, the source code was accessible
and the original RoboMind was adapted to support the Delphi programming language. Adapted RoboMind and B# can assist users to
improve their knowledge of programming language syntax (Table 3) as program solutions are implemented or generated in the Delphi
programming language. In Scratch, drag and drop building blocks are used to created program solutions.
Table 3
Evaluation of PATs using selection criteria.
Criteria
Assists with learning of the Delphi programming language syntax
Constructivist to promote self-study
Assists with the application of programming knowledge
Assists with the understanding of code execution
Programming skills
Promotes problem solving and planning
Provides simple error messages to assist with debugging
Feedback to guide solution creation
Feedback regarding errors
% of criteria met:
B#
PlanAni
C
C
C
C
C
C
C
C
C
C
C
C
C
C
90
n/a
C
n/a
60
Scratch
Alice
RoboMind
C
C
C
C
C
C
C
C
n/a
C
C
n/a
60
C
n/a
C
C
n/a
60
Jeliot
Ville
BlueJ
Greenfoot
jGRASP
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
60
C
60
C
40
C
40
C
40
C
60
168
All three PATs are: constructivist to promote self-study (Table 3) by scholars to improve their understanding of programming concepts;
assist with the development of knowledge of programming principles and concepts; and allow users to step through the program code
solution line by line during execution to interpret the code and see the effects of each line on the program output. Stepping through the code
can assist user understanding of code execution. None of the three PATs demonstrates to users how programming knowledge can be applied
to solve programming problems.
Scratch makes use of the drag-and-drop interface which ensures that users can only use the correct statements and syntax. Scratch thus
indirectly supports error handling but error messages are not explicitly provided to scholars to support debugging. The criteria to provide
simple error messages or feedback regarding errors are thus not applicable to Scratch due to the interface employed. RoboMind and B# use
simple error messages to inform users of syntax errors in the code using language and terms that are simple for novice programmers to
understand. Scratch uses building blocks with statements that indicate to users where conditions or variables must be inserted or if other
statements must be included within a loop or control structure in order to guide the creation of solution. B# allows users to build a solution
using a owchart diagram. Users are able to visualise the execution of the solution using the owchart.
RoboMind is the only one of the three PATs that does not explicitly support problem solving and planning, the development of code
comprehension and the provision of feedback to guide solution creation. The focus of RoboMind is to provide users with an animated, visual
environment in which the program solution is executed.
The programming concepts that IT scholars are expected to know (Table 4) are outlined in the Curriculum and Assessment Policy
Statement (CAPS) for Information Technology published by the South African Department of Basic Education (Department of Basic
Education, 2011). The three PATs support different criteria and programming concepts. Procedures are supported by RoboMind and
Scratch, although only RoboMind supports the passing of parameters in the procedure call. RoboMind does not support text output (Table 4),
however, the program output is visualised when the robot moves through the environment.
3.2. PATs selected as intervention tools
An evaluation of the impact of the three PATs selected as intervention tools rst required an understanding and description of the
programming environments presented by each, including the mapping between programming concepts in the PAT and in the Delphi
programming language. Schools who participated in this study teach the Delphi programming language as prescribed by the Department of
Basic Education (2011). RoboMind (Section 3.2.1) is an open-source PAT which was adapted so that the syntax and semantics more closely
matched the Delphi programming language. The building blocks used in the Scratch environment (Section 3.2.2) are mapped to corresponding programming constructs in the Delphi programming language. Finally, the similarities and differences between B# and Delphi are
presented (Section 3.2.3).
3.2.1. RoboMind
RoboMind is a PAT that can be used as an introduction to automation and programming without any assumed prior knowledge of
programming (RoboMind, 2009). RoboMind allows users to create and execute a text-based program, similar in syntax to that of the Java
programming language, to instruct a robot to interact with objects in a map-based world. A standard set of commands controls the robot's
movement and actions in the world. Basic programming concepts such as looping, nested if-statements and user-dened procedures can be
implemented. The RoboMind source code is freely available thus RoboMind could and was adapted to support syntax similar to Delphi, the
programming language used by participating schools. The programming concepts supported by the original version of RoboMind are
presented (Section 3.2.1.1). Adaptations to RoboMind are discussed (Section 3.2.1.2) to prepare the PAT for evaluations in the participating
schools.
3.2.1.1. Original RoboMind. RoboMind implements basic programming concepts such as looping and nested if-statements. Procedures with
parameters are also acceptable. No variables can be declared or used, except for the procedure parameters. The language syntax used is
similar to Java although there is some variation such as the lack of semi-colons.
Table 4
IT curriculum programming concepts (Department of Basic Education, 2011) supported.
Programming concept
RoboMind
Scratch
Procedures
Functions
One-dimensional arrays
repeat-loops
Objects and classes
Two-dimensional arrays
while-loops
String handling
for-loops
Correct use of parameters
if-statements
SQL statements
File handling
Accessing a database
case-statements
Output
Variables
Input
Total: 18
C (adapted to support)
C
C
C
C
C
C
C
C(adapted to support)
7
B#
C
C
C
C
C
C
11
C
C
C
C
8
169
The user types out the code instructions to navigate the robot through the world (A in Fig. 2) and then selects Run to execute the instructions. The tool will notify the user of any errors in the code. If the code is error-free, the user can watch the robot navigate through its
environment based on the code instructions provided (B in Fig. 2). The current line of code being executed is indicated by a pointer and a
message at the bottom of the screen (C in Fig. 2). Users can stop or pause the execution of the program (D in Fig. 2) as well as change the
execution speed. The RoboMind environment is freely available for individual, educational and commercial use (RoboMind, 2009). The
development environment is open source allowing RoboMind to be adapted.
RoboMind also assists novice programmers to develop knowledge of programming principles and concepts and code execution as users
are able to follow step-by-step execution of the program code as the robot moves through the map world. RoboMind assists novice programmers to improve programming skills by providing simple error messages to assist with debugging. RoboMind provides support for
basic programming concepts including if-statements, looping (determinate and indeterminate), creating user-dened procedures and
calling built-in procedures and functions to control the robot, which includes the use of parameters. The RoboMind microworld and program development environment has been designed to be easy to use and the robot animation provides a tangible output of the textual
representation of the program solution.
3.2.1.2. Adapted RoboMind. RoboMind was adapted to make it suitable for use by IT scholars in South African secondary schools. The adaptations were necessary as the original version of RoboMind did not support the same syntax as the Delphi programming language. Table 5
provides a comparison between the syntax of programming concepts implemented in Delphi, original RoboMind and adapted RoboMind.
The original RoboMind supports if-statements and while-loops, although the syntax differs from Delphi. The original RoboMind supports a
determinate looping structure, repeat(x), which repeats statements a specied (x) number of times. The for-loop is the determinate looping
structure supported by Delphi, whereby the starting and ending values of the looping iterator, are specied.
Table 5 indicates the adaptations to RoboMind and how they compare to the Delphi programming language. Note that the for-loop
implemented in Adapted RoboMind only requires the number of times to repeat the loop; the start value is set to 1 and is not user-specied
as in Delphi.
Limited support for the declaration and use of variables has also been implemented in the adapted version of RoboMind. Shortcomings of
the implementation of variables are that no data types are specied when variables are declared.
3.2.2. Scratch
Similar programming construct blocks are grouped together in colour-coded categories (D in Fig. 3), for example, motion, sound and
variables. Selecting a category lists all the associated blocks (C in Fig. 3), thus allowing users to drag required blocks to the work area (A
Fig. 2. RoboMind interface.
170
Table 5
Comparison of Programming Concept Statements for Delphi, original RoboMind and Adapted RoboMind.
Concept
Delphi
Original RoboMind
Adapted RoboMind
for-loop (determinate)
while-loop (indeterminate)
repeat-loop (indeterminate)
for i: 1 to 10 do
while (condition) do
repeat
until (condition);
if (condition) then
else
begin..end;
repeat(10)
while (condition)
e
for1to(10)
while (condition) do
repeat
until (condition);
if (condition) then
else
begin..end;
if-statements
Code block delimiters
if (condition)
else
{..}
in Fig. 3) to build a script. The program is executed in the window on the top right (B in Fig. 3). The script blocks are highlighted as the
program executes to indicate ow of code execution. In addition, users can set an option to execute blocks one step at a time (singlestep), making it easier for users to follow the execution of code and associate code blocks with actions of the graphical objects and
characters.
The advantages of using Scratch are that it is visually appealing and promotes active learning. Research by Malan and Leitner (2007)
showed a decrease in the number of students dropping or failing an introduction to computer science course when Scratch was used to
introduce programming concepts to the scholars. A drawback of using Scratch is that users may struggle to move to a traditional programming environment without an intermediate software tool to provide a link between the programming concepts introduced in Scratch
and the methods of implementing these concepts in a programming language (Resnick et al., 2009) where syntax becomes relevant.
Scratch does not enforce syntax rules due to the use of the drag-and-drop interface. Users, therefore, do not need to be concerned with
the overhead of programming language syntax. More focus can be given to the planning of programming solutions. Scratch is not opensource software, thus no adaptations could be made to Scratch. There are several differences between the implementation of programming concepts in Scratch and Delphi.
Multiple scripts can be created for different events in Scratch or when a name is broadcast. Broadcasting a specied name is the
equivalent of a procedure call in Delphi. The terminology used in Scratch differs from the Delphi language and syntax. In addition, procedure
parameters are not supported in Scratch. Global variables are used instead of passing parameter values to the procedure. Scratch supports
the implementation of a list, which is similar to a dynamic array in Delphi, but not a one-dimensional array. The length of the list is not
specied when the list is created. Scratch supports the insertion of values at specic positions in the list which would be similar to assigning
values to a static array at specic index positions.
Fig. 3. Creating a script in Scratch by dragging building blocks.
171
Fig. 4. Scratch: Implementation of arrays and looping.
The Delphi repeat-loop equivalent in Scratch is the repeat until block (Fig. 4). The loop terminates when the conditional statement is true,
which is equivalent to the implementation of the repeat-loop in Delphi. However, in Delphi the repeat-loop is post-conditional while in
Scratch it is pre-conditional. The forever if block in Scratch (Fig. 4) is equivalent to the while-loop in Delphi. The loop continues while the
conditional statement is true and the loop is pre-conditional, which is equivalent to the implementation of the while-loop in Delphi. The
wording on the block, however, does not correspond to the syntax of the while-loop in Delphi.
The Scratch equivalent of the Delphi for-loop is the repeat block (Fig. 4). However, only the number of times the code in the block must be
repeated is specied. The repeat block is thus equivalent to a Delphi for-loop in which the counter variable iterates from the value one (1) to
the integer value specied. No counter variable is dened in the Scratch repeat block. In addition, the wording used on the block is not
consistent with the Delphi for-loop syntax.
Scratch supports Delphi string operations such as joining words, determining the length of a string and returning specic characters
within a string. The wording used on the blocks differs from the syntax that is used in Delphi. Scratch also supports if-statements. A simple if
block as well as an if-else block are available for use in program solutions. Nested if-statements are also supported in Scratch.
Variables are supported although no data types are specied. Values are assigned to variables using the set block. Scratch is also able to
accept string input from the user and execute scripts triggered by events such as mouse-clicks or key presses, as in Delphi. Scratch supports
different types of program output. Objects or characters, referred to as sprites, can move, change colours, draw pictures and play music.
String output, which is the most common form of output in Delphi, can also be displayed.
3.2.3. B#
B# (Greyling, Cilliers & Calitz, 2006) is an iconic programming environment designed to simplify programming tasks by assisting novice
programmers with problem-solving strategies and the design of algorithms. B# uses a visual owchart approach that supports programming concepts such as assigning values to variables, conditions, loops, inputs and outputs.
Fig. 5 is a snapshot of the B# user interface currently in the execution state. The iconic owchart representation of the program running is
depicted (A in Fig. 5). The associated code generated by the system is also displayed (B in Fig. 5) allowing users to compare the owchart
created with the code generated. The code generated by B# is Object Pascal (Delphi) code; however, no graphical user interface components
are used. Input and output is received using readln and displayed using write/writeln, respectively. The resulting program is a command line
console program (D in Fig. 5) that accepts text input and displays text output.
Functionality is provided to control execution of the program, such as stepping through the code one line at a time or stopping the
execution at a particular point (E in Fig. 5). The current values assigned to variables are visible at all times during the execution of the code (C
in Fig. 5). The only programming concepts supported by B# that differ with regard to syntax from Delphi are input and output. B# interacts
with users using a console application for the main execution (Fig. 5), however, when tracing through the program Graphical User Interface
(GUI) forms are used to accept user input. The Delphi programming environment can be used to create console applications as well as
programs with a GUI. The subject content outline for programming in the IT subject requires IT scholars to create programs that interact with
users (input and output) using a GUI. IT scholars programming in Delphi are not required to and may not be explicitly taught to use a console
line application interface.
172
Fig. 5. Stepping through code created using a owchart in B#.
4. Experimental procedure
The purpose of the study was to determine the impact of a PAT on IT scholar understanding of programming concepts and perception of
the difculty of learning to program. The research questions are presented again as follows:
RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?
RQ2: What impact do the PATs have on IT scholar perceived difculty of programming?
RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specic programming concepts?
In order to determine the impact of the PATs, an evaluation between the use of a PAT and not using a PAT, was required. Experimental
research propositions are ideal to test the proposed PATs where the desired outcome would be improved understanding of programming
concepts. An experimental approach evaluates the results obtained from a control group in comparison to the results obtained by treatment
group: a group receiving a treatment or some form of intervention. In this study the treatment was the use of a PAT. In true experimental
designs, the two groups should be equivalent where participants are randomly assigned to control and treatment groups (Gribbons &
Herman, 1997; Kenny, 1975). However, in this study it was difcult to randomly assign scholars across different schools to control and
treatment groups. The different teachers and teaching environments were also a factor.
A variation of the non-equivalent groups design of the quasi-experimental research procedure (Gribbons & Herman, 1997; Kenny, 1975)
was thus followed where existing IT classes within the schools were used. Each school's participating treatment group received the same
PAT. Scholars within the same class could not be separated as control and treatment group scholars, thus the study spanned two academic
years (Table 6). Scholars in Grade 10 and Grade 11 of the rst year of the study constituted the control group, while scholars in Grade 10 and
Grade 11 of the second year of the study and who indicated that they used the PAT, constituted the treatment group. Scholars who received
the PAT but did not use it (self-reported) also formed part of the control group of the study. Grade 10 and Grade 11 scholars were evaluated
separately for certain of the data collected, due to the difference in programming knowledge and experience.
173
Table 6
Quasi-experimental research design for this study.
First year
Treatment
Control
Second year
Pretest
Pretest
PAT
Posttest
Posttest
The aim of the study was for IT scholars to have access to the PAT for an extended period during the school academic year, during which
time different programming concepts were taught to the scholars at school. The IT teacher did not provide any assistance with regards to the
use of the PAT. The PAT was evaluated as a tool that IT scholars could use in their own time to improve their understanding of programming
knowledge and skills. The study was completed in three different South African secondary schools offering IT as a subject from Grades 10 to
12 (one of the four schools that originally consented had only four participants in the treatment group). Each school received only one of the
three PATs. The IT teacher at each school had at least 10 years' experience teaching programming at secondary school level. The IT teacher
did not teach any programming content using the PAT.
4.1. Instruments and data collection
Three instruments were used to collect data in order to evaluate the impact of the PAT received on IT scholar understanding of programming concepts. The Perceived Difculty of Programming Questionnaire (PDPQ) was administered as a pre- and post-test to Grade 11
scholars in the control and treatment groups. The PDPQ consisted of items evaluated using a 7-point semantic differential scale. Participants
were required to evaluate their perceived difculty of IT programming concepts as well as their ability to solve programming problems,
create programming solutions and nd errors in code (RQ2). This questionnaire was only administered to Grade 11 scholars (Fig. 6). Grade 10
scholars could not evaluate their experience of programming at the beginning of the study as they had not yet had any programming instruction. Grade 11 control and treatment group participants completed this questionnaire.
Multiple-choice class tests were administered as class tests to the control and treatment group scholars (Fig. 6) after certain respective
programming topics had been completed in class. Grade 10 scholars were assessed on their understanding of if-statements, for-, while- and
repeat-loops. Grade 11 scholars were evaluated on their understanding of procedures, parameters and one-dimensional arrays. Multiplechoice class test results were used to assess the programming knowledge of IT scholars in order to evaluate the impact the use of the
PATs have on IT scholar knowledge of programming concepts (RQ1).
A PAT Evaluation Questionnaire was also administered to treatment group scholars (Fig. 6) to evaluate the PAT received. Treatment group
participants were required to rate the usefulness of the PAT in terms of understanding programming concepts supported. Feedback from the
PAT Evaluation Questionnaire was used to determine the IT scholar perceived usefulness of the PAT received with regards to the understanding of specic programming concepts (RQ3).
4.2. Data analysis
Quantitative data analysis methods, namely descriptive and inferential methods (Cohen, Manion, & Morrison, 2007), were used to
analyse the questionnaires and class tests administered to IT scholars who participated in this research study. Descriptive data analysis
Fig. 6. Quasi-experimental research strategy with questionnaire instruments.
174
Table 7
Number of participants in control and treatment groups.
Control
PAT
RoboMind
Scratch
B#
Total
Treatment
Grade 10
Grade 11
Total
Grade 10
Grade 11
Total
25
24
17
73
27
15
12
59
52
39
29
132
24
5
18
60
4
3
11
24
28
8
29
84
methods such as the calculation of the sample mean (M), as well as frequency distribution of responses, were used to report semantic
differential scale results. Descriptive statistics were also used to describe the relationship between different variables such as control or
treatment group and grade with respect to questionnaire responses in the presentation of results (Section 5).
The questionnaires administered contained items that used a 5- or 7-point semantic differential scale. All semantic differential scale data
referred to in this work was analysed as interval ratio data as only the extreme values had descriptions (e.g. Strongly disagree and
Strongly agree) on the questionnaires.
Inferential statistics were used to determine whether the differences in the results of the control and treatment groups were statistically
and/or practically signicant. Statistical signicance was determined at the 95% condence interval. Analysis of covariance (ANCOVA)
methods were used to examine the differences between the control and treatment groups where pre- and post-test data was available. Data
collected from the PDPQ was evaluated using this analysis. Analysis of variance (ANOVA) methods were used to evaluate the multiple choice
class results as only one dependent variable was available in order to determine signicant differences between control and treatment
groups. One sample t-tests were used to determine the difference between mean scores and the neutral value of rating scale data obtained
from the PAT Evaluation Questionnaire results.
5. Results
Table 7 indicates the number of scholars in the Grade 10 and Grade 11 control and treatment groups at each of the three schools,
evaluating one of the PATs. Grade 10 and Grade 11 scholars were evaluated separately as Grade 10 scholars had no prior programming
experience at the start of the control or treatment year and Grade 11 scholars had at least one year of programming experience (from Grade
10).
The results for each of the three PATs evaluated e RoboMind (Table 8), B# (Table 9) and Scratch (Table 10) e are presented. Each table
presents the results of different programming concepts addressed by the particular PAT, obtained using the three different data collection
instruments (Section 4). In each of the tables, M1 refers to the control group mean, while M2 refers to the treatment group mean.
The results obtained from the different questionnaire instruments completed by participating scholars after evaluating a PAT at each of
the schools were analysed to address each of the three research questions.
5.1. RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?
Overall, the results indicated that the evaluated PATs supported scholar understanding of looping concepts. Not all of the results were
statistically signicant. The multiple choice class test results (Table 8) indicated that RoboMind treatment group test means were higher
than the control group means for all of the programming concepts evaluated. One-way ANOVA analysis showed no statistically signicant
differences between the class test means. The results for RoboMind indicated statistically signicant mean score differences between the
control and treatment group for repeat- and while-loops in terms of scholar perceptions of difculty. The treatment group mean class test
mark for these two concepts was higher than the control group mean, although not statistically signicant. The results from all three instruments indicated RoboMind's support for the understanding of if-statements. For each of these concepts, RoboMind supports the correct
syntax and implementation (Section 3.2.1).
Table 8
RoboMind support of programming concepts results (*p < 0.05).
Concept
PDPQ
M1
M2
ANCOVA
M1
ANOVA
M2
Procedures
2.56
n 27
2.52
n 27
2.22
n 27
2.04
n 27
1.63
n 27
1.44
n 27
1.19
n 27
2.5
n4
3.52
n4
1.00
n4
1.00
n4
1.00
n4
1.00
n4
1.00
n4
F(1; 16) 0.18

p 0.674
F(1; 22) 1.87
p 0.186
F(1; 27) 6.49
p 0.017*
F(1; 27) 5.54
p 0.026*
F(1; 27) 3.49
p 0.073
F(1; 27) 2.76
p 0.108
F(1; 27) 1.65
p 0.210
88.15
90.00
n 27
n4
Not evaluated
F(1; 29) 0.07

p 0.794
2.50
t(5) 0.75
n6
p 0.49
Not evaluated
70.00
74.81
n 22
n 27
77.92
84.13
n 22
n 27
79.09
82.15
n 22
n 27
73.82
79.56
n 22
n 27
Not evaluated
F(1; 47)
p 0.433
F(1; 47)
p 0.247
F(1; 47)
p 0.532
F(1; 47)
p 0.213
2.43
n7
2.00
n6
2.38
n8
2.96
n 25
2.88
n 24
Parameters
repeat
while
for
if
Variables
Multiple choice
M2
PAT evaluation
0.63
1.38
0.40
1.60
T-stat
t(6) -0.93
p 0.386
t(5) 1.94
p 0.386
t(7) -1.05
p 0.329
t(24) -0.17
p 0.870
t(23) -0.55
p 0.588
M2
T-stat
2.00
n8
t(7) 1.53
p 0.170
2.71
n 17
2.65
n 17
2.71
n 17
3.00
n 21
2.70
n 20
t(16) -0.96
p 0.040*
t(16) -1.14
p 0.269
t(16) -1.00
p 0.332
t(20) 0.00
p 1.00
t(19) -1.14
p 0.267
175
Table 9
B# support of programming concepts results (*p < 0.05; **p < 0.01).
Concept
repeat
while
for
if
case
Variables
Input
Output
PDPQ
Multiple choice
M1
M2
ANCOVA
3.57
n7
2.88
n8
2.50
n8
1.88
n8
2.50
n8
2.00
n8
1.75
n8
2.13
n8
2.5
n8
2.13
n8
2.00
n8
1.75
n8
2.50
n8
1.50
n8
1.38
n8
1.38
n8
F(1; 12)
p 0.462
F(1; 13)
p 0.401
F(1; 13)
p 0.821
F(1; 13)
p 0.912
F(1; 13)
p 0.557
F(1; 12)
p 0.535
F(1; 13)
p 0.321
F(1; 13)
p 0.210
M1
0.58
0.75
0.05
0.01
0.36
PAT evaluation
M2
44.00
53.33
n 20
n 18
46.49
62.70
n 20
n 18
60.89
62.22
n 19
n 18
70.33
No marks
n 12
Not evaluated
ANOVA
M2
F(1; 36) 0.87

p 0.357
F(1; 36) 2.77
p 0.105
F(1; 35) 0.02
p 0.879
e
2.20
t(9) -2.45
n 10
p 0.037*
2.00
t(9) -3.00
n 10
p 0.015*
Error in questionnaire
0.41
1.06
1.74
2.00
n 15
2.00
n 12
2.60
n 15
2.40
n 15
2.33
n 15
T-stat
t(14) -4.18
p 0.001**
t(11) -2.87
p 0.015*
t(14) -1.47
p 0.164
t(14) -2.07
p 0.057
t(14) -2.32
p 0.036*
M2
T-stat
2.00
n9
1.78
n9
t(8) -2.45
p 0.040*
t(8) -3.35
p 0.010*
1.78
n9
1.67
n9
2.00
n9
2.11
n9
2.11
n9
t(8) -3.35
p 0.010*
t(8) -4.62
p 0.002**
t(8) -2.00
p 0.081
t(8) -1.57
p 0.154
t(8) -1.57
p 0.154
Scratch provides support for the looping concepts, particularly for-loops, as indicated by Grade 10 and Grade 11 treatment group scholar
evaluations (Table 10) of the difculty of programming concepts (PDPQ) as well as the neutral mean results from the evaluation of the PAT
(PAT Evaluation Questionnaire). The results were not statistically signicant. In Scratch, the for-loop characteristic of nite repetitions is
demonstrated using a repeat block even though the syntax does not match (Section 3.2.2). The multiple choice class test results for the
Scratch treatment group did not provide any indication that scholar understanding of the programming concepts improved.
B# treatment group scholars have higher mean ratings than control group scholars for the looping concepts evaluated using the multiple
choice test results (Table 9). However, none of the mean differences were statistically signicant and treatment group scholars disagree that
B# assisted with the understanding of the looping concepts. Overall, there was no indication from the results that B# assisted with scholar
understanding of programming knowledge.
5.2. RQ2: What impact do the PATs have on IT scholar perceived difculty of programming?
Scholars were required to rate (on a 7-point semantic differential scale) their perception of the difculty of each of the programming
concepts evaluated when using the Perceived Difculty of Programming Questionnaire (PDPQ). A rating of one indicated that the scholars
perceived the concept as extremely easy to understand, while a rating of seven was categorised as extremely difcult. A comparison of the
mean scores of the control and treatment group post-test results for the PDPQ indicate that scholars in the RoboMind treatment group
perceived all the programming concepts to be easier than the control group did (Table 8), except for the use of parameters. The differences
between the control and treatment group means for the repeat- (F(1; 27) 6.49, p 0.017) and while-loop concepts (F(1; 27) 5.54,
p 0.026) were statistically signicant.
The mean ratings (PDPQ) for input, if-statements, string handling and procedures indicated that the Scratch Grade 11 treatment group
respondents perceived these concepts to be more difcult than the Scratch Grade 11 control group respondents (Table 10). A two-way
Table 10
Scratch support of programming concepts results.
Concept
String handling
1-d arrays
Procedures
repeat
while
for
if
Variables
Input
Output
PDPQ
Multiple choice
M1
M2
ANCOVA
1.80
n 15
1.87
n 15
2.50
n 15
2.27
n 15
1.93
n 15
1.73
n 15
1.27
n 15
1.33
n 15
1.27
n 15
1.47
n 15
2.00
n3
1.33
n3
3.30
n3
1.67
n3
1.33
n3
1.33
n3
1.68
n3
1.33
n3
1.33
n3
1.33
n3
F(1; 15)
p 0.400
F(1; 15)
p 0.380
F(1; 12)
p 0.230
F(1; 14)
p 0.534
F(1; 15)
p 0.462
F(1; 15)
p 0.382
F(1; 15)
p 0.317
F(1; 15)
p 0.444
F(1; 15)
p 0.889
F(1; 15)
p 0.914
M1
0.75
Not evaluated
0.82
97.67
n 15
92.00
n 15
70.00
n 20
71.43
n 20
62.84
n 19
72.00
n 20
Not evaluated
1.60
0.41
0.57
0.81
1.07
0.62
0.02
0.01
PAT evaluation
M2
86.67
n2
75.00
n2
46.67
n9
53.97
n9
59.63
n8
68.00
n8
ANOVA
PATs received after test

F(1; 24)
p 0.202
F(1; 24)
p 0.257
F(1; 24)
p 0.422
F(1; 24)
p 0.999
F(1; 24)
p 0.430
1.78
1.35
0.61
0.00
0.65
M2
T-stat
2.00
n4
1.50
n2
2.00
n2
3.00
n4
2.50
n4
3.00
n5
3.60
n5
3.00
n5
3.20
n5
3.00
n5
t(3) -1.41
p 0.252
t(1) -3.00
p 0.205
t(1) -0.50
p 0.500
t(3) 0.00
p 1.00
t(3) -0.77
p 0.495
t(3) 0.00
p 1.00
t(4) 1.18
p 0.305
t(4) 0.00
p 1.00
t(4) 0.30
p 0.78
t(4) 0.00
p 1.00
176
ANCOVA analysis did not indicate statistically signicant differences between the control and treatment group mean ratings for any of the
programming concepts supported (Table 10). The concepts with the lowest treatment group mean (M2) ratings (n 3) and with higher
corresponding control group (M1) ratings (n 15), were the for-loop (M1 1.73, M2 1.33), while-loop (M1 1.93, M2 1.33) and array
concepts (M1 1.87, M2 1.33). Only three treatment group scholars completed the PDPQ, affecting the reliability of the comparison between control and treatment group results. The multiple choice class test mean scores for the treatment group scholars were also lower than
the control group scores for all of the programming concepts supported (Table 10), supporting the perceptions of the participants. None of
the multiple choice class test differences are statistically signicant. An evaluation of the difference between the Scratch treatment and
control group results indicated that treatment group scholars had more difculty understanding programming concepts than control group
scholars (PDPQ and multiple choice test results). The IT teacher was replaced after the rst term of the treatment year. The results suggest
that the scholars' transition to a different IT teacher and teaching environment inuenced the poorer performance (tests) and greater
perceived difculty of programming concepts by treatment group scholars. Based on the results, Scratch was not able to support scholars
who did not have a stable teaching environment.
B# treatment group scholars perceived (PDPQ results) looping concepts to be more difcult when comparing mean ratings to the other
programming concepts evaluated (Table 9). Treatment group scholars, however, perceived the difculty of most of the programming
concepts to be easier to understand, compared to control group scholars. The differences were not statistically signicant.
5.3. RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specic programming
concepts?
The usefulness of the PAT in terms of assisting with the understanding of the programming concepts supported was evaluated using a
semantic differential scale where a rating of 1 indicates that scholars strongly disagreed and a rating of 5 indicates that scholars strongly
agreed that PAT assisted their understanding of the concept. The mean evaluation ratings for RoboMind (Table 8) for all of the concepts were
less than, or equal to the neutral value, 3. One sample t-tests indicated that none of the programming concept mean values were statistically
signicantly different from the neutral value. Evaluation results indicated the highest mean ratings for the support provided by RoboMind
for the understanding of if-statements (M2 2.96).
The Grade 10 and Grade 11 treatment group scholars (n 10) provided qualitative feedback about what they liked and disliked about
RoboMind (PAT Evaluation Questionnaire). Ninety percent of participants indicated that RoboMind helped them to practise basic programming concepts. Forty percent of scholars indicated that they disliked that the syntax of certain of the programming concepts supported
in RoboMind (Section 3.2.1) was slightly different to coding in Delphi. Forty percent of the responses indicated dislike for the fact that
RoboMind supports a limited number of programming concepts.
The Scratch Grade 10 and 11 treatment group scholar mean evaluation ratings (Table 10) for input, output, variables, if-statements, forloops and repeat-loops were greater than or equal to the neutral value, 3. One-sample t-test analysis indicated that none of the ratings were
statistically signicant from the neutral value. Many scholars did not complete the questionnaire due to administrative difculties. The
number of scholars who evaluated the usefulness of Scratch to understand the programming concepts supported, ranged from two to ve
scholars, depending on the concept (Table 10).
The usefulness of B# with regard to the understanding of the programming concepts supported, was evaluated twice (Table 9) by Grade
10 and Grade 11 treatment group scholars in the academic year. In both evaluations, the results indicated that scholars did not agree that B#
was useful to assist with the understanding of repeat-loops, while-loops, if-statements and case-statements. For all of these concepts, one
sample t-test analysis indicated that the mean rating score was statistically signicantly different from the neutral value, 3. The mean rating
for output in the rst evaluation was also statistically, signicantly different from the neutral value.
6. Discussion
The aim of the research study was to evaluate the impact of three PATs, namely an adapted version of RoboMind, Scratch and B#, on
South African IT scholar understanding of introductory programming concepts. The impact of the PATs on IT scholar understanding of
programming concepts was evaluated by addressing three research questions (Section 4). Each of the research questions were evaluated by
comparing the means of control and treatment group multiple-choice class test assessments and perceived difculty of programming. IT
scholars using the PAT (treatment group) also evaluated their perception of the impact the PAT had on their understanding of programming
concepts supported in the PAT.
The ANOVA/ANCOVA analysis results for all three PATs indicated that there were no statistically signicant differences between the
means of control and treatment groups for the multiple choice class tests although the treatment group means were higher than control
group means for scholars using the adapted RoboMind and B#. The use of the three PATs did not have a signicant impact e positive or
negative e on IT scholar knowledge of programming concepts, addressing RQ1 (Section 5.1). This is in line with ndings by Levy et al. (2001).
Quantitative results may show an improvement, however the use of a PAT, specically with animation, does not necessarily improve scholar
performance in long term use. In a study by Al-Imamy et al. (2006), participants in the treatment group performed signicantly better in
examination assessment. However, the use of the web-based tool was included by instructors in the teaching environment.
This study also aimed to determine the impact of the PATs on IT scholar perceived difculty of specic programming concepts (Section
5.2). The Perceived Difculty of Programming Questionnaire (PDPQ) results indicated that the adapted RoboMind assisted with the understanding of repeat- and while-loops, as perceived by IT scholars. The difference between control and treatment group means was statistically signicant for these two concepts only. The adapted RoboMind thus had a positive impact on the IT scholar perceived difculty of
repeat- and while-loops. IT scholar feedback from open questions identied the syntax of RoboMind to be an issue which may have affected
the understanding of the other concepts. The syntax of for-loops and variables in RoboMind differed from the Delphi programming language
syntax (Section 3.2.1). Future improvements to the adapted RoboMind to match the syntax more closely or exactly to the Delphi programming language could result in more programming concepts being perceived as easier to understand by IT scholars.
177
There were no statistically signicant PDPQ results for B# (Table 9) or Scratch (Table 10). Although qualitative feedback from scholars
indicated that Scratch was easy to use, scholars disliked that it was too simple and without purpose. Scholars may not have been motivated
to use Scratch which may have, in turn, impacted the effectiveness of Scratch to assist scholar understanding of programming concepts.
Scholars evaluating B# indicated that they liked the automatic code generation and that the graphical owchart representation explained
programming concepts, particularly loops and if-statements. However, scholars disliked that the generated code differed from Delphi. The
only difference in syntax was with regards to input and output (Section 3.2.3).
An evaluation of participating treatment group IT scholar perception of the usefulness of the PATs with respect to improving their
understanding of programming concepts (Section 5.3) provided mixed results. Feedback for the usefulness of adapted RoboMind and
Scratch did not provide statistically signicant results. Scholars indicated, however, that RoboMind was most useful for the understanding of
if-statements. Scholars using B# did not agree that B# was useful to improve their understanding of programming concepts (results were
statistically signicant). This is a contradiction to qualitative scholar feedback received stating that the owchart representation assisted
their understanding. The impact of a differing syntax on scholar understanding may have had a negative inuence on their perception of the
PAT. The overall feedback indicates that the syntax of the programming environment was important for participants. The results of studies
by de Raadt (2008), Malan and Leitner (2007) and Rongas et al. (2004) indicate that explicit use of a PAT and focus on specic programming
knowledge and skills may be more successful than self-regulated use of a PAT that is not part of the classroom teaching approach.
Another result of the PAT as part of self-regulated learning and not enforced as part of the classroom teaching environment is that not all
participants continued to use the PAT in this study. This was a similar nding in the study by Levy et al (2001) evaluating Jeliot, where
weaker scholars were overwhelmed and stronger scholars did not think the PAT was needed.
7. Conclusions and future work
Quantitative analysis indicated that the PATs had no impact on IT scholar understanding of programming concepts in general. Qualitative
feedback from scholars identied several aspects that could be addressed by future PATs developed for use by IT scholars in South African
schools. The syntax of the programming language used must match the Delphi programming language as closely as possible and the PAT
should support the implementation of as many programming concepts as possible. The programming environment should be easy to use,
however, not too simple as users will not be challenged or motivated to use the PAT.
The research was limited by participating school environments such as IT teacher workload and small IT classes. The results of these
limitations were incomplete questionnaire and class test data. In spite of these limitations, the research ndings and contributions have
formed the basis for future work. Extending the study by increasing the sample size of participating schools and the inclusion of usability
studies would provide additional data making the results more generalisable. Inclusion of the PAT as part of the teaching environment may
allow scholars to be more condent using the PAT. Further adaptations to RoboMind as well as the development of a new PAT specically
designed for IT scholars based on the nding of the research, are also possible. The contributions of this study such as the identication of
programming knowledge and skills, selection criteria for PATs and the evaluation results could be generalised and made applicable to
support novice programmers in general.
References
Al-Imamy, S., Alizadeh, J., & Nour, M. A. (2006). On the development of a programming teaching tool: the effect of teaching by templates on the learning process. Journal of
Information Technology Education, 5, 271e283.
Alexander, P. M., & Twinomurinzi, H. (2012). Changing career choice factors as the economic environment changes. In Proceedings of SAICSIT'12, October 1-3 2012, Pretoria
South Africa (pp. 295e305).
Areias, C., & Mendes, A. (2007). A tool to help students to develop programming skills. In International Conference on Computer Systems and Technologies (CompSysTech07).
Baldwin, L. P., & Kuljis, J. (2000). Visualisation techniques for learning and teaching programming. Journal of Computing and Information Technology, 4, 285e291.
Calitz, A. (2010). A model for the alignment of ICT education with business ICT skills requirements. DBA, Nelson Mandela Metropolitan University.
Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6th ed.). Routledge.
Department of Basic Education. (2011). Curriculum and assessment policy statement. Grades 10e12. Information Technology. Available at: http://www.education.gov.za/
LinkClick.aspx?leticketjIbU9d%2B63O4%3D&tabid420&mid1216 Accessed 13.08.13.
Garner, S. (2007). A program design tool to help novices learn programming. In ICT: Providing choices for scholars and learning (pp. 321e324). Ascilite Singapore 2007.
Greyling, J., Cilliers, C., & Calitz, A. (2006). B#: The development and assessment of an iconic programming tool for novice programmers. In 7th International Conference on
Information Technology Based Higher Education and Training (ITHET06) (pp. 367e375).
Gribbons, B., & Herman, J. (1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5(14). Available at: http://PAREonline.net/getvn.asp?
v5&n14 Accessed 13.08.13.
Guzdial, M. (2004). CS education research, chap. Programming Environments for Novices.
Havenga, M., & Mentz, E. (2009). The school subject information technology: a South African perspective. In Proceedings of SACLA'09 (pp. 77e83). Mpekweni Beach Resort,
South Africa.
Heines, J., & Schedlbauer, M. (2007). Teaching object-oriented concepts through GUI programming. In Eleventh Workshop on Pedagogies and Tools for the Teaching and
Learning of Object Oriented Concepts, held at the 21st European Conf. on Object-oriented Programming. Berlin, Germany.
Hug, S., Guenther, R., & Wenk, M. (2013). Cultivating a K12 computer science community: a case study. In Proceedings of SIGCSE13, March 6-9, 2013, Denver, Colorado, USA.
Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: a taxonomy of programming environments and languages for novice programmers. ACM Computing
Surveys, 37(2), 88e137.
Kenny, D. A. (1975). A quasi-experimental approach to assessing treatment effects in the nonequivalent control group design. Psychological Bulletin, 82(3), 345e362.
Koorsse, M. (2012). An evaluation of programming assistance tools to support the learning of IT programming: A case study in South African secondary schools. Doctoral thesis,
Nelson Mandela Metropolitan University.
Koorsse, M., Calitz, A. P., & Cilliers, C. B. (2010). Programming in SA Schools: the Inside Story. In Proceedings of SACLA. South Africa: Pretoria.
lling, M. (1999). The problem of teaching object-oriented programming, Part 1: Languages. Journal of Object-Oriented Programming, 11(8), 8e15.
Ko
Kunkle, W. M. (2010). The impact of different teaching approaches and languages on student learning of introductory programming concepts. Thesis. Drexel University. Doctor of
Philosophy.
Lahtinen, E., Ala-Mutka, K., & J
arvinen, H. (2005). A study of the difculties of novice programmers. In Proceedings of the 10th Annual SIGCSE Conference on innovation and
Technology in Computer Science Education (pp. 14e18). Caparica, Portugal: ACM.
Levy, R. B.-B., Ben-Ari, M., & Uronen, P. A. (2001). An extended experiment with Jeliot 2000. In Proc. First International Program Visualization Workshop (pp. 131e140). Pavoo,
Finland: University of Joensuu Press.
Lister, R., Adams, E., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., et al. (2004). A multi-national study of reading and tracing skills in novice programmers. In Working Group
Reports From ITiCSE on innovation and Technology in Computer Science Education (pp. 119e150). Leeds, United Kingdom.
178
Ma, L., Ferguson, J., Roper, M., Ross, I., & Wood, M. (2008). Using cognitive conict and visualisation to improve mental models held by novice programmers. In Proceedings of
the 39th SIGCSE Technical Symposium on Computer Science Education (pp. 342e346).
Malan, D., & Leitner, H. (2007). Scratch for budding computer Scientists. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (SIGCSE07) (pp.
223e227).
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., et al. (2007). A survey of literature on the teaching of introductory programming. In Working group
reports on ITiCSE on Innovation and technology in computer science education, ITiCSE-WGR 07 (pp. 204e223). New York, NY, USA: ACM.
Pendergast, M. (2006). Teaching introductory programming to IS students: Java problems and pitfalls. Journal of Information Technology Education, 5, 491e515.
Porta, M., Maillet, K., & Gil, M. (2010). Dec-cs: the computer science declining phenomenon. In Proceedings of the World Congress on Engineering and computer science 2010
(WCECS2010) (Vol. 2). San Francisco, USA.
de Raadt, M. (2008). Teaching programming strategies explicitly to novice programmers. Doctoral dissertation. University of Southern Queensland.
Resnick, M., Maloney, J., Monroy-Hern
andez, A., Rusk, N., Eastmond, E., Brennan, K., et al. (2009). Scratch: programming for all. In Communications of the ACM, 52 pp. 60e67).
Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: a review and discussion. In Computer science education (Vol. 13, pp. 137e172).
RoboMind. (2009). RoboMind website. Available at: http://www.robomind.net/en/index.html Accessed 13.03.10.
Rongas, T., Kaarna, A., & Kalvianen, H. (2004). Classication of computerized learning tools for introductory programming courses: learning approach. In Proceedings of the
IEEE International Conference on Advanced Learning Technologies (ICALT04) (pp. 678e680).
Shuhidan, S., Hamilton, M., & D'Souza, D. (2009). A taxonomic study of novice programming summative assessment. In Proceedings of the Eleventh Australasian Conference on
Computing Education (Vol. 95, pp. 147e156). Darlinghurst, Australia: Australian Computer Society, Inc, ISBN 978-1-920682-76-7. ACE 09, 95.
Strieber, A. (2011). The 10 best jobs of 2011. Available at: http://www.careercast.com/jobs-rated/10-best-jobs-2011 Accessed 10.08.11.
Wilson, C., Sudol, L., Stephenson, C., & Stehlik, M. (2010). Running on empty: The failure to teach K-12 Computer Science in the digital age. Tech. Rep. ACM.
Wood, D., Bruner, J., & Ross, Gsss (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry and Allied Disciplines, 17, 89e100.

Programming Assistance Tools

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Programming Assistance Tools

Transféré par

Droits d'auteur :

Formats disponibles

Computers & Education 82 (2015) 162e178

Contents lists available at ScienceDirect

Computers & Education

Programming assistance tools to support the learning of IT

* Corresponding author. Tel.: 27 41 504 3925.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 1. Knowledge and skills required by a programmer.

Assist with the understanding of code execution

Criteria for PAT

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 2. RoboMind interface.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 3. Creating a script in Scratch by dragging building blocks.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 4. Scratch: Implementation of arrays and looping.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 5. Stepping through code created using a owchart in B#.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Fig. 6. Quasi-experimental research strategy with questionnaire instruments.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

F(1; 16) 0.18

F(1; 29) 0.07

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

F(1; 36) 0.87

PATs received after test

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

M. Koorsse et al. / Computers & Education 82 (2015) 162e178

Vous aimerez peut-être aussi