Running head: VR PEDESTRIAN SAFETY APP

CALIFORNIA STATE UNIVERSITY

MONTEREY BAY

Virtual Reality Pedestrian Safety Training Module

CAPSTONE Report
Submitted in partial satisfaction of requirements of the degree of
MASTER OF SCIENCE in
Instructional Science and Technology

Justin Baffico
December 9th, 2017
Capstone Approvals: (At least one advisor and capstone instructor should approve)

___________________________ _____________
Advisor Name Signature Date

___________________________ _____________
Capstone Instructor Name Signature Date
VR PEDESTRIAN SAFETY APP 2

Table of Contents
Table of Contents 2

Executive Summary 3

Introduction 4
Background 4
Problem Description 4
Audience 4
Literature Review 5

Solution Description 8
Goals 8
Learning Objectives 8
Solution to Gaps 9
Instructional Solutions and Reasoning 10
Media Components 12
Challenges 12

Procedure 14
Prototype 14
Process 14
Final Steps 14

Resources 15

Timeline 16

Evaluation 17
Formative Evaluation 17
Recommendations 18
Summative Evaluation 20

Conclusion 22
VR PEDESTRIAN SAFETY APP 3

Executive Summary
Proper street crossing behavior is a necessary skill for children. Children often omit
proper behavior techniques while crossing the street, increasing the likelihood of a pedestrian
accident. Although formal pedestrian safety programs have had success, far too many children
do not experience formal street safety training. This capstone provides pedestrian safety training
for children ages six to twelve, through the use of an asynchronous virtual reality application.
Although research supports virtual reality (VR) as a viable mode of pedestrian safety
training, and VR technology is available on millions of smartphones across the globe, there has
not been a commercially available VR pedestrian safety training application available on capable
devices. The goal of this project was to develop a virtual reality pedestrian safety training
application that will help children learn the behaviors needed to properly cross the street. The
application features a tutorial mode that focuses on five key behavior-based objectives essential
to crossing the street, an arcade mode that gamifies practice of the objectives, and an exam mode
that tests mastery of the objectives.
Resources and development focused on creating a tutorial instructional videos course, a
practice Arcade mode, and an Exam mode with fully immersive VR Environments. Focus
was also directed to develop resources for parents or guardians of app users, and for testing the
usability and learning effectiveness of the application. Since the project was developed for a
platform of which I had no prior experience, I was limited in my confidence and capability of
developing virtual reality software. I was able to overcome my limitations, and in doing so,
learned the skills necessary to develop other instructional materials for virtual reality.
After completion of a working beta prototype, I was able to perform both summative and
formative evaluations. The formative evaluation included a usability test, as well as a pre and
post-test survey. The summative assessment included a paired t-test of pre-test and post-test
assessments on the learning objectives. Now, upon satisfactory evaluation of the application, the
project is available for free download on the Google Play Store marketplace.
VR PEDESTRIAN SAFETY APP 4

Introduction
Background
Pedestrian safety is an ongoing concern for society. Children are one of the most
vulnerable demographics in pedestrian-vehicle traffic accidents. Although many pedestrian
safety trainings are available for delivery, a high number of children either receive ineffective
training, or do not receive training at all. The development of a virtual reality pedestrian safety
training application, and the release of the application for access on major smartphone
marketplaces as a free download, creates opportunities for children to receive pedestrian safety
training they would have otherwise not received.
Problem Description
Motor vehicle accidents are a leading cause of death for children ages three to thirteen.
One in every five of children killed in that range happens to be a pedestrian (CDC, 2015). Part of
this problem can be attributed to a lack of procedural safety knowledge. Unfortunately, more
than half of young children observed crossing streets engage in unsafe street-crossing behavior
(Percer, 2009). Correcting these behaviors before children attempt to cross the street on their
own should be a major social priority. Children ages six to twelve will benefit most from the
training materials developed.
Since the empirical knowledge required of pedestrians involves dynamic spatial
processing, virtual reality training modules have the potential to be more effective than computer
or video based trainings. Virtual reality is often more engaging than other methods of training,
making it more likely for a child to participate in self-directed training. Children have increased
accessibility to smartphone technology. Virtual reality applications are more affordable to
consumers due to recent development projects (Gear VR, Google Cardboard, etc.). However,
there was not a pedestrian safety course available on the Google Play Store or The App Store
before this application, Crosswalk VR. Desired performance outcomes included a lower
likelihood of child pedestrian accidents, improved pedestrian safety behaviors, and increased
dialogue about pedestrian safety between children and their parents or guardians.
Audience
The target learning population for this project includes children ages six to twelve. Many
children within this age range have not experienced formal pedestrian safety training before.
VR PEDESTRIAN SAFETY APP 5

Although not recommended, children within this age range are likely to start crossing the street
on their own for the first time. It is important for children to get an opportunity to learn the
concepts and behaviors associated with crossing the street properly before they attempt to cross
the street. Children have more frequent access to their own smartphones, and will continue to
gain access, as trends suggest. A 2013 study showed seventy-two percent of American children
age eight and under have used a mobile device for some type of media activity, such as playing
games, watching videos, or using apps, an increase from thirty-eight percent in 2011 (Common
Sense Media, 2013). Having a training application available to this population increases
opportunities for pedestrian safety training. Parents of the target learners can also facilitate the
acquisition of the module by downloading it on their smartphone device, and allowing their child
time to use it. The application is not targeted to children younger than six, since research with
other multimedia training platforms suggests no improvements in this demographic. This
application also appeals to young gamers, and explorers of VR. Since VR is a popular
technology among this demographic, it will have motivational appeal over non VR pedestrian
safety applications, videos, books, or other types of training modules.
The project is marketed as an educational game that improves child pedestrian behaviors,
available on the Google Play Store. This project initially targets learners who speak English, and
who live in a country where driving is done on the right side of the road. The project can later be
expanded to include speakers of other languages, and those who live in areas where people drive
on the left side of the road. The target audience for the formative and summative assessments
was the students from my school district in Oakley, California. It was easy for me to gain access
to students, as their interest was already grounded in virtual reality technology from our
coursework in Computer Science class. A group of students from my class, combined with a
group of students from a Maker Lab intervention class at one of our elementary schools,
represented a population that resembled the target audience for the application (children ages six
to twelve).
Literature Review
Current literature and examination indicates there is not another commercially available
virtual reality pedestrian safety training module. There are several non-VR pedestrian safety-
training modules available for download on the App store and Google Play store. Of these
available training modules, learning effectiveness data is not available. However, there are
VR PEDESTRIAN SAFETY APP 6

numerous other studies of pedestrian safety training modules that have prominent research to the
learning targets of this capstone. The following section identifies different types of pedestrian
safety trainings, explores their relevance, and describes the gaps that need to be addressed for
each training type.
Virtual reality training. There have been several studies indicating that virtual reality
pedestrian training can be effective. Virtual reality happens to be one of the most inclusive
methods for delivering training. Research indicates that it is also an effective platform for
teaching pedestrian safety to children with autism (Josman, Ben-Chaim, Friedrich, & Weiss,
2008). Although this method offers promising results, there is not other commercially available
virtual reality training module.
Other Delivery Methods. There have been a variety of pedestrian safety training
methods developed through different mediums. Research across different mediums indicates that
trainings can be effective for behavioral changes of street crossers. As new technologies emerge,
the training takes the shape in the new technology media. If virtual reality training incorporates
the best practices from the existing trainings, it can have a wide reaching audience, especially as
mobile VR use gains popularity. This section explores different training methods that have been
used to implement pedestrian safety behaviors
Parent administered training. Parents are capable of providing effective pedestrian
safety training when specifically instructed to do so (Schwebel, Davis, & ONeal, 2012).
Unfortunately, parents do not often use crossing the street as an opportunity to instruct the child
on proper behaviors. A study found that only 6% of parents talk to their children while crossing
the street (Zeedyk & Kelly, 2003).
In-school training. In-school pedestrian trainings are found to be highly effective.
Schools offer a setting to gather large numbers of children and place them in front of trained
educators. Children are also accustomed to an environment of learning while in school.
However, not all schools participate in pedestrian safety training. Crosswalk VR can be used to
supplement an in-school training by having a teacher facilitate learning while the application is
being used.
Street side training. Street side training at actual pedestrian crosswalks offers short-term
improvement in safe pedestrian behaviors. There is still a question to the effectiveness of long-
VR PEDESTRIAN SAFETY APP 7

term retention (Barton, Schwebel, & Morrongiello, 2006). Street side training also requires an
available trainer. A low percentage of children have street side training available to them.
Community intervention. Community programs using advertising campaigns can
increase pedestrian safety. Research conducted after a community intervention had taken place
showed that children are more knowledgeable about pedestrian safety, and that children display
safer pedestrian behaviors when crossing streets (Schwebel, Davis, & ONeal, 2012; Preusser &
Bloomberg, 1984). A gap persists for communities that do not have pedestrian safety
intervention programs in place.
Video training. Video training offers an alternative to other time and labor-intensive
trainings. Research suggests that video training increases a childs knowledge of pedestrian
safety, and modestly improves their behaviors (Preusser & Lund, 1988). Younger children seem
have more difficulty learning from video trainings (Zeedyk, 2003).
Interactive multimedia training. There are different interactive multimedia trainings
emerging online, and as application downloads for smartphones. One study suggests that
knowledge of pedestrian safety had increased after children participated in WalkSafe, a CD-
ROM based pedestrian safety training (Glang, Noell, Ary, & Swartz, 2005). A downfall of
interactive multimedia trainings is that they lack the ability to simulate complex spatial dynamics
that are involved when actually crossing the street.
Evaluation of gaps. There are gaps between what is and what should be for every
type of pedestrian safety training listed above. Gaps in parent administered training, in-school
training, street side training, and community interventions can be filled by those in the position
to implement these types of trainings (i.e. parents, teachers, school administrators, community
leaders). There are several gaps that can be met through instructional design. These gaps include
developing effective training modules for any type of pedestrian training. Research indicates that
virtual reality is one of the most effective platforms for pedestrian safety training. There is a gap
in this type of training because there is not already an available VR app for pedestrian safety.
The gap that needs to be filled within this category of training is met by the development of a
training module, and making this module widely available.
VR PEDESTRIAN SAFETY APP 8

Solution Description
The proposed solution to meet the mentioned gaps was to design and develop an
asynchronous virtual reality pedestrian safety training application. The application was intended
to give children several foundational skills need to cross the street, and create dialogue between
parents and children about street safety. The application focuses on five targetable learning
objectives. The different instructional strategies, including instructional videos, immediate
feedback, guided practice environments, and examination, will be used to develop participants
proficiency towards mastery of the learning objectives. Instructional strategies were selected
based on evaluation of instructional theories, including operant conditioning, programmed
instruction, and gamification.
Goals
The overall goal for any pedestrian safety training application is to reduce risk of injury
or death for child pedestrians. A VR pedestrian safety-training module has the potential to reach
a wide audience. Based on an evaluation of the needs and gaps in the area of pedestrian safety
for children, the goals for the project aim to achieve the following:
Have the capability to reach millions of users through placement on a major smartphone
application marketplace.
Provide an immersive virtual reality environment that will engage learners to voluntarily
complete the training module (self-directed).
Create dialogue between children and their parents or guardians about the importance of
using safe behaviors when crossing the street.
Teach behaviors that will decrease the likelihood for a child to be involved in a
pedestrian accident.
Learning Objectives
The above-mentioned learner goals are reinforced by one terminal objective broken down
into five enabling objectives. Although there are more skills a learner must have to fully
participate in crossing the street safely, the learning objectives were chosen because they provide
a good foundation for children to have when they begin to practice crossing the street with adult
supervision. A person using the application can fail or succeed at each enabling objective,
identifying areas to return to on further use of the application. All objectives require spatial
VR PEDESTRIAN SAFETY APP 9

interaction within the VR application. Since research shows that VR pedestrian safety
applications translate to actual acquired pedestrian safety skills, learners will be assessed initially
within a virtual environment. A learner will progress through five levels that target specific
behaviors, one for each enabling objective. The terminal objective and enabling objectives are
described below:
Terminal Objective: Given virtual reality pedestrian traffic scenarios, children ages six to
twelve will be able to demonstrate proper pedestrian safety behaviors with 100%
accuracy after one hour of using the VR Pedestrian Safety application.
Enabling Objective 1: Given a virtual reality street simulation, learners will be
able to identify pedestrian crosswalks with 100% accuracy after one hour of using
the VR Pedestrian Safety application.
Enabling Objective 2: Given a virtual reality street simulation, learners will be
able to initiate pedestrian crossing signals when available with 100% accuracy
after one hour of using the VR Pedestrian Safety application.
Enabling Objective 3: Given a virtual reality street simulation, learners will be
able to cross when crossing signals indicate it is appropriate with 100% accuracy
after one hour of using the VR Pedestrian Safety application.
Enabling Objective 4: Given a virtual reality street simulation, learners will be
able to look left, right, then left again before crossing street with 100% accuracy
after one hour of using the VR Pedestrian Safety application.
Enabling Objective 5: Given a virtual reality street simulation, learners will be
able to cross the street while avoiding traffic with 100% accuracy after one hour
of using the VR Pedestrian Safety application.
Solution to Gaps
The application addresses several gaps in pedestrian safety. Since VR pedestrian safety
has proven effective in its early forms, publishing the application addresses the gap of a lack of
access to VR pedestrian safety training. Another gap is in the quality of instruction for existing
pedestrian safety trainings. This app includes instructional strategies that are effective at
changing behaviors in relation to crossing the street. Since street crossing also deals with three-
dimensional spatial relationships, this form of training offers simulations that other forms of
training (wrote, computer based, etc.) are unable to offer. Trainings offered at actual
VR PEDESTRIAN SAFETY APP 10

intersections and streets can be dangerous if traffic isnt controlled. This app offers an alternative
to street side trainings but is not a substitute for continued, adult supervised training.
Instructional Solutions and Reasoning
Instruction within the app takes form in three different game modes. The game modes
offer a way to progress through the five enabling objectives. The three game modes include
Tutorial Mode, Arcade Mode, and Exam Mode. Specific instructional strategies, and reasoning
based on instructional theories are described below for each game mode.
Tutorial Mode. This game mode links from the application to a Google Form. The
Google Form takes the user through five lessons, one lesson for each of the learning objectives.
Once a user starts a lesson, they are shown an instructional video. The video will state the
desired behavior in a real life traffic scenario, demonstrate how to replicate the desired behavior
(screencast of gameplay), and prompt the user to complete the behavior accurately. The user will
then be faced with a clarifying question that must be answered correctly before the user is able to
move on to subsequent lessons. If a user satisfies the objective by demonstrating knowledge of
the desired behavior, the user will move on to the next lesson. After completing all five
scenarios, the user will is taken back to the menu screen. The users score will be shown
underneath the level they have just completed.
Reasoning behind tutorial mode design is derived from several instructional theories and
methods, including operant conditioning and programmed instruction. Operant conditioning is
based on the notion that behaviors can be learned and controlled by consequences. The user is
rewarded with positive reinforcement by being able to continue on to the next lesson. The user is
also able to navigate back to the arcade and exam modes once the tutorial mode is complete.
Programmed instruction is a method that presents users with information in a series of steps. The
learners are able to work through these steps (tutorial lessons) at their own speed. They are also
shown immediate feedback to their answers at each step of the lessons.
Arcade Mode. The users are able to further practice their skills in the arcade mode upon
successful completion of the tutorial mode levels. This ensures that users practice the new
behaviors in a variety of ways (i.e. direct instruction, simulated practice). By providing no
guidance, other than the lesson objective titles, learners can find this section difficult, unless they
have already completed the tutorial mode. This further motivates the users to go through the
tutorial mode first. The arcade mode randomly places the user in one of twenty-five simulated
VR PEDESTRIAN SAFETY APP 11

street crossing environments (identical to ones featured in the exam mode). The user attempts to
demonstrate the desired behavior of the level. If achieved, the user is placed in another random
simulated street crossing environment. A progress bar indicates the number of current levels
successfully completed. If the user demonstrates an undesired behavior, they are notified, and
returned to the main menu. The app can be expanded to have the user's high score displayed next
to the arcade mode title on the menu, encouraging users to break their own high score. The
layout of this game mode is similar to many popular arcade style games that the learners are
familiar with, including Frogger, and Flappy Bird.
The method of gamification learning is the main reasoning for including the arcade mode.
Gamification learning uses typical elements of game playing to facilitate learning. In this
instance, it is intended to motivate the user to gain more practice at the desired behaviors. App
users find themselves motivated to use the arcade mode, even after completion of the exam
mode. Competition among peers to beat each others high scores makes using the application
more desirable.
Exam Mode. Exam mode features all twenty-five simulated environments in sequence
by learning objective. The user is tested on all five enabling objectives. The user is then shown a
results page that indicates their proficiency on each of the specific learning objectives. The user
can retake portions of the exam that are not at 100% accuracy. If the user passes the exam with
100% accuracy, that specific sub-level remains highlighted in green. They are also prompted to
initiate conversation with an adult guardian about crossing the street and overall street safety.
Both the exam mode and the arcade mode use simulation components. Simulations are
safe and ideal environments for learners to practice skills that they will eventually need in a real-
life scenario. Simulations are suitable for this training purpose, mainly due the danger of
practicing pedestrian behaviors street side, the low cost replication of street crossing scenarios,
and the three dimensional spatial relationships that street safety involves, and which VR can
replicate.
Structure of Behavioral Changing Components. The following outline indicates where
behaviors are targeted for correction throughout use of the application.
A. Pedestrian Safety Training
1. Identify pedestrian crosswalks.
1.1. Proper behavior description
VR PEDESTRIAN SAFETY APP 12

1.2. Improper behavior correctional response

2. Initiate pedestrian crossing signals when available.
2.1. Proper behavior description
2.2. Improper behavior correctional response
3. Cross when crossing signals indicate it is appropriate.
3.1. Proper behavior description
3.2. Improper behavior correctional response
4. Look left, right, then left again before crossing street.
4.1. Proper behavior description
4.2. Improper behavior correctional response
5. Determine when it is safe to cross when there is no crossing signal.
5.1. Proper behavior description
5.2. Improper behavior correctional response
B. Exam Mode
a. Improper behavior correctional response (upon completion)
C. Arcade Mode
a. Improper behavior correctional response
Media Components
Media components include instructional videos, 3D virtual environment traffic situations,
and sounds, all encased in a mobile application. The application is available for direct
downloaded on the smartphone application marketplace, Google Play store, as a self-contained
training program. Inside the application, the user experiences menus, resource links, and
instructional videos (See Appendix A). The app is currently intended for users to have virtual
reality headsets along with their smartphones. The app can be expanded to feature a mode where
it is still responsive to motion detection without being inserted into a headset (360 mode).
Challenges
There were three major challenges in completing this capstone. The first challenge was
using C# scripts in Unity to construct advanced game actions. I was able to create paths for
vehicles within the game, and produce object to VR reticle interaction. However, there were
many game actions that required more advanced scripts, including the tracking of a user's score
on menu screens, calculating percentages scored on the exam mode, locking and unlocking
VR PEDESTRIAN SAFETY APP 13

levels, and features of the game like the progress bar. It was challenging to construct the exam
mode to track the user's behaviors on the practice scenarios. I gained help from YouTube, game
design students at CSUMB, online and message boards.
Another challenge dealt with creating an application that teaches desired pedestrian
behaviors, without promoting unsupervised street crossing by children. This application, and the
successful completion of the exam, does not justify that a child is ready to cross the street
without adult supervision. The solution was adding a disclaimer, and providing links of resources
for children and guardians to explore together on the app download page.
A third challenge involved designing a learning effectiveness test for the application. The
desired changed behaviors are ones that occur in real life street crossing scenarios. The app itself
is only capable of testing behaviors in simulated environments. There are existing studies that
show that virtual reality pedestrian training modules can translate to real life behaviors. I decided
against conducting a similar observational study. The size and scale of such a study was outside
the scope of this project and my resources. I measured learning outcomes solely on a simulated
environment through pre-tests and post-tests.
VR PEDESTRIAN SAFETY APP 14

Procedure
Prototype
The process started in fall of 2016 with the development of my initial instructional design
document for this capstone. After feedback on the document, where I laid out an initial design, I
began the iterative design phase. This included prototyping virtual environments to be used in
the application. The first prototype had a working menu system, a working tutorial mode without
instructional videos, and a working arcade mode. The prototype was executable on a computer
through running the Unity software. There was also an .exe file that was optimized for the
Oculus Rift VR headset. The prototype was developed in the spring of 2017 with the assistance
of a Game Design student at CSUMB. The prototype also featured 3D models built by a graphic
design student at CSUMB.
Process
Originally, the process for completing this capstone was based on the ADDIE model.
However, during the Design phase, it was clear that the focus needed to be on product quality
and meeting deadlines. The remaining instructional design process more resembled the SAM
model, which focused on repeated small steps, rather than perfectly executed big steps (see
figure 1). This allowed for rapid prototyping of the VR application.

Figure 1. The SAM Model. (alleninteractions.com, 2017)

Final Steps
Final steps in this process included turning the alpha prototype into a beta, where it went
through a usability test and learning effectiveness test. After both tests were completed, the
VR PEDESTRIAN SAFETY APP 15

feedback was implemented into the product. The product is now available for download, and will
continue to go through minor updates.
My role in the final steps included finding collaborators and sources of knowledge,
creating instructional videos, creating the tutorial mode, developing the beta prototype and final
product application, designing and conducting a usability test, designing and conducting a
learning effectiveness test, and insuring that the application became publishable to smartphones
via Google Play Store.

Resources
Time allocation for the project took 6-8 hours a week, including meeting with other
collaborators over video conferencing. Total development cost was $0, since the development
software is free, and hardware is already available. I had already purchased an Android
Developers License, so that was not a cost either. Since the training module was designed for a
mobile device, the application was formatted for both the Android and iOS platform. However,
publishing issues prevented the application from being available on iOS devices. Unity for Apple
was used to create the application, and Google Forms was used to create the training mode
within the application. Once the application was published, I hosted the application on the
smartphone application marketplace. This required the developers registration status.
To save development time, I used Yobi3d, and the Unity Asset Store to find portions of
the 3D environment created in Unity. I needed monitor the size of the application package.
Having too large of an application file size would have discouraged people from downloading
the application. I originally attempted to keep the file size to fewer than 250 MB. Anything
larger may have required potential learners to delete other media from their devices in order to
make room. The current version is under 70 MB.
Learners need a smartphone to access the training. Learners also need some type of VR
headset (Samsung Gear, Google Cardboard, or any third party VR headset). The application does
work without a headset, but is more immersive if a headset is used.
External expertise was sought from those advising advanced scripts, and those experienced
with publishing apps created in Unity. Unity is a visual game engine that supports C# scripts for
objects inserted into the game hierarchy. Advanced C# scripts were necessary to track game
progress and utilize other game features. I found much of my help from other members of the
CSUMB community, including Game Design students and a Game Design professor.
VR PEDESTRIAN SAFETY APP 16

Timeline
To launch a successful product by the capstone deadline, milestones were given deadlines
to aid in development. Milestones reflect the completed-by date. Work was completed
throughout time leading up to milestone dates.

Instructional Design Document - November 2016

Literature Review - November 2016
Initial Design Storyboard - February 2017
Alpha Prototype App - May 2017
Unity Assistance Found - September 24th
Finish Storyboards - September 26th
Screencast Tutorial Videos - September 30th
Insert Videos Into App - October 8th
Project Checkpoint - October 24th
Finish Exam Mode - October 29th
e-Portfolio Update / MIST Reflection - October 31st
Beta Prototype / Usability Test - November 5th
Export Application - November 12th
Capstone Project Update - November 14th
Publish Application for Download - December 3rd
VR PEDESTRIAN SAFETY APP 17

Evaluation
Evaluation of the learning objectives occurred in two instances, summative evaluations
and formative evaluations. Continuous Summative evaluation data is not being retrieved from
the application user now that the application is available on the Google Play Store. Retrieving
application data can be seen as invasive to one's privacy, and is not recommended for apps
whose primary users are children. A summative evaluation test was performed during the beta
stage of the application. Formative evaluation data was also collected at the beta test phase. The
data was used to measure the effectiveness of the simulation, adjust instructional information to
learning targets, and recommend improvements to the application as needed.
Formative Evaluation
The formative evaluation of the VR Pedestrian Safety Training application consisted of a
usability test, and pre and post-test survey. Since the app is targeted for children ages six to
twelve, the test consisted of a population resembling the target audience. I had access to children
ages ten through twelve from within my classroom. After gaining parental approval, I observed
five children using the application, and administered a pre and post-test survey. During the
observation, I asked questions about the gameplay, and their understanding of the apps purpose.
A pre and post course survey was administered, asking questions about their interest in use of an
application of that nature, and their comfort level within the simulated environment.
Observations. Five of the twenty participants were observed 1-to-1 by an examiner for
the purpose of a usability test. Examiner would ask clarifying questions to identify what the
participant was struggling with at any given moment. The Examiner would also ask the
participants to read aloud whenever they would read a portion of the module. The observations
were video recorded to document statements, and keep track of recommendations for the
module.
Pre and Post-test Survey. A pre-test survey was administered before the module. A
post-test survey was administered immediately following the summative post-test. The survey
was only required of the five participants in the usability test. The survey asked scaled questions
about self-reported in pedestrian safety, course recommendation, and a general feedback
comments section. The full pre-test and post-test survey can be viewed in Appendix B.
The surveys indicted that users gained interest in pedestrian safety after completing the
module, and were generally comfortable with the modules use of VR. Figure 3 shows the
VR PEDESTRIAN SAFETY APP 18

participants mean pre and post-test survey results of their interest in crosswalk safety, along
with the mean self-indicated comfort level with VR. Figure 4 shows the individual interest levels
and self-indicated comfort levels of each participant.

Figure 3. Mean Survey Results

Figure 4. Individual Survey Results

Recommendations
During the usability portion of the formative evaluation, many issues were identified that
were not previously noted in the design process. Some confusion restricted participants to
VR PEDESTRIAN SAFETY APP 19

complete the course on their own accord without navigating to incorrect parts of the module.
Ultimately, the participants were successfully able to navigate and complete the course after
some guidance from the Examiner.
The following table lays out the location of the issue, the issue identified, and possible
recommendations for the issue based on observations and feedback.

Location Issues Recommendations

Menu System -Confusion about order of -Label with Part 1, 2, and 3

occurrence

Introduction -Some device volume was off -Indicate volume is used

-Do not know how to return to -Insert working deep link back to app, or
app once completed specify to close intro and open app

Arcade Mode -Does not go back to menu -Edit code to return to menu system
system after 25th puzzle

Exam Mode -Some users were confused at -Indicate to start with lesson 1 and work
which lesson to start up, or limit user until lesson 1 is solved

Lesson 1 -Some users identified where to -Include sidewalk portion of crosswalk

start on crosswalk (sidewalk) in reticle view acceptance of sensor.
instead of crosswalk

Lesson 2 -NO ISSUES

Lesson 3 -Some users crossed when farther -Specify to wait for CLOSEST
signal indicated crosswalk sensor

Lesson 4 -Users had trouble identifying the -Expand angle to which event system
left pane. Some looked left, right accepts input, or tint the view panes so
left, but were not looking at the that the user can easily identify which
correct angle directions to look.
VR PEDESTRIAN SAFETY APP 20

Lesson 5 -If user crosses in between cars, -Fix event system from collision system
success is still recorded. Even to a timed system and if they step too
missing a car slightly works soon then the user will fail.

General -Back button menu system is -Remove back button system or fix.
unresponsive from game play

Table 1. Recommendations for Module Improvement

After correcting known issues, further usability tests with members of the target audience
can identify more unknown issues, or provide feedback to improve the application.
Summative Evaluation
Since I had access to the students in my district, I tested twenty students within the target
age range, ranging from ages six to twelve. Summative evaluation included testing the
participants on the target learning objective. The information gathered from this evaluation
provides insight to the skills learned after completing the module. Participants in this assessment
took the exam portion of the module before and after the course, I hypothesized gains in
performance of the learning objectives.
I used a directional two-sample t-test to test my hypothesis that use of the application will
indicate positive gains in overall score on the examination. Since the application is targeting
precursor behaviors for pedestrian safety, the test can be done from within the exam mode of the
application. Since some levels of the exam portion will not result in incorrect answers (lessons 1,
2, and 4), Students were evaluated with a pass/fail at the first moment the soft key trigger was
pressed in each level. Otherwise, all results, pre and post-test would have resulted in 100% for
those individual learning objectives. Further research could be conducted to test if the
application is effective for changing monitored behaviors in actual street crossing scenarios.
Intended Outcomes. It was hypothesized that completion of the module would increase
test results of the participants. The null hypothesis without training is that there would be no
significant difference in test scores. The research hypothesis is that there will be a statistically
significant increase between pre-test and post-test scores of the same participants upon
completion of the training module.
Observed Outcomes. Scores on the pre-test and post-test had a possible range of 0 to 25.
Mean scores from pre-test to post-test increased from 19.2 to 21.65. Since all participants
 VR PEDESTRIAN SAFETY APP 21

completed both a pre-test and a post-test assessment, a paired two-sample t-test for dependent
samples was used to test validity. The test statistic had an absolute value of 4.79. Since our
hypothesis was directional, the t-stat was compared to the t critical one tail value of 1.729. Since
the test statistic was greater than the t critical one tail value, we reject the null hypothesis and
accept the research hypothesis - see table 2 below. The course was a statistically significant
factor in participants increase from pre-test to post-test score.
t-Test: Paired Two Sample for Means

Variable 1 Variable 2

Mean 19.2 21.65

Variance 6.694736842 6.660526316

Observations 20 20

Pearson Correlation 0.6179330618

Hypothesized Mean Difference 0

df 19

t Stat -4.85047171

P(T<=t) one-tail 0.00005555934098

t Critical one-tail 1.729132793

P(T<=t) two-tail 0.000111118682

t Critical two-tail 2.093024022

Table 2. Summative Assessment T-Test

The pre-test and post-test featured identical problems. As the user answered, they
received immediate positive or negative feedback. It is possible that this feature of the exam
alone increased test scores to a degree. If the participants were positively reinforced, then they
would answer in a similar fashion when a similar virtual situation was presented. However, this
feedback system is still a function of the module, and therefore is included as a factor that
increases knowledge of the tested learning objectives.
VR PEDESTRIAN SAFETY APP 22

Conclusion
In fall of 2016, a student within the district of which I teach was struck and killed in a
pedestrian crossing accident. Upon reflection of this tragedy, I considered what could be done to
further spread pedestrian safety. My interest in virtual reality technology, and its applications as
a training platform led me to develop a concept for a VR pedestrian safety-training module,
Crosswalk VR. The creation of this training module encompassed skills and knowledge gained
from participating in Cal State Monterey Bays Masters of Instructional Science and Technology
program. The development of this capstone produced a viable training application available for
public download on the Google Play Store, as well as allowed me to refine the skills and
knowledge gained from MIST courses.
Careful evaluation of the pre-test and post-test scores resulted in an accepted research
hypothesis, one that points to Crosswalk VR having a statistically significant impact on acquired
pedestrian safety behaviors. Additionally, formative evaluations identified issues that, once
fixed, will result in a more desirable, and more effective product. By creating a fully immersive
VR experience that was comfortable for the users, the product has potential to gain popularity.
Also, since there is no other commercially available VR Pedestrian Safety training application
available on either major smartphone marketplace, Crosswalk VR can be considered one of the
first training modules of its kind for that particular purpose.
There were several limitations and challenges during the creation of this capstone. The
main challenge was from a development standpoint. Since I had never developed any VR
application before, I was limited in my ability to effectively create the module. I was able to get
assistance from other students with a game design background at CSUMB, and ultimately create
a product that I am satisfied with. This overcoming challenges involving developing with new
platforms and technologies has given me valuable skills that will prepare me for whatever
technological direction instructional design is heading.
I will continue to improve and implement recommendations created through the
formative evaluation process. Since the application is hosted on the Google Play Store, I can
publish further versions of the application that will automatically update to users who have
already downloaded the application. Overall, Crosswalk VR is a successful training application
that showcases the skills and knowledge that the MIST program can achieve, and furthers my
goal of improving and promoting safe pedestrian behaviors.
VR PEDESTRIAN SAFETY APP 23

References
Barton, B. K., Schwebel, D. C., & Morrongiello, B. A. (2006, 10). Brief Report: Increasing
Children's Safe Pedestrian Behaviors through Simple Skills Training. Journal of
Pediatric Psychology, 32(4), 475-480. doi:10.1093/jpepsy/jsl028
Centers for Disease Control and Prevention. WISQARS (Web-based Injury Statistics Query and
Reporting System). Atlanta, GA: US Department of Health and Human Services, CDC;
2015. Available at http://www.cdc.gov/injury/wisqars. Accessed February 17, 2016.
Glang, A., Noell, J., Ary, D., & Swartz, L. (2005, 09). Using Interactive Multimedia to Teach
Pedestrian Safety: An Exploratory Study. American Journal of Health Behavior, 29(5),
435-442. doi:10.5993/ajhb.29.5.6
Josman, N., Ben-Chaim, H. M., Friedrich, S., & Weiss, P. L. (2008, 01). Effectiveness of virtual
reality for teaching street-crossing skills to children and adolescents with autism.
International Journal on Disability and Human Development, 7(1).
doi:10.1515/ijdhd.2008.7.1.49
Percer, J. (2009). Child pedestrian safety education: Applying learning and developmental
theories to develop safe street-crossing behaviors. Washington, D.C.: U.S. Dept. of
Transportation, National Highway Traffic Safety Administration.
Preusser, D. F., & Lund, A. K. (1988, 12). And keep on looking: A film to reduce pedestrian
crashes among 9 to 12 year olds. Journal of Safety Research, 19(4), 177-185.
doi:10.1016/0022-4375(88)90021-7
Preusser, D. F., & Blomberg, R. D. (1984, 06). Reducing child pedestrian accidents through
public education. Journal of Safety Research, 15(2), 47-56. doi:10.1016/0022-
4375(84)90001-x
Schwebel, D. C., Davis, A. L., & ONeal, E. E. (2012). Child Pedestrian Injury: A Review of
Behavioral Risks and Preventive Strategies. Retrieved October 03, 2016, from
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3467946/#R51
Zeedyk, M. S. (2003, 08). Tackling children's road safety through edutainment: An evaluation of
effectiveness. Health Education Research, 18(4), 493-505. doi:10.1093/her/cyf033
Zeedyk, M., & Kelly, L. (2003, 09). Behavioural observations of adultchild pairs at pedestrian
crossings. Accident Analysis & Prevention, 35(5), 771-776.doi:10.1016/s0001-
4575(02)00086-6
VR PEDESTRIAN SAFETY APP 24

Zero to Eight: Childrens Media Use in America. Common Sense Media. (2013). Retrieved
December 10, 2016, from https://www.commonsensemedia.org/research/zero-to-eight-
childrens-media-use-in-america-2013/key-finding-2:-kids'-time-on-mobile-devices-
triples
VR PEDESTRIAN SAFETY APP 25

Appendix A
Screenshots from Crosswalk VR
Start Menu

Main Menu

Exam Menu
VR PEDESTRIAN SAFETY APP 26

VR Simulated Environment

Tutorial mode of Crosswalk VR. This can be accessed independently of the application at
VR PEDESTRIAN SAFETY APP 27

https://goo.gl/forms/K5JIgcnlSHxl9k5A3
VR PEDESTRIAN SAFETY APP 28

Appendix B
Formative Assessment Pre and Post Course Survey