Running Head: Course Substitution Training Module

Course Substitution Training

Michael Vicent

California State University, Monterey Bay

IST 622

Dr. Bude Su

July 15th, 2019

Running Head: Course Substitution Training Module

Section I Introduction

The Course substitution Training is a module created in Captivate designed to

train Graduate Faculty and Staff on how to process course substitutions for students in

OASIS. Currently there is no formal training in place for this new business process

across graduate departments. For the university, it is important to maintain accurate and

up-to-date records for students. The training consists of brief introduction of the topic

followed by a software simulation designed to simulate the entire process without

negative repercussions to the student’s official records. This module will serve as one of

the components of my Capstone and was part of my IST 526 prototype. Five learners

participated in the evaluation of the learning module.

Section II Methodology

Prototype

The prototype utilized for this evaluation is one component in a series of modules

designed to train graduate faculty and staff on a series of new business processes.

These include training in three core areas: Academic Requirements Report, Course

Substitution, and Classified Standing. The prototype is designed to be asynchronous

and housed on a website for easy access through computer or laptop. The prototype

mimics one for one how to the learner will complete these processes in both OnBase
Running Head: Course Substitution Training Module

and OASIS. It includes both an introduction of new information for learners as well as

narration and prompts for learners to follow as they complete the training.

Learners

The Course Substitution Training was designed to train Graduate Faculty, Staff,

Advisers, and Coordinators. These individuals are often very knowledgeable about

particular sections of OASIS, however they most likely do not have any experience

working with the Academic Requirements Report or modifying it since it has not existed

prior to Fall 2018. They have access to computers at work and actively work in OASIS

on a daily basis. The learners are derived from multiple departments with diverse

backgrounds and skill sets in higher education. Because this is a completely new

process, I do not anticipate any prior knowledge on how to submit course substitutions.

For this term project, the population will consist of staff at California State

University Monterey Bay. Of the five users tested, three will be female and two will be

male. While the learners have worked in OASIS, they are not familiar with the particular

business process in this training module.

Learning objectives:

The objectives defined below state what the learner should be able to accomplish

after completing the instructional task.

Running Head: Course Substitution Training Module

● From memory, graduate faculty and staff will be able to define the meaning of

each term on the course substitution form in OASIS.

● Given a software simulation of OASIS, graduate faculty and staff will be able to

submit a course substitution with 100% accuracy.

Tryout Conditions and Process

The tryout conditions consist of four parts: pre-test, training, post-test, and

usability survey. The pre and post-test surveys were created using Google Forms and

sent out via email along with the training. The learners’ recorded their answers, which

were reported to my CSUMB email once completed. The testing was completed

asynchronously due to several reasons. First due to the time constraints of the project

and the ability to book a common site for testing and observation. While this would have

been ideal, it would have required multiple laptops and small conference room. Second,

the availability of users widely varied due to meetings and workload differences.

Creating an asynchronous testing environment allowed the users to take the test at their

own pace and when it suited them best.

Pre-Test

The goal of the pre-test questions was to gauge the learner’s prior knowledge on

the subject and determine their comfort level using OASIS and the Academic

Requirements Report.
Running Head: Course Substitution Training Module

questions were comprised of multiple choice questions. These questions will be utilized

in both pre and post-test to measure change and determine whether the training was

effective or not. The questionnaires were included in an email to the learners along with

the link to the training.

Training

The training was sent to learners through an email. This email included a link to

the training, the pre and post-test survey, as well as the usability survey. The Adobe

Captivate module was part of the IST 526 Prototype and thus housed on the CSUMB

Myspace Server for anyone to access who had the link made available to him or her.

Post-Test

After the learner completed the training, they proceeded to click the link to the

post-test. To measure change, the post-test was identical to the pre-test.

Usability Survey

The purpose of the usability survey was to determine whether the content was

presented in a manner that was easy to navigate and easy to understand. The

questions were in a multiple-choice format and a Likert scale was used to gauge the

learner’s opinions on the training. In addition, the usability survey ends with three
Running Head: Course Substitution Training Module

Open-ended questions that allow the learners to identify areas of strength, areas of

improvement, and any additional comments they have on the training.

Tryout Conditions

The tryout conditions were designed to simulate similar conditions to how the

actual training would be administered. As stated above, the training is designed to be

asynchronous and independent from any need of an instructor. The learners were

required to have a computer as well as an internet connection to access the training

materials. All training and test items were sent via email and provided as links.

Section III Results

Entry Conditions

The entry conditions exhibited by learners was as anticipated. Learners were

instructed to contact me if any issues arose accessing the training or surveys. All

learners work in the same office as I do and could contact me via email or in-person as

needed. All learners had no issue accessing the training and completing the usability

and pre/post-test survey. Because all learners were CSUMB employees, they were all

familiar with Gmail and Google Forms.

Running Head: Course Substitution Training Module

Instruction

The observed instruction was similar to the intended instruction. Learners were

able to easily access the training and navigate through the software simulation. One

learner had issues getting the audio to an adequate listening level in the training

module. It was later determined that the training module’s audio was not loud enough

for some users.

Summary of Results

Pre/Post Test. Learners were able to receive a total of 22 points in the pre and post-

test surveys. Every learner assessed scored higher in the pre and post-test survey

except for one learner who had a perfect score in both tests. However, this learner was

an outlier due to their knowledge on the subject matter and is someone has been an

employee of CSUMB for some time.

Summary of Pre and Post Test Results

Running Head: Course Substitution Training Module

Average Pre-test score: 15.2 out of 22 points or 69.09%

Average Post-test score: 20 out of 22 points or 90.90%

Each test started out with questions regarding the learners comfort level with the

various software they would interact with. Comfort level with OASIS did not see any

observable change; however, this is most likely because all learners were employees

who use OASIS on a daily basis. Therefore, the training would not have a significant

impact on their feelings regarding this software.

More significant however was the increase in comfort level reported regarding

whether the user would “be comfortable processing a course substitution.” In the pre-

test, learners scored an average of 2.4 out of 5 points on comfort level, however in the

post-test learners scored an average of 4.4 out of 5 points. Comfort level with the

Academic Requirements Report also increased from an average score of 3 out of 5

points to 4 out of 5 points.

In all content related questions, learners scored improved. Specifically questions

4 and 6. Question 4 increase from a 60% pass rate to a 100% pass rate. Question 7,

which had an average score of 60% in the pre-test raised to a score of 95% in the post

test.
Running Head: Course Substitution Training Module

Pre-Test Question 7 Result

Post-Test Question 7 Results

Running Head: Course Substitution Training Module

Usability Survey Results. Users reported that they were satisfied with the usability of

the training overall.

80% of users also found the training easy to navigate.

Running Head: Course Substitution Training Module

10

80% of users found the screenshots useful to the overall experience.

60% of users found that the training was easy to comprehend.

Running Head: Course Substitution Training Module

11

60% of users found the audio narration was easy to follow while 20% of users had

neutral feelings about the audio and 20% “disagreed” with the statement.

100% of users found the software simulation easy to navigate

Running Head: Course Substitution Training Module

12

Outcomes

My hypothesis for this assessment was that training would improve post-test

scores when compared to pre-test scores. Based on the table above, it can be

concluded that there is a statistically significant difference between the pre and post-test

results. The t-stat was 3.28, which is larger than the critical value of 2.13, and the p-

value of .015 is smaller than the .05 conventional alpha level.

Recommendations

1. Improve Audio Narration. Based on the results from the usability survey, users

exhibited some difficulty in getting audio levels to be at an adequate level. This

resulted in difficulty hearing the instruction and following the prompts during the

software simulation. Re-recording the audio is recommended based-on user

feedback so that narration is clearer and audio levels appropriate for users.
Running Head: Course Substitution Training Module

13

2. Add navigation buttons onto all slides. It is recommended that navigation buttons

be added to the slides. While slides transition automatically after narration is

complete at the beginning of the training, user navigation buttons would allow

users to move through slides, as they feel comfortable. Because audio narration

needs to be improved, this was a recommendation so that users could listen to

slides multiple times if needed.

3. Improve prompts in software simulation. While the software simulation does

provide narration and directions on each slide, better signaling was suggested by

users. Users reported that while the prompts that were embedded in the training

were helpful, they did not always stand out. Editing the prompts to highlight

clickable areas and signal the users to the correct button prompts is

recommended

Section IV Summary

The hypothesis for the training was that the learning solution would improve the

post-test scores over the pre-test scores. After reviewing the statistical data collected

from the pre and post assessments, it can determined that there was a statistically

significant difference between pre and post-test scores. Users reported higher scores in

all content-related questions. Areas of improvement include audio narration and user

navigation that will be implemented in future versions of the learning solution. I plan
Running Head: Course Substitution Training Module

14

to utilize these results to improve my overall Capstone Project and develop a more

succinct learning solution for users.

Section V Appendices

Pre-Test Questionnaire

https://forms.gle/fHk9PSZXC8tu9E9eA

Post-Test Questionnaire

https://forms.gle/7utJ56HNhfuudTTM6

Usability Survey

https://forms.gle/8cmKUw9Pk5UzyRVd7

Course Substitution Training

http://myspace.csumb.edu/~mvicent/ist526/IST%20526%20Final%20Project%20Su
bmitting%20Course%20Substitution/