Vous êtes sur la page 1sur 21



Assessment Plan

for Learning Management System in Educational Technology Master Program in Arab Open University (AOU)

Tech6102 | Presented by: Asma AlKalbani| 82934 Fatma Alhinai| 98569 & Mitha Alkhuzami|102477

Table of Content

1. Introduction.3-7

1.1 About the Target Institution (AOU)..3

1.2 Vision & Mission...3-4

1.3 Background About Educational Technology Master Program..4-7

2. Assessment Plan Model... 7-8

3. Analysis Phase.8-11

3.1 Need Assessment..8

3.2 Stakeholder...8-9

3.3 Context.9

3.4 Assessment Plan Scope....9-11

4. Design Phase...11-12

4.1Assessment Plan Goal...12

4.2 Assessment Plan Questions..12

4.3 Assessment Tool..12

5.Development Phase12-13

6. Implementation Phase13

7. Evaluation Phase13-18

7.1 Result Analysis13-18

7.2 Recommendations...18

7.3 Sharing results.....18

8. References..19-20

9. Appendix....21


Program evaluation is carefully collecting information about all or some aspect of a program

in order to make necessary decisions related to the program. It is a systematic method for collecting,

analyzing, and using information. It helps evaluator to guide decision-making, identify where support

is needed, provides a picture of what has been accomplished, what needs to be done and enhances the

effectiveness of programs. Evaluators go over evaluation process using scientific methods and

valuable data to ensure high quality of assessment. On this document, we developed an assessment

plan aimed to measure students' satisfaction toward LMS in Educational Technology master program

in Arab Open University. We measured students satisfaction in terms of in instructional materials,

communication, user interface and technical support.

About the Target Institution (AOU)

The Arab Open University (AOU) is a sustainable development and educational non-

profit project. It was founded by HRH Prince Talal bin Abdul-Aziz, Chairman of the AOU

Board of Trustees. In September 2000, AOU was officially declared in the meeting of Arab

ministers of higher education where five Arab countries offered to host the headquarters of

AOU. Among them was Kuwait that was chosen to be the headquarters of AOU in December

2000. Today AOU has eight branches in Kuwait, Lebanon, Jordan, Saudi Arabia, Egypt,

Bahrain, Oman, and Sudan. The (AOU) in Oman focuses on integrated (LMS) in the learning

process for undergraduate and postgraduate students.


A pioneering Arab Open University dedicated to building the science and knowledge



Develop and disseminate knowledge, and build expertise according to international

quality standards without time or geographical barriers for the sake of contributing and preparing

manpower for development needs, and to build science and knowledge society in the Arab


About Educational Technology Master Program

The Educational Technology Program is a full-time postgraduate program of study

comprising either taught courses only or taught courses and a dissertation. The program was started

in 2012 and graduated three cohorts; first cohort on 2014, second cohort on 2015, and the third cohort

on 2016. The numbers of student were increased in each cohort (18, 30,43 respectively). The

Educational Technology Program aims to:

1. Contribute towards the achievement of the missions and objectives of the AOU, that is

the dissemination of knowledge and contribution to human development in the Arab


2. Meet the high demand in many of the Arab States for qualified well- trained teachers who

employ modern technology and teaching strategies

3. Enhance the quality of teacher preparation and teacher training in general, thereby

contributing to the socio-economic development and improvement of education in the

Arab States

4. Respond to employment market demands for personnel with skills and qualifications in

Instructional Technology, for example, business, mass media, and multimedia production

The program is distinctive in several respects:

It provides students with a flexible open learning opportunity.

It combines both face- to- face instruction (25%) and interactive distance learning (75%).

Students admitted to the Educational Technology Program in Instructional Technology

will avail themselves of excellent upto date teaching and support materials conducive to

self- learning.

Successful candidates will qualify not only for the AOU Educational Technology

Program but for a UK OU validated Educational Technology Program degree as well,

which will provide graduates with opportunities to continue their PhD study locally,

regionally or internationally.

It offers career progression in education and, more generally, in the public and private


The Program is in high demand in all Arab countries, particularly in view of the following facts:

Professionals engaged in teaching, especially school directors and leaders need to be

qualified to master's level in education for promotion and enhanced career prospects.

More and more faculties of education and teacher training institutions are being

established in Arab countries, and thus there is a great demand for qualified Masters

holders to teach Instructional technology skills and basic education courses.

National and regional businesses, the mass media, international organizations and

publishing houses also require Arab personnel who are trained in educational technology

The Program will be delivered through the open learning system, which according to the AOU

regulations - is delivered through two complementary modes:

Face-to-face interactive tutorials, constituting 25% of course credit hours, and;

Interactive distance learning delivered through specially designed teaching and support

materials that are conducive for self-learning, constituting 75% of course credit hours.
This M. Ed. Program in Instructional Technology will qualify students to be able to:

1. Understand the fundamental concepts and skills for professional use of technology in the

classroom as well as distance learning.

2. Understand how well technology-based solutions could help solving instructional


3. Design methods and tools for the development of innovative learning environments.

4. Use various media to communicate and collaborate effectively with students, colleagues,

and others.

5. Evaluate the effectiveness of hardware and software in improving student learning.

6. Use data and current research to promote these practices.

7. Identify suitable areas the progress of the use of (ICT) in education.

8. Improve the understanding of the impact of (ICT) on the organization of teaching and


9. Bring to the classroom all the content, motivational and management expertise to capture

students imaginations and connect learning in the students.

10. Use technology to foster students curiosity and creativity, as well as engage students in

meaningful problem-solving activities.

11. Implement information communication technology effectively, as well as using student

data to assess and modify instruction.

12. Develop technology-rich lesson plans, teaching strategies, and assessments.

13. Stimulate the development of services and systems to ensure access to multimedia

products and Internet-based services for education.

14. Assess the pedagogical and organizational impact of (ICT) on learning processes and


15. Invigorate the involvement of teachers in the conceptualization of educational multimedia

resources and services for E-learning.

Timeline of assessment plan

No. Task Week 8 Week 9 Week 10 Week 12 Week 13

1 Organize your team

2 Define the goals/objectives of

your plan.
3 Plan the Needs Assessment or
4 Determine the standards or
benchmarks you will use

5 Decide Upon Assessment
Tools (note: these may be
different from the needs

assessment tools)
6 Perform your assessment

7 Analyze the results

8 Make Recommendations and

Create an Action Plan
9 Share your results

Assessment Plan Model

We adapted ADDIE model to guide our assessment plan and we match it with the nine steps of

quality assurance plan. Figer1 shows the evaluation model with its steps:

Analysis Needs Assessment (interview with QA staff in AOU)

Stakeholders: ET Master students and faculty.
Context: LMS in AOU.
The Scope of Assessment plan: (LMS), satisfaction, instructional material,
communication, interface and students' support.
Design Define the assessment plan goal
Define the assessment plan Questions
Decide Upon Assessment Tools.
Development Determine the standards.
Develop the Assessment Tools (Survey)
Test instruments validity and reliability (pilot testing)
Implementation Distributing assessment tool (distributed online survey)
Evaluation Analyze the results and discuss them
Report with recommendations
Share the results

Analysis Phase

Determine the Need for the assessment plan

We conducted an interview with Dr. Nader Said (the head of the Educational Technology

department in (AOU) to determine the needs to build the assessment plan. We prepared different

questions and recorded the interview by phone for further information, see appendix (1). We

decided to use this tool because it helps us to collect qualitative data as much as possible. After

analyzing the interview, we decide to do an assessment plan to assess student satisfaction toward

using the Learning Management System (LMS) as the department depends heavily on it as they

use it as distance education tool. Also, they did not measure their students satisfaction toward

using LMS.

The target audience of this plan is master students at the first and second year in

Educational Technology program in(AOU). The sample of assessment covers 53 students. Most

of them are an employee in the governmental and private sector. They came from different

regions and some of them are foreigners who come with the different culture. they have willing

to complete their higher degree in education.


This plan focuses on studying LMS environment in AOU specifically for those master

students who study Educational Technology program.

Assessment Plan Scope

Learning Management System (LMS). It is a software that automates the

administration of training events. The LMS registers users, tracks courses in a catalog and

records data from learners. It also provides reports to management. An LMS is typically

designed to handle courses by multiple publishers and providers like lectures, designer, and

instructional developer. It usually doesn't include its own authoring capabilities; instead, it

focuses on managing courses created by a variety of other sources (Remes, 2005).

Student Satisfaction. Researchers have identified students' satisfaction as an important

factor in measuring the quality of blended learning (Naaj et al, 2012). Students' satisfaction

defined by Sloan Consortium as, Students are successful in the learning and are pleased with

their experience (J. C. Moore, 2009). Another definition of student satisfaction said that

satisfaction is the perception of enjoyment and accomplishment in the learning environment.

(Sweeney & Ingram, 2001). Both definitions focus on accomplishment and success in learning,

pleasure, and enjoyment of the experience. Considering student satisfaction as an indicator of

LMS success and as the ultimate target of LMS use is sensible for several reasons. First, previous
research showed that LMS use supports rather than modifies, existing teaching and learning

approaches. Thus, make sense to redefine LMS success and shift indications of success away

from achieving a pedagogic revolution toward improving student satisfaction. Second,

satisfaction is defined as the ability of a service or a product to address customer needs. Even if

student needs are not fully known, it is reasonable to assume that high student' satisfaction is

indicative of success in the sense of whether the LMS responds well to their needs (Naveh,

Tubin & Pliskin, 2010).

There are many factors that influence student satisfaction while using LMS. In this

assessment plan, we are measuring student satisfaction towards the LMS in terms of;

instructional materials, communication, interface design and technical support.

Instructional Materials. They refer to the content delivered by LMS. Institutions use the

LMS to supplement traditional face-to-face delivery where faculty members develop and share

digital learning materials via the Internet. In this case, the LMS is used as electronic repositories

of learning materials. In addition, it should be noted that learners rely on learning materials as

their major source of information during the learning process. Consequently, they place great

value on content that is well-organized, effectively presented, interactive, clearly written, in the

right length, useful, flexible, and provide an appropriate degree of breath. There is also a strong

positive relationship between quality of learning materials and overall learners perceived

satisfaction with the LMS. Therefore, learners tend to be disappointed with the LMS when they

find out that uploaded materials are of poor quality and do not provide intended educational

objectives (Mtebe, 2015).

Communication. Learning environments in which social interaction and collaboration

are allowed and encouraged lead to positive learning outcomes. Collaborative learning and
communication tools can improve student satisfaction in the online learning environment. These

tools allow for group work and immediate feedback. Students can share viewpoints and discuss

them with one another in a virtual environment and with the student and instructor, thereby

gaining insights and perspectives. This type of environment allows for social interaction and

creates meaningful and active learning experiences (Abou Naaj, Nachouki, & Ankit, 2012).

User Interface. Interface or sometimes called the interface design. The interface of the

LMS significantly influences students satisfaction towards LMS. The result of the user -

friendly interface will directly influence student satisfaction towards the use of LMS in blended

learning (Malik, 2009). Therefore, the interface design of LMS is important as its play a vital

role in user interface interaction. In the online environment, users will utilize the computer to

access the content and interact with other online users and instructors. The LMS must be

designed in an appropriate way so that users would not face any trouble when using it. The

problem in LMS design would decrease the user satisfaction when using the LMS. Thus, a

simple LMS user interface must be included for ease the interaction between users and systems

(Ramakrisnan, Jaafar, Razak & Ramba, 2012).

Technical Support. It enhances the satisfaction of users and has a critical effect on users

believes in accepting or rejecting an LMS. When users receive no help from the assistants while

being faced with a problem, they will get the feeling that working with the system is a waste of

time and hence will quit working with it. Technical support in this case, refers to the existence of

assessing staff or faculty to solve problems students encounter when they are working with LMS

(Baleghi-Zadeh, Ayub, Mahmud & Daud, 2017). In addition, disabled users must be considering

in LMS to provide them with suitable materials that work with their skills.

Design phase

Assessment Plan Goal

Measure the e-learning effectiveness in terms of Student satisfaction toward using the LMS in

ET Master Program in AOU.

Assessment plan questions

How satisfied are the students with the Instructional materials in the LMS?

How satisfied are the students with the Communication in the LMS?

How satisfied are the students with the LMS Interface?

How satisfied are the students with the Technical Support in the LMS?

What are the challenges that students face while using LMS?

Deciding upon assessment plan instrument

Based on students need and purpose of this assessment plan, our team decided to use the survey

as a tool to collect quantitative data to measure students' satisfaction toward LMS in AOU. In

addition, this tool eliminates biases and we can collect much data in less time from a big number

of students.

Development Phase

We developed a survey to assess students satisfaction toward the LMS by adopting American

Society for Training & Development (ASTD) standards for e-learning. Besides that, some items

in the survey, from Bauk, Scepanovic, & Kopp, (2014) study which focused on estimating

students' satisfaction with web-based on learning system in the blended learning environment.

We used Google forms to develop the survey and collect data. The instruments had

been reviewed by our instructor to ensure its validity. In addition, it had been implemented

within a small group of students in AOU to ensure its reliability via SPSS by using Alpha

Cronbachs. The inter-rater reliability of survey achieved an excellent value (0.93). The survey
consisted of 5 major parts. The first four parts are instructional materials, communications,

interface and technical support using three scales (Agree, Natural and Disagree). The final part is

the open question to determine the problems that student face while using LMS. For further

information, see (Appendix 2).

Implementation Phase

We distributed the survey among 53 master students, however, we received 31 responses

only and we see that this amount of responses is reasonable because it exceeded the half of our

target sample.

Evaluation Phase

Result of survey

Table1: Results of students' satisfaction toward LMS in term of instructional material

N Item Agree Neutral Disagree

1 The material in (LMS) are up to date 0.61 0.39 0.0
2 The instructional material is easy to understand 0.74 0.23 0.03
3 The instructional material meets different students 0.42 0.42 0.16
learning styles (example: text, sound, and graphics)
4 The link in the Instructional Materials direct you to the 0.72 0.26 0.02
right references
5 Instructional Materials are user-friendly, convenient and 0.58 0.39 0.03
easy to operate
6 Supplemental resources are available in case of (LMS) 0.42 0.29 0.29
failure. (example; printed materials, PowerPoint)

Sometimes figures speak better than words. To analyze the responses, we calculated

the mean of responses by using SPSS software. Although the students responses to

instructional material in LMS is quite clear in terms of considered features (Table1). We

developed 6 items to measure students satisfaction toward instructional material in (LMS).

Result shows, more than 50% of students agreed that instructional materials in LMS are up to

date, user-friendly, convenient, easy to operate and easy to understand. Naaj, Nachouki, &

Ankit (2012) stated that easy access is one of the most important factors influencing student'

satisfaction. A big number of students considered that instructional material didn't meet

enough different students learning styles. Malik ( 2009) reported that students' satisfaction

and acceptance of LMS is influenced by the teachers teaching style and ways of delivering

lectures in a friendly manner. It is important to stress that 22 students responded positively to

the item; The link in the instructional materials direct you to the right references" but in case

of (LMS) failure some supplemental resources are not available for students. Sometimes,

students couldn't access to supplemental resources due to the failure of the internet connection

or maintenance in LMS itself.

Table2: Results of students' satisfaction toward LMS in term of communication

N Items Agree Neutral Disagree

1 I can easily communicate with other students 0.68 0.23 0.097
2 The announcements through (LMS)are sufficiently clear 0.58 0.35 0.07
and precise
3 I receive teachers' announcements on the courses (LMS) 0.64 0.23 0.13
4 I can easily communicate with teachers 0.71 0.13 0.16
5 The teachers spend time on online communication 0 0. 0.1
with students (feedback, answer questions etc.). .35 45 9
In the second round of the survey, students were asked to assess communication among

LMS via five items. Data summarized that students can easily communicate with others. They

received clear, precise and sufficient announcement through LMS. More than 20 students agreed

that they received a regular announcement from their instructor on the courses (LMS). Malik

(2009) stated that when students dont face any problem with teacher interaction their

satisfaction will be improved. Item (4) related to interaction with teachers assigned the highest

mean (0.71) while the lowest score in this group recorded by item (5), " The teachers spend time

on online communication with students (feedback, answer questions etc.)". The present of

teacher via online helps students to answer their inquiry early. It also affects students'

performance when they find teacher follow them and observe their work online especially when

they know that teacher can see how many students login LMS in details; in which time, for what


Table3: Results of students' satisfaction toward LMS in term of user interface

N Item Agree Neutral Disagree

1 I can navigate (LMS)freely and easily (example; guidance). 0.71 0.23 0.06

2 Interface provide forms of feedback 0.55 0.35 0.097

3 The interface of LMS is attractive 0.35 0.42 0.23

We developed three items in this category. According to students' responses toward the

interface of LMS, they noted that students can navigate (LMS) freely and easily with guidance.

Malik (2009) explained that friendly user interface will directly influence student' satisfaction

towards courses in LMS. For the second item; "Interface provide forms of feedback", 18 students

(55%) answered agree. When students were asked to assess the interactivity of LMS interface, only
11students (35%) agreed. Interactivity could be affected by the organization of the content inside

(LMS), colors, design, and nature of materials.

Table4: Results of students' satisfaction toward LMS in term of students' support

N Items Agree Neutral Disagree

1 I can get help from technician when I 0.26 0.23 0.26
2 I can access technician through 0.52 0.29 0.19
phone, email, and fax
3 The technical assistance offered by 0.39 0.42 0.19
technician outside the class time
4 Assistive technologies are available 0.39 0.35 0.26
for disabled students

We developed four items to measure the level of students' satisfaction toward technical

services in LMS. From the above numbers, it seems that students weren't satisfied with technical

support in LMS. All items in this group had a mean less than 0.39 or equal except item (2); " I

can access technician through phone, email, and fax" which recorded 0.52 and it is still not high

acceptable. Oliveira, Cristiano & Nakayama (2015) stated that there is a need to know where the

students "walk", what they access, what they read when they're doing in LMS to support them

whenever they ask.

According to the responses to the last question: " What are the challenges that students

face while using LMS?". The major of students responses could be categorized into three issues:

Firstly, LMS doesnt work with all file extension format. Secondly, sometimes students face

problems with the internet so they can't access to LMS and they received the notifications lately.

Thirdly, the technician doesn't available 24 hours when needed.


N Category Mean
Current survey Indicator
result items
1 Instructional materials 0.57 0.50 and above

2 Communication 0.59 0.50 and above

3 User interface 0.53 0.50 and above

4 Students support 0.39 0.50 and above

We calculated the mean average for instructional material, communication, user interface

and technical support and compared them with indicator mean to measure the level of students'

satisfaction in each category. The average mean for student' satisfaction in the first group is 0.57

which is above than indicator mean. That means students were satisfied with the instructional

material in LMS. The results of the second category showed that students were generally

satisfied with communication via LMS (mean=0.59). The lowest mean item in this category was

related to the time spent by the teacher to communicate with students (mean= 0.35) in which

students are less satisfied and require improvement. According to Naaj, Nachouki, & Ankit paper

"Evaluating Student Satisfaction with Blended Learning in a Gender-Segregated Environment",

they found that communication in LMS having the highest mean which indicates that students

are satisfied with the level of communication (Naaj, Nachouki, & Ankit, 2012). That study

agreed with our result. More than 50% of students were satisfied with interface services in LMS

but there were also some areas in which the students were less satisfied specifically in the

interactivity of the interface. Malik, 2009 stated that students positive attitude towards interface

of the online environment will automatically impact on their satisfaction positively and increase

the chances of taking courses' activity on time. The mean average of students' satisfaction toward
technical support is very low (0.39) because they didnt get enough support from technician

inside or outside class time and disabled students didnt find enough assistive technologies in

LMS. This result doesnt agree with the result of Naveh, Tubin & Pliskin (2012) study. They

explained that the successful implementation and student satisfaction towards LMS could be

achieved with a very excellent quality of technical assistance.


Based on the previous result, we recommended faculty in Educational technology

program in AOU to

- Differentiate the ways of presenting instructional material in LMS to meet all learning styles.

- Increase the amount of time for online communication to answer all students inquiry in short


- Improve the interactivity of LMS interface to stimulate students attention

- Improve the quality of technical support services for students in LMS.

Share Results

We shared our results with colleagues in the classroom to exchange our experiences,

instructor of this course, educational Technology program at AOU to get benefits from our

recommendations ,and all interested in developing assessment plan by publishing it over social



There are various alternative learning environments in our country. We need to evaluate

these environments to ensure high quality of services, learning and outcomes. In this document,

we developed assessment plan to measure students' satisfaction toward LMS in AOU. We

found that AOU need to empathize and develop technical support services in the LMS.


Abdul Rahman, N., Hussein, N., & Aluwic, A. H. (2015). Satisfaction on Blended Learning in a

Public Higher Education Institution: What Factors Matter? Elsevier, 768- 775. doi:


Bauk, S., Scepanovic, S., & Kopp, M. (2014). Estimating Students Satisfaction with Web Based

Learning System in Blended Learning Environment. Hindawi Publishing Corporation, 1-

11. doi:10.1155/2014/731720

Oliveira, C., Cristiano, C., & Nakayama, K., (2015). Learning Management Systems (LMS) and

e-learning management: an integrative review and research agenda. Journal of

Information Systems and Technology Management, 13(2). doi:10.4301/s1807-


Malik, M., (2009). Students Satisfaction toward E-learning: Influential Role of Key Factors.

Retrieved from http://www.ciitlahore.edu.pk/pl/abrc/Proceedings




Naaj, M. A., Nachouki, M., & Ankit, A., (2012). Evaluating Student Satisfaction with Blended

Learning in a Gender-Segregated Environment. Journal of Information Technology

Education. Retrieved from


Abou Naaj, M., Nachouki, M., & Ankit, A. (2012). Evaluating Student Satisfaction with Blended

Learning in a Gender-Segregated Environment. Journal of Information Technology

Education, 11, 186-200. Retrieved from


Naveh, G., Tubin, D., & Pliskin, N. (2012). Student satisfaction with learning management

systems: a lens of critical success factors. Technology, Pedagogy and Education.

Retrieved from



Naveh, G., Tubin, D., & Pliskin, N. (2010). Student LMS use and satisfaction in academic

institutions: The organizational perspective. Elsevier, 13(3), 127-133.


. Mtebe , J. (2015). Learning Management System success: Increasing Learning Management

System usage in higher education in sub-Saharan Africa. International Journal of

Education and Development using Information and Communication Technology, 11(2),

51-64. Retrieved from https://files.eric.ed.gov/fulltext/EJ1074158.pdf.

Zadeh, S. B., Mohd Ayub, A. F., Mahmud, R., & Daud, S. (2017). The influence of system

interactivity and technical support on learning management system utilization.

Knowledge Management & E-Learning, 9(1), 50-68. Retrieved from http://www.kmel-


Ramakrisnana, P., Jaafar, A., Abdul Razak, F. H., & Ramba, D. A. (2012). Evaluation of User

Interface Design for Leaning Management System (LMS): Investigating Students Eye

Tracking Pattern and Experiences. Elsevier, 67, 527 - 537. doi:



Appendix 1: Interview to determine the need for the assessment plan:

Do you have a strategic plan in ET post program in AOU?

Have you conducted an assessment plan in ET post program in AOU?

What is your need or you going to plan in post-program ET (E-learning) in AOU?

How effective is the use of E-learning to post program in AOU?

How are satisfied students being with the use of E-learning in post-program in AOU?

Is there a significant effect on increasing student achievement when using of E-learning

in post-program in AOU?

What are the most prominent things that focus on the content that is presented in E-

learning to post-program students in AOU?

Is there technical support for students? What kind of support are you provide in E-

learning to post-program students in AOU? What are the challenges that students face in


Appendix 2: Press the link to have a look at the survey