Doctoral Program of Information Systems

PhD Research Proposal

<E-Learning>

Learning Management Systems Evaluation

Supervised by:

Prof. Dr. Alaa El Din M. Riad Prof. Dr. M. Badr Senousy

Prof of Computer and Information Systems Prof of Computer and Information Systems

Submitted by:
Nouran M. Radwan
Faculty of Computer and information Sciences _Mansoura University
2018
1 2/22/2018
 Acknowledgment

In the name of Allah, the Most Gracious and the Most Merciful

 Special appreciation goes to my supervisor Prof. Dr. Alaa Riad for his supervision
and constant support. I also owe Prof. Dr. M Badr Senousy great thanks for his
constructive comments and guidance throughout the dissertation work.
 My Thanks to Prof. Dr. Ibrahim El Henawy and Prof. Dr. Atef Ghalwash for
agreeing to be a discussant for my thesis.
 I am most grateful to Prof. Dr. Florentine Smarandache [Department of
Mathematics, University of New Mexico, USA] and Prof. Dr. Ahmed Salama
[Department of Mathematics and Computer Science, Faculty of Sciences, Port
Said University, Egypt] for lending me their expertise and helping me to
understand neutrosophic approach.
2
 Publications

 "A new expert system for learning management systems evaluation

based on neutrosophic sets." Expert Systems 33.6 (2016): 548-558.
Indexed in ISI/Thomson Reuters, IF:1.18
 "Neutrosophic AHP Multi Criteria Decision Making Method Applied on
the Selection of Learning Management System". International Journal
of Advancements in Computing Technology 8.5 (2016).
Indexed in Scimagojr/SJR, IF:0.115
 "Approaches for Managing Uncertainty in Learning Management
Systems." Egyptian Computer Science Journal 40.2 (2016): 1-7.
 "Neutrosophic Logic Approach for Evaluating Learning Management
Systems." Neutrosophic Sets and Systems 11(2016):3-7.

3 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

4 2/22/2018
 Agenda

Introduction : Problem Statement , Contributions

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

5 2/22/2018
 Problem Statement

 Previous studies conducted to evaluate LMSs

– unilateral view.
– no comprehensive model.
– full information availability condition.

 LMSs described by uncertainty terms.

6 2/22/2018
 Contributions

 Proposing a comprehensive model for LMSs evaluation

under uncertainty.

Software Quality Success

Selection Assessment Measurement

• Neutrosophic • Neutrosophic • Neutrosophic

Analytical Expert System Success
Hierarchy Decision Measurement
Making Method Model

7 2/22/2018
 Introduction

True
Certainty
False

Data or Vagueness
Information

Imprecision
Uncertainty
Ambiguity

Inconsistency

Fig 1. Certain and Uncertain Data or Information

8 2/22/2018
 Introduction
Table 1. Main Uncertainties Types
Main Meaning Example
Uncertainties
Types
Vagueness when available information is The boy is nearly tall.
normally graded.
Imprecision when the obtainable information is The temperature of the machine is
not defined. between 88-92 °C.
Ambiguity when information leads to several The flower color may be red or
possible interpretations. yellow.

Inconsistency when available information is The chance of raining tomorrow is

contradicted and cannot be true at 70%, the chance of not raining is 50%.
the same time.
9 2/22/2018
 Introduction X

Membership Function
X 1

Membership Function
1

0 Y
0 X1 Fig 3. Type 2 Fuzzy Set
Y
Fig 2. Fuzzy Set
X

Membership Function
Membership Function

0
1 1

0
1

0 Y
0 Y
10 Fig 4. Intuitionistic Fuzzy Set Fig 5. Neutrosophic Set
 Introduction

• Weather Forecasting
– Precipitation
– Windspeed
Fuzzy Type 2 Fuzzy
P: 70% P: 50-70%
W: 60% W: 45-65%
 Raining 70%  Raining 50%-65%

Intuitionistic Neutrosophic -Temperature

Fuzzy P:50%, P’:50% -Humidity
P: 50%, P’:20% W:70%, W’:40%
W: 50%, W’:30%  Raining 50%, ???
 Raining 50%, Not Raining 30%,
Not Raining 20% Indeterminacy 30%
11
 Introduction

Table 2. Multivalued Logic Models and Uncertainty Data Types

Multivalued Logic Uncertainty Data types
Models
Vagueness Imprecision Ambiguity Inconsistency
Fuzzy  ------- ------- -------

Type 2 fuzzy   ------- -------

Intuitionistic   ------- -------

Fuzzy

Neutrosophic    

12 2/22/2018
 Introduction

 Neutrosophic logic is proposed by Smarandache in which

variable is described by

₋ t is the degree of truth

₋ f is the degree of false
₋ i is the degree of indeterminacy.

13 2/22/2018
 Introduction

Examples of Indeterminacy

14 2/22/2018
 Introduction

Neutrosophic Applications

Medical Data Mining Image GIS Topology Decision

Domain • Clustering Processing • Spatial Making
• Diagnosis • Forecasting • Image regions with • Recruitment
and Retrieval ambiguity • Investment
prognosis • Image boundary
• Drug test denoising
reliability

15 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

16 2/22/2018
 Related Work

Year Author Title Concern Technique

Jayakumar Website Quality Assessment Model for
Quality
2013 and Developing Efficient E-Learning Questionnaire
Assessment
Banbehari Framework
Expert System for Evaluating Learning Quality
Valdez- Traceability
2013 Management Systems Based on Assessment
Silva et al. Model
Traceability
A Model For Assessing Learning
Mtebe et Management System Success In Success
2014 Questionnaire
al. Higher Education In Sub-Saharan Measurement
Countries
Evaluation Model For Applying An E-
Analytic
Teng-chiao learning System In A Course: An Analytic Selection
2014 Hierarchy
Lin et al. Hierarchy Process–multi-choice Goal Process
Process
Programming Approach
17 2/22/2018
 Related Work

Year Author Title Concern Technique

Bhuiyan Developing and evaluating a desktop- Quality
2014 Questionnaire
and Kundu based learning management system Assessment
Evaluation and measurement of
Quality
2016 Sahid et al. Learning Management System based on Questionnaire
Assessment
user experience

M. Attia et A Model For Assessing And Enhancing Success Dempster-

2016
al. Efficiency Of E-learning Systems Measurement Shafer theory

Kennedy A Model For Evaluating E-learning

Success
2017 Hadullo et Systems Quality In Higher Education in Questionnaire
Measurement
al. Developing Countries

18 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

19 2/22/2018
 Research Methodology

Identifying LMSs evaluation

Literature Review
process

Understanding neutrosophic
Semi-Structured concept and discussing LMSs
Questionnaire selection criteria, quality and
success factors

Implementing and validating

Expert Questionnaire the proposed LMSs evaluation
model

Fig 6. Research steps of the study

20 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

21 2/22/2018
 Learning Management Systems Evaluation

Phase 1: Software Phase 2: Quality Phase 3: Success

Selection Assessment Measurement

Identify LMSs criteria Determine LMSs Quality Define Factors affecting

requirements and factors and attributes LMSs success
alternatives

Compare alternatives Defining memberships Extracts each factor

under each criteria for LMSs quality factors Importance

Ranks the overall weights Develop a comprehensive

and select the best Obtaining and validating model for success
alternative quality results measurement

22 Fig 7. LMSs Evaluation Proposed Model 2/22/2018

 Learning Management Systems Evaluation

Phase 1 : LMSs Software Selection

 LMSs selection is a multi criteria decision making process .

 The AHP (Analytical Hierarchy Process) method provides

help decision makers in analyzing LMSs criteria.

 The main limitation of AHP is

₋ its incapability of reflecting uncertain data

23 2/22/2018
 Learning Management Systems Evaluation

Phase 1 : LMSs Software Selection The procedures of the NAHP are as follows:

1 • Define criteria, sub criteria and alternatives of the problem.

2 • Construct the hierarchy of the considered problem.

3 • Identify the neutrosophic numbers that correspond to the 1–9 Saaty scale.

4 • Collect neutrosophic preference via the pairwise comparison from experts

5 • Check the consistency of each pairwise comparison.

6 • Calculates the relative weight of criteria.

7 • Compares the alternatives under each criterion or sub criterion.

8 • Ranks the overall weights, and a choice is made of the best alternative.
24 2/22/2018
 Learning Management Systems Evaluation

Phase 1 : LMSs Software Selection

Selecting the Most Appropriate LMS Cost

Moodle
Student Tracking
Evaluative
Tools Exam Pool Sakai

Platform
Compatibility Atutor
Content Developing
Tools
Documentation ILIAS
Support
Technical
Dokeos
Sustainability

Fig 8. Decision Hierarchy Model of the LMS

25 2/22/2018
 Learning Management Systems Evaluation

Phase 2 : LMSs Quality Assessment Neutrosophic Expert System Development

• Determine system requirements represented in inputs, rules and outputs.

1

2 • Experts define inputs, rules of knowledge base and output of the system.

3 • Inputs are presented in truth, falsity and indeterminacy membership functions.

4 • Inference engine create the rules which are stored in the neutrosophic inference.

5 • Neutrosophic sets are converted into a single crisp value which has triplet format

6 • Implementing the neutrosophic expert system by using inputs, rules and output

7 • Validate the system to ensure that the output is equivalent to human experts.
26 2/22/2018
 Learning Management Systems Evaluation
Efficiency
Phase 2 : LMSs Quality Assessment Error Tolerance
Learnability
Usability
Memorability

User Satisfaction
Navigability
System Quality Accessibility
Robustness

Understandable
Fault Tolerance
Reliability
Maturity

Recoverability

27 Fig 9. LMSs System Quality of Neutrosophic Expert System 2/22/2018

 Learning Management Systems Evaluation

Phase 2 : LMSs Quality Assessment

Fig 10. Efficiency True Input Membership

Fig 11. Efficiency Indeterminacy Input Membership

Fig 12. Efficiency False Input Membership

28 2/22/2018
 Learning Management Systems Evaluation

Phase 2 : LMSs Quality Assessment

Fig 13. True System Quality Knowledge Base

Fig 14. Indeterminacy System Quality Knowledge Base

Fig 15. False System Quality Knowledge Base

29 2/22/2018
 Learning Management Systems Evaluation

Phase 2 : LMSs Quality Assessment

Fig 16. System Quality System Quality True Membership

Fig 17. System Quality Indeterminacy Membership
Fig 18. System Quality False Membership
30
 Learning Management Systems Evaluation

Phase 3 : LMSs success measurement

Learner
Personal Factors
Instructor
System Quality
System Factors Information Quality
LMSs Success Service Quality

Organizational Management Support

Factors Training Support

Supportive Ethical and Legal Issues

Factors Cost

Fig 19. Factors Affecting LMSs

31 2/22/2018
 Learning Management Systems Evaluation

Phase 3 : LMSs success measurement

System Design

System Factors Organizational Supportive

Personal Factors
- System Quality Factors Factors
- Instructor
- Information - Management - Ethical and
- Learner
Quality Support Legal issues
- Service Quality - Training - Cost

System Usage Perceived Usefulness Perceived Ease of Use

System Outcome User Satisfaction Intention to Use

LMSs Success
32 Fig 20. The Proposed LMSs Success Measurement Model 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

33 2/22/2018
 Results

Phase 1 : LMSs Software Selection

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Evaluative
Cost Compatibility Support Sustainability
Tools
TRUE 0.4292 0.6382 0.5632 0.5011 0.4779
INDETERMINANACY 0.5902 0.3298 0.4087 0.5027 0.5404
FALSE 0.5708 0.3618 0.4368 0.4989 0.5221
Fig 21. Criteria Weight in Neutrosophic Numbers
34 2/22/2018
 Results

Phase 1 : LMSs Software Selection

0.7 0.6542
0.6 0.57725
0.4992
0.5 0.46875
0.4195
0.4
0.3
0.2
0.1
0
Cost Evaluative Tools Compatibility Support Sustainability

Fig 22. Criteria Weight in Deneutrosophied Numbers

35 2/22/2018
 Results

Phase 1 : LMSs Software Selection

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Moodle Atutor Dokeos Sakai ILIAS
TRUE 0.8838 0.8709 0.8315 0.8147 0.802
INDETERMINANACY 0.0949 0.112 0.1655 0.1895 0.2096
FALSE 0.1162 0.1219 0.1685 0.1853 0.198
Fig 23. Overall Score of LMSs Alternatives
36 2/22/2018
 Results

Phase 2 : LMSs Quality Assessment

0.8
0.7
0.6
System Quality

0.5
0.4
0.3
0.2
0.1
0
1 2 3 4 5 6 7
Fuzzy 0.4722 0.5625 0.5952 0.6061 0.6458 0.6944 0.7333
Neutrosophic 0.4931 0.5193 0.5378 0.546 0.5446 0.5837 0.6336
Fig 24. System Quality in Fuzzy and de-neutrosophied numbers
37 2/22/2018
 Results

Phase 3 : LMSs success measurement

Table 3. Factor Importance by Fuzzy Set and Neutrosophic Set
Factors Dimensions Factors Importance
Fuzzy Neutrosophic set Deneutrosophied
set set
Personal Factors F1: Learner Characteristics 64% (0.66, 0.35, 0.27) 67%
F2: Instructor Characteristics 74% (0.75, 0.36, 0.26) 70%
System Factors F3: System Quality 64% (0.66, 0.36, 0.31) 66%
F4: Information Quality 62% (0.64, 0.32, 0.29) 68%
F5: Service Quality 64% (0.66, 0.36, 0.30) 66%
Organizational F6: Management Support 74% (0.75, 0.35, 0.28) 70%
Factors F7: Training 60% (0.61, 0.34, 0.31) 65%
Supportive F8: Ethical and Legal issues 65% (0.67, 0.32, 0.27) 68%
Factors F9: Cost 61% 0.63, 0.36, 0.35) 64%
38 2/22/2018
 Results

Phase 3 : LMSs success measurement

Table 4. Weight Importance by Fuzzy Set and Neutrosophic Set
Dimension Weight Importance

Fuzzy Neutrosophic set Deneutrosophied

set number
W1: Learner effect on perceived usefulness. 62% (0.64, 0.32, 0.20) 70%
W2: Instructor effect on perceived usefulness. 63% (0.53, 0.25, 0.23) 70%
W3: System quality affect perceived usefulness. 70% (0.61, 0.37, 0.20) 67%
W4: Information quality effect on perceived usefulness. 70% (0.72, 0.34, 0.29) 69%
W5: Service quality effect on perceived usefulness. 62% (0.64, 0.32, 0.34) 66%
W6: Management support impact on perceived usefulness. 58% (0.61, 0.37, 0.38) 62%
W7: Training effect on perceived usefulness. 62% (0.64, 0.32, 0.20) 69%
W8: Ethical and legal issues affect perceived usefulness. 74% (0.75, 0.30, 0.20) 73%
W9: Cost affect perceived usefulness of learners. 61% (0.63, 0.35, 0.37) 64%
39 2/22/2018
 Agenda

Introduction : Problem Statement , Contribution

Related Work

Research Methodology

Learning Management Systems Evaluation

Results

Conclusion and Future Work

40 2/22/2018
 Conclusion

 Applying neutrosophic logic as theoretical and practical

aspects of in LMSs evaluation.
 Presenting a comprehensive model to evaluate LMSs under
uncertainty.
 Using neutrosophic sets gives obvious intuition than the
fuzzy logic which is limited in representing paradoxes.
 Neutrosophic logic is needed for suitable description of an
object in uncertain environment such as expert system.

41
 Future Work

 Personal LMSs.

 Adaptive test generation

 Talent LMSs.

42 2/22/2018
 Thanks Experts
There is no way to express sincerely thank to the experts :
 Alberto Caballero [Associate professor, Computer Science, Universidad Católica San Antonio
de Murcia, Spain ]
 Alex Salas [Learning Management Systems Admin]
 David Cook [Lecturer in the School of Science and a member of the ECU Security Research
Institute]
 Krzysztof Nesterowicz [E‐Learning specialist, PhD candidate, National University of Public
Service, Budapest, Hungary]
 Mugenyi Justice Kintu [PhD candidate, Dept. of Educational Sciences, Vrije Universiteit
Brussel]
 Stylianos Sergis [M.Sc. in "Informatics in Education", PhD candidate, University of Piraeus,
Greece]
 Tarik A. Rashid [Associate Professor on Salahaddin University, Kardistan]
 Vjekoslav Hlede [Senior Learning Management System Specialist, PhD candidate, American
Association of Anesthesiologists]
43 2/22/2018
 Thanks

44 2/22/2018