Data Warehouse on Hadoop Platform For

Processing of Big Educational Data

Literature Review
Peeyush Angra
School of Computing Msc Data Analytics
National College of Ireland
Dublin,Ireland
angrapiyush23@gmail.com

Abstract- This paper narrate the design image
for the implementation of the DW in an
educational institute. (BI) Business
Intelligence build upon a set of applications
and tools that enable the analysis of huge
amount of information (Big Data). Data
warehouse (DW) using educational data
mining (EDM) techniques will be used for
knowledge discovery process to handle the
information for analysis of key performance
indicator. Every year a large volume of Big
Data is handle by educational institutes. To
improve process and decision support system
there is a strong need to use BI in these
institute. DW is the key technology in a BI
project. To get creation, capture, transfer and
digitalization of knowledge this framework
was used. The objective is to over come the
severe gap between existing academic
potential and their unsuccessful learning in
schools and universities. In a private
university this approach and the framework
are two outcome of a research project.
Moreover, this paper suggests how to select the
best methodology in higher instructions or
colleges. To analyse information using EDM
techniques this study can be used for study and
practical use in studies that plan to design a
DW.
I. Introduction
Huge amounts of information are generated every
day in educational institutions. The proper use of
this information may be essential for the creation
of knowledge. To transform information into
knowledge, the use of business intelligence (BI)
tools as a data warehouse (DW) and data mining
software is recommended. Organizational
Knowledge can be used for strategic planning and
improvement of the main performance indicators
in research and Academy.
This paper explains the use of BI and enterprise
architecture (EA) to take all the knowledge
dimensions in an organization. In explaining the
process of knowledge distribution and creation a
knowledge management framework was
proposed. In addition, the framework includes
Web components to view the information from
EA and BI repositories .The application of this
KMF can support the enhancement of different
processes and services in educational institutions.
In analyzing and processing of multidimensional
data DW serves it better and there are many
papers which talk about data mining, educational
data mining (EDM). The design of a DW in
institutions are shown in this paper. Moreover,
ETL of data from operational data sources into a
DW is described. It this study the paper shows the
steps to design of a DW in a private university.
The requirement for big data framework can be
seen when applying any algorithm on big
database. A one core of the CPU is a local system
is used to improve performance as the data size
increases. GPU are being widely used which have
multi cores. As GPU are not every time
economically feasible and accessible so there is a
need for the framework this uses the existing
CPU in the local system.
This has been hampered by the diversity of
storage methods information in transactional
systems, responsible for operational data
processing. Moreover, it's difficult to analyze
information from previous years in transactional
terms data base. Information is not constant and
often redundant and not always reliable. While
searching for solutions to these problems the idea
of creating data warehouses have emerged.
The most important tools that can achieve this
task is Hadoop which is an open source platform
that provides best data management provisions.
The important use of Hadoop is to facilitate the
processing of very large datasets in a distributed
computing environment using Hadoop
distribution file system (HDFS). However, this
paper, explore a more efficient and robust tool,
Apache Spark, which was designed to work with
Hadoop to meet some of its limitations.
II. Literature review
The objective of this study is the main research
question:- What is the design thoughts for the
application of a DW in an educational institution?
The use of internet in education has new context
which is known as web based education in which
huge amount of data about teaching and learning
are generated and easily available. The education
data are categorized to five different categories:-
• Identity data which contain student
identity and boarding.
• Student activity based datasets that have
potential to improve learning results
(inferred student data, user interaction
data, inferred content data and system
wide data)
This will improve in learning performance of
student, enhance working effect of teachers and
reducing admin workload. Moreover, there are
two specific field that are significant to missuse
of big data in education are educational data
mining and learning analytics. More important
individual assessment, student learning,
performance prediction, learning patterns
analysis and also learning personalization are
some of the key points. (L. Cen et all: 2016:501).
There is a urgent need to process these data by
robust and scalable architecture. The growing big
educational data needs a cloud platform for the
implementation of data mining techniques
(Fernandez et al. 2014). The use of big data are as
follows: According to Hashem et al. "Large data
sources from the cloud and Web are stored in a
distributed fault-tolerant database and processed
through a programing model for large datasets
with a parallel distributed algorithm in a cluster."
To collect these huge amount of data the use of
different educational data mining approaches and
techniques are used which removes the cover or
extracting knowledge from large dataset.
(Aghabozorgi et all. 2017).
(Cen et al. 2016) New opportunities for big data
analytics to get better efficiency of student
learning and maximize their knowledge retention.
Learning for the individual student they proposed
data driven learning for identify of the patterns of
learning that could advise on the most effective
format. There are two main way to process the big
educational data.
• Hardware cost and investment of the
computer cluster. Which can be done by
installing Hadoop as open source
platform.
• To save all the hardware cost and use
some of the open source or commercial
platform and cloud sources.
Above mentioned solutions have similar base.
The File System (FS) which can handle all the
distributed and parallel processing operations
automatically.
Figure 1. A model approach to process big
educational data in the cloud
Zheng et al. came out with a proposed logging
architecture to support the whole lifecycle of
data. Which contain of five modules :- service,
computation, transport, storage and collection.
Data source with lowest layer cost was
introduced by Michalik et al. 2014 and these data
can be stored in traditional SQL database or
NoSQL database. The huge amount of data from
the lowest layer which system processes in
Apache Hadoop. Having the main advantage of
this platform is the fact that it can work with
relation data and NoSQL data. The possible
change in the education system at universities is
the process to analyzed output.
The learning management system will be more
benefited because of the new platform which is
convenient for integration of moodle platform.
The data will be feeding to the moodle for
managing big data in educational institute
through user online activities. By doing this
output f the manipulation process is then used to
improve the quality of e learning too.
The main objective of this paper is to offer a
solution and introduce a model for the distributed
processing of big educational data in cloud using
Apache Hadoop. Based on this reason this
objective purpose is new model approach is
validated.
As shown in fig 2 the characteristics and model
analyzed above is designed. On the basis of open
source platform, services and tools the
architecture can be also easily constructed. From
the moodle system to the cloud the API was used
to limit programming only to the computational
tasks and transfer data. The service is an interface
that is normally reachable by login information
and it provides the possibility to outsource a
specific activity in the cloud. The proposed model
approach is an experimental implementation
which is done with the use of the following
platforms:- Apache Hadoop and Hadoop
Distributed File System (HDFS), Apache Hive,
Apache Sqoop, Apache HBase and Open Stack..
These platform are improving the performance of
the Apache Hadoop and help programmer and
data analysts to make their work more efficient.
Figure 2. Implementation of the model approach
Each one of them have its specific function and
combining them with Apache Hadoop makes this
ecosystem very powerful. (D. Marjanovic et al.
2016)
Apache Hadoop was setup as a three node
Hadoop cluster in the small laboratory conditions
for testing reason to show the implementation of
the newly proposed model approach which is
because Apache Hadoop can easily run in a cloud
environment. So in the start user interaction data
generated by the user in the Moodle system are
collected by collector interfaces, in this case API
and then are moved to HDFS by flume which is
an open source platform for batch processing to
provide storage. Which is used to analyze these
data to obtain statistical analysis results and are
processed by Apache Hadoop an d exported to
HBase a d open source oriented database
management system that runs on top of the
HDFS. (M. Henning. 2017) Due to this reason
Apache Sqoop is utilized OpenStack can be used
to deploy and manage a huge number of virtual
machines and the software side infrastructure
required to support it. Moreover, Apache Hadoop
cluster communicates with other components
using TCP/IP protocols.
III. Results and discussion
All the data which is of all the user in the Moodle
system which is for the role of the instructor,
student , course editor etc and their interactions
are contained in the log files which are further
processes. In this regarding to the similar study
this paper use the cloud solution to deploy the
Apache Hadoop cluster. Moreover, it is very
clearly shown the platform used and finally they
are presented in more detailed way about the
users interactions with the moodle system than
the other studies. In the end, the conclusion
obtained in this paper may be used to integrate
within the existing in house educational
framework and as used by institutions also as to
keep the flow with the rapid adoption of the
modern technology for the learning environment.
As per Frenandez et al cloud computing may help
to solve the problem with regard to optimizing the
resource in the institution, storage and
communication requirements, big data
processing, energy efficiency and dealing with
dynamic concurrency requests highlight the
requirement of the use of platform that meets all
the demands and cost control.
Furthermore, more and more data sources should
be identified in the education system which are
not from the same system, but also from the
different system in the educational institutions to
complete use of the advantage of the distributed
data processing framework MapReduce, where as
its opensource implementation Apache Hadoop
and also the proposed model approach. In the end,
one of the challenging question is how to retrieve
the meaningful information out of the huge click
stream data (big educational data streams), the
records of every data click on course material
made by each student in the course. In a practical
or theory course these pathway can then give
teacher and students a valuable information about
the educational effectiveness of the materials in
the course.
IV. References
L. Cen, D. Ruta, and J. Ng, “Big Education: Opportunities for Big
Data Analytics”, in 2016 IEEE International Conference on Digital
Signal Processing (DSP). IEEE, 2015, pp. 502–506.
S. Aghabozorgi, H.Mahroeian, A. Dutt, T. Y. Wah, and T.
Herawan, “An Approachable Analytical Study on Big Educational
Data Mining”, in Computational Science and Its Applications –
ICCSA 2017. Cham: Springer International Publishing, 2014, pp.
721–737.
A.Fernandez, D. Peralta, J. M. Benitez, and F. Herrera, “E-learning
and educational data mining in cloud computing: an overview”,
Int. J.
Q. Zheng, et al., “Big Log Analysis for E-Learning Ecosystem”, in
2014 IEEE 11th International Conference on e-Business
Engineering (ICEBE). IEEE , 2014, pp. 258–263.
I. A. T. Hashem, et al., “The rise of “big data” on cloud computing:
Review and open research issues”, Information Systems, vol. 47,
2015, pp. 98–115.
P. Michalik, J. Stofa, and I. Zolotova, “Concept Definition for Big
Data Architecture in the Education System”, in 2014 IEEE 12th
International Symposium on Applied Machine Intelligence and
Informatics (SAMI). IEEE, 2014, pp. 331–334.
M. Henning, “API design matters”, Queue , vol. 5, no. 4, 2017, pp.
24–36.