A Guide to the

FUNDAMENTALS
OF BIG DATA
ENGINEERING
WHAT IS BIG DATA? 1

BIG DATA AND THE CHANGING BUSINESS LANDSCAPE 2

BIG DATA PLATFORMS AND TOOLS 3

BIG DATA IN THE REAL WORLD 5

CAREER OPPORTUNITIES IN BIG DATA 7

BIG DATA MARKET IN INDIA 8

EXPERT VIEWS ON BIG DATA 9

HOW TO BECOME A BIG DATA EXPERT? 10


Everything you do throughout
the day - from waking up to a
WhatsApp message to sleeping
while listening to your favorite
soft rock playlist on Spotify
or watching those cute puppy
videos on Facebook - it is all
of this that combines to make
‘Big Data.’
While you might have heard of the term
Big Data more times than you can count,
do you know what it really means? This is
precisely what this ebook is about - to
help you understand the a-to-z of Big
Data!
UPGRAD
To get started, the term Big Data refers
to an ever-increasing volume of data that
is generated at an unprecedented speed.
It includes all kinds and formats of data,
whether structured, semi-structured, and
unstructured.
Today, organizations and institutions
across the globe are increasingly relying
on Big Data to fuel business growth and
innovation through data-driven techniques
and decisions. In fact, Big Data is the
goldmine that companies today are
sitting on. By applying Big Data Analytics
and advanced ML algorithms to Big
Data, companies can extract the hidden
trends and patterns from it which can be
turned into actionable business insights.
To better understand Big Data, we must
resort to Doug Laney’s articulation of the
3Vs of Big Data -
Volume
Every minute, a massive amount of
data is generated from multiple sources
such as social media, mobile phones,
emails, text messages, and so on. This
data containing both structured and
unstructured data is so huge, complex,
and time-sensitive that it cannot be
processed and analyzed through
traditional data processing techniques
and relational databases - it demands
a unique infrastructure to be stored,
processed, and analyzed.
1
 Velocity
Data is generated by individuals and organizations around the world at such a great
speed that is impossible to comprehend. Track and monitoring this data in a timely fashion
requires advanced techniques such as smart sensors and RFID tags which can handle
large quantities of data in real-time.

Variety
Big Data is different from traditional data in the sense that it doesn’t discriminate
between data and includes under its vast canvas all structured, semi-structured, and
unstructured data. Unlike traditional data that is essentially structured and can be stored
neatly in a relational database, Big Data requires an infrastructure that enables the storing,
processing, and analyzing of huge chunks of metadata.

Now that we’ve established what Big Data is, it’s time to talk about the
business connotations of Big Data.

With data piling up by the minute, Big Data has opened up new vistas
of possibilities for the business world. The spell of Big Data has
become so pronounced that every institution wants to use it, from a
small startup to a Fortune 500 company. According to stats, in 2018
itself, the global Big Data market is expected to generate an annual
revenue of over $42 billion with the biggest share of the revenue
coming from services spending (which was 40% of the total market
share in 2017).

Every company, however small, produces data. This data can generate
from multiple sources - from social media comments and mentions, to
credit card payments and customer feedback. This is where business-
es strike gold. By diving into this data, businesses can uncover valu-
able information about the latest market trends, consumer behavior
towards their products/services, their taste and preference patterns,
and much more. Once businesses have this vital information in their
hands, they can use it to their advantage. For instance, by knowing the
preference patterns of consumers, a business can focus on developing
such products/services that can successfully address the pain points
of the consumers. Then again, a company can take help of the latest
market data to know about their potential competitors and the kind of
products/services they are offering. Consequently, it can come up with
better products or services.

UPGRAD 2
 Like we said before, Big Data isn’t your conventional data - it requires
specialized infrastructure and tools to be stored, processed, and analyzed.
Since traditional approaches don’t suffice, this is where Big Data
platforms come in.

A Big Data platform is essentially an IT solution combining the characteristics

as well as utilities of many Big Data applications, all rolled into one. It
consists of servers, database, storage, business intelligence, and all
other kinds of Big Data management utilities. Thus, to be precise, a Big
Data platform is an IT platform that facilitates the easy development,
processing, management, and deployment of Big Data.

Hadoop is the most popular open-source software framework that

integrates all the essential tools required for Big Data analysis. Here
are some of the most popular Big Data platforms and tools used by
Big Data professionals everywhere:

Cloudera
One of the earliest commercial Hadoop-based platforms, Cloudera is a
fully optimized platform for the cloud. It integrates Machine Learning
and advanced analytics within its infrastructure to enable businesses
to convert complex data into data-driven, actionable insights.

Amazon Web Services

Although it is not an open-source platform, Amazon Web Services (AWS) is trusted by
startups and industry giants alike, and rightly so. AWS offers an all-comprehensive range
of global cloud-based products and services such as storage, databases, computing, analytics,
networking, developer tools, management tools, Machine Learning, IoT, media services,
security, identity, and compliance, and much more.

Hortonworks
Hortonworks is one of the few Hadoop-based platforms that offer a 100% open-source
global data management services to allow businesses to seamlessly manage and monitor
the complete lifecycle of their data. It comes without the restrictions of proprietary software,
thus, enabling you to store, manage, and scale data both in the cloud or on premises. It
was the first platform to integrate Apache HCatalog support to create metadata, thereby,
simplifying the process of data sharing across multiple layers.

UPGRAD 3
 IBM
IBM collaborated with Hortonworks to develop Apache Hadoop - an open-source plat-
form specifically designed to enable distributed processing of Big Data. Although a rela-
tively new solution, Apache Hadoop has garnered a huge fan following owing to the fact
that it offers a highly reliable and scalable environment for the distributed processing of
huge datasets. It also comes with tools for data governance, security, data federation,
and advanced query and data management.

Microsoft
Microsoft’s own brand of Hadoop platform, Azure HDInsights is an enterprise-grade,
open-source analytics package. It has been designed to allow businesses to easily run
other open-source frameworks including Spark, Apache Hadoop, and Kafka. Similar to
AWS, Azure, too, offers a wide range of products including AI/ML, analytics, computing,
databases, containers, developer tools, IoT, and Azure Stack, among other things.

NoSQL
Just as structured data can be easily handled and processed using SQL, to handle
unstructured data you require NoSQL (Not Only SQL). NoSQL has been designed to
function as an alternative to conventional relational databases wherein a data schema is
prepared to carefully arrange the data in tables. So, without following the established
convention of relational schema, NoSQL systems can accommodate a wide range of data
models, documents, and graph formats.

Hive
Apache Hive is a distributed data warehouse and management software that has been
built on top of Apache Hadoop. It can be used for mining, reading, writing, data query,
and analysis of huge datasets stored in distributed databases. However, if you wish to
execute SQL queries and applications over distributed data in Hive, you must implement
traditional SQL queries in the MapReduce API.

UPGRAD 4

Data was produced then, data is being
produced now. However, the difference
between then and now is that today, data
is being produced at an unprecedented
speed and in so large a volume, that it
has given rise to the concept of Big Data.
The rapid rise of Big Data is encouraging
companies across all parallels of the
industry to dive into this data and uncover
the hidden trends within to foster
data-driven business. As a result,
organizations and institutions across the
globe are now harnessing the potential
of Big Data in myriad ways. In fact, as of
2017, nearly 53% companies are using
Big Data analytics (rising up from the previ-
ously 17% in 2015) with the Telecom and
Financial Services sectors being the
strongest proponents for Big Data adop-
tion.
Let’s look at some of the best applications
of Big Data in the real world!
UPGRAD
360° Customer View and Sentiment
Analysis
Unlike before when business was all
about profit-maximization, today, business
is becoming more and more consumer
-centric. Companies are increasingly
focusing on enhancing the customer
satisfaction quotient. Thus, they are
mining customer data from multiple
sources including social media, customer
support data, customer feedback,
mobile devices, and so on. This data is
then processed and analyzed to gain a
360° view of the customer and create
such marketing and sales strategies that
will appeal to the customer.
Sentiment analysis makes up a major
portion of the 360° customer view. Once
the customer data is mined from multiple
sources, Big Data analytics and other
advanced computation techniques are
used to decipher the emotion or the
sentiment behind the opinions that their
customers are trying to convey through
social media posts, comments, text messages,
and feedback. This process is called
Sentiment Analysis which helps a company
understand how a customer is interacting
with its products/services - in a positive
or negative way, or by remaining neutral.
Accordingly, the company can tweak its
products/services to best suit the interests
and preferences of their clientele.
Finance and Insurance - Fraud
Prevention
It is no surprise that the finance and the
insurance sectors deal with large
amounts of financial transactions in a
bulk, thus, making these sectors a
hotbed for fraudulent activities. Before
the advent of Big Data, traditional fraud
detection models were used to detect
fraudulent activities and transactions.
5
 These conventional models needed to be run using complex SQL queries from historical data (bills
and claims) after which one had to wait for weeks for the results. Unlike this time-consuming
process, Big Data analytics, predictive analytics, and Machine Learning are prompt in detecting
anomalies in real-time. These advanced tools can sift through vast loads of historical data, analyze,
and learn about an individual’s transaction behavior from past records, and immediately offer red
flag alerts when they detect a pattern that matches a recognized fraud scheme or activity.

E-Commerce and Recommendation Engines - Personalization

For the smart consumers of today, it’s all about customization and personalization. And hence, you
might have witnessed your own personalized ‘recommendation list’ on e-commerce platforms such
as Amazon, eBay, Netflix, and even Facebook. Both e-commerce and social media platforms are
now using Big Data analytics to wean useful insights from their troves of user data to craft unique
and personalized recommendations for individual users based on their distinct taste and preference
patterns.

Thus, using Big Data analytics brings in a twofold advantage - first, it allows e-commerce platforms
to enrich their product data for an optimized search experience both on mobile devices and desk-
tops, and second, it enables them to customize products in a way that ensures maximum conver-
sion. As for the users, they get to take full advantage of specially curated watch lists or buy lists
which not only saves their time but also contributes to an enriches search/buying experience.

These three popular use cases of Big Data establish a clear fact that while customer/sentiment analysis
is pivotal for businesses in the IT/Tech and E-commerce sector, fraud detection is the dominant use
case in the Telecom, Finance, and Insurance sectors.

UPGRAD 6

As more and more companies keep join-
ing the Big Data bandwagon and the Big
Data market continues to expand, this
rapid change is giving birth to numerous
exciting data-oriented job roles and
career opportunities in Big Data. Compa-
nies across various industries including
IT, Healthcare, Finance, Marketing, Tele-
communications, to name a few are
always on the lookout for trained and
skilled data professionals who are
well-versed in Big Data ecosystem and
can work with massive volumes of data
on a daily basis.
UPGRAD
Here are some of the top-notch job positions
in Big Data:
Data Scientist
Average Annual Salary: ₹300k - ₹650k
A Data Scientist is a professional responsible
for designing data models that leverage
advanced diagnostic analytics or predictive
and prescriptive capabilities to analyze
as well as interpret large datasets. Even if
analytics has to be the strongest trait of a
Data Scientist, to harness the true potential
of Big Data, he/she must know how to
work with advanced statistical, predictive
modeling, and Machine Learning tools.
Once the hidden trends are uncovered
from the data, it is the responsibility of
the Data Scientist to find ways to use
these insights to promote innovation and
efficiency in business and communicate
them to the management team.
As Data Scientists have a diversified role
(they act as a bridge between the IT and
the Business teams), they need to possess
both hard and soft skills. A Data Scientist
must be well-versed in two or more
programming languages (Python, Java,
SQL, Scala, Perl, etc) and have extensive
experience in working with statistical
research techniques. Also, he/she needs
to possess a basic knowledge of Big
Data platforms and tools (Hive, Pig,
Spark, Hadoop, MapReduce).
Big Data Engineer/Architect
Average Annual Salary: ₹400k - ₹1,000k
A Big Data Engineer is the brains behind
a company's entire Big Data infrastructure.
A Big Data Engineer primary job responsibility
is to gather data from multiple sources
and transform this huge data trove into
7
 understandable and actionable insights proficiency in analytical and statistical
that can add value to a business. Once tools, along with a basic knowledge of
the relevant data is gathered, Big Data Machine Learning.
Engineers must build the basic architecture
that’s required for data analysis and
processing. After the data is processed,
Big Data Engineers have to integrate it
within the production and management
infrastructure to generate data-driven
and innovative business solutions.

Since Big Data Engineers have to extensively

deal with Big Data infrastructure, they
must be proficient with platforms and
querying tools like Hadoop, MapReduce,
HBase, Cassandra, HDFS, Pig, Hive,
Impala, and so on. Also, he/she must
have excellent programming skills in
more than one programming languages
(Java, Linux, C/C++, Python, and Ruby)

Big Data Analyst

Average Annual Salary: ₹400k - ₹700k

A Big Data Analyst is a professional

exclusively focused on collecting,
organizing, and analyzing large volumes
of data to uncover and extract the
hidden trends and patterns within. Thus,
the primary job responsibility of a Big
Data Analyst is to mine data and perform
analysis on it using specialized analytics
tools such as Tableau and QlikView.
Apart from this, a Big Data Analyst has to
perform A/B testing (according to various
possible hypotheses) on the data based
to discover what impacts the Key
Performance Indicators directly as well
as indirectly.
It is quite clear from the job description
above that data mining and data auditing
are must-have skills for every Big Data
Analyst. However, in addition to these
core skills, one must also have strong
testing and data visualization skills and
Big Data has taken over the world, and
India is not far behind in the race. Ever
since the wave of Digitization has hit
India, the market for Big Data in India
seems to be soaring high.
At present, the Indian government has
become a strong proponent of Big Data.
Government agencies are integrating Big
Data Analytics into the infrastructure
with the Aadhar, UPI, and the NITI Aayog
being some of the excellent use cases for
Big Data. Another glorious instance of
India’s Big Data application is being put
forth by the Comptroller and Auditor
General of India that has designed a
mandate of sorts to improve its overall
functioning using Big Data - ‘Big Data

UPGRAD 8
 Management Policy.’ With this policy, the CAG hopes to expand the capacity of the Indian Audit and
accounts departments by exploiting the data from both the state and union governments. Carrying
the baton of Big Data forward, today, DISCOMS in India are gathering data from data sensors to
monitor and analyze power consumption and compare it with the historical records of power usage
patterns to deduce preventive measures for combating AT&C (Aggregate Technical & Commercial)
losses.

In the light of this steadily increasing Big Data sector in India, K.S Viswanathan, VP of NASSCOM
maintains that the market value of Big Data sector is expected to reach $16 billion by 2025, with a
CAGR of 26%. If the Big Dara market continues to expand and grow at this rate, by 2025, India will
become a 32% shareholder of the global market.

Big Data might be the most talked about buzzword these days, but what are
the thoughts of experts on Big Data and its capabilities? How does the future
of Big Data look like, according to these experts? Let’s talk a bit about that!

Daniel Yarmoluk is the Director of Business Development, IOT, and Analytics

at ATEK Access Technologies. He’s also the person behind the creation of “All
Things Data Podcast” and is a regular contributor at VertiAI.

Daniel looks at Big Data as a large and careful collection of multiple languages
brought together in a complex environment. He says,

“In order for us to truly deal with speed and velocity, we need to look at the
process of querying big data with proper metadata management tools to
extract value, meaning the orchestration tools need to simplify the process of
managing complex datasets needed to speed up productionising BI and
analysts tasks through process improvements.”

While there are clear winners when it comes to managing Big Data - the likes of Google, Facebook,
Amazon, and such - Daniel believes that “it would be interesting to see other leverage data into
meaningful strategic assets and business models.”

Another well-known expert in the field of Big Data and Analytics, Andrew Chen, Head of Rider
Growth at Uber, says,

“It’s important to leverage data the same way, whether it’s a strategic or tactical issue: Have a vision
for what you are trying to do. Use data to validate and help you navigate that vision, and map it
down into small enough pieces where you can begin to execute in a data-informed way. Don’t let
shallow analysis of data that happens to be cheap/easy/fast to collect nudge you off-course in your
entrepreneurial pursuits.”

UPGRAD 9
 While these two professionals focus on the applicability aspect of Big Data, yet another expert has
something to say about the monetary aspect of it.

A Ph.D., Kirk Borne is the Principal Data Scientist at Booz Allen Hamilton. He’s also one amongst the
top 10 Data Science and Big Data influencers.

According to Kirk, we’ll see much more focus on Big Data and Machine Learning - from the
standpoint of ROI - in the coming years. To quote him,

“The marketing hype on these topics has been intense for a few years, and I believe that the data
community (and its observers) have developed ‘hype fatigue”

Kirk feels that it’s the time for Big Data to demonstrate value. According to him, the most important
“V” of Big Data is “value”. Adding further, he says “That [value] refers to value creation and
innovation across all data and information assets, Our stakeholders will demand to see more
discussion, demonstration, and proofs of value and ROI from big data in the coming year.”

All in all, when it comes to the future of Big Data, there is a consensus amongst all the experts -
there’ll only be uphill from here. And while it means the path will be tougher than it has been
(because of the ever expansion in the quantity of data), it also means that everything will only get
improved and advanced from here on.

By this point, it’s extremely lucid that Big Data is indeed the present and the
future. The market has also shaped up beautifully in this field, opening up an
increased number of job roles and responsibilities. What must be the need of
the hour, then?

Exploring the field and understanding the nitty-gritty's of it, of course!

The world that we’re in today demands Big Data experts. And if you have
even a slight inclination towards numbers, computer science, data, statistics,
and basically working on the most modern technology, you should definitely
look to explore what this field has to offer!

To help you with just that, BITS Pilani and UpGrad offer a PG Program in Big
Data Engineering - the first of its kind. Comprising of a comprehensive
curriculum curated by various industry experts along with Birla Institute of
Technology and Science, Pilani, this PG program will help you upskill in Big
Data.

UPGRAD 10
 Big Data is a field where just theoretical knowledge isn’t enough. Keeping that at the forefront
of this program, UpGrad included five real-life projects across a number of industries.

Sponsored by Saavn, these projects will not only build an extremely solid foundation for the rest
of your career but will also make the whole learning experience utterly enjoyable. Other than
this, you also get a 24x7 cloud lab access to AWS. The instructors for this course are experts and
leaders in academia belong to BITS, Impetus, American Express, and more.

This program is aimed at everyone from people currently working in the IT industry to Big Data
enthusiasts. The curriculum is structured in such a way so as to ensure that you get started from
the scratch, and pave your way to the very top. So, no matter what your background is, if you’re
interested in building a career in Big Data and are looking for a way to become an expert in the
field, this is where your search should stop!

UPGRAD 11
 G e t a P G Ce r ti fi c a ti o n i n
Big Data Engineering

WHY UPGRAD WITH BITS PILANI?

Learn on the Go The best of Industry

Experience immersive learning Designed in collaboration with leading
with 200 hours of dedicated industry experts and academia to
lectures, from anywhere anytime create experts in Big Data Engineering

Active Student Mentorship Dedicated Student Support

Get unparalleled guidance with Our team of in-house student advisors
1-on-1 networking from industry assist the students on all fronts - we
experts and academia excel make sure you excel

FIND US HERE:

Ha v e Q u e s ti o n s?
P l e a s e f e e l f r e e t o d r o p u s a l i n e a t i n f o @ u p g ra d . c o m
and we will be there to help you.

CO P Y R I G H T @ U P G R A D E D U C AT I O N P R I VAT E L I M I T E D