eBOOK

Big Data Explained, Analysed, Solved


Big Data Explained, Analysed, Solved
What you will learn
This eBook gives an overview of what big data
is and its growing importance. It talks about
some of the different kinds of big data, as well
as some of the different things you would
do with it.
Tweet this
The functional section of this book discusses
applications, tools, managed services and
clouds, used together or separately, that
will help you benefit most from big data.
You can skip directly to any section and focus
on what’s most important to you, or read the
book straight through.
2
Canonical is involved in Big Data
Canonical, the company behind Ubuntu,
works closely with its partners on all aspects
infrastructure and partner solutions to
support storing, managing, and analysing
big data.
 3
Big Data Explained, Analysed, Solved

The Author

Bill Bauman, Strategy & Content, Canonical,

began his technology career in processor
development and has worked in systems
engineering, sales, business development,
and marketing roles. He holds patents
on memory virtualization technologies
and is published in the field of processor
performance. Bill has a passion for emerging
technologies and explaining how things work.
He loves helping others benefit from modern
technology.

Bill Bauman
Strategy & Content, Canonical

Tweet this
 4
Big Data Explained, Analysed, Solved

Contents

Overview Functional Partnership

What is Big Data, a general overview 05  esign, deploy, and package
D 13  anonical as a strategic partner
C 22
Big Data solutions for Big Data
The increasing importance of Big Data 07
Juju Big Data Charms 14 Conclusion 23
Different types of Big Data 08
Juju Big Data Frameworks 15 About Canonical 24
Big Data analysis and action 10
Ubuntu for Big Data systems 16
Do I need a cloud for Big Data? 11
OpenStack is a Big Data warehouse 18

BootStack for Big Data 19

Ubuntu Advantage Storage for Big Data 20

Machine Containers for Big Data 21

Tweet this
 5
Big Data Explained, Analysed, Solved

Big Data general overview

Big Data refers to extremely large sets of Traditional Data

data that aren’t easily stored or analysed by
To understand big data, consider some
traditional methods. Typically the data is too
examples of traditional data. Traditional
large, varied in nature, or moves too fast, for
data may be a database of clients, with their
traditional database systems to handle it.
associated contact information. It could be
This often is referred to as volume, variety,
a database of cars, years, makes, models. Structured Database
and velocity.
This sort of data will usually grow gradually
in size and the types of data stored
rarely changes.

Traditional data is generally well-structured

and fits predefined or predictable categories.

Predefined datasets

Tweet this Incremental, predictable growth


Big Data Explained, Analysed, Solved
Big Data
When we look at Big Data, typically the data
is not so neatly organised. Some big data
examples could be random spots on a map,
documents, images, huge lists of named or
unnamed individuals that have happened to
all be in the same general area at a given time,
or the millions of clicks on a web page
in a given week.
Big Data can be structured or unstructured,
but generally the database and analysis tools
are specially designed for a given purpose
and to handle the tremendously large scale,
size, velocity, and variety that most big data
datasets represent.
Tweet this
6
The general purpose
The reason that these gigantic data sets are
being compiled and stored is so that we can
analyse the data. Analysis includes pattern
recognition, trends, associations, etc. The
Often Unstructured outcome of analysis is respective actions
that would otherwise not be possible
without big data.
In the next section of this eBook, we go into
further detail about big data analysis - why
we do it, why it’s important, and the sort
of information for which we’re looking.
Purpose-specific toolset
Rapid growth

Big Data Explained, Analysed, Solved
The increasing
importance of Big Data
Collect
Organisations of all sizes and functions are
increasingly gathering more information
about their interactions and transactions.
They are also looking to third parties to
provide additional data. Regardless of how
they gather data and the types and quantities
are increasing. In a modern, data-driven world,
an organisation that isn’t taking advantage
of big data collection, analytics, and action,
is likely going to become uncompetitive with
those that are.
Tweet this
Analyse
The analysis of big data can have big returns.
The ability to understand the types of data
that are collected, to correlate one type of
data with another, observe trends, identify
outliers, and many other analytic functions,
are increasingly valuable in organisations
of all types.
Without thorough analysis via the use of
modern, big data analytics tools, it can be
easy to miss or overlook important trends,
shifts in perspective, or subtle changes
in customer interaction. Through analysis,
you can learn patterns and predict actions
before they occur and even begin to direct
them via actions discussed here in the
Act section.
7
Act
The ability to do something with the data
that is collected and analysed is the most
compelling part of big data. Corporations
can offer more compelling products and
solutions. Governments can better predict
and serve the needs of citizens. Even small
business can identify short and long term
trends in their sales and interactions with
customers, as well as other businesses.
All of these outcomes are about improved
efficiencies and experiences for everyone
involved, from the provider to the consumer.
 8
Big Data Explained, Analysed, Solved

Types of Big Data

Big data can be structured or unstructured. Structured big data Compilations

New tools and datasets are blurring the lines
Compiled big data is merging existing or
that separate the two. Below are some
otherwise disparate databases into a single
common examples of big data types.
dataset. For example, the data could include
names, locations, demographics, account
balances, credit scores, etc, all combined
into a compiled big data dataset.
Remember, just because it is structured, does
not mean it isn’t big data. Structured big data Transactions
could be compiled from millions or billions of Transactional big data is everything having to
data points, daily or even hourly. do with a transaction, including whether the
transaction was even completed. The data
User Input
could include what was purchased, how long
This is data that is created via a prompt or it took, was it online or in-store, were other
requested action to a user. This could be a items typically purchased together.
ratings system, a survey, a loyalty program, or
any other prompt for the user to input specific
data in specific fields that are then stored in a
structured manner.

Tweet this

Big Data Explained, Analysed, Solved
Unstructured Big Data
Big data is most commonly associated with
unstructured data. Unstructured data, like
photos and IoT datasets, were largely the
genesis of modern big data.
Tweet this
User-generated Content
Every day, millions of Internet users post
pictures, videos, short messages, audio,
and more. Much of this data is completely
unassociated with a category or field.
Essentially, it is completely unstructured
and it is the function of targeted big data
applications to aggregate, cull, present,
and analyze these datasets.
9
Passive Data
This is generally the data that is generated
without specific intent or interaction from
users. For example, cell phones are perpetually
updating GPS coordinates of their users’
respective locations. Logistics information,
bar code scans, delivery information, are
all data that are passively updated but can
provide valuable insights when analyzed.

Big Data Explained, Analysed, Solved
Big Data analysis and action
Predictive Analytics
Probably the most common type of
analysis, using past patterns or performance
to determine future actions is one of the
best known uses of big data. It’s important
to analyse data from a multitude of different
perspectives and to include cross referenced,
sometimes loosely-associated data, to establish
the most comprehensive patterns and future
predictions. Predictive analytics can also be
bolstered by machine learning, whereby, over
time, the system builds its own intelligence
profile on a given a subject, individual, or topic.
Tweet this
Descriptive Analytics
The focus here is on metrics, a summary of
what has happened. This could be views, clicks,
counts, posts, etc. While descriptive metrics
are not necessarily incredibly useful on their
own, they are the underlying data points that
feed more advanced analysis and actions.
Descriptive analytics have been used for many
years now, and are the foundation of the many
graphs and charts we see on the Internet and
in presentations today.
10
Prescriptive Analytics
Largely an intelligent evolution of predictive
analytics, with a prescriptive approach, data
analysis is used to determine recommended
actions. Where predictive analytics looks
at patterns and makes recommendations,
prescriptive analytics looks at patterns,
associates them with additional datasets,
determines where individual data points
coincide or there are recurring common
descriptors or activities, and then prescribes
a potential course of action or solution.
Prescriptive analytics are generally
underutilised but offer great potential
to reduce time to market for solutions
or assessment times for individuals
in various fields.

Big Data Explained, Analysed, Solved
Do I need a cloud for Big Data?
Even though big data was born in the cloud,
it doesn’t mean you need a cloud to take
advantage of big data solutions or to act
on the data. The most important aspects
of working with big data are that you
have chosen the right tools and the right
applications for your solution. Canonical
can help you with both.
Canonical has created an open source solution
for system design and service modeling called
Juju. Juju simplifies the process of designing
your solution, then configuring, associating,
and deploying the applications in it. Having
a tool like Juju means that selecting the right
big data applications for your needs is the
most important remaining factor.
Tweet this
11
For more information on Juju, see section For more on building your own private cloud,
Design, deploy, package Big Data solutions. see the sections OpenStack is a Big Data
warehouse and BootStack for Big Data later
Although it isn’t necessary, a cloud can
in this eBook.
be tremendously beneficial to big data
processing. The nature of big data is that it is
constantly changing, and the purpose of that
data, the analysis of that data, and the storage
of that data can change just as quickly. A tool
like Juju can help you keep up with the change
in usage by deploying new big data charmed
solutions. But Juju can’t do it all.
For system scalability and the ability to easily
access different types of storage for different
needs, a cloud is recommended. Juju can
talk directly to both public and private cloud
solutions, like AWS and Canonical OpenStack,
respectively.
 12
Big Data Explained, Analysed, Solved

Choosing the right applications

There are many ways to go about application
selection. Some people already know which
big data processing solutions they want to
use. Others are looking for advice, or looking
to explore potential new solutions.

In the Juju Big Data Charms section of this book,

we outline many big data software solutions
that are available, and give a brief description
of their purpose. This is a great starting point
to see what’s out there, and Juju makes it easy
to try them all.
Juju is the game-changing service modelling tool BootStack is your OpenStack private cloud, running
that lets you build entire cloud environments with on your hardware, in your choice of datacentre
only a few commands. with Canonical’s experts responsible for design,
deployment and availability.

Additionally, in the BootStack for Big Data

section of this book, we go into detail on how
a BootStack cloud helps to start processing big
data quickly and efficiently.

Tweet this

Big Data Explained, Analysed, Solved
Design, deploy, package
Big Data solutions
Whether in a cloud or on a dedicated system,
managing all the applications in a big data
solution is best handled by a tool that does
more than static configuration management
or orchestration deployment.
Juju is a service modelling product from
Canonical that gives you a blank canvas on
which you can visually lay out all of your big
data apps. Communications and data paths
are defined as relationships between the
applications by connecting the apps on your
canvas. The visual solution design and all the
application relationships can be deployed
immediately, and exported and saved
as a bundle for future use.
Tweet this
13
Juju, Charms, and Bundles Evolving the solution
The use of Charms is what gives Juju its incredible When it comes to big data processing, the
capabilities to manage applications in complex solution is rarely static. Big data deployments
infrastructures. Charms are intelligent scripts evolve over time, and that often involves adding
wrapped around big data applications that or removing components services. The same
allow them to be dynamically configured tool that you used to design and deploy
and deployed without manual configuration. the solution can be used to dynamically
add and remove components within it.
The abstraction of application relationship
Juju’s service modelling approach lets you
management by Juju’s Charms is what allows
evolve your solution and keep pace with
big data solutions to be rapidly deployed and
the rapidly changing big data market.
seamlessly scaled. Without the application
abstraction that Juju provides, big data system
services require manual intervention or iteration
of inflexible, static configuration scripts any
time the solution design needs to be updated
or changed.
 14
Big Data Explained, Analysed, Solved

Juju Big Data Charms

Ingest & Messaging Scale Out Storage Analytics / Search /Visualisation

• Message Processing • Ceph • SpagoBI

• Flume • Swift • Saiku
• Kafka
• Storm
• Message Queues noSQL
• Spark
• RabbitMQ
• Stack
• ZeroMQ • Datafari (ManifoldCF, SolR)
• ElasticSearch
• LogStash • Zeppelin
Structured Data • Kibana
• iPython Notebook
• MySQL • Document Databases
As discussed on the Design, deploy, package
• PostGreSQL • MongoDB Big Data solutions page, these are a sample
• CouchDB of the Charms available for big data. With Juju,
• Percona Cluster
• Couchbase you can readily deploy any combination of
• MariaDB • Column & KV these Charms and define their configurations
and data paths all from a graphical interface,
• Cassandra / DSE CLI, or API.
• quasardb
• memcached
• Redis
Tweet this
 15
Big Data Explained, Analysed, Solved

Juju Big Data Frameworks

Big data frameworks are available for Hadoop Spark

deployment in Juju. You can deploy an entire
• Hadoop Flavours • Spark
Hadoop cluster with a Juju Charm bundle,
or Spark, Docker, or Kubernetes, for example. • Apache Hadoop • Spark Streaming
The Charms listed on the Juju Big Data • Cloudera Hadoop
• Spark SQL
Charms page can all be associated with the
• YARN
frameworks listed here, as appropriate. • SparkML
• Hive
All of these frameworks benefit from Juju’s • GraphX
ability to automatically configure application • Mahout
data paths and relationships.
• HBase Container Ecosystem & Orchestration

• Pig • Docker

• ZooKeeper • LXD / LXC

• Flume • Kubernetes

• Kafka • Mesos

• Tez

• Storm

• Hue
Tweet this
 16
Big Data Explained, Analysed, Solved

Ubuntu for Big Data systems

Ubuntu Server is the most popular cloud
operating system in use. There are many
reasons why Ubuntu is so popular, but one
of the primary reasons is that Canonical
started to focus on OS scalability many years
ago. When you’re working with big data, you
need a cloud-ready platform, like Ubuntu,
that is designed for scalability and reliability.
Ubuntu Server can be used as a traditional
operating system. There are also optimised
variants for low latency and other task-specific
solutions, like big data processing.
Where Ubuntu runs:
• On-premise, in your own cloud
• In an external, private cloud
Ubuntu allows you to process your big
data anywhere. Keep sensitive information
in-house, leverage the public cloud for
unpredictable workloads, and trusted private
cloud partners for both.

• On public clouds, like AWS, Azure, Rackspace,

Google Cloud Platform, IBM, and many
others, please see the Ubuntu Certified
Public Cloud page for more

Tweet this
 17
Big Data Explained, Analysed, Solved

How Ubuntu runs:

The flexibility of Ubuntu to run anywhere
on almost any architecture makes it the ideal
platform choice to execute big data workloads.

 are metal server on - x86,

B  irtual Machine on - KVM,
V Public cloud guest instance
ARM, POWER, or z Mainframe VMware, Hyper-V, and
other hypervisors

Private cloud guest instance Container on bare metal Container as a virtual machine Container as a cloud instance

Tweet this

Big Data Explained, Analysed, Solved
OpenStack is a Big Data warehouse
The section Do I need a cloud for Big Data
in this book addresses some of the benefits
of clouds for big data. Specifically, an OpenStack
cloud is the most popular private cloud solution
for big data.
OpenStack is a community-based private
cloud solution. It is not a single product, but
a collection of individual projects designed
to seamlessly interact to create a functional
cloud. Canonical OpenStack is a production-
ready, supported OpenStack distribution,
and more.
The best way to build an OpenStack cloud
is using Autopilot. Autopilot is a graphical
installation tool that allows you to select
the components of OpenStack you would like
to install and deploys them for you. It can even
deploy them with high availability.
Tweet this
18
Autopilot is designed to work with an The base platform of Canonical OpenStack
extended tool set beyond just OpenStack. is Ubuntu. Ubuntu is not only the most popular
cloud operating system, it is also the most
MaaS, Metal as a Service, automates the
popular OpenStack infrastructure operating
configuration of the physical nodes in your
system. Ubuntu runs on the OpenStack
OpenStack environment. Juju, discussed
physical nodes, providing critical services
further in the Design, deploy, package Big
like compute, networking, and storage.
Data solutions section of this eBook, allows
It is also the platform for your guest instances,
you to automatically deploy applications
whether they are LXD machine containers
and their respective relationships within your
or virtual machines, where you run your big
OpenStack cloud. Landscape manages the
data applications.
Autopilot experience, as well as the cloud
itself, and the guest instances within it. Combining OpenStack with Canonical’s
feature-rich tools and Ubuntu creates a
The comprehensive tool set that comes
scalable, reliable, automated platform for
with a Canonical OpenStack cloud makes it
deploying and managing big data solutions
easier, faster, and more robust to deploy big
for any type of analytics, monitoring, and
data solutions - from the bare metal, to the
more. Canonical even guarantees upgrade
platform operating system to the
ability of your OpenStack Big Data cloud.
applications themselves.

Big Data Explained, Analysed, Solved
Big Data Cloud, quick and easy
BootStack is a unique, managed Canonical
OpenStack offering. It is unique in that you
may choose to run the solution in your own
datacenter, on your own hardware, or in a 3rd-
party hosted facility, like IBM SoftLayer,
an Ubuntu Certified Public Cloud partner.
Canonical’s engineers have years of OpenStack
experience. With BootStack, you can leverage
their knowledge of how-to and best practices
and have a Canonical OpenStack cloud ready
for big data processing in days.
With BootStack, you focus on the data, and
Canonical takes care of the infrastructure.
Additionally, when you want, Canonical can
transfer total control of your OpenStack
environment to you.
Tweet this
19
All of the tools that make Canonical Whether you just want to try it out, don’t
OpenStack the platform of choice for big have the in-house skills, or want to get up
data are included in BootStack. Even better, and running quickly, BootStack can provide
they can be preconfigured for you and ready the answer to a big data cloud. To learn more
for use. As soon as your BootStack cloud is about BootStack, and use the BootStack
ready, you can start using all the big data calculator to calculate potential savings,
solutions in the Juju Charm Store. You’ll find visit the BootStack managed cloud page.
the core big data solutions you expect and can
even start discovering new big data solutions
from all our Charm partners.
BootStack is billed on a pay for use model.
The model is similar to that of Ubuntu
Advantage Storage. These unique and
innovative price models are part of the
initiative to make private cloud usage and
consumption as easy to calculate and predict
as that of public clouds.
 20
Big Data Explained, Analysed, Solved

Ubuntu Advantage Storage

Ubuntu Advantage Storage is a unique and Pay for what you use
ideal storage solution for big data storage

Unused Capacity
Another unique feature of Ubuntu Advantage

Unused Capacity
Your
and real-time processing. It is based on
Storage is its pay for use, metered model. As Content

New
Software Defined Storage (SDS) solutions, Storage
opposed to paying for all the storage in your
allowing for flexibility and modern data
datacenter, you just pay for the storage that’s

Redundant Data
management approaches.

Unused Capacity

Unused Capacity
actively in use. Additionally, you don’t pay

Total Capacity

Total Capacity
for replicas or online backups. The cost savings
Choose the right technology

Total Capacity
compared to other SDS-based and managed
Ceph, NexentaEdge, Swift and SwiftStack are storage solutions can be 2x to 3x,
all supported by Ubuntu Advantage Storage. or even more. What What What

Used

Used

Used
you you you
That means, you choose the right technology pay for pay for pay for
The pay for use model of Ubuntu Advantage
for your solution, and it is all directly supported
Storage is similar to that of our managed
by Canonical. The hardware you choose to Grow your capacity, Increase your redundancy,
OpenStack solution, BootStack. These unique without growing your bill pay the same!
run the solution on is just as important, and
and innovative price models are part of the
Canonical’s partners and engineers can help
initiative to make private cloud usage and
you with that, as well.
consumption as easy to calculate and predict
as that of public clouds.

Tweet this
 21
Big Data Explained, Analysed, Solved

Machine Containers for Big Data

Machine containers are a relatively new
technology in the virtualisation ecosystem.
Delivered by Ubuntu as a technology called
LXD, they provide the management of
traditional virtual machines without the
system overhead.
Many big data solutions execute optimally
when run at bare metal speed. That can limit
the use of virtualisation, though, and restrict
system placement. By using LXD, multiple
services can share a single system and all have
direct hardware access.
LXD isn’t just about performance. There are
big data workloads that run in public clouds
as guest instances. Almost all of those instances
are virtual machines. One of the benefits
of LXD machine containers is that it provides
process isolation and application mobility
(live migration) to running processes. That
means increased manageability for public
cloud instances, as well as bare metal and
private cloud solutions.

Multiple services can share a single system and all have direct
hardware access

Tweet this
 22
Big Data Explained, Analysed, Solved

Canonical as a strategic
partner for Big Data
Working with Canonical as your valued partner will maximise your success with big data.
Some attributes to keep in mind and that Canonical delivers are:

Your strategic big data partner should

understand and have experience designing,
building, deploying, and managing scalable
infrastructures and big data applications.
Ideally that partner brings with it an entire
Scalability Application catalog Prebuilt, intergrated Time to solution ecosystem of additional big data partners.
bundles
Canonical works closely with a multitude
of big data software and platform providers
to ensure choice in solutions while maintaining
quality and integrity in the overall stack.

24/7 Support Existing expertise Managed offerings ...and more

Tweet this

Big Data Explained, Analysed, Solved
Conclusion
There are many kinds of big data.
There are many big data applications, services,
and solutions.
Canonical has domain expertise, understands
big data, has strong industry partnerships, and
can provide a scalable, supported solution.
Tweet this
Your data is important. You need to know how
to store, process, and act on your data. The
overview, explanations, and solutions outlined
in this book will get you started or accelerate
your journey to maximising the benefits of the
data you have and the new data you will start
collecting.
Your best next step is to contact
Canonical today.
23
If you’re excited to hear more and talk
to us directly, you can reach us on our
Contact Us page.
To learn more about a managed solution for
big data, download the paper BootStack Your
Big Data Cloud.
If you want to start trying things out
immediately, we highly encourage you
to visit Juju solutions for big data.

Big Data Explained, Analysed, Solved
About Canonical
At Canonical, we are passionate about
the potential of open source software to
transform business. For over a decade, we
have supported the development of Ubuntu
and promoted its adoption in the enterprise.
Tweet this
24
By providing custom engineering, support
contracts and training, we help clients in the
telecoms and IT services industries to cut
costs, improve efficiency and tighten security
with Ubuntu and OpenStack. We work with
hardware manufacturers like HP, Dell and
Intel, to ensure the software we create can
be delivered on the world’s most popular
devices. And we contribute thousands of man-
hours every year to projects like OpenStack,
to ensure that the world’s best open source
software continues to fulfil its potential.