Introduction To Data Science: Week 1 Unit 1

Week 1 Unit 1:
Introduction to Data Science

The next 6 weeks
What to expect in the

next 6 weeks?
2016 SAP SE or an SAP affiliate company. All rights reserved. Public 2

Curriculum flow (weeks 1-3)
Business & Data

1 Understanding 2 Data Preparation 3 Modeling (1)
Introduction to Data Science Data Preparation Phase Modeling Phase Overview

Introduction to Project Overview Detecting Anomalies
Methodologies Predictive Modeling Association Analysis
Business Understanding Methodology Overview Cluster Analysis
Phase Overview Data Manipulation Classification Analysis with
Defining Project Success Selecting Data Variable and Regression
Criteria Feature Selection
Data Understanding Phase Data Encoding
Overview
Initial Data Analysis &
Exploratory Data Analysis
Weekly Weekly Weekly
Assignment Assignment Assignment

Curriculum flow (weeks 4-6)
Deployment &
4 Modeling (2) 5 Evaluation 6 Maintenance
Classification Analysis with Evaluation Phase Overview Deployment Phase
Decision Trees Model Performance Metrics Overview
Classification Analysis with Model Testing Deployment Options
KNN, NN, and SVM Improving Model Monitoring & Maintenance
Time Series Analysis Performance Automating Deployment &
Ensemble Methods Maintenance
Simulation & Optimization Myths & Challenges
Automated Modeling Data Science Applications
and References
Weekly Weekly Weekly

Assignment Assignment Assignment
Final Exam

Introduction to Data Science Watch the
Cumulative points lead to record of achievement deadlines!
Participate in Weekly Final Exam Record of

Assignment (Weeks1-6) (Week 7) Achievement
6 assignments When results above

180 points 180 points
6 x 30 = 180 points

What is data science?
Data science is an
interdisciplinary field about
processes and systems that
enable the extraction of
knowledge or insights from
data.
Data science employs
techniques and theories
drawn from a wide range of
disciplines.

Data science personas
SAP HANA
Data Analysts / Citizen
Business Users Data Scientists Application
Data Scientists
Developers
Analytics skills from low to high
Business User / Data Analyst Custom Embedded
Embedded Analytics
Driven Analytics Analytics Analytics
SAP Suite / Application Innovation / Industry / LoB / CDP SAP Hybris Marketing, IoT Predictive Maintenance,
Fraud
Application Function
SAP Predictive Analytics Modeler (AFM)
Predictive in SAP HANA PAL, APL, R, AFLs e.g. UDF, OFL
Data Science Solutions from SAP

Data science solutions from SAP
SAP
SAP SAP HANA SAP RDS Partner
Industry &
Predictive SAP Lumira Studio / Analytics Analytical
LoB
Analytics AFM Solutions BI & Tools
Solutions
SAP HANA
Predictive Analysis Business Function Automated
Simulation Optimization
Library (PAL) Library Predictive Library
R
Text Analysis and
Text Search Spatial Analysis Graph Engine Rules Engine
Mining
3rd Party Data SAP Data Data

SAP IQ HADOOP SAP ESP Connectors
Source Services
Data types
Connect to SAP HANA directly or via Sybase IQ / Hadoop / ESP / Data Services
Transaction Unstructured Real-Time Location Machine

Others
Data Data Data Data Data

SAP HANA Predictive Analysis Library (PAL)
Build High-Performance SAP HANA

Hadoop / Sybase IQ, KNN Regression
Predictive Apps
Sybase ASE, Teradata classification
Main Memory
The SAP HANA Predictive C4.5
K-means
Analysis Library (PAL) is a Virtual decision
built-in C++ library for Tables SQLScript ABC tree
Optimized classification Association
performing in-memory data Query Plan Weighted analysis:
mining and statistical score tables market
Spatial, Machine,
calculations. Real-Time Data Text PAL basket
Analysis
PAL is designed to provide R Scripts R Engine
high performance on large
datasets for real-time Spatial Unstructured
Data
analytics. SAP HANA Studio/AFM,
Apps & Tools

SAP HANA Predictive Analysis Library (PAL) algorithms
SAP HANA Predictive Analysis SAP HANA Predictive Analysis Library

Library (PAL) contains a wide range of
Association Analysis
algorithms that can be deployed for
Classification Analysis
in-HANA and standalone data science
applications. Regression
Cluster Analysis
A wide range of algorithms are
Time Series Analysis
available for the following types of
analysis: Probability Distribution
Outlier Detection
Link Prediction
Data Preparation
Statistic Functions (Univariate)
Statistic Functions (Multivariate)

SAP HANA Automated Predictive Library (APL) algorithms
SAP HANA APL is an application

function library (AFL) that lets you Classification Clustering
use the data mining capabilities of Models Models
the SAP Predictive Analytics
automated analytics engine on
your customer datasets stored in Regression
SAP HANA. Models Time Series
APL Analysis
You can create a wide range of
models to answer your business
questions.
Social Network Recommendation
Analysis

R integration for SAP HANA and standalone
Application
R
SAP HANA Database

SQL Interface
R
Calculation Engine
R
Rserve
Trigger R
Font
R R
R Operator
Client
Write Rserve
Rserve
R R Runtime
Results
Tables Tables

SAP Predictive Analytics
SAP Predictive Analytics is built for both

data scientists and business / data
analysts, making predictive analytics
accessible to a broad spectrum of
users.
Automated and expert modes
Used to automate data preparation,
predictive modeling, and deployment
tasks
Rich pre-built modelling functionality
PAL, APL, and R language support
Advanced visualization
Native integration with SAP HANA

Application function modeler (AFM)
Graphical tool to build advanced

applications in SAP HANA
Web-based flow-graph editor
Support for AFL, R, SDI, & SDQ
Used to create procedures or task
runtime operations
Interoperability with SAP HANA studio
AFM
SAP HANA studio-based AFM

PAL function support including time
series, clustering, classification, and
statistics
General usability enhancements for an
easier, simpler, and more functional
experience

Thank you
Contact information:
open@sap.com
2016 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate
company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its
affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and
services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as
constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop
or release any functionality mentioned therein. This document, or any related presentation, and SAP SEs or its affiliated companies strategy and possible future
developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time
for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-
looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place
undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

Week 1 Unit 2: Introduction to
Project Methodologies
Introduction to Project Methodologies
Why should there be a project methodology?
The data science process must be reliable and

repeatable by people with little data science
background. TIME
A project methodology:
Task 1
Provides a framework for recording experience
Allows projects to be replicated Task 2
Provides an aid to project planning and Task 3
management
Is a comfort factor for new adopters Task 4
Reduces dependency on stars

Cross-industry standard process for data mining (CRISP-DM)
Business Data
Understanding Understanding
Data
Preparation
Deployment
Modeling
Data
Evaluation

CRISP-DM Phase 1: Business Understanding
Business Data Data

Modeling Evaluation Deployment
Understanding Understanding Preparation
Business
Determine Business Business
Background Success
Objectives Objectives
Criteria
Requirements
Assess Inventory of Risks & Costs &
Assumptions & Terminology
Situation Resources Contingencies Benefits
Constraints
Data Science
Determine Data Data Science
Success
Science Goals Goals
Criteria Key
Initial TASKS
Produce Project Assessment
Project Plan of Tools & OUTPUTS
Plan
Techniques

CRISP-DM Phase 2: Data Understanding
Business Data Data

Initial Data
Collect Initial
Collection
Data
Report
Data
Describe
Description
Data
Report
Data
Explore
Exploration
Data
Report Key
TASKS
Verify Data Data Quality
Quality Report OUTPUTS

CRISP-DM Phase 3: Data Preparation
Business Data Data

Dataset Dataset Description
Rationale for
Select Data
Inclusion/Exclusion
Data Cleaning
Clean Data
Report
Construct Data Derived Attributes Generated Records

Key
Integrate Data Merged Data TASKS
OUTPUTS
Format Data Reformatted Data

CRISP-DM Phase 4: Modeling
Business Data Data

Select Modeling Modeling

Modeling Technique
Technique Assumptions
Generate Test
Test Design
Design
Build
Parameter Settings Models Model Description
Model Key
TASKS
Revised Parameter OUTPUTS

Assess Model Model Assessment
Settings

CRISP-DM Phase 5: Evaluation
Business Data Data

Evaluate Assessment of Data

Approved Model
Results Mining Results
Review
Review of Process
Process
Determine List of Possible

Decision
Next Steps Actions Key
TASKS
OUTPUTS

CRISP-DM Phase 6: Deployment
Business Data Data

Plan
Deployment Plan
Deployment
Plan Monitoring & Monitoring

Maintenance Maintenance Plan
Produce
Final Report Final Presentation
Final Report Key
TASKS
Review Experience OUTPUTS

Project Documentation

CRISP-DM Update
Business Data
Understanding Understanding
Data
Monitoring Preparation
Modeling
Deployment Data
Evaluation

Thank you
open@sap.com

Week 1 Unit 3: Business
Understanding Phase Overview
Business Understanding Phase Overview
CRISP-DM Phase 1: Business Understanding
Business Data Data

Business
Determine Business Business
Background Success
Objectives Objectives
Criteria
Requirements
Assess Inventory of Risks & Costs &
Assumptions & Terminology
Situation Resources Contingencies Benefits
Constraints
Data Science
Determine Data Data Science
Success
Science Goals Goals
Criteria Key
Initial TASKS
Produce Project Assessment
Project Plan of Tools & OUTPUTS
Plan
Techniques

Phase 1.1: Determine Business Objectives
Task
The first objective of the data analyst is to thoroughly
understand, from a business perspective, what the
client really wants to accomplish.
Outputs
Background
Business Objectives
Business Success Criteria

Phase 1.2: Assess Situation
Task
In the previous task, your objective is to quickly get to the crux of
the situation. Here, you want to flesh out the details.
Outputs
Inventory of Resources
Requirements, Assumptions, & Constraints
Risks & Contingencies
Terminology
Costs & Benefits

Phase 1.3: Determine Data Science Goals
Task
A business goal states objectives in business terminology.
A data science goal states project objectives in technical terms.
Outputs
Describe data science goals.
Define data science success criteria.

Phase 1.4: Produce Project Plan
Task
Describe the intended plan for achieving the data
mining goals and thereby achieving the business goals.
Output
Project plan with project stages, duration, resources,
etc.
Initial assessment of tools & techniques.

Thank you
open@sap.com

Week 1 Unit 4: Defining Project
Success Criteria
Defining Project Success Criteria
Business and data science project success criteria: reminder
Business success criteria

Describe the criteria for a successful or useful
outcome to the project from the business point
of view.
Data science success criteria

Define the criteria for a successful outcome to
the project in technical terms.

Recent industry surveys
In their Third Annual Data Miner Survey, Rexer Analytics, an analytics and
renowned CRM consulting firm based in Winchester, Massachusetts asked the
BI community How do you evaluate project success in data mining? Out of 14
different criteria, a massive 58% ranked Model performance (lift, R2, etc) as
the primary factor.
Model performance (lift, R2, etc) 58%

Improve efficiency 49%
Produced new business insights 48%
Revenue growth 44%
Increased sales 42%
Figure 5. Based on 110 users who have implemented predictive analytics initiatives Return on Investment (ROI) 42%
that offer very high or high value. Respondents could select multiple choices.
Increased profit 38%
Improved quality of product/service 35%
Increased customer satisfaction 34%
Reduce costs of producing products/.. 27%
Customer service improvements 21%
Results were published 13%
Produced new scientific insights 12%

Model success criteria: descriptive or predictive models
Descriptive Models Predictive Models

Descriptive analysis describes or summarizes raw data Predictive analysis predicts what might happen in the
and makes it more interpretable. It describes the past future providing estimates about the likelihood of a
i.e. any point of time that an event occurred, whether future outcome.
it was one minute ago or one year ago.
One common application is the use of predictive
Descriptive analytics are useful because they allow us analytics to produce a credit score. These scores are
to learn from past behaviors and understand how used by financial services to determine the probability of
these might influence future outcomes. customers making future credit payments on time.
Common examples of descriptive analytics are reports Typical business uses include: understanding how sales
that provide historical insights regarding a companys might close at the end of the year, predicting what items
production, financials, operations, sales, finance, customers will purchase together, or forecasting
inventory and customers. inventory levels based upon a myriad of variables.
Descriptive analytical models include cluster models, Predictive analytical models include classification
association rules, and network analysis. models, regression models, and neural network models.

Model success criteria: choosing the algorithm
Anomalies Trends
What anomalies or unusual values What are the trends, both historical
might exist? Are they errors or real and emerging, and how might they
changes in behavior? continue?
Associations
What are the correlations in
the data? What are the
cross-sell opportunities?
? Relationships
What are the main influencers, for
example customer churn, employee
turnover etc.?
Groupings
Are there any clear groupings of the data,
for example customer segments for
specific marketing campaigns?

There is a wide range of algorithms to choose from
Detect anomalies
Classification Regression
or outliers (data Forecasting with time
Association Clustering continuous target
bivariate target variable cleansing or series data
variable
decision support)
Descriptive Models Predictive Models

Which business question do you need to answer?
Classification Regression
Who will (buy | fraud | churn ) next What will the (revenue | # churners) be next
(week | month | year)? (week | month)?
Segmentation or Clustering Forecasting (Time Series Analysis)

What are the groups of customers What will the (revenue | # churners) be over
with similar (behavior | profile )? next year on a monthly basis?
Association or Recommendation
Link Analysis
Engines
Analyze interactions to identify
(communities | influencers) Provides recommendations on web sites or to
retailers basket analysis

Model accuracy and robustness
The accuracy and robustness of the model

are two major factors to determine the
quality of the prediction, which reflects how
successful the model is.
Accuracy is often the starting point for
analyzing the quality of a predictive model,
as well as an obvious criterion for prediction.
Accuracy measures the ratio of correct
predictions to the total number of cases
evaluated.
The robustness of a predictive model
refers to how well a model works on
alternative data. This might be hold-out data
or new data that the model is to be applied
onto.

Training and testing: data cutting strategies
Two-Way Partition Three-Way Partition

Central to developing predictive models
and assessing if they are successful is a
train-and-test regime. Complete Complete
Data Set Data Set
Data is partitioned into training and
test subsets. There are a variety of
cutting strategies
(e.g. random/sequential/periodic).
Training Test Training Validation Test
We build our model on the training Set Set Set Set Set
subset (called the estimation subset)
and evaluate its performance on the test
Develop Evaluate Develop Evaluate Evaluate
subset (a hold-out sample called the models models models models the
validation subset). selected
Select the best model
models
Simple two and three-way data based upon validation
set performance
partitioning is shown in the diagram.

Example success criteria for predictive models
Business Criteria: Model Performance Criteria: Software Usability Criteria:

Models meet business Depends on the algorithm. For a Speed and ease of model development
goals the model meets classification model for example: so the customer can build new models
the business objectives Model accuracy compared to any and update existing models quickly
specified in CRISP-DM previous, similar models. There are a Speed and ease of model deployment
phase 1.1 as defined by variety of accuracy measures that will be
so the customer can create Apply
the customer discussed in this course
datasets easily and deploy models
The models contributing Model robustness Models have quickly with the required outputs
acceptable robustness
variables and the variable (probabilities, deciles, etc.) speed to
categories make business market
sense Ease of model maintenance so the
customer can easily define when models
require refreshing/rebuilding and
undertake this quickly and easily
Integration capability with other systems

Example success criteria for descriptive models
Business Criteria: Model Performance Criteria: Software Usability Criteria:

Models meet business Depends on the algorithm. For a cluster Speed and ease of model development
goals the model meets model for example: so the customer can build new models
the business objectives Determining the clustering tendency of a and update existing models quickly
specified in CRISP-DM set of data (distinguishing whether non- Speed and ease of model deployment
phase 1.1 random structure actually exists in the
so the customer can create Apply
data)
Contributing variables and datasets easily and deploy models
categories make business Comparing the results to given class quickly with the required outputs
labels (comparing model results to
sense (probabilities, deciles, etc.) speed to
existing cluster groups)
market
Evaluating how well the results of the
analysis fit the data without reference to Ease of model maintenance so the
external information customer can easily define when models
Comparing the results of different cluster require refreshing/rebuilding and
models to determine which is better undertake this quickly and easily
Determining the best number of clusters Integration capability with other systems

Thank you
open@sap.com

Week 1 Unit 5: Data Understanding
Phase Overview
Data Understanding Phase Overview
CRISP-DM Phase 2: Data Understanding
Business Data Data

Initial Data
Collect Initial
Collection
Data
Report
Data
Describe
Description
Data
Report
Data
Explore
Exploration
Data
Report Key
TASKS
Verify Data Data Quality
Quality Report OUTPUTS

Phase 2.1: Collect Initial Data
Task
Acquire the data (or access to the data) listed in the project
resources.
This initial collection includes data loading into the data exploration
tool and data integration if multiple data sources are acquired.
Output Initial Data Collection Report

List the following:
The dataset (or datasets) acquired
The dataset locations
The methods used to acquire the datasets
Any problems encountered
Record problems encountered and any solutions.

Phase 2.2: Describe Data
Task
Examine the gross or surface properties of the
acquired data and report on the results.
Output Data Description Report

Describe the data that has been acquired, including:
The format of the data.
The quantity of data, e.g. the number of records and
fields in each table.
The identities of the fields.
Any other surface features of the data that have been
discovered.

Phase 2.3: Explore Data
Task
This task tackles the data mining questions, which can be
addressed using querying, visualization, and reporting.
Output Data Exploration Report

Describe results of this task including:
First findings or initial hypothesis and their impact on the
remainder of the project.
If appropriate, include graphs and plots.

Phase 2.4: Verify Data Quality
Task
Examine the quality of the data, addressing questions such
as:
Is the data complete?
Is it correct or does it contain errors?
Are there missing values in the data?
Output Data Quality Report

List the results of the data quality verification.
If quality problems exist, list possible solutions.

Thank you
open@sap.com

Week 1 Unit 6: Initial Data Analysis
& Exploratory Data Analysis
Initial Data Analysis & Exploratory Data Analysis
Initial data analysis
Initial data analysis (IDA) is an essential part of nearly every analysis

Problem Solving, A Statisticians Guide
Christopher Chatfield
Chatfield defines the various steps in IDA. It includes analysis of:

The structure of the data
The quality of the data
errors, outliers, and missing observations
Descriptive statistics
Graphs
The data are modified according to the analysis:
Adjust extreme observations, estimate missing observations, transform
variables, bin data, form new variables.

Exploratory data analysis
Exploratory data analysis (EDA) is an approach to analyzing data

for the purpose of formulating hypotheses that are worth testing,
and complements the tools of conventional statistics for testing
hypotheses.
It was so named by John Tukey.
Wikipedia
It is important to understand what you CAN DO before you learn to

measure how WELL you seem to have done it.
To learn about data analysis, it is right that each of us try many
things that do not work that we tackle more problems than we
make expert analyses of. We often learn less from an expertly done
analysis than from one where, by not trying something, we missed
an opportunity to learn more.
John Tukey, Exploratory Data Analysis

Example US Stores data demonstration
Demonstration using the

SAP Predictive Analytics expert system.
We will use US Stores retail data to walk you
through this topic.
The dataset contains the following variables:
Store location
Turnover
Margin
Staff
Store size

Example Data visualization

Example Scatter plot matrix

Example Bubble plot

Example Parallel co-ordinate plot

Example Box plot

Example Statistical summary chart

Thank you
open@sap.com

Introduction To Data Science: Week 1 Unit 1

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Introduction To Data Science: Week 1 Unit 1

Transféré par

Droits d'auteur :

Formats disponibles

Week 1 Unit 1:

Introduction to Data Science

What to expect in the

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 2

Business & Data

Introduction to Data Science Data Preparation Phase Modeling Phase Overview

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 3

Weekly Weekly Weekly

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 4

Participate in Weekly Final Exam Record of

6 assignments When results above

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 5

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 6

Predictive in SAP HANA PAL, APL, R, AFLs e.g. UDF, OFL

Data Science Solutions from SAP

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 7

3rd Party Data SAP Data Data

Transaction Unstructured Real-Time Location Machine

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 8

Build High-Performance SAP HANA

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 9

SAP HANA Predictive Analysis SAP HANA Predictive Analysis Library

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 10

SAP HANA APL is an application

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 11

SAP HANA Database

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 12

SAP Predictive Analytics is built for both

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 13

Graphical tool to build advanced

SAP HANA studio-based AFM

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 14

National product specifications may vary.

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 16

The data science process must be reliable and

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 2

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 3

Business Data Data

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 4

Business Data Data

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 5

Business Data Data

Dataset Dataset Description

Construct Data Derived Attributes Generated Records

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 6

Business Data Data

Select Modeling Modeling

Revised Parameter OUTPUTS

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 7

Business Data Data

Evaluate Assessment of Data

Determine List of Possible

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 8

Business Data Data

Plan Monitoring & Monitoring

Review Experience OUTPUTS

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 9

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 10

National product specifications may vary.

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 12

Business Data Data

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 2

2016 SAP SE or an SAP affiliate company. All rights reserved. Public 3