Slides Deep Dive Into Data Science

Deep Dive into Data Science
with
KNIME Analytics Platform
KNIME AG
Copyright © 2019 KNIME AG

Overview
KNIME Analytics Platform
This educational material was produced for the course held at ®

Copyright © 2019 KNIME AG 12 ODSC India 2019 in Bangalore on Aug 10, 2019. Do not copy or
distribute.
Do you recognize this?
https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining

distribute.
Let’s unroll it!
Model
Data Model Model
Optimizatio Deployment
Preparation Training Evaluation
n
It always starts
with some data
…
Data Manipulation Model Training Parameter Tuning Performance Measures Files & DBs
Data Blending Bag of Models Parameter Optimization Accuracy Dashboards
Missing Values Handling Model Selection Regularization ROC Curve REST API
Feature Generation Ensemble Models Model Size Cross-Validation SQL Code Export
Dimensionality Reduction Own Ensemble Model No. Iterations … Reporting
Feature Selection External Models … …
Outlier Removal Import Existing Models
Normalization Model Factory
Partitioning …
…

distribute.
The many Lives of a Dataset
Model
Data Model Model
Optimizatio Deployment
Preparation Training Evaluation
n
Original Partitioning: Training Set Validation Set Test Set New Data from Real
Data Set • Training Set World Applications
with Past • Validation Set
Observation • Test Set
s

distribute.
What is KNIME Analytics Platform?
• A tool for data analysis, manipulation, visualization, and reporting

• Based on the graphical programming paradigm
• Provides a diverse array of extensions:
– Text Mining
– Network Mining
– Cheminformatics
– Many integrations,
such as Java, R, Python,
Weka, Keras, Plotly, H2O, etc.

distribute.
Visual KNIME Workflows
NODES perform tasks on data

Not Configured
Configured
Inputs Outputs Executed
Status Error
Nodes are combined to create

WORKFLOWS

distribute.
Data Access
• Databases
– MySQL, PostgreSQL
– any JDBC (Oracle, DB2, MS SQL
Server)
• Files
– CSV, txt
– Excel, Word, PDF
– SAS, SPSS
– XML
– PMML
– Images, texts, networks, chem
• Web, Cloud
– REST, Web services
– Twitter, Google

distribute.
Big Data
• Spark
• HDFS support
• Hive
• Impala
• In-database processing

distribute.
Transformation
• Preprocessing
– Row, column, matrix based
• Data blending
– Join, concatenate, append
• Aggregation
– Grouping, pivoting, binning
• Feature Creation and Selection

Copyright © 2019 KNIME AG 9
10 ODSC India 2019 in Bangalore on Aug 10, 2019. Do not copy or
distribute.
Analysis & Data Mining
• Regression
– Linear, logistic
• Classification
– Decision tree, ensembles, SVM,
MLP, Naïve Bayes
• Clustering
– k-means, DBSCAN, hierarchical
• Validation
– Cross-validation, scoring, ROC
• Deep Learning
– Keras, DL4J
• External
– R, Python, Weka, H2O, Keras

distribute.
Visualization
• Interactive Visualizations
• JavaScript-based nodes
– Scatter Plot, Box Plot, Line Plot
– Networks, ROC Curve, Decision
Tree
– Plotly Integration
– Adding more with each release!
• Misc
– Tag cloud, open street map,
molecules
• Script-based visualizations
– R, Python

distribute.
Deployment
• Database
• Files
– Excel, CSV, txt
– XML
– PMML
– to: local, KNIME Server, SSH-,
FTP-Server
• BIRT Reporting

distribute.
Over 2000 Native and Embedded Nodes Included:
Data Access Transformation Analysis & Mining Visualization Deployment

MySQL, Oracle, ... Row Statistics R via BIRT
SAS, SPSS, ... Column Data Mining JFreeChart PMML
Excel, Flat, ... JavaScript
Matrix Machine Learning Plotly
XML, JSON
Hive, Impala, ... Text, Image Web Analytics Community / 3rd Databases
XML, JSON, PMML Time Series Text Mining Excel, Flat, etc.
Text, Doc, Image, ... Java Network Analysis Text, Doc, Image
Web Crawlers Python Social Media Analysis Industry Specific
Industry Specific Community / 3rd R, Weka, Python Community / 3rd
Community / 3rd Community / 3rd

distribute.
Overview
• Installing KNIME Analytics Platform

• The KNIME Workspace
• The KNIME File Extensions
• The KNIME Workbench
– Workflow editor
– Explorer
– Node Repository
– Description
• Installing new features

distribute.
Install KNIME Analytics Platform
• Select the KNIME version for your

computer:
– Mac
– Windows – 32 or 64 bit
– Linux
• Download archive and extract the
file, or download installer package
and run it

distribute.
Start KNIME Analytics Platform
• Use the shortcut created by the installer
• Or go to the installation directory and launch KNIME via the knime.exe

distribute.
The KNIME Workspace
• The workspace is the folder/directory in which workflows (and

potentially data files) are stored for the current KNIME session.
• Workspaces are portable (just like KNIME)

distribute.
The KNIME Workbench

distribute.
KNIME Explorer
• In LOCAL you can access your own

workflow projects.
• The Explorer toolbar on the top
has a search box and buttons to
– select the workflow displayed in
the active editor
– refresh the view
• The KNIME Explorer can contain 4
types of content:
– Workflows
– Workflow groups
– Data files
– Shared Components

distribute.
Creating New Workflows, Importing and Exporting
• Right-click in KNIME Explorer to create new workflow or workflow group

or to import workflow
• Right-click on workflow or workflow group to export

distribute.
Node Repository
• The Node Repository lists all

KNIME nodes
• The search box has 2 modes

– Standard Search – exact match
of node name
– Fuzzy Search – finds the most
similar node name
• Nodes can be added by drag and

drop from the Node Repository
to the Workflow Editor.

distribute.
Console and Other Views
• Console view prints out error

and warning messages about
what is going on under the hood
• Click on View and select Other…

to add different views
– Node Monitor, Licenses, etc.

distribute.
Description
• The Description window gives

information about:
– Node Functionality
– Input & Output
– Node Settings
– Ports
– References to literature

distribute.
Workflow Description
• When selecting the workflow,

the Description window gives
information about the
workflow’s:
– Title
– Description
– Associated Tags and Links
– Creation Date
– Author

distribute.
Workflow Coach
• Node recommendation engine

– Gives hints about which node use next in the workflow
– Based on KNIME communities' usage statistics
– Based on own KNIME workflows

distribute.
Tool Bar
The buttons in the toolbar can be used for the active workflow. The most
important buttons:
– Execute selected and executable nodes (F7)
– Execute all executable nodes
– Execute selected nodes and open first view
– Cancel all selected, running nodes (F9)
– Cancel all running nodes

distribute.
KNIME File Extensions
• Dedicated file extensions for Workflows and Workflow groups associated

with KNIME Analytics Platform
• *.knwf for KNIME Workflow Files
• *.knar for KNIME Archive Files

distribute.
More on Nodes…
A node can have 3 states:
Not Configured:
The node is waiting for configuration or incoming data.
Configured:
The node has been configured correctly, and can be executed.
Executed:
The node has been successfully executed. Results may be
viewed and used in downstream nodes.

distribute.
Inserting and Connecting Nodes
• Insert nodes into workspace by dragging them from Node Repository or

by double-clicking in Node Repository
• Connect nodes by left-clicking output port of Node A and dragging the
cursor to (matching) input port of Node B
• Common port types:
Model
Flow Variable
Image
DB Connection DB Data
Data

distribute.
Node Configuration
• Most nodes require configuration

• To access a node configuration window:
– Double-click the node
– Right-click -> Configure

distribute.
Node Execution
• Right-click node
• Select Execute in the context menu
• If execution is successful, status
shows green light
• If execution encounters errors, status
shows red light

distribute.
Node Views
• Right-click node
• Select Views in context menu
• Select output port to inspect execution results
Plot View Data View

distribute.
Curved Connections!

distribute.
Getting Started: KNIME Example Server
• Connect via KNIME Explorer to a public repository with large selection of

example workflows for many, many applications
• Workflows also available on KNIME Hub

distribute.
Sharing Workflows
How to use the KNIME Hub

distribute.
KNIME Hub
A place to share knowledge about

Workflows and Nodes
https://hub.knime.com
distribute.
The KNIME Hub

distribute.
Searching Nodes and Workflows

distribute.
Opening a Workflow from the Hub

distribute.
Open Workflow in KNIME Analytics Platform

distribute.
Saving the Workflow

distribute.
Edit the Workflow
Drag & Drop

distribute.
Sharing the Workflow
1. Save your Edits
2. Connect to KNIME Hub

distribute.
Log in the Hub
KNIME Forum
Account Credentials

distribute.
Publish your Workflow
1. Edit Metadata
2. Drag & Drop

distribute.
Open your Workflow in the Hub

distribute.
Open your Workflow in the Hub

distribute.
Hot Keys (for Future Reference)
Task Hot key Description
Node Configuration F6 opens the configuration window of the selected node
F7 executes selected configured nodes
Shift + F7 executes all configured nodes
Node Execution Shift + F10 executes all configured nodes and opens all views
F9 cancels selected running nodes
Shift + F9 cancels all running nodes
Node Connections Ctrl + L connects selected nodes
Ctrl + Shift + L disconnects selected nodes
Ctrl + Shift + Arrow moves the selected node in the arrow direction
Move Nodes and Ctrl + Shift + PgUp/PgDown moves the selected annotation in the front or in the back of all
Annotations overlapping annotations
F8 resets selected nodes
Ctrl + S saves the workflow
Workflow Operations
Ctrl + Shift + S saves all open workflows
Ctrl + Shift + W closes all open workflows
Metanode Shift + F12 opens metanode wizard

distribute.
Stay connected with KNIME
Blog: knime.com/blog
Follow us on social
media:
Forum: forum.knime.com
KNIME Hub:
hub.knime.com
KNIME E-Learning Course:

www.knime.com/e-learning-course

distribute.
Today’s Example: Next Best Offer (NBO)
• Traditional Direct Marketing advertises a single product to a specific

audience. The Next Best Offer (NBO) approach focuses on taking existing
customers (and their data) and using upsell models to find interesting
new products for them.
• Today we construct a workflow that joins diverse data sources into a set
of complete customer records. Using this, we will build and deploy a
predictive model to find people who might be interested in a newly
available product.

distribute.
The Data

distribute.
The Data

distribute.
Today’s Example: Next Best Offer (NBO)
Explore the final workflow

distribute.
Importing Data
Accessing Files and Databases

distribute.
Data Source Nodes
Typically characterized by:

− Orange color
− No input ports, 1-2 output ports
Output port
Status
Node label

distribute.
New Node: File Reader
Workhorse of the KNIME Source nodes

• Reads all text based files (e.g. csv, txt, etc.)
• Many advanced features allow it to read most ‘weird’ files
– Short lines, inline comments, headers and special encoding
YouTube KNIME TV Channel video:

https://youtu.be/flaHQw-Qhlg
distribute.
File Reader Configuration
File path
Basic
Settings
Advanced
Settings
Preview
Help Button

distribute.
Alternative Faster Way …
Drag & Drop

OR
Copy & Paste

distribute.
Filenames and the knime:// Protocol
Absolute URL
Mountpoint-relative URL
Local path

distribute.
Workflow-Relative File Paths
• Best choice if workflows are to be

shared
• Requires matching folder structure
within workflow group
− Independent of environment outside of
workflow group
• Example: Path to „Sentiment Analysis.table“
− Local path:
C:\Users\rb\knime-workspace\KNIMEUserTraining\data\Sentiment Analysis.table
− Workflow relative:
YouTube KNIME TV Channel: https://youtu.be/U9sP4g4yGwY

distribute.
New Node: Excel Reader (XLS)
• Reads .xls and .xlsx file from Microsoft Excel

• Supports reading from multiple sheets

distribute.
Excel Reader Configuration
File path
Sheet
specific
settings
Preview
distribute.
New Node: Table Reader
• Reads tables from the native KNIME Format.

• Maximum performance, minimum configuration
File path
YouTube KNIME TV channel video:

https://youtu.be/tid1qi2HAOo
distribute.
Database Connectivity
• Read data from any JDBC enabled database

• Write your own SQL or model it using dedicated nodes

distribute.
New Nodes: Database Connectors
• Native: Postgres, MySQL, MS SQL Server, SQLite

• Database Connector (e.g. Oracle, DB2, HANA).
• Big Data: HIVE and Impala

distribute.
Other Useful Data Sources
• PMML Reader – reads standard predictive models

• XML Reader with XPATH support
• Python/R Source nodes
• Tika Parser – extracts textual data from 200+ file types
• REST Web Services, and many more

distribute.
Importing Data Exercise
Start with exercise: Importing Data

Read the following files
• Sentiment Analysis.table
• Sentiment Rating.csv
• Product Data2.xls
Optional: Read table web_activity from the database

WebActivity.sqlite
(hint: drag and drop the files from the KNIME Explorer
panel to get started)

distribute.
Today’s Example

distribute.
Data Manipulation
Clean, Join, Aggregate

distribute.
Data Manipulation Nodes
• Yellow color with a variety of input and output ports

• Apply a transformation to input data
• Many, many nodes!

distribute.
New Node: Concatenate
Combine rows from 2 tables with shared

columns
• Handles duplicate row keys gracefully
• Take the union or intersection of columns

distribute.
New Node: Cell Replacer
Replaces the content of a column based on a

lookup
• Top port references the table to be searched
• Bottom port holds the lookup table (search
keys and replacement values)

distribute.
New Node: String Manipulation
Create and edit values in String columns

• Clean up capitalization (eg.
Lowercase)
• Replace strings
• Modify existing strings or create new
columns

distribute.
Data Manipulation Exercise, Activity I
Start with exercise: Data Manipulation, Activity I

• Concatenate web activity data from old and new systems
• Replace sentiment evaluation (strings) with corresponding numeric
values
• Use String Manipulation to ensure that all entries of the Products
column are lower case from the product data spreadsheet

distribute.
Joining Columns of Data
Left Table Right Table
Join by ID
Inner Join
Left Outer Join Right Outer Join

Missing values in the Missing values in the
right table. left table.

distribute.
Joining Columns of Data
Left Table Right Table
Join by ID
Full Outer Join
Missing values in
the right table
Missing values in
the left table

distribute.
New Node: Joiner
Combines columns from 2 different

tables
• Top port contains “Left” data table
• Bottom port contains “Right” data
table

distribute.
Joiner Configuration – Linking Rows
Joiner mode
Values to join on.

Multiple joining columns
are allowed

distribute.
Joiner Configuration – Column Selection
Columns from left

table to output table
Columns from right

table to output table

distribute.
Data Aggregation
RowID Group Value

r1 m 2
r2 f 3 RowID Group Sum(Value)
r3 m 1 r1+r3+r6 m 8
r4 f 5 r2+r4+r5 f 15
r5 f 7
r6 m 5
Aggregated on “Group” by method:

sum(“Value”)

distribute.
New Node: GroupBy
Aggregate rows to summarize data

• First tab provides grouping options
• Second tab provides control over aggregation details Aggregation columns
Aggregation methods
YouTube KNIME TV video:

https://youtu.be/bDwF-TOMtWw
distribute.
New Node: Duplicate Row Filter
Detect duplicate row and apply a selected treatment

• First tab provides the option to select columns
• Second tab provides options for treating duplicated values Flag or Remove
Duplicates
Select criteria to
keep row

distribute.
New Node: Column Expression
• Append or modify an arbitrary

number of columns using
expressions
• Many different functions are
available
• No restriction on number of
lines per expression allow to
write complex expressions
• Part of the KNIME Labs
extension

distribute.
Workflow Organization and
Documentation

distribute.
Comments & Annotations
Double-click to
Double-click to write
write
Use the
Use thepanel
paneltoto
change properties
change properties
YouTube KNIME TV Channel:

https://youtu.be/AHURYB_O8sA

distribute.
Workflow Organisation – Good Practices
• Workflow annotations
• Node labels
• Metanodes
− Right click -> Create
Metanode...
− Organize workflow by
task
− Hide complexity &
improve readability

distribute.
KNIME WorkflowDiff
• Automates identification and comparison of nodes in a workflow,

metanodes, and two different workflows
• Identifies insertions, deletions, substitutions, and parameter changes

distribute.
Data Manipulation Exercise, Activity II
Start with exercise Data Manipulation, Activity II

• Join all data together using a series of joiner nodes and the
“Customer Key” field
• Resolve duplicates in the joined dataset
• Clean up and document your workflow using annotations, node
labels, and metanodes

distribute.
Today’s Example

distribute.
Data Visualization
Charts and Tables

distribute.
Data Visualization
• Large selection of easy to use

visualization nodes
− Web-based and interactive
− Dedicated nodes,
no scripting required
• Plotly nodes
− Similar but integrated from an
external library
• R and Python View nodes for

highly customizable graphics
− Require scripting

distribute.
Visualizations using 1 Column

distribute.
Visualizations using 2 Columns
1 Column

distribute.
Visualizations using 3 Columns

distribute.
New Node: Scatter Plot
Image
outport
• Plots different columns on X and Y
• Displays data including color
information Interactivity
options
• Produces an interactive view and
an image
• Select data points and publish
selection to other views

distribute.
New Node: Scatter Plot
Four configuration tabs

distribute.
New Node: Color Manager
• Color by nominal or continuous values

• Sync colors between views using the color model port and Color
Appender node
Color range
Discrete
for numerical
colors for
values
nominal
values

distribute.
New Node: Bar Chart
• Show numerical values across

categories
• Vertical or horizontal bars
• Bars can be grouped or stacked

distribute.
New Node: Line Plot
• Plot sequence of values,

e.g. over time
• Useful to identify trends,
also between groups

distribute.
New Node: Stacked Area Chart
• Visualizes numerical values

from multiple columns as
stacked areas
• Great for plotting distributions
over time

distribute.
Selection & Filtering in JavaScript Views
Interactivity allows you to select data points in views

− Selection is propagated to other views
− Highlight selected rows or filter them
− Click “Apply” to add column to data that indicates selection (true/false) for
use in downstream nodes
Apply selection

distribute.
Components – Combined Views
• Multiple JavaScript View nodes Scatter Plot

can be combined in Components
• Selections are transmitted to all
other views
• Also for use on the KNIME
WebPortal
Table View

distribute.
Interactivity across Charts: Selection and Filter Events

distribute.

distribute.
Subscribing to Selection and Filter

distribute.
Subscribing to Selection and Filter

Publishing Selection
distribute.
Configure Content and Views Layout
• Click layout button when inside • Add views and rows via drag&drop
Component to assign views to rows • Add columns using + buttons
and columns

distribute.
Data Aggregation
Aggregation: Count
Sex Hair Age Sex blond brown black red
f blond 31 f 2 1 1 0
m red 22 m 1 1 0 2
f blond 53
m brown 16
f brown 47 Aggregation: Mean(Age)
f black 22 Sex blond brown black red
m blond 13 f 42 53 22 0
m red 55 m 13 16 0 38,5
Solution: Pivoting Node

distribute.
Data Aggregation
Gender Hair Age

f blond 31
m red 22 Aggregation: Mean(Age)
f blond 53 Gender blond brown black red
m brown 16 f 42 53 22 0
f brown 47 m 13 16 0 38,5
f black 22
m blond 13
m red 55
Pivoting Node: Group - Pivot - Aggregate

distribute.
New Node: Pivoting
Performs pivoting on selected columns for grouping and pivoting

• Values of group columns become unique rows
• Values of the pivot columns become unique columns for each set of
column combination together with each aggregation
• Many aggregation methods are provided (similar to GroupBy)

distribute.
New Node: Pivoting
Groups ~ Rows
Pivots ~ Columns
Aggregation

distribute.
Script-based View Nodes
• R View nodes for greater customizability

− Use your favorite libraries, e.g. ggplot2
• If you prefer Python: Python View node

• For JS developers: Generic JavaScript View

distribute.
Visualization Exercise
Start with exercise: Visualization
• Read sales.csv data
• Use a Color Manager to color by Product
• Plot BasketValue against BasketSize using Scatter Plot
• Compare the sum of BasketValue by time and product in a Line Plot and a Stacked
Area Chart (Use the Pivoting node to get the sum of sales per Quarter and Product!)
• Read Fully Joined Data

• Show the number of customers in the web activity categories using Bar Chart
• Show the age distribution of customers using Histogram
• Create a composite view by combining the Bar Chart and Histogram
• Select one web activity class in the Bar Chart. Which age classes are represented in
the selected web activity class?

distribute.
Today’s Example

distribute.
Data Mining
Partition, Learn, Predict, Score

distribute.
Data Mining Strategies
Example Applications:
• Anomaly Detection (fraud, predictive maintenance)
• Association Rule Learning (market basket analysis)
• Clustering (market segmentation)
• Classification (next best offer, churn preventions)
• Regression (trend estimation)

distribute.
Data Mining: Process Overview
Train
Training
Model
Set
Apply Score
Model Model
Original
Data Set
Test
Set
Train and Evaluate

Partition data
apply models performance
distribute.
Data Mining in KNIME
• KNIME has many modeling tools!

− Decision tree, random forest, SVM, regression,
neural networks, clustering, …
− and integrations with other libraries:
R, Python, H2O, WEKA, libSVM, etc.
• And many model evaluation nodes
− ROC, standard, numeric and entropy scorers
− Feature elimination
− Cross validation

distribute.
New Node: Partitioning
• Use to split data into training and evaluation sets

− Partition by count (e.g. 10 rows) or fraction (e.g. 10%)
− Sample by a variety of methods; random, linear, stratified

distribute.
Learner-Predictor Motif
• Most data mining approaches in KNIME

use a Learner-predictor motif.
• The Learner node trains the model with
its input data. Trained Model
• The Predictor node applies the model to

a different subset of data.
New Data!

distribute.
Classification
Predict nominal outcomes on existing data (supervised)

• Applications
− Churn analysis (yes/no)
− Chemical activity (active/inactive)
− Spam detection (spam/not spam)
− Optical character recognition (A-Z)
• Methods
− Decision Trees
− Neural Networks
− Naïve Bayes
− Logistic Regression
distribute.
Target Column
• Target column contains values that are predicted by the

classification model
• Binomial target values are often encoded to 1 and 0
Application Target Target Values
Column
Churn Churn Yes/No or 1/0
analysis
Chemical Active Yes/No or 1/0
activity
Spam Spam Yes/No or 1/0
Detection
Optical Character A-Z
Character
Recognition

distribute.
KNIME’s Decision Tree
J.R. Quinlan, “C4.5 Programs for machine learning”

J. Shafer, R. Agrawal, M. Mehta, “SPRINT: A Scalable Parallel
Classifier for Data Mining”
• C4.5 builds a tree from a set of training data using the concept of
information entropy.
• At each node of the tree, the attribute of the data with the highest
normalized information gain (difference in entropy) is chosen to split the
data.
• The C4.5 algorithm then recurses on the smaller sub lists.

distribute.
New Node: Decision Tree Learner

distribute.
Decision Tree View
Most of the people who

don‘t churn have more
than one contract

distribute.
New Node: Decision Tree Predictor
• Takes a decision tree model &

applies it to new data
• Check the box to append class
probabilities

distribute.
New Node: Scorer
Compare predicted results to known truth in

order to evaluate model quality

distribute.
New Node: Scorer
Confusion matrix shows the distribution of
model errors
An accuracy statistics table provides a

detailed analysis of model quality

distribute.
Confusion Matrix

distribute.
Receiver Operating Characteristics
• Sort by confidence in target class

• Plot true positive rate vs false positive rate
• Ideal models achieve 100% TPR with 0% FPR
• Area under the curve indicates model
quality
− (1=ideal model, 0.5 = random outcome)

distribute.
New Node: ROC Curve
• Requires individual class probabilities from

a preceding predictor
• User must define:
1. Original class column
2. Positive class value
3. Probability for the selected positive class
value for one or multiple models

distribute.
Data Mining Exercise, Activity I
Start with exercise: Data Mining, Activity I:
• Partition the fully joined data
− 50%, Stratified Sampling
• Train a decision tree on the training data
− Predict the Target column
• Use the model to predict the upsell potential (Target =1 or Target=0) for
remaining records
• Evaluate the quality of a model with a Scorer.
• Optional: Find AUC for the model using ROC curve

distribute.
Regression
Predict numeric outcomes on existing data (supervised)

• Applications
− Forecasting
− Quantitative Analysis
• Methods
− Linear
− Polynomial
− Regression Trees
− Partial Least Squares

distribute.
New Nodes: Linear Regression Learner & Regression Predictor
A linear model relating a dependent variable to 1 or more independent

variables
− Model coefficients provided in 2nd output port
− Also available: Polynomial and Tree Ensemble Regression nodes

distribute.
New Node: Numeric Scorer
Similar to scorer node, but for nodes with

numeric predictions (e.g. linear/polynomial
regression)
• Compare dependent variable values to
predicted values to evaluate goodness of fit.
• Report R2, RMSD, SEM etc.

distribute.
Data Mining Exercise, Activity II
Start with exercise: Data Mining, Activity II:

• Read the weather.table
• Split the data into 2016 for training and use 2017 as test data
• Train a linear regression model that predicts the AIR_TEMP as a function
of all other parameters in the data set
• Use the model to predict the temperature in 2017 and evaluate it with
the Numeric Scorer
• Optional: Calculate mean temperature per month on the training data
− Join the mean temperature to the test data set (2017)
− Use the Numeric Scorer to see if the easiest model is better than the Linear
Regression

distribute.
Today’s Example

distribute.
Clustering
Discover hidden structure in unlabeled data (unsupervised)
• Applications
− Market Segmentation
− Diversity picking
• Methods
− K-means/medoids
− Hierarchical
− DBScan
− OPTICS
− Neighbourgrams

distribute.
New Nodes: k-Means Clustering
• Looks at n observations to define

the means for k clusters.
• Each observation is then assigned
to its closest cluster center.
• You must provide k.

distribute.
Data Mining Exercise, Activity III
Start with exercise: Data Mining, Activity III

• Read the location_data.table file
• Filter to entries from California (region_code = CA)
• Train a k-means model with k=3. Use only position data for clustering
(latitude and longitude)
• Optional: Plot latitude and longitude in a view (OSM Map or Scatter Plot)
and use that to help you visually optimize k.

distribute.
Model Optimization

1 142 ODSC India 2019 in Bangalore on Aug 10, 2019. Do not copy or
distribute.
KNIME’s Tree Ensemble Models
• The general idea is to take X

advantage of the “wisdom of the
crowd” 1 4
…
1
5 2 2 7 7 6
• Ensemble models: Combining 2 9 6 7 6 8 9 3 3 9 5 7
predictions from a large number P1 P2 … Pn
of weak predictors, e.g. decision y

trees
Typically: for classification the
• Leads to a more accurate and individual models vote and the
robust model majority wins; for regression, the
individual predictions are
• This is called ”bagging” averaged

distribute.
How Does Bagging Work?
• Pick a different random subset of the training data for each model in the
ensemble (bag)
Build tree Build tree Build tree
1 4 1
5 2 5 7
… 7 6
2 9 6 7 2 8 9 3 3 9 5 7

distribute. 3
An Extra Benefit of Bagging: Out of Bag Estimation
• Allows testing the model using the training data: when validating, each
model should only vote on data points that were not used to train it
X1 X2
1 4 1 1 4 1
5 2 2 7 … 7 6 5 2 2 7 … 7 6
2 9 6 7 6 8 9 3 3 9 5 7 2 9 6 7 6 8 9 3 3 9 5 7
P1 P2 … Pn P1 P2 … Pn
y1OOB y2OOB

distribute. 4
Random Forest
• Bag of decision trees, with an extra element of

randomization when building the trees: each
node in the decision tree only “sees” a subset of
the input columns, typically 𝑁
• Random forests tend to be very robust w.r.t.
overfitting (though the individual trees are almost
certainly overfit) Build tree
• Extra benefit: training tends to be much faster
1
5 2
2 9 6 7

distribute. 5
New Nodes: Random Forest Learner
• The output model describes a

random forest and is applied in the
corresponding predictor node using a
simple majority vote
• The statistics table on the attributes
tells how often each attribute…
– … is used in the first three splits
– … was a possible candidate in the first
three splits

distribute.
Tree Ensembles
• Random Forest variant

• More options to set
• Trees may be trained using
subsets of rows and/or columns
and this approach may lead to
greater accuracy

distribute.
New Nodes: Tree Ensemble Learner/Predictor
• Choose which columns to include

• Configure a prototype tree (depth, split criteria etc.)
• Setup ensemble parameters (model count, row/column subsampling)

distribute.
Advanced Data Mining Exercise, Activity I
Start with exercise: Advanced Data Mining, Activity I

• Read the data file CurrentDetailData.table
• Partition the data 50/50 using stratified sampling on the Products column
• Create a Tree Ensemble model to predict the “Products” column
− Use a tree depth of 5, 50 models, and 75% of rows and columns for each
iteration.

distribute.
Parameter Optimization
• Some modeling approaches are very sensitive to their configuration

• Calculating optimum settings is not always possible
• Parameter Optimization loops may help find a good configuration

distribute.
The Loop Block
• A loop block is defined by appropriate loop start and loop end nodes
• Loop body = Nodes in between and side branches
Loop body
Loop
end
Loop start node
node

distribute.
New Node: Parameter Optimization Loop Start
• Define some parameters to

optimize
• Set upper/lower bounds and
step sizes (and flag integers)
• Choose an optimization method
− Brute force for maximum accuracy
but slower computation
− Hillclimbing for better faster
runtimes but may get stuck in local
optimum settings
− Random search to randomly search
for parameter values within a given
range
− Bayesian Optimization (TPE)
distribute.
New Node: Parameter Optimization Loop End
• Collects some value to optimize

in a flow variable.
• Value may be maximized
(accuracy) or minimized (error)

distribute.
Loop Start and End Nodes
• For different tasks, there are different loop start

and end nodes
• Nodes with circular arrows (green) = Start node
• Nodes with a closed circle (red) = End node
• Flow Variables are really helpful to build the
loop body

distribute.
Flow Variables
What is a Flow Variable?

• Flow Variables are workflow parameters
• Flow Variables are used to overwrite existing node settings
• Flow Variables can be of type String, Integer, or Double
How can we create a Flow Variable?

• Special nodes have Flow Variables as output
– Parameter Optimization Loop Start node
– Table Row to Variable node
• In the “Flow Variables” tab of any node

distribute.
Flow Variable Ports

distribute.
Apply a Flow Variable (Button)
The Flow Variable button

distribute.
Apply a Flow Variable (Advanced)
The Flow Variable

tab
List of available Flow

Variables

distribute.
Create a Flow Variable (Button)
Name of the new

Flow Variable

distribute.
Create a Flow Variable (Advanced)
Converting a setting value into a Flow Variable
Name of the new

variable

distribute.
Advanced Data Mining Exercise, Activity II
Start with exercise: Advanced Data Mining, Activity II

• Read the data file CurrentDetailData.table
• Partition the data 50/50 using stratified sampling on the column Products
• Build a parameter optimization loop for a Tree Ensemble Model to predict the
Prodcuts
• Use Hill climbing to determine the optimum number of models (min=10,
max=200, step=10, int = yes)
• Maximize the accuracy in the Loop End node
• What were the optimal settings?
(Hint: don’t forget to use the Flow Variable in your learner node)
distribute.
Integrating External Tools

distribute.
Goal of This Session
• This session gives a quick overview of the external tools that can be called
within KNIME, e.g.:
− Java, R, Python
− Web services

Copyright © 2019 KNIME AG 164 ODSC India 2019 in Bangalore on Aug 10, 2019. Do not copy or 2
distribute.
KNIME Labs
• KNIME Labs enables you to preview new KNIME

features and plug-ins that are still under
development
• The nodes provided in KNIME Labs are not (yet)
part of the official KNIME version because the
functionality and/or API may not be finalized
• You can get these plug-ins by installing the
extension from the KNIME Labs extensions
category

Copyright © 2019 KNIME AG 165 ODSC India 2019 in Bangalore on Aug 10, 2019. Do not copy or 3
distribute.
Java Snippet
• Fastest running scripting node in

KNIME
• Syntax highlighting, auto
completion, error checking
• Templates allow you to save scripts
for later re-use
• Import custom libraries

distribute.
Java Edit Variable
• Same as Java snippet, but

with flow variable ports

distribute.
R Integration
• Run any R code from

KNIME Train model
Apply model
• Works with existing R
installations
• Nodes for many tasks
Run any R code
• First run:
install.packages(‘Rserve‘)
and Pass one
workspace to
install.packages(‘Cairo‘)* multiple nodes
*mac only
Create plot

distribute.
R Integration
Syntax highlighting
Create and
store
templates
R workspace
Show
results
Evaluate
script R console
output

distribute.
Python Integration
• Run Python inside KNIME

• Works with existing installations
• UI modeled after R integration

distribute.
Python Scripting UI

distribute.
RESTful Web Services
• Use KNIME nodes to interact with

RESTful web services
• Send requests using standard
HTTP methods
JSON Response:
XML Response:

distribute.
RESTful Web Services
Provide authentication
Enter URL, or use if necessary
from column
Add delay between

individual requests
https://www.knime.com/blog/a-restful-way-to-find-and-retrieve-data
https://www.knime.com/blog/OSM-meets-CSV-file-and-Google-API

distribute.
KNIME Server as a REST Resource
https://www.knime.org/blog/giving-the-knime-server-a-rest
distribute.
KNIME Server as a REST resource
• Use cURL, SOAPUI or Chrome

extension Postman to explore the
REST API

distribute.
JSON and JSON Path
• Use the JSON Reader (or the GET Resource) nodes

to get an JSON cell
• Use JSONPath nodes to query the JSON and extract
certain parameters
• Editor window simplifies construction of JSON
queries by auto-generating them (click on
properties)

distribute.
JSON Path

distribute.
XML and XPath
• Use the XML Reader (or the GET Resource) nodes to get
an XML cell
• Use XPath nodes to query the XML and extract certain
parameters
• Editor window simplifies construction of XPath queries by
auto-generating them (click on XML elements)

distribute.
XPath

distribute.
Remote File Handling – Cloud Storage
• Integrate remote data sources from

Amazon AWS and Microsoft Azure
− Upload files
− Download files, or read their content
directly into KNIME
− List files in remote directories
− Create directories
− Delete files / directories

distribute.
Remote File Handling – Cloud Storage
Example: Upload all files from a local directory to Amazon S3
Enter
credentials
Upload!
Create
Create new
directory in
bucket
bucket
Create URIs
of local file
paths

distribute.
Google Sheets
• Access your data stored in

Google Services
− Read data from Google Sheets
− Write data to new sheets
− Modify existing sheets
• Makes collaboration and sharing

of data easy
− (especially vs. sending Excel
sheets via email...)
Authenticate via pop-
up window (Oauth2)
distribute.
Google Sheets
• Select from available sheets on Google Drive

• Transform data in KNIME, or enrich with new data
• Create new sheet or update existing sheets
− Allows to read from / write to specific range of sheet (e.g. A1:G10)
Authenticate via pop-

up window (Oauth2)
Select from available sheets, Specify target sheet, select

open in browser for preview which columns to write, etc.

distribute.
Exercises (Optional)
Start with exercise: Integrating External Tools

• Use the GET Request node to call an external web service
− https://raw.githubusercontent.com/tamingtext/book/master/apache-
solr/example/exampledocs/books.json
• Use the JSON Path node to output subparts of the JSON query
– Extract all possible subparts
– Output the subparts as a list
• Use the Ungroup node to split the single JSON queries into separate rows
• Use the JSON Path node to separate “name“, “author“, and “price“ from
the JSON queries
– Hint: Right click on a value in the JSON query and select “Add JSONPath“

distribute.
Exporting Data & Deployment

distribute.
Exporting Data
After an analysis is completed, what next?

− Write results to a file
− Create/update a database
− Save the model for use elsewhere
− Generate a rich report
− Deploy via KNIME WebPortal
− Deploy via workflow as RESTful web service

distribute.
Input/Output in Deployment
Input Output
• File (CSV, Table, XLS, …) • Report (BIRT, Tableau,
• Database Spotfire)
• JSON for REST API • Email
• File (CSV, Table, XLS, …)
• WebPortal

distribute.
To Report / Email
To BIRT Report
Also available:
Nodes for Tableau
and Spotfire

distribute.
To File / Database

distribute.
REST API (Available on KNIME Server)

distribute.
To Dashboard on WebPortal
Step 3
Step 1 Step 2 Step 4 Step 5
Customize
Upload File Select Columns Interactive View Download Image
Column Domains

distribute.
Workflow on KNIME WebPortal
WebPortal Page
(Step 1)
Upload File
Available in
WebPortal Page
KNIME Server (Step 4)
Interactive View

distribute.
Components to Produce Dashboard on Web Page
File
Selection Column
Selection
Stacked
Area Chart
Filter by
Row Filter
Range

distribute.
Data Export Nodes
Typically characterized by:

− Magenta color
− 1 input port, no output ports
− Create file on file system or
write to database

distribute.
New Node: Table Writer

distribute.
New Node: XLS Writer

distribute.
New Node: Database Writer

distribute.
Automation: Call Local Workflow
• Use Call Local Workflow node to send

data and parameters to other
workflows and trigger execution
− Send results back to caller-workflow
− Include report from called workflow Path to workflow
• Create modular workflows

Add report to output
− E.g. separate workflows for ETL and
prediction
• Alternative: Call Remote Workflow Click to query the
expected input(s)
− Trigger execution of workflows on
KNIME Server via REST API
Specify source column(s)
with input data / parameters

distribute.
Automation: Call Local Workflow
Convert output data

to KNIME table
ETL
Calls workflow once
for each input row
Send results back to

caller-workflow
Prediction
Enter default format

for incoming data
distribute.
Use Call Local Workflow to Send Conditional Emails with Report
Sometimes, report should be sent under specific circumstances

• E.g. if some KPI is below threshold
Provide email
credentials, host, etc.
Convert binary
column to file and
save to temp dir
Workflow creates Define rule, only

Path to file
report, sends back send email if
with report
binary column conditions apply

distribute.
Reporting in KNIME

distribute.
Reporting in KNIME
• Reporting in KNIME is done via a 3rd

party application named BIRT
(Business Intelligence Reporting Tool)
• Data is sent to BIRT from KNIME using
special nodes.
• Reports in BIRT are constructed from
report items, which may include
images, tables, charts and labels.
• Reports may be generated in a
variety of formats (html, pdf, pptx,
xlsx, docx, …)

distribute.
Installation
• Can be installed via KNIME -> Install KNIME Extension

• Install the KNIME Report Designer

distribute.
New Node: Data to Report
Send a data table to BIRT
Set the node

label!
Hint: The node label will be used to identify the

data source in the reporting view -> Make sure
to use understandable labels if you have more
than one data source

distribute.
New Node: Image to Report
Send an image to BIRT

− PNG and SVG are supported
formats (see node description
for details)
Hint: Customize the image size in the Data to

Report node to fit the report

distribute.
Edit the Report
Open the workflow and click the Report Editor button in the tool bar

distribute.
Reporting Perspective
Click button to
create report
Data from KNIME

- names of data Report layout – only
sources are taken structure, data is filled in
from node label when creating the report
View tabs
Add report items

via drag and drop

distribute.
Charting in BIRT
• Many chart types

• Fine control of plot appearance
• Familiar ‘Excel Like’ interface
• Supports interactivity

distribute.
Tips & Tricks
• Use an underlying grid to structure the report

• Names of columns should not change
• Use the grouping function to combine results
• Use the Master Layout Tab (For footers etc.)

distribute.
Exporting Data Exercise
Start with exercise: Exporting Data

• Send a heatmap that shows the confusion matrix values to report via
Image to Report node
• Send a table that shows the model accuracy statistics to report via Data to
Report node
• Create a report that includes the following elements:
– A report title
– A table with the overall accuracy
– The heatmap image
• Generate a PDF of your report

distribute.
Today’s Example

distribute.
The End
education@knime.com

Slides Deep Dive Into Data Science

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Slides Deep Dive Into Data Science

Transféré par

Droits d'auteur :

Formats disponibles

Deep Dive into Data Science

Copyright © 2019 KNIME AG

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

• A tool for data analysis, manipulation, visualization, and reporting

This educational material was produced for the course held at ®

NODES perform tasks on data

Nodes are combined to create

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

Data Access Transformation Analysis & Mining Visualization Deployment

This educational material was produced for the course held at ®

• Installing KNIME Analytics Platform

This educational material was produced for the course held at ®

• Select the KNIME version for your

This educational material was produced for the course held at ®

• Use the shortcut created by the installer

• Or go to the installation directory and launch KNIME via the knime.exe

This educational material was produced for the course held at ®

• The workspace is the folder/directory in which workflows (and

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

• In LOCAL you can access your own

This educational material was produced for the course held at ®

• Right-click in KNIME Explorer to create new workflow or workflow group

This educational material was produced for the course held at ®

• The Node Repository lists all

• The search box has 2 modes

• Nodes can be added by drag and

This educational material was produced for the course held at ®

• Console view prints out error

• Click on View and select Other…

This educational material was produced for the course held at ®

• The Description window gives

This educational material was produced for the course held at ®

• When selecting the workflow,

This educational material was produced for the course held at ®

• Node recommendation engine

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

• Dedicated file extensions for Workflows and Workflow groups associated

• *.knwf for KNIME Workflow Files

• *.knar for KNIME Archive Files

This educational material was produced for the course held at ®

A node can have 3 states:

This educational material was produced for the course held at ®

• Insert nodes into workspace by dragging them from Node Repository or

This educational material was produced for the course held at ®

• Most nodes require configuration

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

Plot View Data View

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®

• Connect via KNIME Explorer to a public repository with large selection of

This educational material was produced for the course held at ®

This educational material was produced for the course held at ®