Académique Documents
Professionnel Documents
Culture Documents
Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Champaign
www.cs.uiuc.edu/~hanj
2006 Jiawei Han and Micheline Kamber, All rights reserved
12/17/16
12/17/16
A data warehouse is a subject-oriented, integrated, timevariant, and nonvolatile collection of data in support of
managements decision-making process.W. H. Inmon
Data warehousing:
12/17/16
Data WarehouseSubject-Oriented
12/17/16
Data WarehouseIntegrated
12/17/16
12/17/16
Techniques
Data WarehouseNonvolatile
12/17/16
12/17/16
12/17/16
OLAP
users
clerk, IT professional
knowledge worker
function
decision support
DB design
application-oriented
subject-oriented
data
current, up-to-date
detailed, flat relational
isolated
repetitive
historical,
summarized, multidimensional
integrated, consolidated
ad-hoc
lots of scans
unit of work
read/write
index/hash on prim. key
short, simple transaction
# records accessed
tens
millions
#users
thousands
hundreds
DB size
100MB-GB
100GB-TB
metric
transaction throughput
usage
access
12/17/16
complex query
10
12/17/16
11
12/17/16
12
12/17/16
13
Multidimensional Data
Product
Re
gi
o
Month
12/17/16
14
country
date
product,country
1-D cuboids
date, country
2-D cuboids
12/17/16
3-D(base) cuboid
15
0-D(apex) cuboid
item
time,location
time,item
location
supplier
item,location
time,supplier
location,supplier
item,supplier
time,location,supplier
time,item,location
time,item,supplier
1-D cuboids
2-D cuboids
3-D cuboids
item,location,supplier
4-D(base) cuboid
time, item, location, supplier
12/17/16
16
12/17/16
Techniques
17
item
time_key
day
day_of_the_week
month
quarter
year
branch
branch_key
branch_name
branch_type
location_key
units_sold
dollars_sold
avg_sales
item_key
item_name
brand
type
supplier_type
location
location_key
street
city
state_or_province
country
Measures
12/17/16
18
item
Sales Fact Table
time_key
item_key
branch_key
branch
location_key
branch_key
branch_name
branch_type
units_sold
dollars_sold
avg_sales
Measures
12/17/16
item_key
item_name
brand
type
supplier_key
supplier
supplier_key
supplier_type
location
location_key
street
city_key
city
city_key
city
state_or_province
country
19
Example of Fact
Constellation
time
time_key
day
day_of_the_week
month
quarter
year
item
Sales Fact Table
time_key
item_key
item_key
item_name
brand
type
supplier_type
branch_key
location_key
branch
branch_key
branch_name
branch_type
units_sold
dollars_sold
avg_sales
location
to_location
location_key
street
city
province_or_state
country
dollars_cost
Measures
12/17/16
units_shipped
shipper
shipper_key
shipper_name
location_key
shipper_type 20
12/17/16
21
22
12/17/16
23
12/17/16
Techniques
24
12/17/16
25
all
Europe
region
country
city
office
12/17/16
Germany
Frankfurt
...
...
...
Spain
North_America
Canada
Vancouver ...
L. Chan
...
...
Mexico
Toronto
M. Wind
26
Multidimensional Data
Sales volume as a function of product,
month, and region
Re
gi
o
Industry Region
Year
Product
City
Office
Month Week
Day
Month
12/17/16
27
TV
PC
VCR
sum
1Qtr
2Qtr
Date
3Qtr
4Qtr
sum
Country
Pr
od
uc
sum
12/17/16
28
country
date
product,country
1-D cuboids
date, country
2-D cuboids
12/17/16
3-D(base) cuboid
29
Pivot (rotate):
Other operations
12/17/16
30
12/17/16
31
12/17/16
32
Top-down view
12/17/16
33
12/17/16
Techniques
34
Other
sources
Operational
DBs
Metadata
Extract
Transform
Load
Refresh
Monitor
&
Integrator
Data
Warehouse
OLAP Server
Serve
Analysis
Query
Reports
Data mining
Data Marts
Data Sources
12/17/16
Data Storage
OLAP Engine Front-End Tools
Data Mining: Concepts and
Techniques
35
Enterprise warehouse
collects all of the information about subjects
spanning the entire organization
Data Mart
a subset of corporate-wide data that is of value to
a specific groups of users. Its scope is confined
to specific, selected groups, such as marketing
data mart
Virtual warehouse
A set of views over operational databases
Datapossible
Mining: Concepts
and
Only some of the
summary
views may
12/17/16
Techniques
36
Data extraction
get data from multiple, heterogeneous, and
external sources
Data cleaning
detect errors in the data and rectify them when
possible
Data transformation
convert data from legacy or host format to
warehouse format
Load
sort, summarize, consolidate, compute views,
check integrity, and build indicies and partitions
Refresh
propagate the updates
Data Mining: Concepts
fromandthe data sources to
12/17/16
Techniques
37
Metadata Repository
Operational meta-data
12/17/16
38
Greater scalability
12/17/16
Concepts
and over star/snowflake
Specialized supportData
forMining:
SQL
queries
Techniques
39
12/17/16
40
n in an n-dimensional cube
How many cuboids
T ( Li 1)
with L levels? i 1
12/17/16
41
Cube Operation
city,
year]:
(item)
(year)
FROM SALES
CUBE BY item, city, year
(city, item)
Need compute the following Group-Bys
(date, product, customer),
(date,product),(date,
customer),
customer),
Data Mining: Concepts and
(date),
(product),
(customer)
12/17/16
Techniques
(city, year)
(item, year)
(product,
(city,
item, year)
42
Iceberg Cube
Motivation
Only a small portion of cube cells may be above
the water in a sparse cube
Only calculate interesting cellsdata above
certain threshold
Avoid explosive growth of the cube
12/17/16
43
Base table
Cust
C1
C2
C3
C4
C5
12/17/16
Region
Asia
Europe
Asia
America
Europe
Index on Region
Index on Type
44
12/17/16
Techniques
45
12/17/16
Techniques
46
12/17/16
47
Information processing
Analytical processing
Data mining
12/17/16
48
49
Mining result
Layer4
User Interface
OLAM
Engine
OLAP
Engine
Layer3
OLAP/OLAM
MDDB
Filtering&Integration
Database API
MDDB
Meta
Data
Filtering
Layer1
Databases
12/17/16
Data cleaning
Data
Warehouse
Data Data
integration
Mining: Concepts and
Techniques
Data
Repository
50
Summary
12/17/16
51
12/17/16
52