Tackling The Challenges of Big Data

Introduction to Big Data

Samuel Madden
Professor and Director of Big Data at CSAIL
Massachusetts Institute of Technology

BIG

Data

Transport Medicine Social Media Education Industry

Government Retail Science Personal

Tackling the Challenges of Big Data

2014 Massachusetts Institute of Technology

Example 1: Medical Costs

MGH Cancer Center !
Super-Database!

Largest cancer database in the

world (173,301 patients)!
Based on national tumor registry!
Cross linked with death registry!
Includes billing, reports, labs,
imagery, genome SNPs!

!
!
Question: !
What are the factors
driving costs for lung
cancer patients?!
!

Source: Dr. James Michaelson, PhD, MGH, Harvard Medical School!

Tackling the Challenges of Big Data

2014 Massachusetts Institute of Technology

Example 2: Shopycat

Recommend Things My Friends Like That Are Compatible With

My Interests
Tackling the Challenges of Big Data

2014 Massachusetts Institute of Technology

Example 2: Flight Prediction

EV
EV
EV
K5
K5
NGF
NGF
WIG

How to Predict Flight Delays?

5647PIT
7/5/13 2:10 PM 7/5/13 2:21
Historical Information
5648ABE 7/5/13 12:10 PM 7/5/13 12:00

5651RIC
7/5/13 3:40 PM 7/5/13 3:56
Weather
96PDT
7/5/13 8:15 PM 7/5/13 11:35

96PDX
7/5/13 6:25 PM 7/5/13 6:26
Availability of Planes
4BFM
7/5/13 3:10 PM 7/5/13 3:18
64ISW
7/5/13 4:10 PM 7/5/13 4:16
Other Concurrent Flights
8416PWM 7/5/13 12:01 PM 7/5/13 12:37

Radar

Tackling the Challenges of Big Data

PM
PM
PM
PM
PM
PM
PM
PM

2014 Massachusetts Institute of Technology

Example: Flight Inconsistency!

Source: Luna Dong!
FlightView!

FlightAware!
Orbitz!

6:15 PM!
6:22 PM!
6:15 PM!

9:40 PM!

8:33 PM!
Tackling the Challenges of Big Data

9:54 PM!

2014 Massachusetts Institute of Technology

Example 2: Flight Prediction!

EV
EV
EV
K5
K5
NGF
NGF
WIG

How to Predict Flight Delays?

5647PIT
7/5/13 2:10 PM 7/5/13 2:21
Historical Information
5648ABE 7/5/13 12:10 PM 7/5/13 12:00

5651RIC
7/5/13 3:40 PM 7/5/13 3:56
Weather
96PDT
7/5/13 8:15 PM 7/5/13 11:35

96PDX
7/5/13 6:25 PM 7/5/13 6:26
Departure Time
4BFM
7/5/13 3:10 PM 7/5/13 3:18
64ISW
7/5/13 4:10 PM 7/5/13 4:16
Concurrent Flights
8416PWM Other
7/5/13
12:01 PM 7/5/13 12:37

Radar

Tackling the Challenges of Big Data

PM
PM
PM
PM
PM
PM
PM
PM

2014 Massachusetts Institute of Technology

When Do You Have a Big Data Problem?!

Too many bytes (Volume)!
Too high a rate (Velocity)!
Too many sources (Variety)!
Non-scalable analysis!
Human-intensive!
Computer Intensive!

Tackling the Challenges of Big Data

Flight Data!

Mass General
Hospital Data!

2014 Massachusetts Institute of Technology

Sources of Big Data

learning

science

transportation

retail

Big
Data

finance

social

entertainment

government
health/medical

Tackling the Challenges of Big Data

2014 Massachusetts Institute of Technology

What is this course going to cover?

Taught by experts
in the field! !

Case Studies!
Ingesting and Integrating Data!
Storage and Compute Platforms!
Presentation and Visualization!
Algorithms & Analytics!
!
Tackling the Challenges of Big Data

Regina Barzilay!
John Guttag!
Piotr Indyk!
Tommi Jaakkola!
David Karger!
Security !
Andrew Lo!
& Privacy!
Sam Madden!
Ron Rivest!
Ronitt Rubinfeld!
Daniela Rus!
Michael Stonebraker!
Matei Zaharia!
Nickolai Zeldovich!
!

2014 Massachusetts Institute of Technology

Tackling The Challenges of Big Data

Introduction to Big Data

THANK YOU