The Mythology of Big Data

OReilly Strata Conference February 2, 2011

Mark R. Madsen http://ThirdNature.net @markmadsen

Everytechnologycarries withinitselftheseeds ofitsowndestruc5on.

Codeisacommodity

http://www.flickr.com/photos/ecstaticist/1120119742/

Whatsthecentralmythunderlyingbigdata?

Themyththatdrovethegoldrush
All we need is a fat pipe and pans working in parallel

Youchangeanorgbyac.ngwith,throughothers,notalone.

Evolu5onofdata
50s60s:dataasproduct 70s80s:dataasbyproduct 90s00s:dataasasset 2010s+:dataassubstrate
Therealdatarevolu.onisin businessstructureand processesandhowtheyuse informa.on.

Everythingissodierentnow

Your grandmother, the data scientist.

Manycurrentapproachesmissthepoint

UsingBigData

Itsnotaboutbig

UsingBigData
And big is often not as big as you think it is.

Itsnotreallyaboutdata,either

UsingBigData
If theres no process for applying information in a specific context then you are producing expensive trivia.

Wheredoesthevalueindatacomefrom?
Formostofusinnondata businesses,thistranslates toHowcanweuse informa.ontoimprove thedecisionsmadeinour organiza.on? Weneedtofocusonthat singularlybaddecision makingenDty,thegroup. OrganizaDonsseemto amplifyinnatedecision makingaws.

Decisionmakingreali5es
TheoperaDngmodelinsenior managementisprimarilyintuiDonand paKernbased. Themodeformiddlemanagementis poliDcal,bureaucraDc. Newdataisdestabilizing,whichiswhy youmayhitawalltryingtopushyour datadrivenagenda. Dataiscontextual,soweneedstories toexplainhowwethinktheworld works,whymydataisbeKerthan yours,andwhyyourtheorysucks. CogniDvebiascreatesamorassfor interpretaDon.

Averyabstractbusinessintelligencemodel
Whoarethepeoplemakingdecisions?
Strategic TacDcal OperaDonal

Whatisthenatureoftheirdecisions?
Scope,Dmeframeofdecision,Dmescaleofdata,data volume,breadthofdata,frequency,paKernvsfactbased

Analytic complexity

Strategic TacDcal OperaDonal

Months Days Weeks Mins Days

PaMernbased Broadscope Factbased Moderate scope Rulebased Narrowscope

Theprocessaspectofdecisions5estopeople
ScopeofcontrolforpeopleinmostorganizaDonsaligns: inprocess,onprocess,overprocess

Strategic TacDcal OperaDonal

The exceptions not handled at one level due to rule / procedure / policy deficiency are escalated to the next.

Whatkindofsupportdotheyhavetoday?

Strategic TacDcal OperaDonal

Other people

Email, meetings

Reports, dashboards Realm of traditional BI

Reality of most reports and dashboards is that they provide basic monitoring at best.

Howandwherecanyouapplydatasolu5ons?
Highsinglevalue,less frequent,soimprovethe eecDvenessofindividual decisions. Fuzzymiddleground Lowsinglevalue,frequent, canimprovetheeciency ortheeecDvenessforlarge aggregateimprovement.

Analytic complexity

Strategic TacDcal OperaDonal

Whatdopeopledowithdata?
1. Describe:usedatatocharacterizeacurrentorpriorstateofthe system,forexamplemonitoringandidenDfyingexcepDons 2. Inves5gate:exploredatatodiscovertheboundariesand characterisDcsofasystem,frameaproblemornd supporDng/discrediDngevidence. 3. Explain:usedataandanalyDcmethodstodeterminecauses andeects,buildmodelsandconstructstories. 4. Predict:applyanalyDcmodelstodeterminepossible/probable futurestatesofthesystem 5. Prescribe:usedatainmodelstodenepolicy,procedure,and rulesfortakingacDon,andpossiblyautomatethem Datainfrastructureandtoolsupportfortheseac.vi.esinmost organiza.onsisunevenatbest,decreasingasyoumovedown.

Ifyouwanttobeadatascien1st,orbuildso5waretosupportthem,readthispaper

Structure

Eort

Figure:PirolliandCard,2005

Atoolmakersucceedsas,andonlyas,theusersofhis toolssucceedwithhisaid.Howevershiningtheblade, howeverjeweledthehilt,howeverperfectthehe_,a swordistestedonlybycu`ng.Thatswordsmithis successfulwhoseclientsdieofoldage.

FrederickBrooks

About the Presenter

Mark Madsen is president of Third Nature, a technology research and consulting firm focused on business intelligence, analytics and performance management. Mark is an award-winning author, architect and former CTO whose work has been featured in numerous industry publications. During his career Mark received awards from the American Productivity & Quality Center, TDWI, Computerworld and the Smithsonian Institute. He is an international speaker, contributing editor at Intelligent Enterprise, and manages the open source channel at the Business Intelligence Network. For more information or to contact Mark, visit http://ThirdNature.net.

About Third Nature

Third Nature is a research and consulting firm focused on new and emerging technology and practices in business intelligence, data integration and information management. If your question is related to BI, open source, web 2.0 or data integration then youre at the right place. Our goal is to help companies take advantage of information-driven management practices and applications. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in product and technology analysis, so we look at emerging technologies and markets, evaluating the products rather than vendor market positions.