INFORMATION HAZARDS: A TYPOLOGY OF POTENTIAL HARMS FROM KNOWLEDGE

(2009) Draft 1.11

Nick Bostrom Future of Humanity Institute Faculty of Philosophy & James Martin 21st Century School Oxford University hethatincreasethknowledgeincreasethsorrow Ecclesiastes1:18 Abstract Informationhazardsarerisksthatarisefromthedisseminationorthepotentialdisseminationoftrue informationthatmaycauseharmorenablesomeagenttocauseharm.Suchhazardsareoftensubtler thandirectphysicalthreats,and,asaconsequence,areeasilyoverlooked.Theycan,however,be important.Thispapersurveystheterrainandproposesataxonomy. 1.Introduction Thereis,thesedays,acommonlyheldpresumptioninfavorofknowledge,truth,andtheuncovering anddisseminationofinformation.Itisraretofindsomebodywhoselfidentifiesasanobscurantistor whoopenlyespousesobscurantismasalegitimatepolicyinstrumentofwideutility. Evenreactionariesrarelyobjecttothisgeneralfavoringofinformation.Considersomeparticularly intransigentcreationistwhoopposestheteachingofevolutiontheoryinpublicschools.Hedoesnot constituteacounterexample.Forhedoesnotbelievethatevolutiontheoryisatruthtobeconcealed. Rather,hebelievesevolutiontheoryanerrorthatoughttobereplacedwithmoreaccurate information.Therefore,althoughhehappensunwittinglytostandinthewayoftruth,heneednot disagreewiththeclaimthatthetruthshouldbepromoted.Thecreationist,too,isatruthlover,albeit onewhoseaffectionsareunreciprocated. Althoughnobodymakesabriefforignorancegenerally,therearemanyspecialcasesinwhichignorance iscultivatedinorder,forexample,toprotectnationalsecurity,sexualinnocence,juryimpartiality;to preserveanonymityforpatients,clients,reviewers,andvoters;tocreatesuspenseinfilmsandnovels; toprotecttradesecrets;tomeasuretheplaceboeffectandavoidvariousresearchbiases;andtocreate 1

mentalchallengesforgamingandstudy.Thesecasesarecommonlyacceptedexceptionstothegeneral ruleofknowledgefavoring. 1 Inthispaper,wewillnotbeconcernedwithpostmodernistcritiquesoftheideaofobjectivetruthnor withskepticaldoubtsaboutthepossibilityofknowledge.Ishallassumesomebroadcommonsensical understandingaccordingtowhichtherearetruthsandwehumanssometimesmanagetoknowsomeof thesetruths. Thispaperwillalsonotdiscussthewaysinwhichharmcanbecausedbyfalseinformation.Manyof thosewaysareobvious.Wecanbeharmed,forinstance,byfalseinformationthatmisleadsusinto believingthatsomecarcinogenicpharmaceuticalissafe;or,alternatively,thatsomesafepharmaceutical iscarcinogenic.Wewilllimitourinvestigationtothewaysinwhichthediscoveryanddisseminationof trueinformationcanbeharmful. Letusdefine Informationhazard:Ariskthatarisesfromthedisseminationorthepotentialdisseminationof (true)informationthatmaycauseharmorenablesomeagenttocauseharm. 2 Relativetotheirsignificance,andcomparedtomanydirectphysicaldangers,sometypesofinformation hazardareundulyneglected.Itwillthereforebeusefultodistinguishsomedifferenttypesof informationhazard.Thiswillservetodrawattentiontosomeeasilyoverlookedrisksandwillhelpus createavocabularyfordiscussingthem. Theaimofthispaperistocataloguesomeofthevariouspossiblewaysinwhichinformationcancause harm.Wewillnothereseektodeterminehowcommonandserioustheseharmsare,norhowthey stackupagainstthemanybenefitsofinformationquestionsthatwouldneedtobeengagedinorder toreachaconsideredpositionaboutpotentialpolicyimplications.Itmaybeworthstressing,however, thatevenifonehasanextremelystrongintellectualcommitmenttotruthseekingandpubliceducation, onecanstilllegitimatelyandingoodconscienceexplorethequestionofhowsomeknowledgemightbe harmful.Infact,thisverycommitmentdemandsthatonedoesnotshyawayfromsuchanexploration orfromreportingopenlyonthefindings. 2.Sixinformationtransfermodes
1

Thenotionofdangerousorforbiddenknowledgeisalsoacommontropeinliteratureandinmanymythologies; seee.g.Shattuck1996.

Wesetasidethetrivialwayinwhichutterancescancauseharmviatheirperformativefunctions;cf.Austin1962. Thus,adictatorwhoproclaimsInvadethatcountry!disseminatesinformationthancanobviouslycausealotof harm,butweshallnotcountthispossibilityasaninformationhazard.

Wecandistinguishseveraldifferentinformationformats,ormodesofinformationtransfer.Eachcan beassociatedwithrisk.Perhapsmostobviously,wehave: Datahazard:Specificdata,suchasthegeneticsequenceofalethalpathogenorablueprintfor makingathermonuclearweapon,ifdisseminated,createrisk. 3 Butalso: Ideahazard:Ageneralidea,ifdisseminated,createsarisk,evenwithoutadatarichdetailed specification. Forexample,theideaofusingafissionreactiontocreateabomb,ortheideaofculturingbacteriaina growthmediumwithanantibioticgradienttoevolveantibioticresistance,maybealltheguidancea suitablyprepareddeveloperrequires;thedetailscanbefiguredout.Sometimesthemere demonstrationthatsomething(suchasanuclearbomb)ispossibleprovidesvaluableinformationwhich canincreasethelikelihoodthatsomeagentwillsuccessfullysetouttoreplicatetheachievement. Eveniftherelevantideasanddataarealreadyknown,andpublishedintheopenliterature,an increasedriskmaynonethelessbecreatedbydrawingattentiontoaparticularlypotentpossibility. Attentionhazard:Themeredrawingofattentiontosomeparticularlypotentorrelevantideas ordataincreasesrisk,evenwhentheseideasordataarealreadyknown. Becausetherearecountlessavenuesfordoingharm,anadversaryfacesavastsearchtaskinfindingout whichavenueismostlikelytoachievehisgoals.Drawingtheadversarysattentiontoasubsetof especiallypotentavenuescangreatlyfacilitatethesearch.Forexample,ifwefocusourconcernandour discourseonthechallengeofdefendingagainstviralattacks,thismaysignaltoanadversarythatviral weaponsasdistinctfrom,say,conventionalexplosivesorchemicalweaponsconstituteanespecially promisingdomaininwhichtosearchfordestructiveapplications.Thebetterwemanagetofocusour defensivedeliberationsonourgreatestvulnerabilities,themoreusefulourconclusionsmaybetoa potentialadversary. Itwouldbeamistaketosupposethatbecausesomeideaisalreadyinthepublicdomaintherecanbeno furtherharmindiscussingtheideaandreferringtoitinotherpublications.Suchfurtherdiscussionscan createoraggravateanattentionhazardbyincreasingtheideassalience.Oneindexofhowmuch mileagethereisinanideaistheamountbuzzsurroundingit. Clumsyattemptstosuppressdiscussionoftenbackfire.Anadversarywhodiscoversanattemptto concealanideamayinferthattheideacouldbeofgreatvalue.Secretshaveaspecialallure.The

Thetermdatahazardalsohasanarrowtechnicalmeaninginthecontextofcomputerprocessordesign,which isnotrelevanthere.

adversarymayinvestmoreinunderstandingandexploringanideathatheknowsthathisenemyis tryinghardtokeepsecret.Abookthatiscensoredoftenbecomesmorewidelyread. 4 Itispossiblethateffortstocontemplatesomeriskareasay,existentialriskwilldomoreharmthan good.Onemightsupposethatthinkingaboutatopicshouldbeentirelyharmless,butthisisnot necessarilyso.Ifonegetsagoodidea,onewillbetemptedtoshareit;andinsodoingonemightcreate aninformationhazard.Still,onelikestobelievethat,onbalance,investigationsintoexistentialrisksand mostotherriskareaswilltendtoreduceratherthanincreasetherisksoftheirsubjectmatter. Sometimesitisrighttoharm.Ifinformationenablesanagenttoharmrightly,thiscanbeagoodthing; andthepossibilityofthathappeningshouldthereforenotbeclassifiedasarisk.Forexample,the policesobtainmentofcertaininformationmightharmsomecriminalbyenablingthepolicetoarrest him;andthiscanbegood.However,wecouldsaythatfromthecriminalspointofviewthereisan informationhazard.Hefacesariskthathiswhereaboutswillbereported. Notalltypesofinformationtransferarebestthoughtofintermsofdata,ideas,orattention.Wecan alsoconsiderimplicitformsofinformation,suchasprocessesororganizationalstructures,whichcan giveonefirmanadvantageoveranother,andwhichmightbeimitatedorreplicatedbyacompetitor.5 Similarly,individualsoftenlearn,andshapetheirownpersonality,byemulatingsomeotherperson. Suchemulationcanhappenunintentionallyandevenwithoutawarenessthatemulationistakingplace. Abadrolemodelcanposeatemplatehazard. Templatehazard:Thepresentationofatemplateenablesdistinctivemodesofinformation transferandtherebycreatesrisk. Wecanalsoregisterasadistinctmodeofcommunicationsocialsignaling,wherethefocusisnotonthe contentthatisbeingtransmittedbutonwhatthiscontentandthefactthatitisbeing communicatedsaysaboutthesender.Nonverbalactionscanalsoserveasocialsignalingrole,ifthey areaimednotonlyatdirectlyachievingthesomepracticaloutcometowhichtheyarenominallygeared butalsotosignalsomehiddenqualityoftheagentthatperformstheaction.Forexample,onecould givealmstothepoornotonlybecauseonewishestosatisfytheirneedsbutalsobecauseonewishesto beseenbyonespeersasakind,caring,andgenerousperson.Onemightutterpatrioticstatementsnot onlytoconveytothelistenerinformationaboutvariousattributesofonesnation,butalsotosignal onesstatusasaloyalcitizen,oronesaffiliationwithsomeparticularpoliticalgrouping. Signalinghazard:Verbalandnonverbalactionscanindirectlytransmitinformationaboutsome hiddenqualityofthesender,andsuchsocialsignalingcreatesrisk.

AgoodexampleofthisistheratherdullSpycatcherbyPeterWright,whichbecameabestsellerinthe1980safter Thatchertriedtobanit,seeZuckerman1987. NelsonandWinter1990andPorter2004.

Sometopicsareespeciallyattractivetocrackpots.Seriousacademicsmightshyawayfromsuchtopics becausetheyfearthatworkingonthosetopicssignalsintellectualflakiness.Atleasttwosignaling hazardsariseinthiscontext.Oneistherisktoindividualthinkerswhomightsufferundeserved reputationaldamagemerelyforworkinginanareawhichalsohappenstoattractlotsofcrackpots. Anotheristherisktosocietythatimportantareasofresearchwillremainuncultivatedbecausethe ablestresearchers(andtheirsponsors)protecttheirreputationseitherbyshunningthoseareasinfavor ofmoresociallyacceptable,highstatusfieldsofstudyorbyadoptingrelativelylesseffectivemeansof exploration,suchashypertrophicformalismandexpensivetechnicalapparatus,whichareharderfor crackpotstomimic. Finally,wealsocountasadistinctmodethetransferofinformationcontainedintheparticularway somecontentisformulatedandexpressed.Avividdescriptionofsomeevent,forexample,canactivate psychologicalprocessesthatliedormantwhenthesameeventisrecountedindryprose. Evocationhazard:Therecanbeariskthattheparticularmodeofpresentationusedtoconvey somecontentcanactivateundesirablementalstatesandprocesses. Eachoftheseinformationtransfermodesdata,idea,attention,template,signaling,andevocation canplayaroleincreatingvariouskindsofrisk.Thelatterfour,especially,areeasilyoverlooked. Thefollowingfivesectionsintroduceanothercategorizationschemewhich,whensuperimposedupon thedivisionintoinformationtransfermodes,rendersamorefinegrainedpictureofthewaysinwhich informationcanbehazardous(summarizedintable1,below). 3.Adversarialrisks Previousexamplesfocusedonadversarialsituationsinwhichsomefoeisintentoncausingusharm.A burglarwhoknowswherewekeepourmoneyandwhenwewillreturnhomeisinastrongerpositionto succeedwithhiscrime. Enemyhazard:Byobtaininginformationourenemyorpotentialenemybecomesstrongerand thisincreasesthethreatheposestous. Oneparadigmaticcontextforthistypeofhazardisnationalsecurity.Withinthedefensesector, activitiesaimedatreducingenemyinformationhazardrangefromcounterintelligenceworktothe applicationofcamouflagetoconcealtroopsinthefield. Enemyhazarddependsontheexistenceofvaluableinformationthatanenemymightobtain.Indirectly, therefore,ourownactivitiescanbehazardousiftheycontributetotheproductionofsuchinformation. Militaryresearchoffersmanyexamples.Weinvestinresearchanddevelopmentofnewweaponsand newtactics.Thisactivityproducesinformationthatisvaluabletoourenemy.Theenemyobservesour improvedtactics.Hisspiesobtaintheblueprintsofourimprovedtechnology.Ortherelevant informationleaksoutinotherways,perhapsintheformofideas,attention,ortemplates.Asa 5

consequence,ourenemysoonreplicatesourachievement.Whenhostilitieserupt,webattleourown inventions. Rationalstrategyformilitaryresearchwouldgivesignificantconsiderationtosucheffects.TheUnited States,forexample,mightrefrainfromaggressivelypursuingdevelopmentofelectromagneticpulse weapons.Becauseofthecountrysunusuallyheavyrelianceonelectronics,theexistenceofeffective EMPweaponswoulddifferentiallybenefititsadversaries. Conversely,anaggressiveapproachtoresearchcouldhelpprotectacountryagainstitsenemies.A countrymightpursuemilitaryresearchtocatchupwiththetechnologicalleader.Aleaderinmilitary technologymightinvestinresearchinordertomaintainitslead.Theleadermightreasonthat,whileits discoverieswilleventuallyleakoutandbenefititsenemies,itcanproduceasteadystreamof discoveriesandcontinuallykeepafewstepsahead. Therearesituations,though,inwhichacountryisbetteroffretardingitsprogressinmilitary technology,evenwhenthenewtechnologieswouldnotdifferentiallybenefititsenemiesandevenif considerationssuchascostandforegoneopportunitiesforbuildinginternationaltrustaresetaside. Supposesomecountryhasgreatsuperiorityinmilitarypowerandmilitarytechnology,andasa consequencefaceslittleseriousthreat.Byinvestingheavilyinmilitaryresearch,itcouldincreaseits leadandtherebyfurtherenhanceitssecuritysomewhat.Yetsuchinvestmentmightunderminesecurity inthelongerterm.Therateofinformationleakagemightbeafunctionofthesizeofthetechnological gapsuchthatincreasingthegapincreasestherateoftheleakage.Perhapsweaponssystemscomein generationsanditmaybeinfeasibletokeepsecretmorethanaboutonegenerationbeyondthe enemyslevel.Ifso,introducingnewgenerationsatafasterratemightnotincreasethetechnological lead,butserveonlytoacceleratebothcountriesascentofthemilitarytechnologytree,perhapsto levelswherewarsbecomemoredestructive.Ifyouarealreadyutterlysuperiorinconventional weapons,thenrushingtoinventafissionbomblongbeforeyourenemiescouldhavegottheremightbe counterproductive.Similarly,hasteningtheintroductionofthefusionbombmightbebadstrategyif youcouldhavebeenconfidentofremainingsuperiorinfissionbombs. Acceleratingtheascentofthetechnologytreecouldalsobebadifthetreeisoffiniteheight,sothatat somepointtheleaderrunsoutofopportunitiesforinnovation.Someweaponssystemsmightreacha levelofperfectionfromwhichfurtherimprovementsaredifficultorimpossible.(Inthecategoryof weaponsofmassdestructionfordeterrentuse,forinstance,thehydrogenbombmightrepresenta nearmaximum.)Eventuallyeverybodymayplateauatthislevel,andthepreviousleaderwillloseits advantage.Inordertomaintainatechnologicalleadforaslongaspossible,theleadermightwishto pushthetechnologicalfrontierattheslowestpossiblepacethatisconsistentwithmaintainingan acceptableleadateverypointintimeuntiltechnologicalmaturityisreached. Themilitaryillustrationshowshowinformationhazardsariseinsomesituationsfromonepartys (potential)intenttoinflictharmonanother.However,informationrisksstemmingfromanadversarial relationshiparisemuchmorewidely.Incompetitivesituations,onepersonsinformationcancause

harmtoanotherevenifnointentiontocauseharmispresent.Example:Therivaljobapplicantknew moreandgotthejob. Competivenesshazard:Thereisariskthat,byobtaininginformation,somecompetitorofours willbecomestronger,therebyweakeningourcompetitiveposition. Exclusivepossessionofinformationiscentraltothebusinessmodelofmanyfirms.Acompetitorcan gainvaluableinformationbyobservingtheproductionandmarketingmethodsofarivalfirm,reverse engineeringitsproducts,orheadhuntingitsemployees. 6 Firmsgotogreatlengthstoprotecttheir intellectualcapital,relyingonawidevarietyofmethodsincludingpatenting,copyright,nondisclosure agreements,physicalsecurity,inhouseproductioninsteadofoutsourcing,compensationschemesthat discourageemployeeturnover,andsoforth.Wecanidentifythreattointellectualpropertyasaspecial caseofcompetitivenesshazard: Intellectualpropertyhazard:AfacestheriskthatsomeotherfirmBwillobtainAsintellectual property,therebyweakeningAscompetitiveposition. Anothertypeofadversarialhazardariseswhenanagentsownpossessionofsomeinformationhasthe potentialtorenderherlessabletoprevailinsomecompetitivesituation.Inorderforablackmailerto besuccessful,histargetmustbelievethathehassomeincriminatingorembarrassinginformation, informationthathecouldthreatentorelease.Solongasthetargetremainsunaware,noblackmailcan takeplace.Whenshelearnsaboutthethreat,sheisdeliveredintotheextortionistsclutches.Similarly, inthegameofchicken:Twodriversspeedtowardsoneanotherfromoppositedirections;thefirstto swerveloses.Ifonedrivercouldcrediblycommittonotswerving,hewouldwin,sincehisopponent(it isassumed)wouldratherlosethegamethancrash.Agametheoristengaginginthispastimecould havehimselfblindfolded,preventinghimselffromacquiringinformationabouttherelativedistanceof thetwocars,thusrenderinghimselfincapableofreliablyswervingatthelastpossiblemoment;and therebyconvincinghis(hopefullyrational)counterparttoyieldtheroad. Commitmenthazard:Thereisariskthattheobtainmentofsomeinformationwillweakenones abilitycrediblytocommittosomecourseofaction. Commitmenthazardsaresometimesinstancesofenemyhazardandsometimesofcompetitiveness hazards.(Commitmenthazardscanalsoariseinasingleagentcontext,asweshallseelater.) Insomesituationsitcanbeadvantageoustomakeaprobabilisticthreat,athreatthatleaves somethingtochanceintheterminologyofThomasShelling.7 Athreat,tobeeffective,mustbe credible.Yetthereasonforissuingathreatisdeterrenceexante,notrevengeexpost;andcarryingout athreatisoftencostly.Considersomepossiblepunitiveactionthatissocostlytocarryoutthatathreat todosowouldscarcelybecredible,suchasanuclearfirststrikebyonemajorpoweronanother.A
6

Porter2004. Schelling1981.

nucleararsenalcouldneverthelessbeusedtomakeathreat.SideAcanthreatenthatunlessthesideB makessomeconcession,Awilltakesomeactionthatincreasestheriskofnuclearwar.Forinstance,A couldthreatentoinitiateskirmisheswithconventionalweaponsthatwouldcreatesomeriskof escalation.Alternatively,Acouldthreatentomakeitsownnuclearcontrolandcommandsystemless safeagainstaccidentallaunch,forinstancebycreatingacrisisandputtingitsforcesonhighalert.The ideahereisthatitismuchlesscostlyforAtocarryoutathreattomoderatelyincreasetheriskofa nuclearwarthanitwouldbeforAtoactuallylaunchanuclearattack.Theprobabilisticthreatcan thereforebemorecredibleandmoreeffective. If,however,newinformationcametolightthatdispelledtherequisiteuncertaintyuncertainty,for example,abouthowkeyactorswouldreactduringacrisisthentheabilitytomakeprobabilisticthreats wouldbeundermined.Thepossibilityofsuchinformationbeingreleasedcanthusconstituteakindof informationhazard.DuringtheColdWar,kindredconsiderationsmayhaveledthesuperpowersto maintainsomeambiguityintheirstrategicpostures.Thiskindofinformationhazardmightinvolvea combinationofenemyhazardandcommitmenthazard.8 Wecanalsoidentifyanothertypeofriskthatcanarisefromourownknowledgewhenthereisa possibilitythatsomebodyelsewillbecomeouradversarybecauseofthisknowledge: Knowingtoomuchhazard:Ourpossessingsomeinformationmakesusapotentialtargetor objectofdislike. NadezhdaSergeevnaAlliluyeva,Stalinssecondwife,wasknowntohavemisgivingsaboutthe CommunistPartypurgesandtheconcomitantfamine.FollowingaspatwithStalinin1932,shewas founddeadinherbedroom,apparentlyhavingtakenherownlife. 9 Thesuicidecouldbeinterpretedas akindofdesperaterebukeofStalinandhispolicies;andsincethatwouldbepoliticallyembarrassing, thedeathwasofficiallyattributedtoappendicitis.Theindividualswhohaddiscoveredthebodyand whoknewtherealcauseofdeathfoundthemselvesingravedanger.Inalaterallegedlyunrelatedtrial, twodoctorswhohaddeclinedtosignthefalsedeathcertificateweresentencedtodeathandexecuted. InthewitchhuntsoftheEarlyModernperiodinEurope,awomansallegedpossessionofknowledgeof theoccultorofbirthcontrolmethodsmayhaveputheratincreasedriskofbeingaccusedof witchcraft. 10 InthegenocideperpetratedbyPolPotandtheKhmerRougeregime,theentire
8

IfsideAknewhowAwouldbehaveinacrisis;andifsideB,whilenotknowinghowAwouldbehavebutknowing thatAknewhowAwouldbehave;thenAcouldbecomelessabletoissueaneffectiveprobabilisticthreat.Bcould reasonthatifAknewthatAwouldlaunchanuclearattackinacrisisthenAwouldbelesslikelytothreatento createacrisis(assumingthatBknewthatAwasuncertainastowhetherBwouldyieldtoAsthreat).Thus,B couldinferthatifAdoesinfactissueathreattocreateacrisis,itwouldlikelybebecauseAknewthatacrisis wouldnotescalateintoanuclearwar.ThiswouldmakeBlesslikelytoyieldtothethreat. Montefiore2005. Levack1987.

10

intellectualclasswasslatedforextermination. 11 Someculturesplaceahighvalueonsexualinnocence, particularlyingirls,andayoungwomanmightfindhermarriageprospectsdimmedifsheappearsto knowtoomuchaboutsexorifhergeneraleducationintimidatesprospectivehusbands. 12 Inmany schools,nerdychildrenwhoappeartohavetoomuchacademicknowledgeareostracizedand bullied.Knowingtoomuchhazards,ofvaryingdegreesofseverity,seemtoariseinmanydifferent contexts. 4.Riskstosocialorganizationandmarkets Wehavesofarfocusedonthepossibilityofanadversarygaininganadvantageasaresultofinformation obtainedbyeithertheadversaryorourselves.Theadversarymightthenharmusdeliberatelyand directly,asinamilitaryattack;orindirectlyandperhapsunwittinglybyweakeningourcompetitive position. Yettherearemanyothertypesofinformationhazard.Inparticular,informationcansometimesdamage partsofoursocialenvironment,suchascultures,norms,andmarkets.Suchdamagecanharmsome agentswithoutnecessarilystrengtheningorbenefittingtheiradversariesoranybodyelse. Normhazard:Somesocialnormsdependonacoordinationofbeliefsorexpectationsamong manysubjects;andariskisposedbyinformationthatcoulddisrupttheseexpectationsforthe worse. Behaviorinsomegivendomaincanbeguidedbydifferentnorms,withtheresultofdifferentsocial equilibriabeinginstantiated.Normsaresometimesformulatedandimposedfromabove,withlegal backing;forexample,anormthatonemustdriveontherightsideoftheroad.Buteveniftherehad beennosuchlaw,motoristsmighthavespontaneouslydevelopedthenormofdrivingontherightside, justasthereisanormofextendingtherighthandinahandshake. Withregardtowhichsidetodriveon,thereisnointrinsicbenefittoleftorright,solongasthereis someclearrulethateverybodyfollows.Inotherdomains,however,differentpossiblesocialequilibria canhavewidelydivergentconsequencesforhumanwelfare.Inasocietywithlowcorruption, individualsmightprospermostbybeinghonest,trusting,andplayingbytheofficialrules;whileina highcorruptionsociety,individualsfollowingthosestrategieswouldbesuckers.Theoptimalstrategy foroneindividualdependsonthestrategiespursuedbyotherswhochosetheirstrategiesonthebasis theirexpectationsabouthowotherswillreact.Informationthatalterstheseexpectationscantherefore changebehavior.Thiscanleadtoachangeinnormsthatmovesagrouporawholesocietyfromone

11

FawthropandJarvis2005.Inanyactuallargescalehistoricalepisode,ofcourse,multiplecausesarelikelyto havebeeninvolved,possessionofdangerousknowledgebeingatmostonecontributingfactor. Schlegel1991.

12

equilibriumstatetoanother.Thepossibilityofmovingtoaworsesocialequilibriumcanbe characterizedasanormhazard. Locallysuboptimalpoliciesaresometimesjustifiedfromawiderperspectiveasapriceworthpayingto protectnormsthatservetoblockaslideintoaworsesocialequilibrium.Asociallyconservativeoutlook mightbebasedonthebeliefthatsuchslidesareamajordangerandthatstrictenforcementofexisting normsisanecessarycountermeasure. 13 Evencallingintoquestionaparticularnorm,ormakingsmall adjustmentsofsomenorm,mightunderminetheauthorityofandtherebyweakentheoverall structureofextantnorms,increasingtheriskofmoraldecayorsocialunraveling. 14 Similarly,onecan objecttosomejudicialdecisionsbecauseofthelegalprecedentstheyset;tosomeforeignpolicy decisionsbecauseoftheireffectoncredibility;andsoforth. 15 Ifwetakethewordnorminitswidesense,wecanalsothinkofmoneyasanormorabundleof norms.Thefunctionsthatmoneyservesintheeconomydependonpeoplehavingcertainexpectations aboutotherpeoplesbeliefsandattitudestowardsmoneyanditsspecificforms,suchascash. Counterfeitingandexcessivemoneyprintingcanundermineacurrency,destroyingitsabilitytoserveas amediumofexchangeandastoreofvalue.Thisisanotherexampleofnormhazard. Itisobvioushowsomekindsoffalseinformationcandamagebeneficialnorms.Butnormscanalsobe damagedbytrueinformation.Wehavealreadyalludedtothephenomenonofselffulfilling prophesiespeopleactingmorehonestlyiftheybelievethemselvestobeinalowcorruptionsociety, andviceversa;driversdrivingontherightsideiftheybelievethatotherswillmakethesamechoice. Anotherphenomenoninwhichtrueinformationcandamagenormsisinformationcascades. Informationcascadescanarisewhenagentsmakechoicessequentially,andeachagenthas,inaddition tosomenoisyprivateinformation,theabilitytoobservethechoices(butnottheinformation)ofsome oftheagentsinfrontofherinthequeue. 16 Ithasbeensuggestedthatinformationcascadesplayan
Hirschman1991. Cf.Schellingsconceptofafocalpoint(Schelling1960). RizzoandWhitman2003;Volokh2003.

13

14

15

16

SupposethathundredsofrockfansaredrivingtotheGlastonburymusicfestival.Atsomepointeachdriver reachesanintersectionwheretheroadsignshavebeenvandalized.Asaresult,thereisuncertaintyastowhether toturnleftorright.Eachdriverhassomeprivateinformation,perhapsadimdrugcloudedrecollectionfromthe previousyear,whichgiveshera2/3chanceofpickingthecorrectdirection.Thefirstcararrivesatthe intersection,andturnsright.Thesecondcararrives,andalsoturnsright.Thedriverinthethirdcarhasseenthe firsttwocarsturnright,andalthoughhisprivateintuitiontellshimtoturnleft,hefiguresitismorelikelythathis ownintuitioniswrong(1/3)thanthatboththeprecedingcarswentthewrongway(1/9);soheturnsrightaswell. Asimilarcalculationisperformedbyeachsubsequentdriverwhocanseeatleasttwocarsahead.Everycarends upturningright. Inthisscenario,thereisa1/9chancethatalltherockfansgetlost.Letussupposethatifthathappens,the festivaliscancelled.Hadtherebeenadensefog,preventingeachdriverfromseeingthecarinfront(thus reducinginformation),then,almostcertainly,approximately2/3ofallthefanswouldhavereachedGlastonbury,

10

importantexplanatoryroleinaccountingforfaddishbehaviorinmanydomains,includingfinance, zoology,politics,medicalpractice,peerinfluenceandstigma. 17 Informationalcascadingmightalso contributetoaMatthew(therichgetricher)effect.Forexample,eminentscientiststendtogetmore creditthanunknownresearchersforsimilarcontributions. 18 Partofthereasonmightbethatwhen thereisuncertaintyastowhomadethebiggercontribution,itis,ceterisparibus,morelikelytohave beenmadebythemoreeminentscientist,whoconsequentlygetsthecredit;butwiththeresultthatthe fameofthealreadyslightlyfamouscansnowballtoundeservedproportionswhileothersareunfairly ignored. Anotherimportantwayinwhichtrueinformationcandamagesocialorganizationisthroughinformation asymmetries.Whenonepartyhasinformationthatotherslack,theinformationasymmetrysometimes preventsmutuallybeneficialtransactionsfromtakingplace. Informationasymmetryhazard:Whenonepartytoatransactionhasthepotentialtogain informationthattheotherslack,amarketfailurecanresult. Economicmodelsofadverseselectionandmoralhazardillustratesomeofthepossibilities.Inthe marketforusedautomobiles,theselleroftenhasmoreinformationaboutthequalityofthecarthanthe prospectivebuyer.Ownersofbadcars,lemons,aremorewillingtopartwiththeirvehiclethan ownersofgoodcars.Buyers,knowingthis,suspectthatthecarbeingofferedthemisalemon,andare willingtopayaccordingly.Thisbuypriceistoolowtointerestpotentialsellersofgoodcars,withthe resultthathighqualityusedcarsarewithheldfromthemarket,leavingpredominantlylemons.The informationasymmetryinhibitsthemarketinhighqualityusedcars.Thishelpsexplainwhythevalueof abrandnewvehicleplummetsthemomentitdisembarksthedealership.19 Insuranceoffersmanyillustrationsofthepotentialfornegativeeffectsofinformationasymmetry.For example,incountrieswithprivatehealthcare,considerascenarioinwhichtheavailabilityofgenetic testingcombinedwithvastlyimprovedknowledgeabouthowinterpretthetestsprovidebuyersof healthinsurancewithawealthofnewinformationabouttheirpersonalriskprofile.Ifprivacylegislation prohibitedinsurancecompaniesfromaccessingthesameinformation,theresultinginformation asymmetrycouldunderminetheinsurancemarket.Adverseselectionwouldleadthesubjectswiththe highestriskprofilestobuymoreinsurance.Insurancecompanies,anticipatingthis,wouldraise
enablingthefestivaltotakeplace.Oncethefestivalstarts,anylostfancanhearthemusicfromafarandfindtheir waythere.Wecouldthushaveasituationinwhichreducinginformationavailabletoeachdriverincreasesthe chancethathewillreachhisdestination.Clearweathercreatesaninformationalcascadethatleadstoan inefficientsearchpattern.
17

Bikhchandani,HirshleiferandWelch1992. Merton1968.

18

19

Akerlof1970.Here,asthroughoutthispaper,wearenotconcernedtogiveadetailedaccountofsome particularempiricalphenomenon;ourgoalistoilluminatesomefeaturesoftheconceptuallandscape.

11

premiums.Thehigherpremiumswoulddetermorelowrisksubjects,amplifyingtheadverseselection effectuntil,inanextremescenario,thehealthinsurancemarketcollapses. 20 Relativetosucha scenario,bothbuyersandsellersofinsurancemightbetteroffifneitherobtainstheextrainformation. Thepossibilityofreleaseofnewinformationtoonepartyofapotentialtransactioncanthus,under certaincircumstances,beahazard. Althoughasymmetricinformationisparticularlycorrosive,insurancemarketscanalsocollapsebecause ofsymmetricinformation,informationthatissharedbetweenallparties.Insuranceispredicatedon uncertainty.Itmakesnosenseforyoutoinsureagainstalossthatyouarecertainyouwillnotincur, anditmakesnosenseforaninsurancecompanytoofferyouinsuranceagainstalossthatitknowsthat youwillincuratsomeknowndate;thepremiumwouldhavetoexceedthecoverage. Iftheonlyusefulroleofinsuranceweretoreduceuncertaintyaboutfuturerevenueorwelfare,then informationthatincreasedpredictabilitywouldremovetheneedforinsuranceatthesametimeasit removedthepossibilityofinsurance:noharmwouldbedone.However,insuranceservesother functionsaswell.Oneisredistributivejustice.Ininsurance,thefortunatesubsidizetheunfortunate. Selfishagentsjointheschemebecausetheydonotknow,exante,towhichgrouptheybelong. Entirepoliticalphilosophieshavebeenconstructedaroundthenotionofinsurance.Forexample,in JohnRawlstheoryofjustice,thejustsocialorderisdefinedwithreferencetowhatpeoplewould hypotheticallychoosefrombehindaveilofignorance,i.e.iftheywereignorantaboutwhichsocial roletheythemselvesoccupy. 21 ARawlsianmightattributemanyofthepracticaldifficultiesingetting thisconceptionofjusticeimplementedtothefactthatvotersandpoliticaldecisionmakersareinreality notbehindaveilofignorance.Selfishpeoplewhoknowtheirowncircumstancestheirsocioeconomic class,race,occupation,andsoforthmayfavorpoliciesthatpromotetheirselfinterestratherthanthe allegedlyfairerpoliciesthattheywouldhavechosenhadtheybeenignorantabouttheirownactual situation.Knowledgeofonespresentandfuturesituation,though,isamatterofdegree.Onecan thinkofscenariosinwhichincreasingtheavailabilityofinformationaboutthesethingswouldmakethe implementationofajustsocialordermoredifficult.Forinstance,elitesupportforasocialsafetynet mightslackenifelitescouldknowwithcertaintythatneithertheynortheirchildrenorgrandchildren wouldeverneedtouseit. 22 Supportforprotectionoffreedomofspeechandminorityrightsmight weakenifmostindividualscouldbesurethattheywouldneverfindthemselvesinaprosecutedminority andthattheiropinionswouldneverbeamongtheonesthatthecensorswouldsilence. Thepossibilityofsucheffectsofsymmetricinformationcanbeviewedasarisk:
20

Itisunrealistictosupposegeneticinformationtoproducesuchanextremeconsequencesincemuchofthe varianceinhealthoutcomesisduetonongeneticvariablesandchance. Rawls2005.

21

22

AsimilarpointismadeinKavka1990.Kavkaalsoarguesthatintensesocialconflictwouldariseifthose individualsandgroupsthatwouldsuffer(possiblynoncompensable)harmfromsomeproposedpolicycouldknow thisexante.

12

Unveilinghazard:Thefunctioningofsomemarkets,andthesupportforsomesocialpolicies, dependsontheexistenceofasharedveilofignorance;andtheliftingofwhichveilcan underminethosemarketsandpolicies. Thisphenomenoncanalsobeinstantiatedintheiteratedprisonersdilemma,whereagentsfacea choicebetweencooperatinganddefectinginanunknownnumberofrepeatencounters.Agentsmight cooperateineachroundinordertosecuretheotherplayerscooperationinthefollowingrounds.Yet cooperationcanunravelifplayersknowhowmanyroundstherewillbe.Whentheyknowtheyarein thefinalroundhencefacingtheequivalentofatraditionaloneshotprisonersdilemmatheyboth faceincentivestodefect.Worse,inthepenultimateroundtheycanforeseethattheywillnextbeinthe finalroundinwhichtheywillbothdefect;soincentivesfavordefectinginthepenultimateroundtoo andsoon,allthewaybacktothefirstround.Theopportunityforlongtermmutuallybeneficial cooperationcouldthusberuinedthroughthelossofignoranceaboutthefuturedurationofthe relationship. Wehavealreadydiscussedintellectualpropertyhazardasanexampleofadversarialrisk.Intellectual propertytheft,though,isaproblemnotonlyforindividualfirmsthatrisklosingouttotheir competitors.Threatstointellectualpropertycanundermineentiresectorsoftheeconomicsystemby makingitharderforfirmsandindividualstointernalizethebenefitsoftheirresearchandproduct development.Thelegalsystemprovidesonlypartialprotectionandimposesbigadministrative, transaction,andenforcementcostswhichcanthemselvesimpedeinnovation.Defenseofintellectual assetsthereforetendstodependalsoonvariousformsofsecrecyandphysicalbarrierstoaccessand copyingofsensitivedata.Thepotentialfordevelopmentsthatwouldreducetheseobstacles,whenthat wouldhavenegativeconsequences,constitutesanunveilinghazard.23 Consider,finally Recognitionhazard:Somesocialfictiondependsonsomesharedknowledgenotbecoming commonknowledgeornotbeingpubliclyacknowledged;butpublicreleaseofinformationcould ruinthepretense. Twogentlemen,AandB,areinasmallroomwhenAbreakswind.Eachknowswhathashappened. Eachmightalsoknowthattheotherknows.Yettheycancolludetopreventanembarrassingincident. First,Bmustpretendnottohavenoticed.Second,Amight,withoutlettingonthatheknowsthatB knows,provideBwithsomeexcuseforescapingoropeningthewindow;forexample,Acouldcasually remark,afterashortdelay,thattheroomseemstoberatheroverheated. 24 Therecognitionhazard consistsinthepossibilityofdisseminationofsomeinformationthatwouldconstituteorforceapublic acknowledgement;onlythenwouldtheflatusbecomeasociallypainfulfauxpas.
Theclaimhereisnotthattheeasieritistoprotectintellectualassets,thebetter.Insomeareastheremightfor examplebeaninefficientlyhighleveloflegalprotection.Developmentsthatmakeintellectualpropertytheft easiertocarryout,andhardertodetectandpunish,couldthenbesociallybeneficial. Goffman1959.

23

24

13

 5.Risksofirrationalityanderror Thenthereareinformationhazardswhich,bycontrasttothosementionedabove,dependoneither irrationalityorfalsebeliefs.Thisdependency,ofcourse,doesnotconsignthecorrespondinghazardsto amarginalstatus. Consider Ideologicalhazard:Anideamight,byenteringintoanecologypopulatedbyotherideas, interactinwayswhich,inthecontextofextantinstitutionalandsocialstructures,producea harmfuloutcome,evenintheabsenceofanyintentiontoharm. SupposethatBobbelievesthatscriptureSconsistsofexclusivelyliteraltruths,andthatheiscommitted todoingwhateverSsaysoughttobedone.Suppose,furthermore,thatScontainsthe(presumably false)moralstatementThoushaltdrinkseawater,butthatBobisunawareofthis.Thepotential disseminationtoBobofthispartofthecontentofSconstitutesaninformationhazard.Theinformation couldharmBobbyinducinghimtodrinkseawater.(Notethattheconveyanceoftrueinformation harmsBobhere;inthiscase,thetrueinformationthatScallsfordrinkingseawater.) Intheprecedingexample,thehazardposedbytherelevantinformationistightlycoupledtoBobs idiosyncraticbeliefsystem.Itistruethattheideaofanuclearbombisalsoahazardonlywhencoupled withalargerbeliefsystemforinstance,beliefsaboutphysicsandtechnologyrequiredtobringabomb intoexistence.Yetitseemspossibleandusefultodistinguishthiskindofinstrumentalinformation hazardfromideologicalinformationhazard.Ideologicalhazard,wemightsaybywayofexplication, referstothepossibilitythatthatsomebodywillbemisleadtoheadinsomebaddirectionbecauseofthe waythatsomeinformationinteractswithfalsebeliefsorincompleteknowledge. Thatbadideologiescanbeextremelydangerousisamplyevidencedbytwentiethcenturyhistory.What islessclearishowideologicalhazardcanbestbereduced.Partofthereasonwhythisisadifficult problemisthatideologycanalsobeaforceforgood.TheideologyoftheAmericancivilrights movement,forexample,helpedpushbackracialdiscriminationintheU.S.Inawidesense,ideologyis perhapsaninevitablepartofthehumancondition,andtheproblemofdistinguishinggoodfrombad ideologymaybenoeasiertosolvethantheproblemofdistinguishinggoodfrombadpolicy:nosimple, generallyacceptablealgorithmexists.Moreover,whileradicalideologiesmaybeespeciallydangerous, theymayalsodependingonwhatthestatusquoisrelativetowhichthealternativestheypresentare radicalbeespeciallyappropriateforthesituation.Ifthestatusquoisslaveryandreligious prosecution,thenitwouldbearadicalideologythatproposesnotmerelyameliorationoftheworking conditionsforslavesandreductionofthepenaltiesforheresy,butcompleteabolitionandunlimited religiousfreedom. Nextweturntothefactthathumanbeingsarenotperfectlyrationalnordowehaveperfectself control.Wecanbedistractedagainstourwillandwecansuccumbtotemptationagainstourbetter

14

judgment.Exposuretoinformationcanhaveeffectsonusotherthansimplyimprovingtheaccuracyof ourrepresentationsoftheworld. Someinformationisdistracting.Itinvoluntarilydrawsourattentiontosomeideaorthemewhenwe wouldprefertofocusourmindselsewhere.Anadvertizingjinglemightloopinourmindsanddistract usfromsomethingwewouldratherbethinkingabout.Onetechniqueweusetofighttemptationisto putsomethingoutofourmind;yetinformationaboutthetemptingobjectcanundermineoureffort andmakeusmorelikelytocave.Arecoveringalcoholiccanbeharmedbyexposuretoavividaccount oftheattributesofChateauPetrusPomerol1990. Distractionandtemptationhazards:Informationcanharmusbydistractingusorpresentingus withtemptation. Inmostindividualcasesthedamagedonebydistractingortemptinginformationisperhapsminor.Yet itisnotunreasonabletowonderwhetherthereadyavailabilityofcertainkindsofinformationmight potentiallycausedamageonawiderscale.Perhapsitcouldbearguedthattelevisionhasanaggregate effectonthecontemporaryhumanconditionnottoodissimilarfromthatwhichwouldbeproducedby thewidespreadrecreationaluseofopiatedrugs.Inthefuture,evenmorecompellinglypresented informationandhyperstimulimightbecomeavailableandproveenormouslyaddictive;forexample, newformsofhighlyimmersiveorinteractivevirtualrealityenvironments.Druglikeeffectsonour psychescanbeproducednotonlythroughinjection,ingestion,andinhalationbutalsothroughthe intakeofinformationpresentedincertainmannerstooursenses. Wecanalsobeharmedbyexposureto(thetemplatehazardof)badrolemodels.Evenwhenweknow thatamodelisbad,andwewouldprefernottobeinfluencedbyit,prolongedexposurecan neverthelessbedetrimentalbecauseofakindofsocialosmosis.Someonewhoaspirestoagood writingstylemightbewelladvisedtoavoidreadingtoomuchtrash.Onewhoseekstocultivatealofty sentimentmightwanttoavoidthecompanyofthemeanandpetty.Andthosewhohopethattheir childrenwillbecomeuprightcitizensshouldkeepthemawayfromdelinquentpeers.25 Recentstudies indicatethatsubjectivewellbeingandevenbodymassaresignificantlyinfluencedbyourassociates. 26 Thus, Rolemodelhazard:Wecanbecorruptedanddeformedbyexposuretobadrolemodels. OneexampleofthisistheWerthereffect,namedafterthewaveofsuicidesamongyoungmenwhich sweptEuropeafterthepublicationin1774ofGoethesnovelDieLeidendesjungenWerthers.Several
25

Otherthingsbeingequal,thatis;whichofcoursetheyseldomare.Whendecidingwhattodo,oneshouldalso takeintoaccountthatexposuretoawiderangeofrolemodelscouldprovidemoreopportunitiesforchoice;and thatonecanbecomewiserbyalsoknowingsomethingaboutthedarkside.Whenexcessive,thefearof contaminationbybadinfluencesisstultifying.Initsextremeforms,aloveofpuritycanproducedangerous intoleranceandbigotry. HallidayandKwak2007.

26

15

studieshavecorroboratedtheexistenceofsuchaneffect,findingalinkbetweenmediareportingof highprofilecasesandensuingcopycatsuicides. 27 Informationrisksariseoutofoursusceptibilitytovariouscognitivebiasesthatcanbeaggravatedbythe provisionofcertainkindsofdata.Anchoringbiasresultsfromapplicationoftheanchoringand adjustmentheuristicinwhichpeopleestimatesomeunknownquantitybyfirstanchoringonsome figurethathappenstocometomindandthenadjustingthispreliminaryestimateeitherupordownin anattempttoreflecttheirtotalinformation.Thisleadstobiaswhenpeopleinitiallyanchoronan irrelevantquantityandthenunderadjustintheadjustmentphase.Inonestudysubjectswereaskedto estimatethenumberofcountriesinAfrica.Beforeproducingtheirestimate,awheeloffortunewas spun.Subjectswhoobservedalargernumberonthewheeltendedtogiveahigherestimateofthe numberofAfricancountries,despitethetransparentirrelevanceoftheformerfact.Theextrapieceof trueinformationaboutthenumberonthefortunewheeldiminishedtheaccuracyofgeographical judgment. 28 Manypeopleoverestimatetheirownvirtuesandabilities.Supposesuchapersonreceivessome additionalweakcueoftheirsupposedexcellence,suchasagoodscoreonatriviaquiz.Thisbitof evidence,whichwecansupposetobetrueandinaverylimitedwayinformative,couldaggravatetheir selfoverestimationandconceitedness. 29 Evenknowledgeofhumanbiasesandcriticalphilosophycanleadtheunwarydeeperintoerror,and reducehisabilitytolearn,byarminghimwithcleverargumentswithwhichtorebutobjectionsand rationalizeinconvenientfacts. 30 Aspecialkindoffoolisbornwhenintelligencethusoutwitsitself. Biasinghazard:Whenwearebiased,wecanbeledfurtherawayfromthetruthbyexposureto informationthattriggersoramplifiesourbiases. Methodology,suchasdoubleblindingindrugtrials,canhelpreducetheriskofbiasesentering uninvitedintoourthinkingandacting.Forsimilarprecautionaryreasons,thegullibleoftenhavereason toavoidthehighlypersuasive.Andifoneplanstoexperiencetransportsandecstasiesthatwill temporarilyincreaseonessusceptibilitytodangerousillusionsandimpulses,oneshouldfirsthave oneselftiedtothemast. Conversely,informationcouldalsoharmusbyreducingourbiasesinsofarasourbiasesservesome usefulpurpose.Forexample,atendencytooverestimateourownabilitiesmightnotonlymakeusfeel happierandconfident;astrongbeliefinourownabilitymightalsosignalcompetenceandleadothersto
27

Seee.g.Phillips1982;Stack1996;Jonas1992. TverskyandKahneman1974.

28

29

Dittoforthosewhounderestimatetheirownvirtuesandabilities:feedbackthatconfirmsthistendstobe internalizedwhilefeedbackthatcontradictsittendstobeignored(Brown,Duttonetal.2007). Yudkowsky2008.

30

16

allywithus,promoteus,orvoteforus.Informationthathelpsusseeourselvesforwhatwereallyare coulddepriveusofthesebenefits.Itisalsopossiblethatsocietybenefitsfromexcessindividualrisk takinginsomedisciplines;forexampleifentrepreneurs,inventors,andyoungacademicsoverestimate theirownchancesofsuccess.Iftheseoccupationshavenetpositiveexternalities,itcouldbebeneficial thatbiasesandunrealisticexpectationsoffame,fortune,orhighachievementseduceadditional entrantsintothesefields. Debiasinghazard:Whenourbiaseshaveindividualorsocialbenefits,harmcouldresultfrom informationthaterodesthesebiases. Thereisalsoawiderphenomenonofwhichrolemodelinfluenceisbutaspecialcase.Ourbrainsare constantlyreshapedbywhatwelearnandexperience.Informationgleanedisnotsimplystoredaway asinertdatapackages,asthoughitwerenewvolumessuperaddedtosomeinternalbookshelf.Rather, theincominginformationinteractscomplexlywithpreexistingcognitivestructuresinwaysthatarenot alwayseasytocharacterizeinfolkpsychologicalterms.Newconceptsmightform;boundariesofextant conceptsmightchange;neuronalwiringpatternsarealtered;somecorticalareasmightexpand,causing otherareastocontract;andsoforth.Thereisariskthatsomeofthesechangeswillbefortheworse. Neuropsychologicalhazard:Informationmighthavenegativeeffectsonourpsychesbecauseof theparticularwaysinwhichourbrainsarestructured,effectsthatwouldnotariseinmore idealizedcognitivearchitectures. Toomuchknowledgecanbebadforsometypesofmemory.31 Perhapssomementalillnessesresult frominappropriatecrosstalkbetweencognitivemodulesdesignedtooperateasmoreencapsulated unitsakindofundesirableinternalinformationdissemination.Arecurringideainliteratureand mythologyisthemotifofharmfulsensation,whereapersonsuffersmentalorphysicalharmmerely byexperiencingwhatshouldnormallybeabenignsensation(themythofMedusa,beliefsaboutthe evileyeetc.).Arealworldexampleofharmfulsensationisphotosensitiveepilepsywhichcanbe triggeredinsomesensitiveindividualsflickeringlightsorspecificgeometricpatterns. 32 Irrelevantinformationcanmakevaluableinformationhardertofind.Thisfactisusedinsteganography, thecryptographictechniqueofhidingsecretmessageswithinrepresentationsthatappeartobeof somethingelsesothateventheexistenceofcoverttextisconcealed.Forexample,someofthepixelsin animagefilecanbesubtlymodifiedsoastoencodeaverbalmessageinwhatlookslikeanordinary touristpicture.Inasimilarvein,legaldefenseteamssometimesconcealincriminatingdocumentation
31

RobinsonandSloutsky2007. WhenthecartoonepisodeDennSenshiPorygonairedinJapanin1997,onescenefeaturinganexplosion renderedwithstrobelightingeffectcausedmildsymptomsin510%oftheviewers(thoughsomeofthesemight insteadhavebeenafflictedwithepidemichysteria)andsent685childrentohospitalinambulance.Nolongterm damagewasreported.Therehasalsobeenatleastonemaliciousattempttodeliberatelytriggerphotosensitive epilepsyonline,butitappearsnottohavebeenverysuccessful.SeeRadfordandBartholemew2001;Takada,Aso etal.1999.

32

17

thathasbeensubpoenaedbytheprosecutionbyoverwhelmingitwithsuchmassiveamountsof archivalmaterialthattherelevantdocumentscannotbelocatedintimeforthetrial. Informationburyinghazard:Irrelevantinformationcanmakerelevantinformationharderto find,therebyincreasingsearchcostsforagentswithlimitedcomputationalresources. 33 Onagranderscale,anoverabundanceofinformationalaffordancesmightdeflectourthinkingfrom topicsthataremorecentraltousandrelativelymoreworthyourcontemplation,sothatweshalllive,as inT.S.Eliotscharacterizationofthemodernpredicament,Distractedfromdistractionby distraction. 34 Thiskindofpossibilityleadsustothenextsection. 6.Riskstovaluablestatesandactivities Wehavelookedathowinformationcancauseharmbyaffectingbehavior.Informationcanalsoharm throughitsdirectpsychologicaleffects,forexamplebycausingdisappointment.Moreover,accordingto atleastsomeaccountsofwellbeing,informationcancauseharmevenasidefrompsychological spillovereffectsbyaffectingsomepartofsomeepistemicorattentionalstatethatplaysaconstitutive roleinsomesubjectswellbeing.Thuswecandefine Psychologicalreactionhazard:Informationcanreducewellbeingbycausingsadness, disappointment,orsomeotherpsychologicaleffectinthereceiver. Andwecandistinguishthisfromthefollowingmorephilosophicallyintricatenotion: Beliefconstitutedvaluehazard:Ifsomecomponentofwellbeingdependsconstitutivelyon epistemicorattentionalstates,theninformationthataltersthosestatesmighttherebydirectly impactwellbeing. Considerfirsttheobviousexampleofapsychologicalreactionhazard:badnews,thereceiptofwhich makesussad. Disappointmenthazard:Ouremotionalwellbeingcanbeadverselyaffectedbythereceiptof badnews. Insomecases,ifsomethinggoeswrong,weareboundtohearofiteventually.Insuchcases,the disappointmentisinasensealreadycommittedwhentheadverseeventtakesplace,eventhoughit mighttakeawhilebeforetheeffectisknownandfelt.

33

Andpotentiallyresultinworsesolutions;foradiscussionofhowexcessiveknowledgecandegradeperformance insomeartificialintelligencesystems,seeMarkovitchandScott1988. Eliot2001.

34

18

Inothercases,however,thereisarealchancethatifasubjectavoidshearingofhermisfortunenow, shewillremainignorantandwillbesparedthedisappointmentthatthebadnewswouldoccasion.Such casesmakeiteasiertodisentanglethedisappointmenthazardfromotherpossibleharmsthatmightbe involved.Takethecaseofamotheronherdeathbed,whoseonlysonisfightinginsomefarawaywar. Themotherfacesatleasttwodistinctrisks.First,thereistheriskthathersonwillbekilledorinjured; thisisnotnecessarilyaninformationrisk.Supposethatthesonisinfactkilled.Thenthereisasecond risk,whichisthatthemotherwillfindoutaboutherloss.Supposethatthenewsiscontainedinaletter, whichmightreachherquicklyoritmightbedelayed.Ifitreachesherquickly,shewillspendherlast daysinagonyanddespair;ifitisdelayed,shewilldieinpeace.Herewemightsaythatthemotheris exposedtoaseveredisappointmenthazard. Spoilersconstituteaspecialkindofdisappointment.Manyformsofentertainmentdependonthe marshallingofignorance.Hideandseekwouldbelessfuniftherewerenowaytohideandnoneedto seek.Forsome,knowingthedayandthehouroftheirdeathlonginadvancemightcastshadowover theirexistence. Beforehisretirement,myfatherwouldsometimesmissapivotaltelevisedsoccergamethattookplace duringworkinghours.Planningtowatchtherepriselater,hewouldmeticulouslyavoidanynewssource thatmightdisclosetheresults.Hisdesign,however,wasthwartedbymygrandfather,whohad watchedthegameliveandwhoinvariablyfoundhimselfunabletorefrainfrommakingnotquite subtleenoughallusionstothematch,enablingmyfathertoguesswhohadwon. Spoilerhazard:Funthatdependsonignoranceandsuspenseisatriskofbeingdestroyedby prematuredisclosureoftruth. Knowledgecanalsoexertmoregeneraleffectsonourpsychesandpersonalities.Perhapsanunwanted cynicismispromotedbyanexcessofknowledgeaboutthedarksideofhumanaffairsandmotivations. Nietzschewarnedofthemisusesofhistory:howhistoricalknowledge,approachedandvaluedina certainway,cansapourzestforlifeandinhibitartisticandculturalauthenticityandinnovation.The dangerNietzschepointedtowasnottheeffectsofanyoneparticularpieceofinformationbutrather theconsequencesofacertainexcessofhistorywhichcancauselivingtocrumbleaway:esgibteinen Grad,Historiezutreiben,undeineSchtzungderselben,beiderdasLebenverkmmertundentartet (thereisadegreeofdoinghistoryandvaluingofitthroughwhichlifeatrophiesanddegenerates).35 If Nietzscheisrightaboutthis,andifthedisseminationof(variouskindsof)informationaboutthepast can,underunfavorablecircumstances,contributetosuchanatrophyofspirit,thenwehaveherean exampleofanothertypeofpsychologicalreactionhazard,namely Mindsethazard:Ourbasicattitudeormindsetmightchangeinundesirablewaysasa consequenceofexposuretoinformationofcertainkinds.

35

GermanquotationtakenfromNietzsche1984;EnglishtranslationtakenfromNietzsche2007.

19

Alongsimilarlines,somepeopleworrythatscientificreductionism,akintostripmininginanancient forest,despoilslifeofitsmysteryandwonder. Letusturntobeliefconstitutedvaluehazard.Inpractice,thiscategorycanbedifficulttodistinguish frompsychologicalreactionhazard. Consideragaintheexampleofthemotheronherdeathbedwhoriskshearingthathersonhasbeen killed.Thereisclearlyonerespectinwhichhearingthiswouldbebadforher:itwouldcauseher extremepsychologicaldistress.Thisissufficientfortheretobeapsychologicalreactionhazard.Itdoes notrequirethatitwouldbebadforthemother,allthingsconsidered,tohearofhersonsdeath. Thereareseveralreasonsforthis.First,thereareofcoursevariouspracticalmattersthatwouldneedto beconsideredinanactualsituationlikethis:Perhapsthemotherneedstoknowsothatshecanalter herwill?Perhapsconcealmentofunpleasantnewswouldtendtoerodesocialtrust?Butevenaside fromsuchpragmaticconsiderations,thereisasecondtypeofreasonwhyitmightbebetterforthe mothertoknowofhersonsdeathdespitethesufferingthisknowledgewouldcauseher.Such knowledge,accordingtosomemoraltheories,canbeacomponentofapersonswellbeing(thegood foraperson)evenifitaffectsthesubjectivecomponentofwellbeingfortheworse.Onemighthold thatalifeismadeworse,otherthingsequal,byitsbeinglivedinignoranceofimportantfactsaboutthe centralconcernsofthatlife.Lifeinafoolsparadise,evenifitscoreshighonthehedonicdimension, mightneverthelessscorequitelowinoverallwellbeingonsuchatheory. Justasonemightholdthatthereissomeknowledgethepossessionofwhichisanimportantconstituent ofagoodlife,onemightalsoholdthatthereisknowledge(atleastforsomepeople,insome circumstances)thatmakesadirectnegativecontributiontotheirwellbeing.Thiscanmostobviouslybe seentobethecaseaccordingtoapreferencesatisfactionaccountofwellbeing;fortherewecan generateexamplestriviallysimplybysupposingsomebodytohaveabasicpreferenceagainstknowing aboutsomeparticularsubjectmatter.Butmanyotheraccountsofwellbeingmightalsopermitof examplesofsuchdirectlyburdensomeknowledge.Innocencemightbevaluedforitsownsakeand mightberuinedbyknowledge.Wemighttreasureourprivacyandfinditinfringedbyotherpeoples knowingthingsaboutusthatwewouldratherhavekepttoourselvesorsharedexclusivelywithchosen intimates.Orwemightbebetteroffnotknowingsomepersonaldetailsaboutothers,notjustbecause suchknowledgemightexposeustoriskofworsetreatmentfromothers(asinknowingtoomuch hazard)butalsobecauseourawarenessofthesedetailswouldstandinthewayofourconceivingof othersinmannersthataremoreappropriateormoretoourliking.Withregardtoourfriendsbowels andourparentsbedrooms,thelessweknowthebetter.36 Onecommonlyfearedriskfrominformationis Embarrassmenthazard:Wemaysufferpsychologicaldistressorreputationaldamageasaresult ofembarrassingfactsaboutourselvesbeingdisclosed.
36

AndofcourseBismarckclaimedoflawsandsausagesthatitisbetternottoseethembeingmade.

20

Embarrassmenthazards(whichoftentaketheformofsignalinghazard)commonlycombineelementsof psychologicalreactionhazard,beliefconstitutedvaluehazard,andcompetitivenesshazard.Wemay evenfeartoembarrassourselvestoourselves,perhapsbecauseselfesteemisnotawhollyprivate matterbutisalsoasocialsignalthatinfluencesothersopinionsofus. 37 Somepsychologistsbelievethat aconcerntoprotectselfesteemfromunderminingbyselfrelevantfailurescanleadindividualsto engageinselfhandicappingbehavior. 38 Thiscouldhelpaccountforsomeinstancesofphenomenasuch asprocrastination,hypochondria,substanceabuse,andpracticeavoidance. 39 Supposethatthinkingof yourselfasintelligentisimportantforyourselfesteemandthatyouhaveanimportantexamcoming up.Ifyoupracticehardandfailontheexam,yoursenseofselfcompetencewilltakeahit.Butifyou putoffpracticinguntilthenightbeforetheexam,yourriskisreduced;evensmartpeoplecandopoorly onexamswhentheyhavenotstudiedenough.Andifdespitethehandicapofinsufficientpreparation youstillmanagetogetahighmark,whythenyoumustbetrulybrilliant.Suchperceptionmanagement canimposesignificantcosts. Riskofembarrassmentcansuppressfrankdiscussion.AstudyondeliberationintheFederalReserves FederalOpenMarketCommitteefoundevidencethatanewlyadoptedpolicyoftransparencyinvolving thepublicationofdetailedtranscriptsfrommonetarypolicymeetingsstifledthevoicingofdissenting opinionsandseemedtoreducethequalityofdebate. 40 Intangibleassets,suchasreputationandbrandname,constitutealargepartofthevalueofmanyfirms. Embarrassmentsthatnegativelyimpacttheseassetscancausebillionsofdollarsindamage.Foran exampleonanevengranderscale,considertheColdWarsuperpowerrivalry,inwhichbothcontenders wereengagedinstatuscontestaswellasamilitaryarmsrace.TheApolloprojectwasadirectresponse totheembarrassmentcausedtotheUnitedStatesbytheSovietUnionslaunchofSputnik1,an accomplishmentthatchallengedtheAmericasclaimtotechnologicalsuperiority.TheVietnamandthe Afghanwarswerebothprolongedbecauseofreluctancetosufferthereputationaldamagethatleaders believedwouldresultfromadmittingdefeat. 7.Risksfrominformationtechnologysystems Itisnotonlyanimatebeingsthatprocessanddisseminateinformation;ourinformationtechnological systemsdosoaswell.Distinctiveinformationhazardsariseinrelationtoourcomputersandnetworks. Informationtechnologysystemsarevulnerabletounintentionallydisruptiveinputsequencesorsystem interactionsaswellastoattacksbydeterminedhackers.Hereweconsideronlyriskoccasionedby
Hobden1997. BerglasandJones1978. Smith,SnyderandPerkins1983;Stone2002;ThompsonandRichardson2001. MeadeandStasavage2008.

37

38

39

40

21

informationaleffectsunanticipatedsysteminteractions,worms,viruses,Trojanhorses,denialof serviceattacks,andsoforth.Thismeansweexcluderisksarisingfromthepossibilityofflooding,power outages,andsomebodyattackingyourcomputerwithasledgehammerexceptinsofarasarisk consistsinthepossibilityofinformationalamplificationoftheeffectsofsomesuchnoninformational trauma.Thus,theriskthatyoumightdropyourbrandnewlaptoponahardfloorsothatitbreaksand youincurthecostofbuyingareplacementisnotaninformationhazard.Noristheriskthatsome criticalinformationsystemmightgodownnecessarilyaninformationhazardasdefinedhere.Themere cessationoffunctioningofsomeusefulinformationprovidingsystemisnotenoughunlessthecauseof thecessationisinformationalortheharmarisesfromsomekindofundesirablepropagationof information. Informationsystemhazard:Thebehaviorofsome(nonhuman)informationsystemcanbe adverselyaffectedbysomeinformationalinputsorsysteminteractions. Thiscategorycanbesubdividedinvariousways:onecould,forexample,distinguishcomputerhazards fromnetworkhazards;orsingleoutthreatstocriticalinformationinfrastructure;oronecouldmakea separationbetweenscenariosinvolvinglossofdata,corruptionofdata,disseminationofdatatothe wrongparties;andsoforth.Quitealotofattentionisalreadygiventoinformationsystemhazards,and muchofthisattentionisfocusedonwhatwemayterm Informationinfrastructurefailurehazard:Thereisariskthatsomeinformationsystemwill malfunction,eitheraccidentallyorasresultofcyberattack;andasaconsequence,theowners orusersofthesystemmaybeinconvenienced,orthirdpartieswhosewelfaredependsonthe systemmaybeharmed,orthemalfunctionmightpropagatethroughsomedependentnetwork, causingawiderdisturbance. Risksofthistypecanbequiteseverewhensomecomplexsystemornetworkisusedtocoordinate importanthumanactivities.Forinstance,acorruptionofthesoftwarethatundergirdsimportant financialsystemscouldhaveseriousconsequences. Adifferenttypeofinformationsystemhazardisthatsomeinformationsystemwillinfactfunctionas intended,butbydoingsoitwillcauseharmoramplifysomeriskofharm. Aprivacyadvocatemightobjecttosomegovernmentdatabaseprojectthatwillamassvastquantitiesof informationaboutthecitizenry,notonlybecauseoftheriskthatthesystemmightmalfunctionorbe hacked,butalsobecauseoftheriskthatitwillperformtospecificationandtherebystrengthenthe statesabilitytomonitortheactivitiesofitspeopleandshouldthegovernmentonedayseeaneedto dosototakeactionagainstelementsdeemedundesirableordisloyal.Evenifitwereadmittedthat thegovernmentthatbuildsthesystemcanbetrustedtouseitonlyforgood,onemightfearthatlater governmentswhichinheritthesystemcannotbesotrusted,orthatsomemoreperniciousgovernment elsewherewillseeinthesystemaninspiringprecedent(cf.,ideahazard,andattentionhazard)or justification(cf.,normhazard)forbuildingitsowncomparablesystemandapplyingittoitsown nefariousends.

22

Similarconcernscanapplytoprivatefirms,suchasGoogle,thatcollectpersonalinformationabout hundredsofmillionsofusers.ConsiderhowusefulitwouldhavebeenforatwentyfirstcenturyStalin tobeabletohavehissecurityservicedataminethecitizenrysemailcorrespondenceandsearchengine queriesnotleasttextwrittenbeforehisascenttopowerandatatimewhenhisenemiesmighthave communicatedtheirmostincriminatingthoughtsunguardedly. 41 Informationinfrastructuremisusehazard:Thereisariskthatsomeinformationsystem,while functioningaccordingtospecifications,willservicesomeharmfulpurposeandwillfacilitatethe achievementofsaidpurposebyprovidingusefulinformationinfrastructure. Asystemcanalsobedangerousbypresentinganeasyopportunityforunintentionalmisuse.Considera poorlydesignedemailprogramthatmakesistooeasyfortheunwaryuseraccidentallytoforwardan emailreplytoalltheaddresseesinhercontactlist;anembarrassmentwaitingtoblush.Thishazardlies ontheborderbetweeninformationinfrastructurefailureandinformationinfrastructuremisuse,itbeing unclearwhethersuchanemailprogramisfunctioningaccordingtoitsintendedspecificationsand arguablehowapportionblamebetweenthesystemsdesignersanditsusers. Forcomparison,wemayalsonotetwoothertypesofhazardpotentiallyarisingoutofinformation technology(butwhicharetypicallynotinformationsystemhazards)wheretheharmisnotsomucha consequenceofthegeneralinformationinfrastructureservicesthatasystemprovidesorfailstoprovide butinsteadismoredirectlyrelatedtotheagencyofthesystemitself: Robothazard:Therearerisksthatderivesubstantiallyfromthephysicalcapabilitiesofarobotic system. Anautonomousvehicle,loadedwithexplosivemissiles,andabletolaunchonitsowninitiative,could constitutearobothazard.Wecancontrastthiswith Artificialintelligencehazard:Therecouldbecomputerrelatedrisksinwhichthethreatwould deriveprimarilyfromthecognitivesophisticationoftheprogramratherthanthespecific propertiesofanyactuatorstowhichthesysteminitiallyhasaccess. Anartificialintelligencewouldneedtobeveryadvancedinordertoposeanysignificantthreatinvirtue ofitsowningenuityandagency.Thecreationofartificialgeneralintelligence,withgeneralpowersof reasoningexceedingthoseofhumanbeings,wouldbeassociatedwithaserious,indeedexistential, risk. 42 Asuperintelligence,evenifinitiallyrestrictedtointeractingwithhumangatekeepersviaatext interface,mighthackortalkitswayoutofitsconfinement.Itcouldthengaincontrolovereffectorsto conductoperationsintheexternalworldforexample,byusingitspowersofpersuasiontogethuman beingstodoitsbiddings,orbyassumingcontrolofroboticmanipulators.Itcouldusetheseeffectorsto
41

Ofcoursetherearebigpotentialupsidestoo;e.g.,agoodgovernmentcouldsubpoenathisinformationforuse inagoodcause. Bostrom2002;Yudkowsky2008.

42

23

developnewtechnologiesandtosecureamorecomprehensivegraspofitsphysicalsurroundings.The threatposedbyasufficientlyadvancedartificialintelligencemaydependmuchmoreonitscognitive capabilitiesanditsgoalarchitecturethanonthephysicalcapabilitieswithwhichitisinitiallyendowed. Notallrisksrelatedtorobotsorartificialintelligencesaretobeclassifiedasinformationsystem hazards.Ariskwouldcountassuchahazardif,forexample,itarosefromthepossibilityofacomputer virusinfectingtheoperatingsystemforarobotoranAI.Butasidefromsuchspecialcases,weshallnot countrobothazardsandartificialintelligencehazardsasinformationsystemhazards. 43 Thereis,however,anotherwayforrobotandAIrelatedriskstoenterourinformationhazard taxonomy.Theycanenteritinthesamewaysasanyriskrelatingtopotentiallydangeroustechnological development. 8.Risksfromdevelopment Bothtechnologicalinnovationandeconomicdevelopmentmorebroadlyarisefromtheaccumulationof information,ideas,andinsights;andthiscanresultinarangeofrisksthatwecangrouptogetherunder therubricofdevelopmenthazards. Developmenthazard:Progressinsomefieldofknowledgecanleadtoenhancedtechnological, organizational,oreconomiccapabilities,whichcanproducenegativeconsequences (independentlyofanyparticularextantcompetitivecontext). WhenthemushroomcloudsroseoverHiroshimaandNagasaki,physicists,manyofwhomhadentered theirprofessionforthesheerjoyofdiscovery,foundthemselvescomplicitinthedeathsofperhaps 200,000people. 44 IfthecoldwarhadendedinanalloutnuclearexchangebetweenNATOandthe SovietUnion,asitmighteasilyhavedone,thenmorethanabillioncivilianscouldhavediedasafairly directconsequenceofthedevelopmentofnuclearweapons. 45
43

Thereis,ofcourse,asenseinwhichbothrobotsandadvancedmachineintelligencesareinformationsystems. Thereisalsoasenseinwhichthehumanbrainisaninformationsystem.Yettherisksthatarisefromintelligence ingeneral,orfromthephysicalequipmentofsomerobot,areextremelyheterogeneous;whereforeitwouldseem notveryilluminatingtoconstructaninformationsystemhazardcategorythatlumpedthemalltogether. TheAtomicArchiveestimatesthedeathsinHiroshimaandNagasakiimmediatelyfollowingthebombingsat 105,000,withafurther94,000injured(TheManhattanEngineerDistrict1946).Manyhavelaterdiedofcanceror birthdefectscausedbyradiationexposure,buttheexactfiguresareasubjectofdebate.

44

45

PresidentKennedyissaidtohaveatonepointduringtheCubanmissilecrisisestimatedtheprobabilityofa nuclearwarbetweentheU.S.andtheUSSRtobesomewherebetweenoneoutofthreeandevenKennedy 1968;Leslie1996.JohnvonNeumann,whoaschairmanoftheAirForceStrategicMissilesEvaluationCommittee wasoneofthearchitectsofearlyU.S.nuclearstrategy,isreportedtohavesaiditwasabsolutelycertain(1)that therewouldbeanuclearwar;and(2)thateveryonewoulddieinit(Putnam1979,114).SeealsoCirincione2008.

24

RobertOppenheimer,thescientistwhohadspearheadedtheManhattanproject,acknowledged afterwardsthatthephysicistshaveknownsin;andthisisaknowledgewhichtheycannotlose. 46 Of course,reachingamoralverdictonthescientistswhoworkedontheManhattanprojectisnotassimple astotingupthenumberofdeathsthatwerelatercausedbytheweapontheyinvented.Manyofthese scientistsdevotedthemselvestotheprojectbecausetheyfearedthatHitlermightgetthebombfirsta fearwhich,althoughitturnedouttobeunfounded,wasreasonablegiventheinformationavailable whentheprojectbegan.RichardFeynman,anotherphysicistwholaterreflectedonhisinvolvement, regardedhisinitialdecisiontoparticipateasmorallyjustifiedforjustthisreason;buthethoughtthathe hadfailedmorallyinnotreconsideringhisinvolvementafteritbecameclearthatHitlerhadbeenunable toacquirethebombandthatGermanycouldbedefeatedwithoutit.Furthermore,thedecisiontouse thebombwasnottakenbyphysicistsbutbyPresidentTruman,whomayhaveactedonavarietyof motivesinacomplexstrategicsituation;andsoforth. 47 Thepointhereisnotthatsomeparticularpastactionwasorwasnotjustified,butthatthiskindof consequencecanresultfromtheinformationgatheringworkofphysicistsincludingbasicresearch suchastheearlierworkinquantumandparticlephysicsthatlaidthetheoreticalfoundationsforthe Manhattanproject.Toproceedblithelyandwithoutscruple,asthoughnothingverybadcouldcome fromsuchresearch,wasperhapsexcusablenavetbackthen. 48 Forourowngeneration,whichisable toobservemorehistoricalprecedent,suchnegligencewouldmorelikelyamounttoculpableabrogation ofmoralresponsibility. WhatwastrueofphysicsinthedecadesleadinguptotheSecondWorldWarmaybetrueofother academicdisciplinestoday.Biologyandbiotechnology,whileprovidingurgentlyneededmunitionsfor useinourbattleagainstdisease,malnourishment,andagerelateddebility,alsothreatentoarmthe humanspecieswithweaponsofmassdestructionthatmightbedeployedagainstourownkind. Recentdevelopmentspointtodisturbingpossibilitiesdowntheroad.Considerthesteadilyimproving capacityandavailabilityofDNAsynthesismachines.Thistrendisworrisomewhenoneconsidersthat thegenomesofmanyextremelydangerouspathogensresideinthepublicdomain,includingEbola, Marburg,smallpox,andtheSpanishfluvirus(believedtohavekilledmorethan50millionpeopleduring 19181919).Theknowledgeandtechnologyrequiredtogeneticallymodifymicroorganismssoasto enhancetheirpathogenicityandtheirresistancetocountermeasuresarealsoadvancing.Technological barrierstotheproductionofsuperbugsarebeingsteadilyloweredwhilebiotechnologicalknowhow andequipmentdiffuseevermorewidely. 49
46

Oppenheimer1947. Foroneattemptatamoralassessment,seeGlover2001.

47

48

AlthoughLeoSzilardsexamplesuggeststhatmuchofthisnavetwasavoidableatleastasearlyas1933. Rhodes1995.

49

Seee.g.NouriandChyba2008.Ofcourse,therearealsoriskmitigatingbenefitsfromsuchresearch,for examplebetterprophylacticsandtherapeutics,andbetterknowledgeofourownvulnerabilities.

25

Dangerousinformationcouldalsoarisefromotherfieldsofinquiry.Advancedfutureformsofmolecular nanotechnologymightbeusedtobuildweaponssystemevenmorepowerfulthanhydrogenbombsand supergerms. 50 Artificialintelligencemightonedaysurpassbiologicalintelligenceandtherebybecome extremelypowerful. 51 Technologiesformonitoringandmodifyinghumanbehaviormightadvanceonseveralfrontssuchas ubiquitoussurveillancesystems,automatedfaceandvoicerecognitionsoftware,effectiveliedetection, psychopharmacology,geneticengineering,orneuralimplants.Socialsciencemightmakeprogresson understandingandpredictingthetriggersofsocialunrestandinsurrection.Suchcapabilitiescouldbe usedforgoodorill.Inaworstcasescenariotheycouldfacilitatetheemergenceofnewandpermanent formsoftotalitarianisms,possiblyonaglobalscale. Thepossibilitiesreferredtoaboveconstitutesomeofthemostsignificantexistentialrisksthatmay confronthumanityinthefuture. 52 Otherpotentialtechnologicaldevelopmentssomeforeseeable, othersperhapsnotmayalsocreateexistentialrisks.Becauseoftheextremevaluesatstakein existentialrisks,theycandeservesubstantialconcerneveniftheycouldbeshowntobebothvery unlikelyandveryremoteneitherofwhichisclearlythecasefortherisksjustmentioned. 53 Thesetechnoscientificareasdonotfunctioninisolation.Bioweaponsengineerswoulddrawondata andtechniquesdevelopedbyawiderangeofresearchersinfieldssuchasvirology,medicine,genetics, andbiochemistry.Nanotechnologistsdrawondrawonfieldssuchasmaterialsscience,chemistry, proteinengineering,biotechnology,andsystemsengineering.Artificialintelligencepioneersmay benefitfromadvancesinneuroscience,cognitivescience,computerscience,foundationsofprobability theory,andsemiconductorphysics.Furthermore,alloftheseareasareinfluencedtosomeextentby generaleconomicgrowth,whichtendstoleadtoincreasedfundingforresearch,bettersupporting infrastructure,andamoreeducatedworkforce. Developmenthazardsthusariseinmanyareas,andtheyrangeinseverityfromtrivialtoexistential.Itis importanttorecognizethatdevelopmenthazardsarenotconfinedtoespeciallysinisteror Prometheantechnologicalbreakthroughs.Evensomethingasinnocentasmedicaloragricultural advancesthathelpreduceinfantmortalitycanposesignificantdevelopmenthazards,suchastheriskof overpopulationandpotentiallynegativeknockoneffectsonconflict,percapitaincome,andthe environment.(Obviously,thefactthatsomepotentialdevelopmentisassociatedwithsomeriskdoes
50

Drexler1987;Freitas2000;Gubrud1997.

51

Moravec2000;Bostrom1998;Vinge1993;Kurzweil2006;BostromandSandberg2008.Aselfenhancing generalintelligencethatbecamesuperintelligentwouldbecomeextremelypowerfulandwould,unlessrightly designed,constituteaseverethreattohumanity.Bostrom2003;Yudkowsky2008. AnexistentialriskisonethatthreatstocausetheextinctionofEarthoriginatingintelligentlifeortopermanently anddrasticallydestroyitspotential;seeBostrom2002;Rees2004. Bostrom2003;Matheny2007;Leslie1996;Posner2005.

52

53

26

notentailthatthisdevelopmentwouldonbeonbalancebadorthatitoughtnotbevigorously pursued.) Thedistinctionbetweendevelopmenthazardandthevarioushazardslistedaboveasadversarialrisksis somewhatvague.Developmenthazards,bycontrasttoadversarialrisks,arenottiedtoanyparticular extantcompetitivecontext.Forexample,ariskofsometechnologicaldevelopmentthatconsistsinthe potentialharmtousthatcouldresultfromthedifferentialstrengtheningofourenemyorrivalshould beclassifiedasanenemyhazardoracompetitivenesshazardratherthanadevelopmenthazard.Buta riskofsometechnologicaldevelopmentthatconsistsinthepotentialforharmthatarisesfromthefact thatthistechnologywouldbelikelytocausesomesevereaccidentorwouldgenerallylenditselfto abusesbyawiderangeofindividuals,groups,orstateswouldposeadevelopmenthazard.Some technologicaldevelopmentsmightposebothadversarialanddevelopmentalrisks. 9.Discussion Theconsiderationsadducedabove,althoughnotontheirowndeterminativeofwhatistobedonein anyparticularactualcase,canhelpinformourchoicesbyhighlightingthesometimessubtlewaysin whicheventrueinformationcanhaveharmfulaswellasbeneficialeffects(table1).

27

 TYPOLOGYOFINFORMATIONHAZARDS I.Byinformationtransfermode Datahazard Ideahazard Attentionhazard Templatehazard Signalinghazard Evocationhazard II.Byeffect TYPE SUBTYPE ADVERSARIALRISKS Competivenesshazard Enemyhazard Intellectualpropertyhazard Commitmenthazard Knowingtoomuchhazard RISKSTOSOCIAL Normhazard Informationasymmetry ORGANIZATONAND hazard MARKETS Unveilinghazard Recognitionhazard RISKSOFIRRATIONALITY Ideologicalhazard ANDERROR Distractionandtemptation hazards Rolemodelhazard Biasinghazard Debiasinghazard Neuropsychologicalhazard Informationburyinghazard RISKSTOVALUABLESTATES Psychologicalreaction Disappointmenthazard ANDACTIVITIES hazard Spoilerhazard Mindsethazard Beliefconstitutedvalue hazard (mixed) Embarrassmenthazard RISKSFROMINFORMATION Informationsystemhazard Informationinfrastructure TECHNOLOGYSYSTEMS failurehazard Informationinfrastructure misusehazard Artificialintelligencehazard RISKSFROMDEVELOPMENT Developmenthazard
Table1

28

Therearemanywaysofrespondingtoinformationhazards.Inmanycases,thebestresponseisno response,i.e.,toproceedasthoughnosuchhazardexisted.Thebenefitsofinformationmaysofar outweighitscoststhatevenwheninformationhazardsarefullyaccountedfor,westillunderinvestin thegatheringanddisseminationofinformation.Moreover,ignorancecarriesitsowndangerswhichare oftentimesgreaterthanthoseofknowledge.Informationrisksmightsimplybetolerated.Insome contextstheycouldbeinsuredorhedgedagainstusingavarietyoffinancialinstruments. 54 Whenmitigationiscalledfor,itneednottaketheformofanactiveattempttosuppressinformation throughmeasuressuchasbans,censorship,disinformationcampaigns,encryption,orsecrecy.One responseoptionissimplytoinvestlessindiscoveringanddisseminatingcertainkindsofinformation. Somebodywhoisworriedaboutthespoilerhazardoflearningabouttheendingofamoviecansimply refrainfromreadingreviewsandplotsummaries. Sometimes,suchasinthecasesofsomeideologicalhazardsandsomeinformationasymmetryhazards, thedangerliesinpartialinformation.Thebestresponsemaythenbetoprovidemoreinformation,not less.Someproblemscanbesolvedthroughpolicymeasurestheproblemofasymmetriesinhealth informationcanbeobviated,forexample,byinstitutingpubliclyfundeduniversalhealthcare.Inother cases,suchasdistractionhazardandsomebiasinghazards,thesolutionmaybetocarefullyselectan appropriateformatandcontextfortheinformationthatistobepresented. Whencontemplatingtheadoptionofsomepolicydesignedtorestrictinformation,itisworthreflecting thathistoricallysuchpolicieshaveoftenservedspecialinterests.InTheWeaponofOpenness,ashort essayonroleofsecrecyandopennessinnationalsecurity,ArthurKantrowitzwrote: [S]ecrecyinsiderscomefromaculturewhereaccesstodeepersecretsconveyshigherstatus. Thosewhogetaheadinthecultureofsecrecyunderstanditsusesforpersonaladvancement. Knowledgeispower,andformanyinsidersaccesstoclassifiedinformationisthechiefsourceof theirpower.Itisnotsurprisingthatsecrecyinsidersseethepublicationoftechnological informationasendangeringnationalsecurity.55 Outsidersoftenhavereasonforskepticismwheninsidersinsistthattheirinnerdealingsmustbe protectedfrompublicscrutiny.Secrecybreedscorruption.Kantrowitzarguedthatevenwithrespectto thenarrowcriterionofmilitarystrength,aunilateralpolicyofopenness(atleastinpeacetime)leadsto betterresults. Atthesametime,however,weshouldrecognizethatknowledgeandinformationfrequentlyhave downsides.Futurescientificandtechnologicaladvances,inparticular,maycreateinformationwhich, misused,wouldcausetremendousharmincluding,potentially,existentialcatastrophe.Ifweaddin themanylesserhazardsthatcanbecreatedbysuchadvances,forexamplebytechnologiesthat facilitatecommercialfraudorthatintroduceinsidiousnewchemicalsintothehumanbody,therange
54

Seee.g.Petratos2007. Kantrowitz1992.

55

29

andcomplexityofpotentialinformationhazardsgrowsevengreater.Ifwefurtherexpandourpurview andconsiderthemanyindirectandreciprocalinfluencesbetween,forinstance,scientificinformation andeconomicgrowth,andif,moreover,wealsogiveattentiontothenumerousways,cataloguedin precedingsections,inwhichinformationoutsidetherealmsofscienceandtechnologycancause harmthenweshallcometoappreciatethatinformationhazardsareubiquitous,potentiallyserious, andoftennonobvious. Ananalysisofthepolicyimplicationsofthisresultisbeyondthescopeofthispaper. 56 Bywayof conclusion,though,wemayadumbratetwocontrastingpotentialresponses.Giventhecomplexityof theissuesinvolved,andtheirentanglementwithmanystrategic,philosophical,andpolitical considerations,itisnottrivialtoascertainwhichoftheseresponseshasthemosttorecommendit. 57 Onepossibleresponse,then,wouldbetotaketoheartthemanifoldwaysinwhichthediscoveryand disseminationofinformationcanhavenegativeeffects. 58 Wecouldaccepttheneedtoqualifythe fawningadmirationandunquestioningcommitmenttothepursuitoftruththatcurrentlyconstitutes officialpolicyifnotalwaysconsistentpracticeinmostuniversitiesandlearnedbodies.Amottolike HarvardsVeritas!couldbeviewedasnaveandreckless.Instead,onemightconclude,weoughtto thinkmorecarefullyandopenmindedlyaboutwhichparticularareasofknowledgedeservetobe promoted,whichshouldbeletbe,andwhichshouldperhapsevenbeactivelyimpeded. Sincescholarsareverylikelytobebiasedinfavorofthinkingthattheirownfielddeservestobe promoted,outsiderswhoarelessprejudicedshouldbebroughtintoparticipateinthesedeliberations. TheoldEnlightenmentmodelofscientificresearch,whichpicturesscienceasagoosethatlaysgolden eggsbutonlyifallowedfullautonomyandifshieldedfromexternalsocialcontrol,wouldperhapshave tobereplacedwithadifferentmodelinwhich,forexample,democraticprocessesandpreferencesare allowedgreaterinfluenceoverresearchdirectionsandpriorities. Anotherresponsewouldnotethegreatbenefitsthathistoricallyhavecomefromthepursuitof knowledgeandenlightenment,andfastenonthedangersinherentinanyattempttocurtailfreeinquiry ortoyokescientificresearchtosomepreconceivednotionofthesocialgood.Thoseinclinedtogivethis responseneednotdenythattrueinformationcaninmanyinstancesbeharmfulorhazardous;they needonlymaintainthatonbalancewearebetteroffasloyalsubjectstothecauseofenlightenment.It canalsobehopedthatnewinformationtechnologieswillbringaboutavastlymoretransparentsociety, inwhicheverybody(thewatchmenincluded)areunderconstantsurveillance;andthatthisuniversal

56

Ihopetoaddresssomeoftheseissuesinacompanionpaper. Wemaybelikelytooverlookatleastonecrucialconsideration;seeBostrom2006.

57

58

OnefairlyrecentandwellknownattempttoarguethisisBillJoysarticleinwhichheadvocatesselective relinquishmentofresearchincertainfieldswithinartificialintelligence,nanotechnology,andbiotechnology becauseofdangersheforeseesinthefutureifsuchresearchispursued(Joy2000).

30

transparencywillpreventtheworstpotentialmisusesofthenewtechnologicalpowersthathumanity willdevelop. 59 Evenifourbestpolicyistoformanunyieldingcommitmenttounlimitedfreedomofthought,virtually limitlessfreedomofspeech,anextremelywidefreedomofinquiry,weshouldrealizenotonlythatthis policyhascostsbutthatperhapsthestrongestreasonforadoptingsuchanuncompromisingstance woulditselfbebasedonaninformationhazard;namely,normhazard:theriskthatpreciousyetfragile normsoftruthseekingandtruthfulreportingwouldbejeopardizedifwepermittedconvenient exceptionsinourownadherencetothemoriftheirviolationwereingeneraltooreadilyexcused. Itissaidthatalittleknowledgeisadangerousthing.Itisanopenquestionwhethermoreknowledgeis safer.Evenifourbestbetisthatmoreknowledgeisonaverageagoodthing,weshouldrecognizethat therearenumerouscasesinwhichmoreknowledgemakesthingsworse. 60 References 1. Akerlof,G.A.1970.TheMarketforLemons:QualityUncertaintyandtheMarketMechanism. TheQuarterlyJournalofEconomics84(3):488500. 2. Austin,J.L.1962.InJ.O.Urmson,ed.HowtoDoThingswithWords:TheWilliamJamesLectures DeliveredatHarvardUniversityin1955.Oxford:Clarendon. 3. Berglas,S.andE.E.Jones.1978.Drugchoiceasaselfhandicappingstrategyinresponseto noncontingentsuccess.JournalofPersonalityandSocialPsychology36(4):405417. 4. Bikhchandani,S.,Hirshleifer,D.andWelch,I.1992.ATheoryofFads,Fashion,Custom,and CulturalChangeasInformationalCascades.TheJournalofPoliticalEconomy100(5):992. 5. Bostrom,N.andA.Sandberg.2008.WholeBrainEmulation:ARoadmap.Oxford:Futureof HumanityInstitute. 6. Bostrom,N.2006.TechnologicalRevolutions:EthicsandPolicyintheDark.InNanoscale:Issues andPerspectivesfortheNanoCentury,eds.N.M.D.S.CameronandM.E.Mitchell,129152. Hoboken,N.J.:Wiley. 7. Bostrom,N.2003.AstronomicalWaste:TheOpportunityCostofDelayedTechnological Development.Utilitas15(3):308314. 8. Bostrom,N.2003.EthicalIssuesinAdvancedArtificialIntelligence.Cognitive,Emotiveand EthicalAspectsofDecisionMakinginHumansandinArtificialIntelligence2:1217.
59

Cf.Brin1999.

60

ForcommentsanddiscussionsIamgratefultoWalterSinnottArmstrong,AllenBuchanan,TylerCowen,Robin Hanson,TobyOrd,PythagorasPetratos,RebeccaRoache,AndersSandbergandNickShackel.Iwouldalsoliketo thankNancyPatelandRachelWoodcockforassistance.

31

9. Bostrom,N.2002.ExistentialRisks:AnalyzingHumanExtinctionScenariosandRelatedHazards. JournalofEvolutionandTechnology9. 10. Bostrom,N.1998.HowLongbeforeSuperintelligence?InternationalJournalofFutureStudies2. 11. Brin,D.1999.TheTransparentSociety:WillTechnologyForceUstoChooseBetweenPrivacyand Freedom?NewYork:PerseusBooks. 12. Brown,J.D.,K.A.DuttonandK.E.Cook.2001.FromtheTopDown:SelfEsteemandSelf Evaluation.CognitionandEmotion15:615631. 13. Cirincione,J.2008.TheContinuingThreatofNuclearWar.InGlobalCatastrophicRisks,ed.N. BostromandM.Cirkovic,381401.Oxford:OxfordUniversityPress. 14. Drexler,E.1987.Chapter3:11EnginesofDestruction.InEnginesofCreation:TheComingEraof Nanotechnology,171190.GardenCity:Anchor. 15. Eliot,T.S.2001.FourQuartets.London:FaberandFaber. 16. Fawthrop,T.andH.Jarvis.2005.GettingAwayWithGenocide:CambodiasLongStruggle AgainsttheKhmerRouge.Sydney:UNSWPress. 17. Freitas,R.2000.SomeLimitstoGlobalEcophagybyBiovorousNanoreplicators,withPublic PolicyRecommendations.ForesightInstitute.http://www.foresight.org/nano/Ecophagy.html. 18. Glover,J.2001.Humanity:AMoralHistoryoftheTwentiethCentury.NewHaven:YaleUniversity Press. 19. Goffman,E.1959.ThePresentationofSelfinEverydayLife.GardenCity:DoubledayAnchor. 20. Gubrud,M.1997.NanotechnologyandInternationalSecurity.ForesightInstitute. http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/index.html. 21. Halliday,T.J.andS.Kwak.2007.IdentifyingEndogenousPeerEffectsintheSpreadofObesity, WorkingPaper727.UniversityofHawaiiatManoa,DepartmentofEconomics. http://www.economics.hawaii.edu/research/workingpapers/WP_0727.pdf. 22. Hirschman,A.O.1991.TheRhetoricofReaction:Perversity,Futility,Jeopardy.Cambridge,M.A.: BelknapPress. 23. Hobden,K.L.1997.BehaviouralVersusClaimedSelfHandicapping:UnderlyingMotivationsand AttributionsFollowingFailure.Toronto:UniversityofTorontodoctoralthesis. 24. Hurka,T.1993.Perfectionism.Oxford:ClarendonPress. 25. JonasK.1992.Modellingandsuicide:atestoftheWerthereffect.BrJSocPsychol.,Dec31(Pt 4):295306. 26. Joy,B.2000.WhytheFutureDoesn'tNeedUs.Wired8.04,April:238245,248263. 27. Kahneman,D.,Slovic,P.andA.Tversky,eds.1982.JudgmentunderUncertainty:Heuristicsand Biases.NewYork:CambridgeUniversityPress. 32

28. Kantrowitz,A.1992.TheWeaponofOpenness.InNanotechnologyResearchandPerspectives, eds.B.C.CrandallandJ.Lewis,303311.Cambridge,M.A.:MITPress. 29. Kavka,G.S.1990.SomeSocialBenefitsofUncertainty.MidwestStudiesinPhilosophy15(1): 311326. 30. Kennedy,R.1968.13Days.London:Macmillan. 31. Kurzweil,R.2006.TheSingularityIsNear:WhenHumansTranscendBiology.NewYork:Penguin. 32. Leslie,J.1996.TheEndoftheWorld:TheScienceandEthicsofHumanExtinction.London: Routledge. 33. Levack,B.P.1987.TheWitchHuntinEarlyModernEurope.London:Longman. 34. Markovitch,S.andP.D.Scott.1988.KnowledgeConsideredHarmful,ResearchPaper#030788. AnnArbor,Michigan:CenterforMachineIntelligence. 35. Matheny,J.G.2007.ReducingtheRiskofHumanExtinction.RiskAnalysis27(5):13351344. 36. Meade,E.E.andD.Stasavage.2008.PublicityofDebateandtheIncentivetoDissent:Evidence fromtheUsFederalReserve.EconomicJournal118(528):695717. 37. Merton,R.K.1968.TheMatthewEffectinScience.Science159(3810):5663. 38. Montefiore,S.S.2005.Stalin:TheCourtoftheRedTsar.NewYork:Vintage. 39. Moravec,H.2000.Robot:MereMachinetoTranscendentMind.NewYork:OxfordUniversity Press. 40. Nelson,R.andG.Winter.1990.AnEvolutionaryTheoryofEconomicChange.Cambridge,M.A.: HarvardUniversityPress. 41. Nietzsche,F.1984.VomNutzenundNachteilderHistoriefrdasLeben.Zurich:DiogenesVerlag. 42. Nietzsche,F.2007.OntheUseandAbuseofHistoryforLife.SiouxFalls,S.D.:NuVision Publications. 43. Nouri,A.andC.F.Chyba.2008.BiotechnologyandBiosecurity.InGlobalCatastrophicRisks, eds.N.BostromandM.Cirkovic,450480.Oxford:OxfordUniversityPress. 44. Oppenheimer,R.1947.PhysicsintheContemporaryWorld,LectureatM.I.T.,November25. 45. Petratos,P.2007.Weather,InformationSecurity,andMarkets.IEEESecurityandPrivacy5(6): 5457. 46. Phillips,D.1982.TheImpactofFictionalTelevisionStoriesonU.S.AdultFatalities:NewEvidence ontheEffectoftheMassMediaonViolence".TheAmericanJournalofSociology87(6):1340 59. 47. Porter,M.E.2004.CompetitiveAdvantage.NewYork:FreePress. 48. Posner,R.2005.Catastrophe:RiskandResponse.NewYork:OxfordUniversityPress. 33

49. Putnam,H.1979.Theplaceoffactsinaworldofvalues.InTheNatureofthePhysicalUniverse, eds.D.HuffandO.Prewett,113140.NewYork:JohnWiley. 50. Radford,B.andR.Bartholomew.2001.PokmonContagion:PhotosensitiveEpilepsyorMass PsychogenicIllness?SouthernMedicalJournal94(2):197204. 51. Rawls,J.2005.ATheoryofJustice.Cambridge,M.A.:BelknapPress. 52. Rees,M.2004.OurFinalCentury:WilltheHumanRaceSurvivetheTwentyFirstCentury?Arrow BooksLtd. 53. Rhodes,R.1995.TheMakingoftheAtomicBomb.Simon&Schuster. 54. Rizzo,M.J.andD.G.Whitman.2003.TheCamel'sNoseisintheTent:Rules,Theoriesand SlipperySlopes.UCLALawReview51:539592 55. Robinson,C.W.andV.M.Sloutsky.2007.LinguisticLabelsandCategorizationinInfancy:Do LabelsFacilitateorHinder?Infancy11(3):233253. 56. Schelling,T.C.1981.TheStrategyofConflict,pp.187ff.Cambridge,M.A.:HarvardUniversity Press. 57. Schlegel,A.1991.Status,Property,andtheValueonVirginity.AmericanEthnologist18(4):719 734. 58. Shattuck,R.1996.ForbiddenKnowledge:FromPrometheustoPornography.NewYork:St. Martin'sPress. 59. Smith,T.W.,C.R.SnyderandS.C.Perkins.1983.Theselfservingfunctionofhypochondriacal complaints:physicalsymptomsasselfhandicappingstrategies.JournalofPersonalityandSocial Psychology44(4):787797. 60. StackS.1996.Theeffectofthemediaonsuicide:evidencefromJapan,19551985.SuicideLife ThreatBehav.26(2):13242. 61. Stone,J.2002.BattlingDoubtbyAvoidingPractice:TheEffectsofStereotypeThreatonSelf HandicappinginWhiteAthletes.PersonalityandSocialPsychologyBulletin28(12):16671678. 62. Takada,H.,K.Aso,K.Watanabe,A.Okumura,T.NegoroandT.Ishikawa.1999.EpilepticSeizures InducedbyAnimatedCartoon,"PocketMonster.Epilepsia40(7):9971002. 63. TheManhattanEngineerDistrict.1946.TotalCasualties.TheAtomicBombingsofHiroshima andNagasaki,June29.http://www.atomicarchive.com/Docs/MED/med_chp10.shtml. 64. Thompson,T.andA.Richardson.2001.Selfhandicappingstatus,claimedselfhandicapsand reducedpracticeeffortfollowingsuccessandfailurefeedback.BritishJournalofEducational Psychology71(1):15170. 65. Tversky,A.andKahneman,D.1974.JudgmentunderUncertainty:HeuristicsandBiases.Science 185(4157):11241131 66. Vinge,V.1993.Thecomingtechnologicalsingularity:Howtosurviveintheposthumanera. 34

Vision21:InterdisciplinaryScienceandEngineeringintheEraofCyberspace1122.United States:NASALewisResearchCenter. 67. Volokh,E.2003.TheMechanismsoftheSlipperySlope.HarvardLawReview116:10261134. 68. Yudkowsky,E.2008.ArtificialIntelligenceasaPositiveandNegativeFactorinGlobalRisk.In GlobalCatastrophicRisks,eds.N.BostromandM.Cirkovic,308345.Oxford:OxfordUniversity Press. 69. Yudkowsky,E.2008.CognitiveBiasesPotentiallyAffectingJudgementofGlobalRisks.InGlobal CatastrophicRisks,eds.N.BostromandM.Cirkovic,91119.Oxford:OxfordUniversityPress. 70. Zuckerman,L.,P.HofheinzandS.Naushad.1987.HowNottoSilenceaSpy.Time,August17. http://www.time.com/time/magazine/article/0,9171,965233,00.html

35