The Anatomy of a

Large-Scale
Human-Computation
Engine
Shailesh Kochhar, Stefano Mazzocchi, Praeen Paritosh
free!ase"com, google"com
HC#MP$%&
'uly (), (&%& HC#MP$%&
%* +ree!ase , Human Computation
(* E-ample . Stanfor/ Li!rary
0* 1A2'
3* Lessons

+ree!ase
Structure/ /ata!ase
%( MM entites, 0&& MM triples4facts
'uly (), (&%& HC#MP$%&
5here /oes the /ata come from6
'uly (), (&%& HC#MP$%&
Community contri!utions
Mass 7ata Loa/s
'uly (), (&%& HC#MP$%&
Human 'u/gments 8mproe 2oth
'uly (), (&%& HC#MP$%&
Community
Simplifying contri!ution through games
'uly (), (&%& HC#MP$%&
Mass 7ata Loa/s
Precision* 9A for :;;< accuracy
1ecall* increase coerage
'uly (), (&%& HC#MP$%&
%* +ree!ase , Human Computation
(* E-ample . Stanfor/ Li!rary
0* 1A2'
3* Lessons
'uly (), (&%& HC#MP$%&
1econcile Stanfor/ Li!rary Catalog =ith
free!ase"com
'uly (), (&%& HC#MP$%&
Stanfor/ Li!rary Catalog
3"3MM !oo> e/itions
%"0MM English !oo> e/itions
%"(MM English !oo>s
?&&K authors
'uly (), (&%& HC#MP$%&
+or free!ase, i/entity is >ey
match !oo>s, match authors
'uly (), (&%& HC#MP$%&
Automatic matching insufficient
Traine/ @u/ges nee/e/ to /eci/e har/ cases
'uly (), (&%& HC#MP$%&
Ho= to get this6
'uly (), (&%& HC#MP$%&
1A2'
1e/un/ant Array of 2rains in a 'ar
'uly (), (&%& HC#MP$%&
A!straction
Po=ers human @u/gment applications
%"AMM @u/gments in %? months of
operation
'uly (), (&%& HC#MP$%&
Proi/es primities for more sophisticate/
H' applications
'uly (), (&%& HC#MP$%&
9uestions
'u/gments
9ueues
Agents
'uly (), (&%& HC#MP$%&
7esign Constraints
'uly (), (&%& HC#MP$%&
Content-agnostic
7ynamic /ata
Lo= latency
'uly (), (&%& HC#MP$%&
Architecture
'uly (), (&%& HC#MP$%&
9uestions contain pointers to /ata, pushe/
to a store
9uestions a//e/ to Bueues
Meta/ata allo=s slicing an/ /icing
'uly (), (&%& HC#MP$%&
'S applications pull Buestions from !ro>er
2ro>er matches @u/ge to =or>
Apps ren/er Buestion, collect @u/gment
2ro>er =rites @u/gments !ac> to store
'uly (), (&%& HC#MP$%&
7eclaratie consensus
Ces* 0, Do* 0, S>ip* (, 2a/* (, Ma-* 3
2ro>er notifies agents of consensus
'uly (), (&%& HC#MP$%&
Applications
'uly (), (&%& HC#MP$%&
matchma>er
http*44matchma>er("free!aseapps"com4
'uly (), (&%& HC#MP$%&
2oo> E/ition 9A
'uly (), (&%& HC#MP$%&
type=riter
http*44type=riter"free!aseapps"com4
'uly (), (&%& HC#MP$%&
Scale
'uly (), (&%& HC#MP$%&
%"A MM Buestions
("AMM @u/gments
)&& Bueues
(&E applications
'uly (), (&%& HC#MP$%&
%* +ree!ase , Human Computation
(* E-ample . Stanfor/ Li!rary
0* 1A2'
3* Lessons
'uly (), (&%& HC#MP$%&
1elationships, relationships, relationships
This is not controersial
'uly (), (&%& HC#MP$%&
Spam, collusion, gaming* F&
S>ill /eelopment
Communication, /ocumentation
'uly (), (&%& HC#MP$%&
7on$t hae to pay per-@u/gment
Ces, this is controersial
'uly (), (&%& HC#MP$%&
'uly (), (&%& HC#MP$%&
There are al=ays leftoers
'uly (), (&%& HC#MP$%&
5or>ing on formalizing =or>flo=s
'uly (), (&%& HC#MP$%&
More in the paper
1A2' Architecture
Learning through fee/!ac> loops
'uly (), (&%& HC#MP$%&
http*44ra!@"free!aseapps"com4
http*44=i>i"free!ase"com4=i>i41A2'GAP84
9uestions6