A Lock-Free Hash Table

The Art of (Java) Benchmarking

Dr. Cliff Click
Chief JVM Architect !isting"ishe# $ngineer blogs%a&"ls'stems%com(cliff A&"l )'stems J"ne *+, -.*.

Benchmarking is $as'000
///%a&"ls'stems%com

1 1 1 1

An# F"n000 2M' Java is faster than 'o"r C0003 An# generall' /rong%%% Without exception every microbenchmark 45ve seen has ha# serious fla/s
6 $7ce8t those 45ve ha# a han# in correcting

1 Serious 9
6 2)core3 is "nrelate# to inten#e# meas"rement or 6 error bars e7cee# meas"re# val"es

)8lit o"t micro-bench vs macro-bench

///%a&"ls'stems%com

1 Micro-benchmarks are things 'o" /rite 'o"rself

6 Attem8t to #iscover some narro/ targete# fact 6 =enerall' a time# tight loo8 aro"n# some 2/ork3 6 >e8ort score as iterations(sec 6 e%g% allocations(sec ? ob@ect 8ooling vs =C

1 Macro-benchmarks are s"88ose# to be realistic

6 Larger, longer r"nning 6 e%g% Aeb)erver, !B caching(front-en#, Bortal A88 6 )8ecJBB, )8ecJA88)erver, CMLMark, Tra#eD 6 Loa# testing of Eo"r >eal A88

;-..< A&"l )'stems, 4nc%

)ome Fl#er B"ste# Micro-Benchmarks

///%a&"ls'stems%com

1 CaffeineMark 2logic3
6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8

1 )ciMark- Monte-Carlo
6 H.I of time in s'nc5# >an#om%ne7t 6 )everal foc"se# tests #ea#G infinite s8ee#"8

1 )8ecJVMJH K-.JK#b ? 8"r8orts to be a !B test

6 >eall'L HMI of time in )tring shell-sort

1 )8ecJVMJH K--<Kmtrt
6 >"ntime is m"ch less than * sec

;-..< A&"l )'stems, 4nc%

)ome Fl#er B"ste# Micro-Benchmarks

///%a&"ls'stems%com

1 CaffeineMark 2logic3
6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8

1 )ciMark- Monte-Carlo
6 H.I of time in s'nc5# >an#om%ne7t 6 )everal foc"se# tests #ea#G infinite s8ee#"8

1 )8ecJVMJH K-.JK#b ? 8"r8orts to be a !B test

6 >eall'L HMI of time in )tring shell-sort

1 )8ecJVMJH K--<Kmtrt
6 >"ntime is m"ch less than * sec

;-..< A&"l )'stems, 4nc%

!ea# Loo8s
///%a&"ls'stems%com

1 TimelineL

// how fast is divide-by-10? long start = Sys.CTM() for( int i=0 i!" i## ) int $ = i/10 ret%rn "&1000/(Sys.CTM()-start)

6 *- 4nter8ret a /hile, ass"me *.ms

6 -- J4TG 273 not "se#, loo8 is #ea#, remove#, *.ms

6 N- 24nstantl'3 e7ec"te rest of loo8

1 Time to r"nL -.ms ? Independent of N! 6 Var' O 99P var' score 99P 2!ial-o-)core03

;-..< A&"l )'stems, 4nc%

!ea# Loo8s
///%a&"ls'stems%com

1 )ometimes J4T 8roves 2res"lts not nee#e#3

6 Then thro/s o"t /hole /ork-loo8 6 After r"nning long eno"gh to J4T 6 )o loo8 r"ns at least a little /hile first

1 )core 2o8s(sec3 not relate# to tri8 co"nt 5O5

6 Larger O 99P larger score

1 )core can be infinite- or OaO

6 =enerall' re8orte# as a ver' large, b"t vali# Q 6 An# mi7e# in /ith other n"mbers, conf"sing things 6 (e%g% geomean of infinite5s an# other more real n"mbers)

;-..< A&"l )'stems, 4nc%

)ciMark- Monte-Carlo
///%a&"ls'stems%com

1 H.I of time in syn'hroni(e5# >an#om%ne7t 1 N-letter-com8an' 2s8amme#3 it b' re8lacing /ith intrinsic
#oing a Com8areAn#)/a8 (CA))

1 1 1 1 1

4 /as or#ere# to follo/ s"it (match 8erformance) !o"g Lea sai# 2/aitL @"st make a CA) from Java3 Hence s"n%misc%AtomicLong /as born >a8i#l' re8lace# b' Rnsafe%com8areAn#)/a8%%% %%%an# event"all' @ava%lang%AtomicS

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL Aarm"8

///%a&"ls'stems%com

1 Co#e starts inter8rete#, then J4T5#

6 J4T5# co#e is *.7 faster than inter8reter

1 J4T5ing ha88ens 2after a /hile3

6 Hot)8ot -serverL *.,... iterations 6 Bl"s com8ile time

1 Aarm"8 co#e /ith some trial r"ns

6 Tee8ing testing "ntil r"n-times stabili&e

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL Aarm"8

///%a&"ls'stems%com

1 Oot allo/ing /arm"8 is a common mistake 1 Bo8"lar fail"re-mo#e of C-vs-Java com8arisons

6 Fo"n# on man', man', man' /eb 8ages 6 $ntire benchmark r"ns in fe/ milli-secon#s 6 There are #omains reU"iring milli-secon# reboots%%%

1 But most #eskto8(server a88s e78ectL

6 >eboots are min"tes long an# #a's a8art 6 )tea#'-state thro"gh8"t after /arm"8 is ke' 6 )o a benchmark that en#s in V*.sec

8robabl' #oes not meas"re an'thing interesting

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL 2Com8ile 8lan3

///%a&"ls'stems%com

1 J4T makes inlining

other com8le7 #ecisions

6 Base# on ver' volatile ran#om #ata 6 4nline #ecisions var' from r"n-to-r"n

1 Berformance varies from r"n-to-r"n

6 )table n"mbers /ithin a single JVM invocation 6 B"t co"l# var' b' P-.I /ith ne/ JVM la"nch 6 Bigger a88s are more 8erformance-stable

1 Micro-benchmarks ten# to be 2fragile3 here

6 e%g% * JVM la"nch in M /ill be -.I slo/erS

*S tatistically R igorous Java P erformance E valuatio n OOPSLA 2007

: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL 2Com8ile 8lan3

///%a&"ls'stems%com

AL loo8*L Y loo8-L Y call C Y @ne loo8Y @ne loo8* ret"rn CL Y ret"rn

: ;-..< A&"l )'stems, 4nc%

8"blic int a() W for(%%%) b()G X 8"blic int b() W for(%%%) c()G X 8"blic int c() W Y/ork%%% X

20% chance A inlines B B calls C

80% chance A calls B B inlines C

AL loo8*L Y call B @ne loo8* ret"rn BL loo8-L %%% @ne loo8Y ret"rn

Micro-bench A#viceL 2Com8ile 8lan3

///%a&"ls'stems%com

1 La"nch the JVM man' times

6 Toss *st la"nch to remove F) caching effects 6 Average o"t 2goo#3 r"ns /ith the 2ba#3 6 !on5t other/ise toss o"tliers 6 ("nless 'o" have goo# reasonL i%e% "nrelate# loa#)

1 $no"gh times to get statisticall' relevant res"lts

6 Might reU"ire N.Z r"ns

1 >e8ort average and stan#ar# #eviation

6 4n this case, e78ect to see a large st#%#ev

*Statistically rigorous Java performance evaluation OOPSLA 2007

: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL 2*st fast, -n# slo/3

///%a&"ls'stems%com

1 Timing harness nee#s to invoke man' targets

6 4n a loo8, re8eate#l' a fe/ times 6 $lse J4T sees * hot target in a loo8 6 An# then #oes a g"ar#e# inline 6 An# then hoists the time# /ork o"tsi#e of timing loo8

'lass ben'h1 i)*le)ents ben'h + void s,rt(int i) 'lass ben'h. i)*le)ents ben'h + void s,rt(int i) stati' final int "=1000000 // )illion ... stati' int test( ben'h / ) + long start = Syste).'%rrentTi)eMillis() for( int i=0 i!" i## ) /.s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste).'%rrentTi)eMillis()-start)
: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL 2*st fast, -n# slo/3

///%a&"ls'stems%com

Bass in one of t/o #ifferent classes

'lass ben'h1 i)*le)ents ben'h + void s,rt(int i) 'lass ben'h. i)*le)ents ben'h + void s,rt(int i) stati' final int "=1000000 // )illion ... stati' int test( ben'h / ) + long start = Syste).'%rrentTi)eMillis() for( int i=0 i!" i## ) /.s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste).'%rrentTi)eMillis()-start)
: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 First callL test(new ben'h1)

long start = Sys.CTM() for( int i=0 i!" i## ) /.s,rt(i) ret%rn "&1000/(Sys.CTM()-start)

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 First callL test(new ben'h1)

6 )ingle target callsiteG J4T #oes g"ar#e# inlining 6 4nlines ben'h1.s,rt

long start = Sys.CTM() for( int i=0 i!" i## ) /.s,rt(i) ret%rn "&1000/(Sys.CTM()-start) long start = Sys.CTM() g"ar#e# for( int i=0 i!" i## ) inline Math.s,rt(i) // inline ben'h1.s,rt ret%rn "&1000/(Sys.CTM()-start)

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 First callL test(new ben'h1)

6 )ingle target callsiteG J4T #oes g"ar#e# inlining 6 4nlines ben'h1.s,rt 6 Hoists loo8-invariants, #ea#-co#e-remove, etc 6 $7ec"tion time #oes OFT #e8en# on O000 6 !rea#e# 2!ial-o-)core03

long start = Sys.CTM() long start = Sys.CTM() // 01T2d loo* re)oved for( int i=0 i!" i## ) ret%rn "&1000/(Sys.CTM()-start) /.s,rt(i) ret%rn "&1000/(Sys.CTM()-start) long start = Sys.CTM() g"ar#e# for( int i=0 i!" i## ) #ea# co#e inline Math.s,rt(i) // inline ben'h1.s,rt ret%rn "&1000/(Sys.CTM()-start)

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 )econ# callL test(new ben'h.)

6 -n# target of callG g"ar#e# inlining fails 6 Co#e is incorrectG m"st be re-J4T5# 6 Meas"res overhea# of O calls to bench-%sUrt 6 Bl"s g"ar# fail"re, #eo8timi&ation 6 Bl"s J4T5ing ne/ version of test() 6 Bl"s virt"al call overhea#

long start = Syste).CTM() for( int i=0 i!" i## ) /.s,rt(i) ret%rn "&1000/(Syste).CTM()-start)
: ;-..< A&"l )'stems, 4nc%

int ben'h..s,rt( int i ) + // alternate i)*l -

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 >eversing or#er of calls reverses 2goo#3

6 e%g% 2test(new

ben'h.)

2ba#3 test(new ben'h1) 3

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 >eversing or#er of calls reverses 2goo#3

6 e%g% 2test(new

ben'h.)

2ba#3 test(new ben'h1) 3

1 Timing harness nee#s to invoke all targets

6 4n a loo8, re8eate#l' a fe/ times

'lass ben'h1 i)*le)ents ben'h + void s,rt(int i) 'lass ben'h. i)*le)ents ben'h + void s,rt(int i) ... // war)%* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h. ) // now try ti)ing *rintf(test(new ben'h1))
: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL v-call o8timi&ation

///%a&"ls'stems%com

1 >eversing or#er of calls reverses 2goo#3

6 e%g% 2test(new

ben'h.)

2ba#3 test(new ben'h1) 3

1 Timing harness nee#s to invoke all targets

6 4n a loo8, re8eate#l' a fe/ times

'lass ben'h1 i)*le)ents ben'h + void s,rt(int i) 'lass ben'h. i)*le)ents ben'h + void s,rt(int i) ... // war)%* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h. ) // now try ti)ing *rintf(test(new ben'h1))
: ;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL =C
///%a&"ls'stems%com

1 Avoi# =C or embrace it 1 $ither no (or trivial) allocation, or "se verbose3g' to

make s"re 'o" hit stea#'-state

1 )tatisticsL not @"st average, b"t also st#-#ev 1 Look for tren#s
6 Co"l# be cree8ing =C behavior

1 Co"l# be 2leaks3 ca"sing more-/ork-8er-r"n

6 e%g% leak' HashTable gro/ing hea8 or 6 =ro/ing a Linke#List slo/s #o/n searches

;-..< A&"l )'stems, 4nc%

Micro-bench A#viceL )'nchroni&ation

///%a&"ls'stems%com

1 Acco"nt for m"lti-threa#e# locking 1 4 do see 8eo8le testing, e%g% locking costs on singlethrea#e# 8rograms

1 Never conten#e# lock is ver' chea8

6 ZBiase#Locking makes it even chea8er

1 Very slightly conten#e# lock is 8robabl' +7 more 1 Real contentionL Am#ahl5s La/
6 Bl"s lots and lots of F) overhea#

1 @ava%"til%conc"rrent is 'o"r frien#

;-..< A&"l )'stems, 4nc%

Micro-bench A#vice
///%a&"ls'stems%com

1 >ealistic r"ntimes
6 Rnless 'o" nee# s"b-milli-sec reboots

1 1 1 1 1

Aarm-"8 loo8s - give the J4T a chance )tatisticsL 8lan for variation in res"lts !ea# loo8s ? look for 2!ial-o-)core03, #eal /ith it *st r"n fast, -n# r"n slo/ ? look for it, #eal /ith it =CL avoi# or embrace

;-..< A&"l )'stems, 4nc%

Macro-bench /arnings
///%a&"ls'stems%com

1 JVMJH is too small an'more

6 $as' targetG cache-resi#entG =C ignore#

1 JBB-..., -..M
6 Oot m"ch har#er target 6 V$>E 8o8"lar, eas' eno"gh to 2s8am3 6 )core rarel' relate# to an'thing real

1 )8ecJA88)erver, !aCa8o, )8ecJVM-..H, CMLMark

6 Bigger, har#er to s8am, less 8o8"lar

;-..< A&"l )'stems, 4nc%

Macro-bench /arnings
///%a&"ls'stems%com

1 Bo8"lar ones are targete# b' com8anies 1 =eneral i#eaL JVM engineers are honest
6 B"t /ant the best for com8an' 6 )o #o targete# o8timi&ations 6 e%g% intrinsic CA) for >an#om%ne7t 6 Brobabl' "sef"l to somebo#' 6 Oever incorrect 6 !efinitel' hel8s this benchmark

;-..< A&"l )'stems, 4nc%

T'8ical Berformance T"ning C'cle

///%a&"ls'stems%com

1 1 1 1

Benchmark C becomes 8o8"lar Management tells $ngineerL 24m8rove C5s score03 $ngineer #oes an in-#e8th st"#' of C !eci#es o8timi&ation 2E3 /ill hel8
6 An# E is not broken for an'bo#' 6 Bossibl' hel8s some other 8rogram

1 4m8lements shi8s a JVM /ith 2E3 1 Management anno"nces score of 2C3 is no/ -SC 1 Rsers 'a/n in #isbeliefL 2E3 #oes not hel8 them

;-..< A&"l )'stems, 4nc%

)8ecJBB-...
///%a&"ls'stems%com

1 $mbarrassing 8arallel - no conten#e# locking 1 Oo 4(F, no #atabase, no ol#-gen =C

6 OFT t'8icall' of an' mi##le-/are 6 Very high allocation rate of 'o"ng-gen ob@ects, #efinitel' not

t'8icall' 6 B"t ma'be 'o"r 8rogram gets close[

1 Te' to 8erformanceL

having eno"gh Hea8 to avoi# ol#-gen =C #"ring +-min time# /in#o/

;-..< A&"l )'stems, 4nc%

)8ecJBB-...L )8amming
///%a&"ls'stems%com

1 !rove TFO) of s8eciali&e# =C behaviors

flags

6 Across man' ven#ors 6 Man' rolle# into 2-CCLZAggressiveF8timi&ations3 6 =oalL no ol#-gen =C in + min"tes

1 N-letter-com8an' 2s8amme#3 - /ith a D+-bit VM an# *-=ig

hea8 (in an era of N%M= ma7 hea8s)
*-=ig /as not t'8ical of the time 6 Force# )"n to make a D+-bit VM
6 M"ch more allocation, hence 2score3 before =C 6 Oote that /hile h"ge hea8s are genericall' "sef"l to somebo#',

;-..< A&"l )'stems, 4nc%

)8ecJBB-...L )8amming
///%a&"ls'stems%com

1 !rove TFO) of s8eciali&e# =C behaviors

flags

6 Across man' ven#ors 6 Man' rolle# into 2-CCLZAggressiveF8timi&ations3 6 =oalL no ol#-gen =C in + min"tes

1 N-letter-com8an' 2s8amme#3 - /ith a D+-bit VM an# *-=ig

hea8 (in an era of N%M= ma7 hea8s)
*-=ig /as not t'8ical of the time 6 Force# )"n to make a D+-bit VM
6 M"ch more allocation, hence 2score3 before =C 6 Oote that /hile h"ge hea8s are genericall' "sef"l to somebo#',

;-..< A&"l )'stems, 4nc%

Ahat can 'o" rea# from the res"lts[

///%a&"ls'stems%com

1 The closer 'o"r a88s resemble benchmark 2C3

6 The closer im8rovements to C5s score im8act 'o"

1 H"ge im8rovements to unrelated benchmarks

6 Might be worthless to you

1 e%g% )8ecJBB-... is a 8erfect-'o"ng-gen =C test

6 4m8rovements to JBB score have been tie# to better 'o"ng-gen

behavior 6 Most /eb-servers s"ffer from FL!-gen =C iss"es 6 4m8roving 'o"ng-gen #i#n5t hel8 /eb-servers m"ch

;-..< A&"l )'stems, 4nc%

)8ecJBB-..M
///%a&"ls'stems%com

1 4nten#e# to fi7 JBB-...5s =C iss"es

6 Oo e78licit =C bet/een time# /in#o/s 6 Benali&e score if =C 8a"se is too m"ch

(CTOs are #ela'e# too long) 6 )ame as JBB-..., b"t more CML 6 Oee#s some JavaD-isms o8timi&e#

1 )till embarrassing 8arallel ? 'o"ng-gen =C test 1 A&"l ran "8 to *<.. /areho"se(threa#s
on a NM.=ig hea8, allocating -.=igab'tes(sec for N%M #a's an# )T4LL no ol#-gen =C

;-..< A&"l )'stems, 4nc%

)8ecJBB-..M
///%a&"ls'stems%com

1 4nten#e# to fi7 JBB-...5s =C iss"es

6 Oo e78licit =C bet/een time# /in#o/s 6 Benali&e score if =C 8a"se is too m"ch

(CTOs are #ela'e# too long) 6 )ame as JBB-..., b"t more CML 6 Oee#s some JavaD-isms o8timi&e#

1 )till embarrassing 8arallel ? 'o"ng-gen =C test 1 A&"l ran "8 to *<.. /areho"se(threa#s
on a NM.=ig hea8, allocating -.=igab'tes(sec for N%M #a's an# )T4LL no ol#-gen =C

;-..< A&"l )'stems, 4nc%

)ome Bo8"lar Macro-Benchmarks

///%a&"ls'stems%com

1 )8ecJVMJH ? too small, no 4(F, no =C

6 --<Kmtrt ? too short to sa' anything 6 $sca8e Anal'sis 8a's off too /ell here 6 -.JK#b ? string-sort OFT #b, 8erformance tie# to TLB

str"ct"re, not JVM 6 ---Km8ega"#io ? s"b@ect to o## FB o8timi&ations 6 --HK@ack ? thro/s heav' e7ce8tions ? b"t so #o man' a88serversG also 8arsers are 8o8"lar% 4m8rovements here might carr' over 6 -*NK@avac ? genericall' "sef"l metric for mo#est CBR bo"n# a88lications

cache

;-..< A&"l )'stems, 4nc%

)ome Bo8"lar Macro-Benchmarks

///%a&"ls'stems%com

1 )8ecJVMJH ? too small, no 4(F, no =C

6 --<Kmtrt ? too short to sa' anything 6 $sca8e Anal'sis 8a's off too /ell here 6 -.JK#b ? string-sort OFT #b, 8erformance tie# to TLB

str"ct"re, not JVM 6 ---Km8ega"#io ? s"b@ect to o## FB o8timi&ations 6 --HK@ack ? thro/s heav' e7ce8tions ? b"t so #o man' a88serversG also 8arsers are 8o8"lar% 4m8rovements here might carr' over 6 -*NK@avac ? genericall' "sef"l metric for mo#est CBR bo"n# a88lications

cache

;-..< A&"l )'stems, 4nc%

)8ecJA88)erver
///%a&"ls'stems%com

1 1 1 1

Ver' har# to set"8 r"n Ver' net/ork, 4(F !B intensive

Oee# a #ecent (not great) JVM (e%g% =C is V MI) B"t 8eak score #e8en#s on an "ber-!B an# fast #isk or net/ork 1 Oot so heavil' o8timi&e# b' JVM $ngineers 1 Lots of 2fle73 in set"8 r"les (!B net/ork config)

1 )o har# to rea# the res"lts "nless 'o"r e7ternal (non-JVM)

set"8 is similar

;-..< A&"l )'stems, 4nc%

)8ecJA88)erver
///%a&"ls'stems%com

1 1 1 1

Ver' har# to set"8 r"n Ver' net/ork, 4(F !B intensive

Oee# a #ecent (not great) JVM (e%g% =C is V MI) B"t 8eak score #e8en#s on an "ber-!B an# fast #isk or net/ork 1 Oot so heavil' o8timi&e# b' JVM $ngineers 1 Lots of 2fle73 in set"8 r"les (!B net/ork config)

1 )o har# to rea# the res"lts "nless 'o"r e7ternal (non-JVM)

set"8 is similar

;-..< A&"l )'stems, 4nc%

!aCa8o
///%a&"ls'stems%com

1 1 1 1 1 1 1

Less 8o8"lar so less o8timi&e# >ealistic of mi#-si&e# BFJF a88s OFT t'8ical of a88-servers, J-$$ st"ff $78ect *...5s of classes loa#e# metho#s J4T5# )ome 4(F, more t'8ical =C behavior M"ch better score re8orting r"les !aCa8o "8gra#es coming soon0
6 Oe/ version has /eb-servers

8arallel co#es

;-..< A&"l )'stems, 4nc%

!aCa8o
///%a&"ls'stems%com

1 1 1 1 1 1 1

Less 8o8"lar so less o8timi&e# >ealistic of mi#-si&e# BFJF a88s OFT t'8ical of a88-servers, J-$$ st"ff $78ect *...5s of classes loa#e# metho#s J4T5# )ome 4(F, more t'8ical =C behavior M"ch better score re8orting r"les !aCa8o "8gra#es coming soon0
6 Oe/ version has /eb-servers

8arallel co#es

;-..< A&"l )'stems, 4nc%

)ome Bo8"lar Macro-Benchmarks

///%a&"ls'stems%com

1 CMLMark
6 Berf varies b' *.7 base# on CML 8arser J!T version 6 Too-/ell-behave# 'o"ng-gen allocation 6 Like !aCa8o ? more realistic of mi#-si&e# BFJF a88s
6 Ver' 8arallel (not a contention benchmark)

"nlike most a88-servers

1 )8ecJVM-..H
6 Also like !aCa8o ? realisticall' si&e# BFJF a88s 6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors

;-..< A&"l )'stems, 4nc%

)ome Bo8"lar Macro-Benchmarks

///%a&"ls'stems%com

1 CMLMark
6 Berf varies b' *.7 base# on CML 8arser J!T version 6 Too-/ell-behave# 'o"ng-gen allocation 6 Like !aCa8o ? more realistic of mi#-si&e# BFJF a88s
6 Ver' 8arallel (not a contention benchmark)

"nlike most a88-servers

1 )8ecJVM-..H
6 Also like !aCa8o ? realisticall' si&e# BFJF a88s 6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors

;-..< A&"l )'stems, 4nc%

2Bo8"lar3 Macro-Benchmark Broblems

///%a&"ls'stems%com

1 Rnrealistic treatment of =C
6 e%g% Oone in time# /in#o/ 6 Fr 8erfect 'o"ng-gen collections 6 >eal a88s t'8ical trigger f"ll =C ever' ho"r or so

1 Rnrealistic loa# generation

6 Oot eno"gh loa# to stress s'stem 6 Fr ver' sim8le or re8etitive loa#s 6 Bottlenecks in getting loa# to server

;-..< A&"l )'stems, 4nc%

2Bo8"lar3 Macro-Benchmark Broblems

///%a&"ls'stems%com

1 Benchmark too short for f"ll =C

6 Man' real a88lications leak 6 Broken Nr# 8art' libs, legac' co#e, etc 6 Leaks acc"m"late in ol#-gen 6 Ahich makes ol#-gen f"ll =C e78ensive 6 B"t benchmark never triggers ol#-gen f"ll =C

1 4(F

!B not benchmarke# /ell

6 B"t make a h"ge #ifference in >eal Life 6 Eo"r a88 might share 4(F !B /ith others

;-..< A&"l )'stems, 4nc%

)"mmar'
///%a&"ls'stems%com

1 Macrobenchmarks
6 Targete# b' JVM $ngineers 6 Buyer Beware! 6 The closer the benchmark is to 'o"r 8roblem 6 The more likel' im8rovements /ill im8act 'o" 6 =C is likel' to not be typical of real a88lications 6 Your a88lications ever go N%M #a's /itho"t a f"ll =C[ 6 4(F !B loa# also 8robabl' not t'8ical

;-..< A&"l )'stems, 4nc%

)"mmar'
///%a&"ls'stems%com

1 Microbenchmarks
$as' to Arite, Har# to get >ight Easy to be Fooled Aon5t tell 'o" m"ch abo"t macro-co#e an'/a's ar!u" ? *5s of secon#s to *.5s of secon#s Statistics ? average lots of r"ns 6 $ven o"t variations in the 2com8ile 8lan3 6 Call out to man' metho#s in the hot loo8 6 Be /ar' of dead#code s"8er-score res"lts
6 6 6 6 6

;-..< A&"l )'stems, 4nc%

///%a&"ls'stems%com

B"t Micro-Tr"st in a Micro-Benchmark0

;-..< A&"l )'stems, 4nc%

///%a&"ls'stems%com

B"t Micro-Tr"st in a Micro-Benchmark0

;-..< A&"l )'stems, 4nc%