8lackbox vs WhlLebox 1esLlng

Chooslng Lhe 8lghL Approach Lo uellver CuallLy AppllcaLlons

CVLkVILW
WlLhln Lhe auLomaLed LesLlng world Lhere are Lwo predomlnaLe LesLlng meLhodologles blackbox and whlLebox 1hls paper seeks Lo
explore Lhe pros and cons of boLh approaches and Lo ldenLlfy when each approach should be used Lo ensure quallLy appllcaLlons are
dellvered Lo markeL
ln Lhe end Lhls paper concludes LhaL whlle blackbox LesLlng has had lLs drawbacks ln Lhe pasL lnnovaLlve approaches Lo blackbox
LesLlng make lL Lhe llkely cholce Lo deal wlLh Lhe ever lncreaslng complexlLy of appllcaLlons and dellvers lower 1oLal CosL of
Cwnershlp (1CC) and a beLLer 8eLurn on lnvesLmenL (8Cl) Lo organlzaLlons
DLIINI1ICNS
8/ockbox 1hls LesLlng meLhodology looks aL whaL are Lhe avallable lnpuLs for an appllcaLlon and whaL Lhe expecLed ouLpuLs are LhaL
should resulL from each lnpuL lL ls noL concerned wlLh Lhe lnner worklngs of Lhe appllcaLlon Lhe process LhaL Lhe appllcaLlon
underLakes Lo achleve a parLlcular ouLpuL or any oLher lnLernal aspecL of Lhe appllcaLlon LhaL may be lnvolved ln Lhe LransformaLlon
of an lnpuL lnLo an ouLpuL MosL blackbox LesLlng Lools employ elLher coordlnaLe based lnLeracLlon wlLh Lhe appllcaLlons graphlcal
user lnLerface (Cul) or lmage recognlLlon An example of a blackbox sysLem would be a search englne ?ou enLer LexL LhaL you wanL
Lo search for ln Lhe search bar press Search" and resulLs are reLurned Lo you ln such a case you do noL know or see Lhe speclflc
process LhaL ls belng employed Lo obLaln your search resulLs you slmply see LhaL you provlde an lnpuL a search Lerm and you
recelve an ouLpuL your search resulLs
whitebox 1hls LesLlng meLhodology looks under Lhe covers and lnLo Lhe subsysLem of an appllcaLlon Whereas blackbox LesLlng
concerns lLself excluslvely wlLh Lhe lnpuLs and ouLpuLs of an appllcaLlon whlLebox LesLlng enables you Lo see whaL ls happenlng
lnslde Lhe appllcaLlon WhlLebox LesLlng provldes a degree of sophlsLlcaLlon LhaL ls noL avallable wlLh blackbox LesLlng as Lhe LesLer
ls able Lo refer Lo and lnLeracL wlLh Lhe ob[ecLs LhaL comprlse an appllcaLlon raLher Lhan only havlng access Lo Lhe user lnLerface An
example of a whlLebox sysLem would be lnclrculL LesLlng where someone ls looklng aL Lhe lnLerconnecLlons beLween each
componenL and verlfylng LhaL each lnLernal connecLlon ls worklng properly AnoLher example from a dlfferenL fleld mlghL be an
auLomechanlc who looks aL Lhe lnnerworklngs of a car Lo ensure LhaL all of Lhe lndlvldual parLs are worklng correcLly Lo ensure Lhe
car drlves properly
1he maln dlfference beLween blackbox and whlLebox LesLlng ls Lhe areas on whlch Lhey choose Lo focus ln slmplesL Lerms black
box LesLlng ls focused on resulLs lf an acLlon ls Laken and lL produces Lhe deslred resulL Lhen Lhe process LhaL was acLually used Lo
achleve LhaL ouLcome ls lrrelevanL WhlLebox LesLlng on Lhe oLher hand ls concerned wlLh Lhe deLalls lL focuses on Lhe lnLernal
worklngs of a sysLem and only when all avenues have been LesLed and Lhe sum of an appllcaLlon's parLs can be shown Lo be
conLrlbuLlng Lo Lhe whole ls LesLlng compleLe
1nL kCS AND CCNS
8lackbox 1here are many advanLages Lo blackbox LesLlng Pere are a few of Lhe mosL commonly clLed
1 Lase of use 8ecause LesLers do noL have Lo concern Lhemselves wlLh Lhe lnner worklngs of an appllcaLlon lL ls easler Lo creaLe
LesL cases by slmply worklng Lhrough Lhe appllcaLlon as would an end user
2 u|cker test case deve|opment 8ecause LesLers only concern Lhemselves wlLh Lhe Cul Lhey do noL need Lo spend Llme
ldenLlfylng all of Lhe lnLernal paLhs LhaL may be lnvolved ln a speclflc process Lhey need only concern Lhemselves wlLh Lhe varlous
paLhs Lhrough Lhe Cul LhaL a user may Lake
3 S|mp||c|ty Where large hlghly complex appllcaLlons or sysLems exlsL blackbox LesLlng offers a means of slmpllfylng Lhe LesLlng
process by focuslng on valld and lnvalld lnpuLs and ensurlng Lhe correcL ouLpuLs are recelved
8uL for all of Lhe beneflLs of blackbox LesLlng many aLLempLs Lo creaLe blackbox LesL sysLems resulLed ln several drawbacks
LhaL caused people Lo quesLlon Lhe vlablllLy of Lhe blackbox approach Some of Lhe mosL commonly clLed lssues were
1 Scr|pt ma|ntenance Whlle an lmagebased approach Lo LesLlng ls useful lf Lhe user lnLerface ls consLanLly changlng Lhe lnpuL
may also be changlng 1hls makes scrlpL malnLenance very dlfflculL because blackbox Lools are rellanL on Lhe meLhod of lnpuL
belng known
2 Irag|||ty lnLeracLlng wlLh Lhe Cul can also make LesL scrlpLs fraglle 1hls ls because Lhe Cul may noL be rendered conslsLenLly
from LlmeLoLlme on dlfferenL plaLforms or machlnes unless Lhe Lool ls capable of deallng wlLh dlfferences ln Cul renderlng
lL ls llkely LhaL LesL scrlpLs wlll fall Lo execuLe properly on a conslsLenL basls
3 Lack of |ntrospect|on lronlcally one of Lhe greaLesL crlLlclsm of blackbox LesLlng ls LhaL lL lsn'L more llke whlLebox LesLlng
LhaL lL doesn'L know how Lo look lnslde an appllcaLlon and Lherefore can never fully LesL an appllcaLlon or sysLem 1he reasons
clLed for needlng Lhls capablllLy are ofLen Lo overcome Lhe flrsL Lwo lssues menLloned 1he reallLy ls qulLe dlfferenL
whitebox lke blackbox LesLlng Lhere are dlsLlncL advanLages Lo whlLebox LesLlng Pere are a few of Lhe mosL commonly
clLed
1 Introspect|on lnLrospecLlon or Lhe ablllLy Lo look lnslde Lhe appllcaLlon means LhaL LesLers can ldenLlfy ob[ecLs
programmaLlcally 1hls ls helpful when Lhe Cul ls changlng frequenLly or Lhe Cul ls yeL unknown as lL allows LesLlng Lo
proceed lL also can ln some slLuaLlons decrease Lhe fraglllLy of LesL scrlpLs provlded Lhe name of an ob[ecL does noL change
2 Stab|||ty ln reallLy a byproducL of lnLrospecLlon whlLebox LesLlng can dellver greaLer sLablllLy and reusablllLy of LesL cases lf
Lhe ob[ecLs LhaL comprlse an appllcaLlon never change
3 1horoughness ln slLuaLlons where lL ls essenLlal Lo know LhaL every paLh has been Lhoroughly LesLed LhaL every posslble
lnLernal lnLeracLlon has been examlned whlLebox LesLlng ls Lhe only vlable meLhod As such whlLebox LesLlng offers LesLers
Lhe ablllLy Lo be more Lhorough ln Lerms of how much of an appllcaLlon Lhey can LesL
uesplLe Lhese beneflLs whlLebox LesLlng has lLs drawbacks Some of Lhe mosL commonly clLed lssues are
1 Comp|ex|ty 8elng able Lo see every consLlLuenL parL of an appllcaLlon means LhaL a LesLer musL have deLalled
programmaLlc knowledge of Lhe appllcaLlon ln order Lo work wlLh lL properly 1hls hlghdegree of complexlLy requlres a much
more hlghly skllled lndlvldual Lo develop LesL case
2 Irag|||ty Whlle lnLrospecLlon ls supposed Lo overcome Lhe lssue of appllcaLlon changes breaklng LesL scrlpLs Lhe reallLy ls
LhaL ofLen Lhe names of ob[ecLs change durlng producL developmenL or new paLhs Lhrough Lhe appllcaLlon are added 1he
facL LhaL whlLebox LesLlng requlres LesL scrlpLs Lo be LlghLly Lled Lo Lhe underlylng code of an appllcaLlon means LhaL changes
Lo Lhe code wlll ofLen cause whlLebox LesL scrlpLs Lo break 1hls Lhen lnLroduces a hlgh degree of scrlpL malnLenance lnLo
Lhe LesLlng process
3 Integrat|on lor whlLebox LesLlng Lo achleve Lhe degree of lnLrospecLlon requlred lL musL be LlghLly lnLegraLed wlLh Lhe
appllcaLlon belng LesLed 1hls creaLes a few problems 1o be LlghLly lnLegraLed wlLh Lhe code you musL lnsLall Lhe whlLebox
Lool on Lhe sysLem on whlch Lhe appllcaLlon ls runnlng 1hls ls okay buL where one wlshes Lo ellmlnaLe Lhe posslblllLy LhaL Lhe
LesLlng Lool ls whaL ls causlng elLher a performance or operaLlonal problem Lhls becomes lmposslble Lo resolve AnoLher lssue
LhaL arlses ls LhaL of plaLform supporL uue Lo Lhe hlghly lnLegraLed naLure of whlLebox LesLlng Lools many do noL provlde
supporL for more Lhan one plaLform usually Wlndows Where companles have appllcaLlons LhaL run on oLher plaLforms
Lhey elLher need Lo use a dlfferenL Lool or resorL Lo manual LesLlng
NLCLSSAk VS NICL 1C nAVL
When approachlng auLomaLed LesLlng lL ls lmporLanL Lo undersLand whaL ls necessary vs whaL ls nlce Lo have 8oLh LesLlng
meLhodologles have Lhelr merlL 1o deLermlne whaL approach should llkely be used Lhere are a few quesLlons LhaL every
company should ask
1 Who wlll use Lhe appllcaLlon?
2 WhaL parLs of Lhe appllcaLlon musL be LesLed prlor Lo release and why?
3 WhaL language wlll my appllcaLlon be wrlLLen ln?
4 When are slgnlflcanL changes Lo Lhe ul llkely Lo be made and wlll Lhe underlylng code be affecLed?
3 Where ls Lhe appllcaLlon llkely Lo be deployed?
6 Pow wlll Lhe appllcaLlon be used?
7 Whlch plaLforms does Lhe appllcaLlon need Lo supporL?
ooklng aL each of Lhese quesLlons some of Lhe answers LhaL you mlghL come up wlLh are as follows
1 Lnd users
2 Any parL of Lhe appllcaLlon LhaL wlll be exposed Lo Lhe end user
3 !ava uslng an Lcllpse framework
4 Annually because our cusLomers are always looklng for a new fresh experlence
3 Cn Lhe web
6 CusLomers wlll logln vla Lhe web and wlll enLer lnformaLlon Lhrough a serles of screens
7 Web browsers llke llrelox lL Cpera and Safarl
8ased on Lhe answers glven one can make an educaLed declslon around Lhe Lype of Lool LhaL should be used ln Lhe case of Lhe
appllcaLlon descrlbed above a blackbox Lool would llkely be more effecLlve as lL ls usercenLrlc ln lLs approach focuses on
LesLlng Lhe lnLerface raLher Lhan Lhe underlylng code and a blackbox Lool ls more llkely Lo supporL Lhe mulLlple plaLforms
requlred
8evlewlng Lhe quesLlons suggesLed above does expose a more fundamenLal quesLlon abouL Lhe naLure of LesLlng lLself Why
does anyone LesL? 1he quesLlon ls noL as sllly as lL sounds and lL reveals a loL abouL Lhe longLerm vlablllLy of Lhe Lwo approaches
belng explored hereln
All companles LesL Lhelr appllcaLlons prlor Lo release because Lhelr cusLomers are lnLoleranL of bugs So lL ls ouL of a need Lo
saLlsfy cusLomers LhaL LesLlng ls underLaken ln Lhe flrsL place Clven Lhls loglc follows LhaL Lo LesL ln a manner LhaL reflecLs how a
cusLomer wlll use an appllcaLlon should be a prerequlslLe lf Lhls ls Lrue Lhen only when blackbox LesLlng has been applled Lo
an appllcaLlon can LesLlng Lruly be sald Lo have been compleLed
1hls ls an lnLeresLlng argumenL and one LhaL ls unllkely Lo slL well wlLh whlLebox Lool vendors Powever leL's revlew whaL we
know CusLomers use Culs noL code 1hey enLer lnformaLlon or lnLeracL wlLh an appllcaLlon (Lhe lnpuL) ln some fashlon and walL
Lo geL someLhlng back (Lhe ouLpuL) lf Lhe process works and Lhey geL an accepLable resulL Lhey are happy lf noL Lhey
experlence a problem 1hls ls ldenLlcal Lo Lhe approach employed by blackbox LesLlng Lools As such lL would seem loglcal LhaL a
blackbox Lool should be used now LhaL ls noL Lo say LhaL whlLebox Lools Lo do noL have Lhelr place lL ls slmply Lo say LhaL Lhey
do noL provlde sufflclenL LesL coverage on Lhelr own for an organlzaLlon Lo say LhaL an appllcaLlon has been fully LesLed
WhlLebox Lool vendors would ob[ecL sLrongly Lo Lhe prevlous sLaLemenL 1hey would clLe Lhelr lnLrospecLlve capablllLles as
belng superlor Lo a blackbox LesLlng approach Powever whaL a whlLebox vendor cannoL guaranLee ls LhaL whaL occurs aL Lhe
codelevel wlll be properly dlsplayed on Lhe ul AfLer all Lhe properLy of an ob[ecL mlghL be seL Lo vlslble" buL due Lo a faulL ln
anoLher parL of Lhe appllcaLlon or even due Lo a problem ouLslde of Lhe appllcaLlon Lhe ob[ecL mlghL noL appear Lo Lhe user A
whlLebox Lool would record Lhe sLaLe change as havlng passed because Lhe vlslble" properLy was successfully updaLed 1he
facL LhaL Lhe user cannoL acLually see whaL Lhey are supposed Lo ls largely overlooked Lven where whlLebox vendors have
lnLroduced elemenLs of lmage recognlLlon Lo overcome Lhe vlslble" problem hlghllghLed above whlLebox Lools sLlll Lend Lo
suffer from an lnablllLy Lo deal wlLh perlodlc dlfferences ln lmage renderlng
AlLhough lssues wlLh whlLebox LesLlng have been ldenLlfled above lL should noL be LreaLed as a foregone concluslon LhaL black
box LesLlng provlde comprehenslve appllcaLlon LesLlng coverage CulLe Lhe conLrary whaL ls hopefully evldenL aL Lhls sLage ls
LhaL only when blackbox and whlLe box LesLlng meLhodologles are comblned ls comprehenslve LesL coverage achleved 1he
reason for Lhls ls LhaL boLh meLhodologles address dlfferenL aspecLs of appllcaLlon LesLlng lLself lor blackbox LesLlng Lhe focus
ls on Lhe user experlence whereas whlLebox LesLlng focuses on Lhe lnLernal and maklng sure LhaL Lhe appllcaLlon works as
efflclenLly as posslble (or aL leasL as deslgned) 1herefore Lhese Lwo meLhodologles can be seen as compllmenLary and for
organlzaLlons LhaL have Lhe budgeL and Llme avallable Lo Lake advanLage of boLh Lhey should cerLalnly do so
LACkCk 1LS1ING A NLCLSSAk S1L
So far Lhls poslLlon plece has hlghllghL Lhe pros and cons of boLh blackbox and whlLe box approaches Lo LesL
auLomaLlon and dlscussed when each approach mlghL be mosL approprlaLe ln Lhe lasL secLlon Lhe ldea of Llme and money was
lnLroduced recognlzlng LhaL organlzaLlons have llmlLed quanLlLles of boLh WlLh Lhose consLralnLs ln mlnd lL ls prudenL Lo revlslL
Lhe dlscusslon of whlch approach provldes Lhe besL LesL coverage for Lhe lowesL 1CC and greaLesL 8Cl
ooklng aL Lhe merlLs of boLh blackbox and whlLebox LesLlng whaL seems Lo sLand ouL ls LhaL blackbox LesLlng ls focused on Lhe
end user LhaL underLaklng blackbox LesLlng ls Lhe besL way of ensurlng LhaL Lhose parLs of Lhe appllcaLlons LhaL wlll be exposed
Lo Lhe user work correcLly Comblned wlLh Lhe ease of use qulcker LesL case developmenL and slmpllclLy blackbox LesLlng
represenLs a lower lnlLlal cosL Lhan whlLebox LesLlng and dellvers 8Cl ln a shorLer perlod of Llme
8ecognlzlng LhaL Lo be Lhe case ln Lhe face of budgeLary and Llme consLralnLs blackbox LesLlng musL be consldered a necessary
sLeps ln quallLy assurance process whlle whlLebox LesL represenLs a nlce Lo have
SLLLC1ING A LACkCk 1LS1ING 1CCL
When selecLlng a blackbox LesLlng Lool Lhere are several Lhlngs Lo look for
1 WhaL plaLforms does lL supporL?
2 Pow reslllenL ls lL Lo dlfferences ln Lhe way lmages are rendered?
3 WhaL faclllLles exlsL Lo ald ln lmage recapLure?
4 Pow does lL ldenLlfy lmages (by coordlnaLes or lmage recognlLlon)?
8y asklng Lhese quesLlons and many more you wlll be able Lo ldenLlfy Lhe blackbox Lool LhaL ls rlghL for you We would
recommend LhaL you Lry LggplanL as lL dellvers all of Lhe beneflLs of blackbox LesLlng whlle overcomlng nearly all of Lhe clLed
llmlLaLlons
Whlchever Lool you choose hopefully you recognlze Lhe lmporLance of blackbox LesLlng ln ensurlng you are dellverlng
appllcaLlons LhaL have been LesLed as Lhey wlll be used and meeL cusLomer needs (or aL leasL as deslgned) 1herefore Lhese Lwo
meLhodologles can be seen as compllmenLary and for organlzaLlons LhaL have Lhe budgeL and Llme avallable Lo Lake advanLage
of boLh Lhey should cerLalnly do so
LACkCk 1LS1ING A NLCLSSAk S1L
So far Lhls poslLlon plece has hlghllghL Lhe pros and cons of boLh blackbox and whlLe box approaches Lo LesL
auLomaLlon and dlscussed when each approach mlghL be mosL approprlaLe ln Lhe lasL secLlon Lhe ldea of Llme and money was
lnLroduced recognlzlng LhaL organlzaLlons have llmlLed quanLlLles of boLh WlLh Lhose consLralnLs ln mlnd lL ls prudenL Lo revlslL
Lhe dlscusslon of whlch approach provldes Lhe besL LesL coverage for Lhe lowesL 1CC and greaLesL 8Cl
ooklng aL Lhe merlLs of boLh blackbox and whlLebox LesLlng whaL seems Lo sLand ouL ls LhaL blackbox LesLlng ls focused on Lhe
end user LhaL underLaklng blackbox LesLlng ls Lhe besL way of ensurlng LhaL Lhose parLs of Lhe appllcaLlons LhaL wlll be exposed
Lo Lhe user work correcLly Comblned wlLh Lhe ease of use qulcker LesL case developmenL and slmpllclLy blackbox LesLlng
represenLs a lower lnlLlal cosL Lhan whlLebox LesLlng and dellvers 8Cl ln a shorLer perlod of