IBM Beaverton Linux Change Team

Using SystemTap with Linux on System z

Mi e !"#ei$$y

2012 IBM Corporation

IBM Beaverton Linux Change Team

SystemTap
S%ripting $anguage an& too$s

&ynami% tra%ing'pro(ing
) ) ) ) erne$ *un%tions system %a$$s +erne$,spa%e events User,spa%e events - newest versions .

I&enti*ying the un&er$ying %ause o* a (ug /er*orman%e pro($ems

0$iminate instrument1 re%ompi$e1 insta$$1 an& re(oot se2uen%e

2012 IBM Corporation

IBM Beaverton Linux Change Team

Insta$$ing SystemTap ) #e&3at

systemtap systemtap,runtime erne$,&e(ugin*o erne$,&e(ugin*o,%ommon,ar%h erne$,&eve$ Test

stap ,v ,e "pro(e v*s4rea& 5print*-6 rea& per*orme&7n8.9 exit-.:"

/ass 1; parse& user s%ript < < /ass =; starting run4 #ea& per*orme& /ass =; run %omp$ete& in 10usr'>0sys'?2@ rea$ ms4
2012 IBM Corporation

IBM Beaverton Linux Change Team

Insta$$ing SystemTap , Aove$$

systemtap +erne$,sour%e +erne$,&e*au$t,&e(ugin*o Beve$opment pa% ages CC erne$ (ui$& environment Test

stap ,v ,e "pro(e v*s4&oDsyn%Drea& 5print*-6rea& per*orme&7n8.9 exit-.:"

2012 IBM Corporation

IBM Beaverton Linux Change Team

SystemTap"s s%ripting $anguage

/ro%e&ura$ C,$i e Integers1 Strings1 Esso%iative arrays1 Statisti%s aggregates Imp$i%it$y type& Base& on Two main *un%tion %onstru%ts;
/ro(es Fun%tions ) statements an& expressions use C,$i e operator syntax an& pre%e&en%e

2012 IBM Corporation

IBM Beaverton Linux Change Team

/rimary %onstru%t; pro(e

pro(e GeventH 5 han&$er :

event is
) ) ) ) erne$4*un%tion1 pro%ess4statement1 timer4ms1 (egin1 en& -tapset a$iases.4

han&$er %an have;

) varia($es ) *i$tering'%on&itiona$s -i* 444 next. ) %ontro$ stru%tures -*orea%h1 whi$e.

2012 IBM Corporation

IBM Beaverton Linux Change Team

/ro(e examp$e
I %at simp$e4stp
IJ'usr'(in'stap pro(e (egin 5print*-K/ro(e starte&7nK.9: pro(e timer4se%-@. 5exit-.9: pro(e en& 5print*-K/ro(e en&e&7nK.9:

0vents; (egin1 timer4se%1 en& 3an&$ers; print*-.1 exit-.

2012 IBM Corporation

IBM Beaverton Linux Change Team

pro(e examp$e %ont4

I stap simp$e4stp /ro(e starte& < @ se%on&s $ater 444 /ro(e en&e& The stap program The *ront,en& to the SystemTap too$4

Trans$ates the s%ript into C %o&e Compi$es C an& Lenerates a erne$ mo&u$e Inserts the mo&u$e9

!utput to stap"s st&out CT#L,C un$oa&s the mo&u$e1 terminates stap

2012 IBM Corporation

IBM Beaverton Linux Change Team

pro(e examp$e
I %at siga$tsta% 4stp
pro(e erne$4*un%tion-KsysDsiga$tsta% K. 5
print*-KsysDsiga$tsta% %a$$e&7nK.9 :

0vent; erne$4*un%tion-KsysDsiga$tsta% K. 3an&$er; 5 print*-KsysDsiga$tsta% %a$$e&7nK.9 :

To spe%i*y the return o* the erne$ *un%tion 0vent; erne$4*un%tion-KsysDsiga$tsta% K.4return

2012 IBM Corporation

IBM Beaverton Linux Change Team

pro(e 0vent examp$es

sys%a$$4rea&
when entering rea&-. system %a$$

sys%a$$4%$ose4return
when returning *rom the %$ose-. system %a$$

mo&u$e-KC&as&CK.4*un%tion-KCK.
when entering any *un%tion in the K&as&K mo&u$e

erne$4*un%tion-KCMnet'so% et4%K.4return
returning *rom any *un%tion in *i$e net'so% et4%

erne$4statement-KCM erne$'s%he&4%;2N1>K.
when hitting $ine 2N1> o* *i$e erne$'s%he&4%
2012 IBM Corporation

IBM Beaverton Linux Change Team

/ro(e 0vent ex4 %ont4

I %at &as&D%a$$graph4stp pro(e mo&u$e-KC&as&CK.4*un%tion-KCM&rivers's@N0'($o% '&as&4%K.4%a$$ 5 print* -KOs ,H Os7nK1 threa&Din&ent-1.1 pro(e*un%-.. : pro(e mo&u$e-KC&as&CK.4*un%tion-KCM&rivers's@N0'($o% '&as&4%K.4return 5 print* -KOs G, Os7nK1 threa&Din&ent-,1.1 pro(e*un%-.. : I stap &as&D%a$$graph4stp 0 (ash-@?NPN.; ,H &as&Dgeneri%DsetDo**$ine P0 (ash-@?NPN.; ,H &as&DsetDtargetDstate 100 (ash-@?NPN.; ,H &as&D%hangeDstate @@Q (ash-@?NPN.; ,H &as&D*$ushD($o% D2ueue @=> (ash-@?NPN.; G, &as&D*$ushD($o% D2ueue 21PNP (ash-@?NPN.; ,H &as&Dre$ease 21N2= (ash-@?NPN.; G, &as&Dre$ease =@Q1= (ash-@?NPN.; ,H &as&D($o% D%$earDtimer =@Q?0 (ash-@?NPN.; G, &as&D($o% D%$earDtimer

2012 IBM Corporation

IBM Beaverton Linux Change Team

pro(e 0vents %ont4

timer4ms-200.
every 200 mi$$ise%on&s

pro%ess-K'(in'$sK.4*un%tion-KCK.
entering any *un%tion in '(in'$s -not its $i(raries or sys%a$$s.

pro%ess-K'$i('$i(%4so4QK.4*un%tion-KCma$$o%CK.
entering any g$i(% *un%tion with Kma$$o%K in its name

erne$4*un%tion-KCinitCK.1 erne$4*un%tion-KCexitCK.4return
entering any erne$ *un%tion whi%h has KinitK in its name or returning *rom any erne$ *un%tion with KexitK in its name

2012 IBM Corporation

IBM Beaverton Linux Change Team

pro(e 0vents %ont4

!ptiona$ pro(es

erne$4*un%tion-6may not exist8. R 5 444 : erne$4*un%tion-6this might exist8. J1 erne$4*un%tion-6i* not then this shou$&8. J1 erne$4*un%tion-6i* a$$ e$se *ai$s8. 5 444 :

Con&itiona$ pro(es

pro(e erne$4*un%tion-Ksome *un%K. i* - va$ H 10.

Fi$ter

stap ,e ,x /IB "pro(e sys%a$$4C 5 i* -pi&-. SS target-.. print*-6Os7n81name.:

2012 IBM Corporation

IBM Beaverton Linux Change Team

3an&$er Constru%ts
Taria($es

S%ript Target

Con&itiona$ Statements Loops

2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es
S%ript varia($es

L$o(a$ or $o%a$ Eutomati%a$$y type&; type in*erre& *rom assignment ) Integers -Q?,(it signe&. ) Strings ) Esso%iative arrays -g$o(a$ on$y.1 ) Statisti%s aggregates -g$o(a$ on$y. Eutomati%a$$y initia$ize& to zero'empty

2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es examp$e
I %at vars4stp g$o(a$ xSQ? g$o(a$ arr pro(e (egin 5 i S 10 name S KMi eK arrU0V S 1 arrU2V S @ print*-K x; O&7n i; O&7n name; Os7nK1 x1i1name. *orea%h- y in arr . 5 print*-Ky; O& arrUyV; O&7nK1 y1 arrUyV. : exit-. : I stap vars4stp x; Q? i; 10 name; Mi e y; 0 arrUyV; 1 y; 2 arrUyV; @
2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es ) Erray examp$e

I %at p*au$tBy/ro%ess4stp g$o(a$ numFau$ts pro(e vm4page*au$t5 numFau$tsU exe%name-. V WS 1 : pro(e timer4s-=. 5 print* -KO1Qs7tO10s7t7nK1 K/ro%essK1 KAum page*au$tsK. *orea%h -name in numFau$ts, . 5 print* -KO1Qs7tO&7nK1 name1 numFau$tsUnameV . : exit-. : I stap p*au$tBy/rot%ess4stp /ro%ess Aum page*au$ts ps @00 (ash Q> stapio 10
2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es ) Stat Eggregate examp$e

g$o(a$ Aum#ea&s pro(e v*s4rea& 5 Aum#ea&sUexe%name-.V GGG X%ount : pro(e timer4s-=. 5 *orea%h -name in Aum#ea&s . 5 print* -KO1Qs7tO&7nK1 name1 M%ount-Aum#ea&sUnameV. . : *orea%h -name in Aum#ea&s . 5 print* -KO1Qs7tO&7nK1 name1 Msum-Aum#ea&sUnameV. . : exit-. : /ro%ess %ron& rsys$og& /ro%ess %ron& rsys$og& Aum(er rea&s ? 1 Tota$ Bytes rea& 1Q@P? ?0N=
2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es Cont4
Target varia($es

Taria($es &e*ine& in the sour%e %o&e at event $o%ation

int 2ethDsetupD%hanne$-stru%t 2ethD%hanne$C %hanne$. 5 int %nt9 X%hanne$1 X%nt

spe%ia$ varia($es ) e4g41 Xreturn1 XXparms1 XXvars For pointers to (ase types su%h as integers an& strings ) erne$D$ong-a&&ress.1 erne$Dstring-a&&ress. *or sa*e a%%ess to varia($e va$ues4

I stap ,L "mo&u$e-KC&as&CK.4*un%tion-K&as&Da$$o%D2ueueK." mo&u$e-K&as&Dmo&K.4*un%tion-K&as&Da$$o%D2ueueM&rivers's@N0'($o% '&as&4%;2 1P2K. X($o% ;stru%t &as&D($o% C

2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es Cont4
I stap ,e "pro(e mo&u$e-KC&as&CK.4*un%tion-K&as&Da$$o%D2ueueK. 5print*-KOs7nK1 XXparms.9 exit-.9 :" ($o% S0x@eee@P00

6X8 su**ix to pretty print the &ata stru%ture4 I stap ,e "pro(e mo&u$e-KC&as&CK.4*un%tion-K&as&Da$$o%D2ueueK. 5print*-KOs7nK1 XXparmsX.9 exit-.9 :" ($o% S54g&pS0x01 4re2uestD2ueueS0x01 4441 4(aseS0x@ee%@?001 4%%wD2ueueS5444:1 42ueueD$o% S5444:1 444: 6XX8 su**ix wi$$ print the va$ues within the neste& &ata stru%tures I stap ,e "pro(e mo&u$e-KC&as&CK.4*un%tion-K&as&Da$$o%D2ueueK. 5print*-KOs7nK1 XXparmsXX.9 exit-.9 :" ($o% S54g&pS0x01 4re2uestD2ueueS0x0 14441 4%%wD2ueueS54nextS0x@*1@PP?01 4prevS0x@*1@PP?0:1 42ueueD$o% S54rawD$o% S54ownerD%puS0::1444
2012 IBM Corporation

IBM Beaverton Linux Change Team

Taria($es Cont4
I stap ,L " erne$4*un%tion-KsysDsiga$tsta% K." erne$4*un%tion-KSySDsiga$tsta% Mar%h's@N0' erne$'signa$4%;10QK. Xuss;$ong int Xuoss;$ong int uss -%onst sta% Dt C. points to a signa$sta% stru%ture I %at siga$tsta% 4stp pro(e erne$4*un%tion-KsysDsiga$tsta% K. 5 print*- KOs7nK1 XXparmsX .9: I stap siga$tsta% 4stp ussS2102012Q?0 uossS0 ussS210202N02? uossS0 ussS21??01012P uossS0 ussS?@0201NQPP uossS0

2012 IBM Corporation

IBM Beaverton Linux Change Team

S%ript an& Target varia($es g$o(a$ openFai$s1 hugeDrea&s pro(e erne$4*un%tion-KsysDopenK.4return 5 i* -Xreturn G 0. openFai$sWW9 : pro(e erne$4*un%tion-KsysDrea&K. 5 i* -X%ount H ?C102?. hugeDrea&sWW9 :

S%ript varia($es; openFai$s1 hugeDrea&s Target varia($e; X%ount ) sysDrea&-."s @r& arg Spe%ia$ %ontext varia($e; Xreturn
2012 IBM Corporation

IBM Beaverton Linux Change Team

Con&itiona$'Loop statements
Lroup %ompoun& statements with 5 : Bran%hing
i* -%on&ition. statement1 Ue$se statement2V Looping;

whi$e -%on&ition. statement *or -initia$9 %on&ition9 iteration. statement *orea%h -UTE#11 TE#2V in E##EY U$imit AUMV. statement

(rea 9 %ontinue9 !ther;

return UTELV9 next9 &e$ete TE#9

2012 IBM Corporation

IBM Beaverton Linux Change Team

Con&itiona$'Loop Statements 0xamp$e

i* -*$ag Z C#0ETDFLEL. return 1 e$se return 0 *or-iS09iG109iWW. 5 444 : whi$e -iG10. 5 444 : *orea%h -item in myarr. 5 myarrUitemVWW :

2012 IBM Corporation

IBM Beaverton Linux Change Team

/rimary %onstru%t; *un%tion

*un%tion AEM0;type-E#L1;type1 E#L2;type. 5 'C %o&e to run when AEM0 is %a$$e& C'

return TELU0
:

The ;types are optiona$1 an& may (e

;string ;$ong4

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e; *un%tion
I %at *un%4stp *un%tion isDopenD%reating;$ong -*$ag;$ong.5 C#0ETDFLEL S ? '' 0x? S 00000100( i* -*$ag Z C#0ETDFLEL. return 1 e$se return 0 : pro(e erne$4*un%tion-KsysDopenK.5 %reating S isDopenD%reating-Xmo&e. i* -%reating. print*-KCreating *i$e Os7nK1 userDstring-X*i$ename.. e$se print*-K!pening *i$e Os7nK1 userDstring-X*i$ename.. : I stap *un%4stp !pening *i$e pu($i%'pi% up !pening *i$e mai$&rop !pening *i$e '$i(Q?'$i(wrap4so40 Creating *i$e 'et%'se$inux'%on*ig Creating *i$e 'pro%'mounts 444

2012 IBM Corporation

IBM Beaverton Linux Change Team

Use*u$ he$per *un%tions

pi&-. , whi%h pro%ess is thisR ui&-. , whi%h user is running thisR exe%name-. , what is the name o* this pro%essR ti&-. , whi%h threa& is thisR gettimeo*&ays-. , epo%h time in se%on&s pro(e*un%-. , what *un%tion are we inR printD(a% tra%e-. , print sta% (a% tra%e
See 6man stap*un%s8 *or &etai$s an& many more

2012 IBM Corporation

IBM Beaverton Linux Change Team

Tapsets , pre,written pro(e $i(raries

Tapsets

provi&e easy to use a$iases o* %ommon pro(epoints1 provi&e va$ues o* interest *rom those pro(epoints1 &e*ine those he$per *un%tions not runna($e -pro(e a$iases1 not pro(es. insta$$e& in 'usr'share'systemtap'tapset'

Tapsets are SystemTap s%ripts4

Typi%a$$y en%apsu$ate now$e&ge a(out a parti%u$ar app$i%ation or

erne$ su(system

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e o* a tapset *un%tion

pro(e vm4page*au$t S erne$4*un%tion-Khan&$eDmmD*au$tK. 5 name S Kpage*au$tK writeDa%%ess S -M&e*ine&-X*$ags. R X*$ags Z FEULTDFLELD[#IT0 ; XwriteDa%%ess. a&&ress S Xa&&ress :

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e o* a tapset *un%tion

without sys%a$$ tapset;
pro(e erne$4*un%tion-Khan&$eDmmD*au$tK. 5 numFau$tsUpro(e*un%-.VWW :

using sys%a$$ tapset;

pro(e vm4page*au$t 5 numFau$tsUnameVWW :

2012 IBM Corporation

IBM Beaverton Linux Change Team

Tapset examp$es
sys%a$$4C

/ro(es ea%h system %a$$1 provi&es name an& argstr Use& to pro(e memory,re$ate& events /ro(es so% et,re$ate& events

vm4C

so% et4C

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e; sys%a$$ tapset %ont4

For every system %a$$1 sys%a$$s4stp provi&es;
name; sys%a$$ name argstr; argument va$ues en%o&e& in a string in&ivi&ua$ arg va$ues #etstr; - *or return . return va$ue en%o&e& in a string

pro(e sys%a$$4C 5 print*-KOs-Os.7nK1 name1 argstr. : pro(e sys%a$$4C4return 5 print*-6Os returns Os7n81 name1 retstr. :

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e; sys%a$$ tapset %ont4

pro(e sys%a$$4rea& S erne$4*un%tion-KsysDrea&K. 5 name S Krea&K *& S X*& (u*Dua&&r S X(u* %ount S X%ount argstr S sprint*-KO&1 Os1 O&K1 X*&1 <

/ro(e in s%ript;
pro(e sys%a$$4rea& 5 r&D(ytesDre2ueste& WS %ount :
2012 IBM Corporation

IBM Beaverton Linux Change Team

Tapsets
tapset;;ios%he& , systemtap I! s%he&u$er pro(e points tapset;;ir2 , Systemtap pro(es *or I#\1 wor 2ueue1 et% tapset;; pro%ess , systemtap erne$ pro%ess pro(e points tapset;;net&ev , systemtap networ &evi%e pro(e points tapset;;n*s , systemtap AFS %$ient si&e pro(e points tapset;;n*s& , systemtap AFS server si&e pro(e points tapset;;page*au$t , systemtap page*au$t pro(e points tapset;;per* , systemtap per* pro(e points tapset;;rp% , systemtap Sun#/C pro(e points tapset;;s%si , systemtap s%si pro(e points tapset;;signa$ , systemtap signa$ pro(e points tapset;;snmp , Systemtap simp$e networ management proto%o$ pro(e points tapset;;t%p , systemtap t%p pro(e points tapset;;u&p , systemtap u&p pro(e points

2012 IBM Corporation

IBM Beaverton Linux Change Team

0m(e&&e& C %o&e
0m(e&&e& C is %opie& un%hange& *rom your s%ript to the mo&u$e 4% *i$e4 0m(e&&e& C is a$$owe& on$y in tapsets or in s%ripts %ompi$e& with stap
,g -guru mo&e.4 0m(e&&e& C %o&e is usua$$y use& insi&e a *un%tion that starts with O 5 an& en&s with O:

2012 IBM Corporation

IBM Beaverton Linux Change Team

0xamp$e; 0m(e&&e& C %o&e

O5 Iin%$u&e Gnet'so% 4hH Iin%$u&e Gnet't%p4hH Iin%$u&e Gnet'ip4hH Iin%$u&e Gasm'(yteor&er4hH O: *un%tion s Din*o;string-so% ;$ong. O5 stru%t inetDso% Cinet S -stru%t inetDso% C.--$ong.T3IS,Hso% .9 unsigne& %har sa&&rU?V1 &a&&rU?V9 mem%py-sa&&r1 Zinet,Hsa&&r1 sizeo*-sa&&r..9 mem%py-&a&&r1 Zinet,H&a&&r1 sizeo*-&a&&r..9 sprint*-T3IS,HDDretva$ue1 KO&4O&4O&4O&;O& ,H O&4O&4O&4O&;O&K1 sa&&rU0V1 sa&&rU1V1 sa&&rU2V1 sa&&rU@V1 ntohs-inet,Hsport.1 &a&&rU0V1 &a&&rU1V1 &a&&rU2V1 &a&&rU@V1 ntohs-inet,H&port..9 O:

2012 IBM Corporation

IBM Beaverton Linux Change Team

#e*eren%es

SystemTap &o%umentation
) Tutoria$ ) Beginner"s Lui&e ) Language #e*eren%e ) Tapset #e*eren%e http;''sour%eware4org'systemtap'&o%umentation4htm$
#e&(oo ; SystemTap; Instrumenting the Linux +erne$ *or Ena$yzing
/er*orman%e an& Fun%tiona$ /ro($ems

http;''www4re&(oo s4i(m4%om're&papers'p&*s're&p??QN4p&*

2012 IBM Corporation

IBM Beaverton Linux Change Team

#e*eren%es Cont4

IBM SystemTap B$ueprints

http;''pu($i(4(ou$&er4i(m4%om'in*o%enter'$nxin*o'v@r0m0'topi%'$iaai'$iaaiSys temTap4htm

#30LQ SystemTap Beginners Lui&e #30LQ SystemTap Tapset #e*eren%e

http;''&o%s4re&hat4%om'&o%s'en,US'#e&D3atD0nterpriseDLinux'in&ex4htm$

2012 IBM Corporation

IBM Beaverton Linux Change Team

#e*eren%es
There are man pages;

stap
systemtap program usage1 $anguage summary stappaths your systemtap insta$$ation paths stap*un%s *un%tions provi&e& (y tapsets stappro(es pro(es ' pro(e a$iases provi&e& (y tapsets stapex some examp$e s%ripts

2012 IBM Corporation