Analyzing Microsoft SharePoint

Products and Technologies Usage

Author:
Mike Wise
Date published:
January 2009
Summary:
This white aer will hel ad!inistrators gather and analyze Microsoft SharePoint Products and
Technologies usage and erfor!ance data"
The infor!ation contained in this docu!ent reresents the current #iew of Microsoft $ororation
on the issues discussed as of the date of u%lication" &ecause Microsoft !ust resond to
changing !arket conditions' it should not %e interreted to %e a co!!it!ent on the art of
Microsoft' and Microsoft cannot guarantee the accuracy of any infor!ation resented after the
date of u%lication"
This white aer is for infor!ational uroses only" M($)*S*+T MA,-S .* WA))A.T(-S'
-/P)-SS' (MP0(-1 *) STATUT*)2' AS T* T3- (.+*)MAT(*. (. T3(S 1*$UM-.T"
$o!lying with all alica%le coyright laws is the resonsi%ility of the user" Without li!iting the
rights under coyright' no art of this docu!ent !ay %e reroduced' stored in or introduced into a
retrie#al syste!' or trans!itted in any for! or %y any !eans 4electronic' !echanical'
hotocoying' recording' or otherwise5' or for any urose' without the e6ress written er!ission
of Microsoft $ororation"
Microsoft !ay ha#e atents' atent alications' trade!arks' coyrights' or other intellectual
roerty rights co#ering su%7ect !atter in this docu!ent" -6cet as e6ressly ro#ided in any
written license agree!ent fro! Microsoft' the furnishing of this docu!ent does not gi#e you any
license to these atents' trade!arks' coyrights' or other intellectual roerty"
2009 Microsoft $ororation" All rights reser#ed"
Microsoft' S80 Ser#er' and SharePoint are either registered trade!arks or trade!arks of
Microsoft $ororation in the United States and9or other countries"
The na!es of actual co!anies and roducts !entioned herein !ay %e the trade!arks of their
resecti#e owners"
Table of Contents
INTRODUCTION 4
INSTALLIN AND CON!IURIN LO "ARS#R $
"reparin% the analysis ma&hine $
"reparin% the ser'er lo%s $
LO "ARS#R (U#RI#S $
#numeratin% re&ords $
Countin% users )
Load balan&in% *
User type distribution +
Re,uest -R"S. distribution o'er time /0
Distin&t users o'er time /1
User a%ent distribution /4
2ro3ser usa%e /)
Offi&e &lient 4eb Ser'i&e usa%e 10
Slo3 pa%es 1/
Importin% lo%s into S(L Ser'er 11
R#!#R#NC#S 15
Introdu&tion
Microsoft: SharePoint: Products and Technologies gi#e an ad!inistrator few ossi%ilities to look into what is actually
haening on a far!; for e6a!le' how !any users are acti#e' how hea#ily they are using the syste!' what kind of
re<uests are co!ing in' and fro! what kind of clients they originate" While this will %e rectified in future #ersions' in
fact !uch of this infor!ation is otentially a#aila%le' %ut locked u in the ((S logs"
The <uestion is' how do we e6tract this data= *ne of the easiest ways is to si!ly use 0og Parser' an under>
recognized %ut #ery owerful tool a#aila%le free for download fro! Microsoft" The download link can %e found at the
end of this article" &e ad#ised that 0og Parser is not an officially suorted roduct' thus any %ugs or errors will not %e
handled %y the Microsoft suort channels" &ut this #ersion has %een around a while and it is <uite sta%le"
0og Parser can read and write to a nu!%er of te6tual and %inary for!ats' including all the ((S for!ats" (n fact' it
ro%a%ly ro#ides the %est way to con#ert %etween these for!ats"
0og Parser allows the following tyes of data to %e generated?
Installin% and &onfi%urin% Lo% "arser
Preparing the analysis machine
2ou ro%a%ly will not want to install 0og Parser on your ser#er !achine" )ather' you will want to install it on a client
!achine with ade<uate free sace to hold the logs 4calculate a%out 2 @& er ser#er er day of log files you want to
analyze5"
1ownload 0og Parser 2"2 and install it on your !achine" See the )eferences section at the end of this article for the
link"
2ou !ay want to add the e6ecuta%le directory to your ath en#iron!ent #aria%le to !ake 0og Parser easier to use
fro! the co!!and ro!t" The default ath on a AB>%it client is $?CProgra! +iles 46DA5C0og Parser 2"2"
Preparing the server logs
*f course' the logs !ust %e configured and collected' refera%ly fro! all acti#e We% ser#ers in the far!" While it is
ossi%le to analyze usage fro! a single We% ser#er and draw conclusions a%out the entire far!' it can %e so!ewhat
!isleading %ecause it deends on how the load %alancer is di#iding u the re<uests" (n fact' we can use the results of
this analysis to draw conclusions a%out how E%alancedF the load %alancing actually is"
0og settings for ((S can %e found in the ((S 0og Manager running on your We% ser#er" (n this article' we assu!e that
you ha#e selected the WG$ setting and ha#e ena%led at least the following fields?
!ield Column name
1ate date
Ti!e ti!e
$lient (P Address c>i
User .a!e cs>userna!e
Method cs>!ethod
U)( Ste! cs>uri>ste!
Protocol Status sc>status
Protocol Su%Status sc>su%status
&ytes Sent sc>%ytes
&ytes )ecei#ed cs>%ytes
Ti!e Taken ti!e>taken
User Agent cs>user agent
*f course' you !ight want to ena%le !ore of the!" 3owe#er' this will deter!ine the size of the files that will take u
disk sace on the We% ser#er' and i!act rocessing ti!e in the following" 0arge' hea#ily loaded We% ser#ers can
easily roduce giga%ytes of log files er day"
*nce you ha#e the logs' you will want to collect the! together into a directory on a client co!uter so!ewhere for
analysis" 2ou will need to rena!e the files to reflect the na!e of the We% ser#ers' %ecause log files fro! the sa!e
date are likely to ha#e the sa!e na!e"
Lo% "arser ,ueries
Enumerating records
(n the following e6a!le' we ha#e rena!ed four log filesC" We also created a "tst file 4coying and deleting !ost of the
lines using .otead5 out of a log file' and will use it to test our co!!and %efore running said co!!and on the all the
large ones' which will take !inutes to e6ecute" .ote that one of our log files is <uite a %it s!aller 42BH M& instead of
IJ"2 @&5 than the others" We donKt know why this is yet' %ut we will use the following rocedure to in#estigate"
So' letKs get started %y enu!erating our records" +irst we try with the EtstF file' and when we see it works' we can scroll
%ack with the u>arrow 4co!!and>line editing5 and relace the EtstF with ElogF" We enter the co!!and?
logparser -i:IISW3C "select count(*) as ct from *.log"
And we see this?

.ote how !uch longer it takes' L9"9 seconds instead of less than J" This is why we use the "tst file to try our
co!!ands out first and see if we like the results %efore roceeding to the "log files" 3owe#er' fro! now on' we will
only show the "log results"
So we see we ha#e around JL !illion records in one day" 3!!' see!s like a lot" 0etKs see how successful they were
%y checking out the status codes"
logparser -i:IISW3C "select count(*) as ct,sc-status from *.log group by sc-
status"
This <uery yields the following 4after we Eressed a keyF5"
Took a lot longer this ti!e 4there was !ore to do5" There is an 6, otion to !ake it outut without a ause' %ut then
you lose the headers too" An o%#ious loy is to ie the outut to a file and then load it u into .otead or -6cel for
further analysis' as in the following?
logparser -i:IISW3C "select count(*) as ct,sc-status from *.log group by sc-
status" - !o.t"t
.ow' %ack to the analysis" There are o#er se#en !illion B0J authentication errors" 0ea#ing aside the consideration as
to whether or not this is e6cessi#e 4it is a fact of life that SharePoint Products and Technologies do not kee a user
cookie and re>authenticate e#ery re<uest5' we see that essentially !ost re<uests are either 200s or B0Js" There are
so!e G0Bs 4a%out J"A ercent5' %ut the relati#ely s!all nu!%er indicates that we are handling deendent 4or static5
re<uests rationally; our %lo% cache is working retty well and client caching is working for the !ost art" (f this nu!%er
were larger' say J0 ercent or a%o#e' then it would !erit in#estigation"
Counting users
+ro! now on' we will often not show screen shots of the co!!and ro!t 4unless the result is short5; we will 7ust
show the lo%parser co!!and we are using and the results as a .otead or -6cel ta%le"
0etKs look at how !any users we ha#e logging in"
logparser -i:IISW3C "select count(#istinct cs-username) from *.log" - !o$.t"t
+$74/) different users used this ser#er that day M <uite a few' really"
Load balancing
.ow we are curious as to how the load %alancing is working and how user re<uests are distri%uted o#er the log files"
+or this we will need our first t3o8sta%e ,uery" +irst we <uery %y user and log file and store the results in a "cs# file 4it
could %e an ((SWG$ file too5' and then we run a <uery on that file"
J" )un the following?
logparser -i:IISW3C -o:CS% "select count(*) as ct,cs-username,logfilename
from *.log group by cs-username,logfilename" !out.cs&
After a%out D0 seconds' this results in a A"2 M& file in our directory" .ow we <uery that to see how !any
users there are in each log file" This file has three colu!ns? ct' cs>userna!e' and logfilena!e" We now
<uery it directly in order to disco#er how !any uni<ue users there are in each file"
2" )un the following?
logparser -i:CS% "select sum(ct) as sum,count(*) as users,logfilename from
out.cs& group
The outut looks like?
sum users 'og(ilename
------- ----- -------------------------------
$)*$+,, ,--. C:/S0p'ogs/S12-34,-e"4-$$43.log
*++43.- 3++35 C:/S0p'ogs/S12-34.-e"4-$$43.log
,+5)54- 3-)34 C:/S0p'ogs/S12-34+-e"4-$$43.log
,*$$44- 3-4$3 C:/S0p'ogs/S12-34--e"4-$$43.log
&ecause HDDAIGLLG9IGD2G0IGD0JG N JJ9DAD' and this is %igger than the nu!%er of distinct users we saw %efore' we
know that so!e users !ust %e in !ore than one log" (f you e6a!ine the out"cs# file' you will note that the !ain user is
the E>E user; howe#er' this is !ostly due to all the B0Js' which we could eli!inate %y adding a 3here clause 43here
s&8status9:40/5"
The fact that so!e users are in !ore than one log file indicates that those users are %eing load %alanced to !ore than
one ser#er" Which ones' and does it occur fre<uently= The following <uery will show us?
logparser -i:CS% "select sum(ct) as ct$,count(*) as logcount,cs-username from
out.cs& group by cs-username or#er by logcount,ct$ #esc"
.ote the Eorder by lo%&ount7&t/ des&F" This will order the records so that the ones with the %iggest lo%&ount co!e
first' with ctJ %reaking the tie" And des& is short for EdescendingF; if it is o!itted' the list will sort fro! s!all>to>large"
We donKt outut the users here in order to rotect the user na!es" 3owe#er' if we want to see how !any there are in
each class' we can do the following?
logparser -i:CS% -o:CS% "select sum(ct) as ct$,count(*) as logcount,cs-username
from out.cs& group by cs-username or#er by logcount,ct$ #esc" - !cnt.cs&
And then?
logparser -i:CS% "select count(*), logcount from cnt.cs& group by logcount or#er
by logcount"
*utut?
ct logcount
----- --------
+*))- $
$-3,* )
)*$) 3
*)* *
Statistics:
-----------
6lements processe#: 5,*$-
6lements output: *
6"ecution time: 4.$) secon#s
The users who ended u on !ultile We% ser#ers are ro%a%ly those whose (P address has changed during the day'
!ay%e %y changing co!uters' !ay%e due to a 13$P renewal" More interesting !ight %e to in#estigate the file %y (P
address 4using the &8ip field instead of the &s8username5' %ut that is left as an e6ercise to the reader"
User type distribution
)eturning to the "log files' we !ight now ask oursel#es' E3ow were the to users distri%uted= EThe following si!le
<uery answers that?
logparser -i:IISW3C "select top )4 count(*) as ct,cs-username as user from *.log
group by user or#er by ct #esc"
May%e we want to see what kinds of users we ha#e 4hea#y' light' and so forth5" Try this series of co!!ands?
J" @et the users and their fre<uencies into a "cs# file 4note that we e6clude the re<uests with B0J status codes
here to eli!inate the EdashF user with their se#en !illion re<uests5?
logparser -i:IISW3C -o:CS% "select count(*) as ct,789I:7('8;(count(*))) as
bin,cs-username from *.log <0ere sc-status=!*4$ group by cs-username or#er
by ct #esc" - !userfre.cs&
2" .ow we can #iew the! %y %in" The following co!!and?
logparser -i:CS% "select sum(ct),count(*),bin from userfre.cs& group by
bin or#er by bin #esc" >
yields the following ta%le?
3*)*+* $ $3
3$-*3) ) $)
3$,3,$ . $$
55)$.- *- $4
$44+.-$ $34 5
$4.++3+ 3-$ -
$,-*3+$ $,,$ +
$-+-,5, *-45 .
$*+,-+3 5533 ,
+*,44) $3$+3 *
).3$5, $)*$3 3
$4-5)$ $3--3 )
+3.+$ ).5*3 $
$)$3) $)$3) 4
We i!ort this into -6cel' add a colu!n for A'% R"; 4%y di#iding the first colu!n %y 2B and then %y the second
colu!n5"
We see that our tyical -6tre!e' 3ea#y' Tyical and 0ight users corresond roughly to %ins A' L' and D" (t see!s that
real user %eha#ior see!s to san a !uch wider range' na!ely %uckets J0 to B inclusi#e' which ranges fro! a%out 2 to
D00 )P3"
While this articular analysis includes deendent re<uests' we can do the sa!e without the!' the results are !uch
the sa!e' users show acti#ity o#er a !uch wider range than the current ser#er usage !odel indicates"
(n this case' we E%innedF 4ut the users into %uckets5 on the natural logarith! of the userOs )P3" (f you would rather
%in on factors of two' then you will want to di#ide LO %y LO-1<0.' as in?
789I:7(?I%('8;(count(*)),4..53$,))
Si!ilarly' if you want to %in on owers of J0' use LO-/0<0.' as in?
789I:7(?I%('8;(count(*)),).34),))
Request (RPS) distribution over time
.ow we would like to see how usage is #arying %y ti!e" To do this co!forta%ly' we really need to ha#e the data in a
!ore reada%le for!' so now we co!%ine all our data into one %ig "cs# file" We also crack out the seconds' !inutes
and hours in a way that is !ore easily accessi%le" &ecause the <uery is long' we ut it into a file 4load"t6t5 and use a
co!!and>line ara!eter in 0og Parser to access it?
*ur <uery 4in load"t6t5?
select 6@73AC79(I'6:AB6('og(ilename),'og3o<,
#ate, time, cs-met0o#, cs-uri-stem, cs-username, c-ip, cs(Cser-Agent), cs-0ost,
sc-status, sc-substatus, sc-bytes, cs-bytes, time-taDen,
a##(
a##(
mul(3.44,to9int(to9string(to9localtime(to9timestamp(#ate,time)),E00E))),
mul(.4,to9int(to9string(to9localtime(to9timestamp(#ate,time)),EmmE)))
),
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EssE))
) as secs,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EyyE)) as yy,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EBBE)) as mo,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),E##E)) as ##,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),E00E)) as 00,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EmmE)) as mi,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EssE)) as ss,
to9lo<ercase(6@73AC792A71(cs-uri-stem)) as fpat0,
to9lo<ercase(6@73AC79(I'6:AB6(cs-uri-stem)) as fname,
to9lo<ercase(6@73AC796@76:SI8:(cs-uri-stem)) as fe"t
from *.log
<0ere sc-status=!*4$
And the 0ogParser in#ocation we use on it?
logparser -i:IISW3C file:loa#.t"t -o:cs& - !bigo.cs&
After a few !inutes this yields a G"H @& file' E%igo"cs#F" We will use this file in the future for a nu!%er of things' %ut first
we reduce it further to a distri%ution %y seconds?
logparser -i:CS% >o:CS% "select count(*) as ct,secs,ma"(ss) as ss,ma"(mi) as
mi,ma"(00) as 00 from bigo.cs& group by secs or#er by secs" - !secs#ist.cs&
logparser -i:CS% -o:CS% "select count(*) as ct,#i&(secs,.4) as minu,ma"(ss) as
ss,ma"(mi) as mi,ma"(00) as 00 from bigo.cs& group by minu or#er by minu" -
!min#ist.cs&
.ow it is a si!le !atter to e6tract the eaks we want" +irst we want the hourly a#erage )PS?
logparser >i:CS% "select 00,a&g(ct) from min#ist.cs& group by 00 or#er by 00"
logparser -i:CS% "select 00,ma"(#i&(ct,.4)) from min#ist.cs& group by 00 or#er by
00"
logparser -i:CS% "select 00,ma"(ct) from sec#ist.cs& group by 00 or#er by 00"
$ollecting these results together and lotting the! in -6cel yields?

.ote that if we wanted to ha#e only user oerations 4a !easure used in lanning that nor!ally only includes user>
initiated re<uests5' then we would want to filter out all the static and deendent re<uests first 4"gif' "ng' "%!' "7s' "css'
and "a6d5" The easiest way to do this would %e to change the 3here clause to?
F.
<0ere sc-status=!*4$ an# fe"t=!GgifG an# fe"t=!GpngG an# fe"t=!GbmpG
an# fe"t=!GHsG an# fe"t=!GcssG an#
fe"t=!Ga"#G
3owe#er' %elow we show that the ratio in this case is a%out G>J for %rowsers' so we can 7ust di#ide the a%o#e nu!%ers
%y three for an aro6i!ate #alue"
Distinct users over time
We start out with a new "cs# file' one containing the user distri%ution?
logparser -i:CS% -o:CS% "select count(*) as ct,cs-username,secs,ma"(ss) as
ss,ma"(mi) as mi,ma"(00) as 00 from bigo.cs& group by secs,cs-username or#er by
secs,cs-username" - !user#ist.cs&
Then we consolidate %y hour?
logparser -i:CS% -o:CS% "select 00,cs-username,sum(ct) as re from user#ist.cs&
group by 00,cs-username or#er by 00,cs-username"
logparser -i:CS% "select 00,count(*) from user00#ist.cs& group by 00" >
This reorts distinct users %y hour?
(f we are interested in the !inute' then we ha#e to consolidate %y !inute?
logparser -i:CS% -o:CS% "select #i&(secs,.4) as minu,cs-username,sum(ct) as re
from user#ist.cs& group by minu,cs-username or#er by minu,cs-username"
!usermi#st.cs&
and then?
logparser -i:CS% "select count(*) from usermi#ist.cs& group by minu"
This of course gets us a !uch !ore detailed grah?
User agent distribution
The user agent is the rogra! that was used to access the We% ser#er" +or e6a!le' it !ight %e a %rowser' an *ffice
client' or a rogra! like Microsoft: SharePoint: 1esigner" The user agent is' in rincile' identified %y a string that the
%rowser deli#ers with e#ery re<uest" 3owe#er' this logic is clouded %ecause so!eti!es alications !ake their
%eha#ior deendent on that string' and the agents ha#e to retend to %e another agent 4%y sulying a different string5
in order to get so!ething to work"
Unfortunately' different li%raries used %y the sa!e rogra! !ight use different user agents" Thus' for e6a!le' when
an *ffice client downloads a file' it uses !ore than one user agent string' so!e co!ing fro! the !ain rogra!' and
others co!ing fro! the We%1AP li%rary" .onetheless' agents are a useful way to categorize usage" 3owe#er' as we
will see' it is a %it co!licated"
We start with?
logparser -i:IISW3C "select count(*) as ct,cs(user-agent) from *.log group by
cs(user-agent) or#er by ct #esc"
This yields a !essy outut 4note that we ha#e truncated so!e of the lines with EQF5?
ct cs(Cser-Agent)
-------
---------------------------------------------------------------------------------
------------------
34-4345 BicrosoftI8fficeI6"istenceI?isco&ery
)$,-+$5 Bicrosoft-Web?A%-Bini3e#irJ..4..44$
$$,*$,5 Bicrosoft-Web?A%-Bini3e#irJ..4..444
+-4$.4 BS(ront2ageJ$).4
.,$)3* BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIS'CC$LI.:67IC'3I).4.,4+)+.:67IC'3I3.4.4*,4.LII$.$F
,-$*+5 BoKillaJ*.4I(compatibleL
IBSI6I..4LIBSIWebISer&icesIClientI2rotocolI).4.,4+)+.$*3*)
*-+-3, BicrosoftI8fficeJ$).4I
(Win#o<sI:7I..4LIBicrosoftI8fficeI8ne:oteI$).4..3)4LI2ro)
3*4*,. BoKillaJ*.4I(compatibleL
IBSI6I..4LIBSIWebISer&icesIClientI2rotocolI).4.,4+)+.-3))
3))44+ BoKillaJ*.4I(compatibleL
IBSI6I..4LIBSIWebISer&icesIClientI2rotocolI).4.,4+)+.$*33)
),5*-$ BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIS'CC$LI.:67IC'3I).4.,4+)+LI.:6C'3I3.4.4*,4.LI.:67F
2ress a Dey...
ct cs(Cser-Agent)
------
---------------------------------------------------------------------------------
-----------------
)44-3+ BicrosoftI8fficeJ$).4I
(Win#o<sI:7I..4LIBicrosoftI8fficeI8utlooDI$).4..3)4LI2ro)
$.*4+4 Bicrosoft-Web?A%-Bini3e#irJ,.$.).44
$,4)-+ BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIS'CC$LI.:67IC'3I).4.,4+)+LI.:67IC'3I3.4.4*,47IC'3I$.$.
*F
BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIS'CC$LI.:67IC'3I).4.,4+)+LI.:67IC'3I3.4.4*,4$.$.*3))LI
InF BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIW8W.*LIS'CC$LI.:67IC'3I).4.,4+)+LI.:67IC'3I367IC'3I$.$
.*F.
BoKillaJ*.4I(compatibleL
IBSI6I+.4LIWin#o<sI:7I..4LIS'CC$LI.:67IC'3I).4.,4+)+LI.:67IC'3I3.4.4*,4.$.*3))LII
nF
54-43 BoKillaJ*.4I(compatibleLIBSI6I*.4$LIWin#o<sI:7LIBSISearc0I,.4I3obot)
2ress a Dey...
7asD aborte# by user.
Statistics:
-----------
6lements processe#: $+)$.435
6lements output: $$4
6"ecution time: $+-.,, secon#s (44:4):,-.,,)
The ro%le! is that all the %rowser re<uests add e6tra ara!eters infor!ing the alication what li%raries are installed
and a#aila%le on the %rowser client" +ortunately' 0og Parser has a function that can arse strings nicely'
#=TRACT>TO?#N" We use it as follows"
logparser -i:IISW3C "select count(*) as ct,6@73AC7978M6:(cs(user-agent),4,EJE) as
agent from *.log group by agent or#er by ct #esc"
Which yields the following !ore !anagea%le result 4headers and other irrele#ancies deleted5?
ct agent
------- ---------------------------------------------------------------------
-4+3,5+ BoKilla
3,-4455 Bicrosoft-Web?A%-Bini3e#ir
34-4345 BicrosoftI8fficeI6"istenceI?isco&ery
$$.4)-, BicrosoftI8ffice
+--.-) BS(ront2age
$4$-.+ Win#o<s-3SS-2latform
5*..+ 8C7'88MIS7S
-*$++ -
.+,)4 BicrosoftI8fficeI2rotocolI?isco&ery
,,)4$ BicrosoftI?ataIAccessIInternetI2ublis0ingI2ro&i#erI2rotocolI?isco&ery
3+$54 C(S
33,$5 non-bro<ser
$,*4+ ;388%69WSS9SN:C1
$),*5 SystemICenterI8perationsIBanagerI)44+I..4..)+-.4
$4+3+ 8utlooDConnector
---+ S8A2I7oolDitI3.4
3,$5 Bicrosoft6"c0angeSer&er-1ttpClient
$-4. :S2layer
$,,- 7estIforIWebI(ormI6"istence
$$** BSI6I..4
$435 BS8ffice
.$$ BS-WebSer&ices
*5) 3ssOan#it
))3 Info2at0?A
)4- BicrosoftI?ataIAccessIInternetI2ublis0ingI2ro&i#erI?A%
5$ (?BI)."
+. Win#o<s-Be#ia-2layer
,+ 8pera
*- InternetI6"plorer
*, ?a&Clnt
*) BicrosoftI%isioIBSI6
3- BicrosoftI?ataIAccessIInternetI2ublis0ingI2ro&i#erICac0eIBanager
3* Bin#HetIBin#Banager
33 Pa&a
3) %OI2roHect
34 0ttp.e"e
)- Sleipnir
)+ :e<KieI4.55.5I(<<<.ne<Kie.comL:e<sIAggregatorLI)
)$ Wi#commIOtSen#toII6
)4 contype
$5 I?A
$- :e<s;atorInbo"
$3 BicrosoftIWin#o<sI:et<orDI?iagnostics
$) (ee#?emon
$) P:'2
- 8ffice'i&eIWebISer&iceIClientI2rotocolI$.,
. @enuI'inDISleut0I$.)H
. (ee#rea#erI3.$3I(2o<ere#IbyI:e<sbrain)
. (irefo"
. @B'ISpy
3 &eo0-QRSQRT4Iser&iceI(:7I..4LII6I+.4..44$.$-444LIen-CSIWin#o<s)
3 Cser-Agent
3 Bicrosoft-A7'-:ati&e
) WS?A2I
) %CSoapClient
) PaDartaICommons-1ttpClient
$ 8utlooD-6"press
$ SlimOro<ser
$ %OI8penCrl
Statistics:
-----------
6lements processe#: $+)$.435
6lements output: ,5
6"ecution time: -..*4 secon#s (44:4$:)..*4)
*f course this data co!es fro! a de#eloer sho 4erhas the %iggest in the world M Microsoft5' so we are likely to
see all kinds of weird %rowser agent strings" &ut the !a7ority will %e the usual susects"
We coy the results into -6cel for an analysis"
We note the following?
The large nu!%er of @i&rosoftAOffi&eA#Bisten&eADis&o'ery results is due to a *ne.ote issue that
caused e6cessi#e olling" The actual nu!%er with correct %eha#ior should not %e significant and it would end u in
Misc"
4ebDAC is !ostly used %y *ffice clients to !aniulate files"
So!e of the SharePoint 1esigner calls are ro%a%ly We%1AP calls initiated %y an *ffice client %ecause the
latter uses so!e of the sa!e rotocol and li%raries as SharePoint 1esigner"
So if we further si!lify this' reducing the *ne.ote downloads and get the following distri%ution"
&ut letKs look !ore closely"
ro!ser usage
We are interested in characterizing %rowser usage atterns now" +irst we get all the %rowser re<uests into a "cs# file to
reduce the total size and seed u the rocessing"
logparser -i:IISW3C "select * into ie.cs& from *.log <0ere 6@73AC7978M6:(cs(user-
agent),4,EJE)UEBoKillaE an# sc-status=!*4$"
*utut?
Statistics:
-----------
6lements processe#: $+)$.435
6lements output: ,*)-34,
6"ecution time: $++..4 secon#s (44:4):,+..4)
.ote that we ha#e droed the B0Js' and' as an aside' we can see that fro! the original D0LGH9L Mozilla re<uests'
2AB'000 were B0Js' or G2"L ercent' which is al!ost a third"
.ow we want to look at the e6tensions' and classify the! as follows?
logparser -i:CS% "select count(*) as ct,789'8W63CAS6(6@73AC796@76:SI8:(cs-uri-
stem)) as e"t from ie.cs& group by e"t or#er by ct #esc"
)esults?
ct e"t
------- ----
)$.)53) gif
+4*$55 Hpg
.5)$). asp"
.4+).$ asm"
3*+*-. css
)*,*,. png
$-3-+* Hs
$443)-
-)-3. a"#
,*+,3 one
Statistics:
-----------
6lements processe#: ,*)-34,
6lements output: 34
6"ecution time: 5,.3. secon#s (44:4$:3,.3.)
*f course we are not interested in all of this data" &asically' we want the results shown in the ta%le %elow" So we ie
the results into a te6t file 4adding Re6t"t6t M<5 to the end' i!ort it into a sreadsheet and add u the nu!%ers"
#Btension Class Number "er&ent
Total HB2DG0H
"as6 We% ages A92JJB J2"L
"gif' "ng' "%!' "7s' "css' "a6d 1eendent or static re<uest GLGJ2LJ AD"L
"doc' "doc6' "6ls' "6ls6' "t'
"t6' "dot6' "t6t' "zi' "#sd' "e6e'"
7eg' "dll' "!sg' "onetoc2' "6a!l'
"6ls!' "!G
+ile downloads J2AJGG 2"G
"as!6 We% Ser#ices A0L2AJ JJ"J
%lank )edirects J00G2D J"D
"ht!' "ht!l LGLBD J"B
Also note we ha#e a ri!ary9deendent re<uest ratio of H"B' calculated %y the ration of "as6 ages to the static
re<uest ages" The o#erall %rowser user oeration to re<uest ratio is a%out G>J or a %it less' deending on whether or
not you include the "ht!' "ht!l' and %lank re<uests into consideration"
Also interesting is the %reakdown %y file download?
Ext Count Pct
one 54753 39.5%
docx 15948 11.5%
xlsx 15831 11.4%
doc 11673 8.4%
pptx 9912 7.2%
dll 7392 5.3%
xls 4896 3.5%
pdf 4786 3.5%
ppt 3165 2.3%
msg 2121 1.5%
zip 1893 1.4%
vsd 1866 1.3%
onetoc2 997 0.7%
xaml 924 0.7%
xlsm 800 0.6%
txt 629 0.5%
mpp 266 0.2%
dotx 261 0.2%
exe 188 0.1%
mp3 78 0.1%
jpeg 66 0.0%
Total 138445 100.0%
.ow letKs look and see what "as6 ages are called?
logparser -i:CS% "select count(*) as ct,789'8W63CAS6(6@73AC796@76:SI8:(cs-uri-
stem)) as e"t,789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-stem)) as fname from ie.cs&
<0ere e"tUEasp"E group by e"t,fname or#er by ct #esc"
This yields no less than JJ0HJ different ages" *f course it has a long flat ta%le" 0ooking at the to 20 we see the
following?
.ow we take a closer look at the ASM/ We% ser#ice calls" What We% Ser#ices are %eing called=
logparser -i:CS% "select count(*) as ct,789'8W63CAS6(6@73AC796@76:SI8:(cs-uri-
stem)) as e"t,789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-stem)) as fname from ie.cs&
<0ere e"tUEasm"E group by e"t,fname or#er by ct #esc"
And we get the following" These are ro%a%ly asynchronous calls e!%edded in the "as6 age' and there are far fewer
different kinds" *f course' we cannot see the actual na!e of the !ethod in#oked %ecause it is inside the %ody of the
re<uest and not recorded in the ((S logs"
ct e"t fname
------ ---- ----------------------
***.44 asm" site#ata.asm"
$$354) asm" lists.asm"
3)+)) asm" e"celser&ice.asm"
.+*- asm" searc0.asm"
3-)3 asm" <ebpartpages.asm"
3))* asm" publis0ingser&ice.asm"
$))+ asm" sites.asm"
,*$ asm" usergroup.asm"
),$ asm" #spsts.asm"
+* asm" copy.asm"
+$ asm" <ebs.asm"
), asm" business#atacatalog.asm"
)* asm" userprofileser&ice.asm"
)3 asm" spellc0ecD.asm"
) asm" #<s.asm"
) asm" searc0a#min.asm"
$ asm" #ocuments.asm"
$ asm" &ie<s.asm"
Statistics:
-----------
6lements processe#: ,*)-34,
6lements output: $-
6"ecution time: .+.5. secon#s (44:4$:+.5.)
"##ice client $eb Service usage
.ow we do the sa!e thing for the *ffice client calls' that is' the non We%1AP calls" +irst we generate a "cs# file to
analyze"
logparser -i:IISW3C "select * into oc.cs& from *.log <0ere 6@73AC7978M6:(cs(user-
agent),4,EJE)UEBicrosoftI8fficeE an# sc-status=!*4$"
*utut?
Statistics:
-----------
6lements processe#: $+)$.435
6lements output: ,+,5$.
6"ecution time: -+..* secon#s (44:4$:)+..*)
As %efore' we see that only HLH9JA of the original JJA02DH re<uests were not B0J' or a%out H0"2 ercent"
.ow' we %reak out the We% ser#ices with the following co!!and?
logparser -i:CS% "select count(*) as ct,789'8W63CAS6(6@73AC796@76:SI8:(cs-uri-
stem)) as e"t,789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-stem)) as fname from oc.cs&
<0ere e"tUEasm"E group by e"t,fname or#er by ct #esc"
)esults?
ct e"t fname
------ ---- --------------------------
**34,, asm" lists.asm"
+$54$ asm" <ebs.asm"
*,+5) asm" <orDflo<.asm"
$.*5 asm" social#ataser&ice.asm"
$33$ asm" imaging.asm"
++, asm" #<s.asm"
34- asm" publis0e#linDsser&ice.asm"
).. asm" &ie<s.asm"
$)5 asm" meetings.asm"
.3 asm" alerts.asm"
*) asm" people.asm"
3. asm" sli#elibrary.asm"
), asm" &ersions.asm"
* asm" <ebpartpages.asm"
Statistics:
-----------
6lements processe#: ,+,5$.
6lements output: $*
6"ecution time: ..3+ secon#s
These are #arious We% Ser#ices called %y the *ffice client"
Slo! pages
.ow we ha#e a look at EslowF ages" We will look only at ho!e ages 4default"as6 ages5 and look at the a#erage
Eti!e>takenF field in ((S" This is the a!ount of ti!e that ((S needed to render that age in ht!l' including the necessary
%ackend S80 calls" We use the %igo"cs# file we generated earlier in this aer as a %asis"
logparser -i:CS% -o:CS% "select a&g(time-taDen),#i&(secs,344) as
minu,ma"(mi),ma"(00) from bigo.cs& <0ere 789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-
stem))UE#efault.asp"E group by minu or#er by minu" - !0omea&gmin.cs&
We take these #alues and lot the!?
.ow' this a#erage does not really tell us a whole lot" We see that there are so!e ti!es where the a#erage age is
twice as slow' %ut what ercentage of users are actually e6eriencing slow ages= We e6a!ine this using two
<ueries' getting two strea!s' one with the total nu!%er of sa!les' and one with the nu!%er of ages with #alues
higher than B000 !s 4B seconds5?
logparser -i:CS% -o:CS% "select count(*) as ct,#i&(secs,344) as
minu,ma"(mi),ma"(00) from bigo.cs& <0ere 789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-
stem))UE#efault.asp"E group by minu or#er by minu" - !0omepagect.cs&
logparser -i:CS% -o:CS% "select count(*) as ct,#i&(secs,344) as
minu,ma"(mi),ma"(00) from bigo.cs& <0ere 789'8W63CAS6(6@73AC79(I'6:AB6(cs-uri-
stem))UE#efault.asp"E an# time-taDen!*444 group by minu or#er by minu" -
!slo<pagect.cs&
So we see at certain ti!es of the day a large ercentage of our users do e6erience slow age load ti!es' articularly
fro! B until J0 in the !orning"
.ote that !erging the two strea!s was not an entirely tri#ial task in e6cel since so!e !inute #alues in the slow age
count could %e !issing" To a#oid this you could use larger inter#als 4say A00 seconds5' which would also gi#e a
s!oother lot"
Alternati#ely' you could use an -6cel looku function" To generate our grah' we used the -6cel CLOO?U" function
to find the right #alue' and the ISNA function to turn !issing #alues into zero" There !ay %e an easier way' erhas
in#ol#ing a !odified 0og Parser <uery that kees the strea!s together in the first lace"
%mporting logs into S&L Server
*f course there are ti!es when you !ay need to read the data into S80 Ser#er or e#en another data%ase" 0og Parser
can do that too" (t is actually e6tre!ely easy to do' and the following co!!and will read our test log into a ta%le called
Sh0ogTest in the 0ogTest data%ase"
logparser -i:IISW3C "select * into S0p'og7est from *.tst" -o:sl -ser&er:Sea#ra
-#atabase:'og7est -#ri&er:"SV' Ser&er" -create7able:8:
Statistics:
-----------
6lements processe#: 55.5$
6lements output: 55.5$
6"ecution time: ,*..5 secon#s
This has the disad#antage that you lose all the useful 0og Parser functions that are occasionally suerior to what S80
offers" +ortunately' we can re!edy that and ha#e the %est of %oth worlds"
The following inut file and 0og Parser in#ocation does 7ust that' adding so!e e6tra colu!ns to ease <ueries %y date
arts' file na!e' file e6tension' or file ath' and e6cluding B0Js"
Note: The current user is %eing used i!licitly in the connection" Also' the ti!e is %eing con#erted to the local ti!e"
This only works correctly if the local ti!e on the analysis !achine is the sa!e as the ser#ers where the log files
originated fro!"
logparser -i:IISW3C file:loa#S0p$$43.t"t -o:sl -ser&er:Sea#ra -#atabase:'og7est
-#ri&er:"SV' Ser&er" -create7able:8:
The loadShJJ0G"t6t file?
select 6@73AC79(I'6:AB6('og(ilename),'og3o<,
#ate, time, cs-met0o#, cs-uri-stem, cs-username, c-ip, cs(Cser-Agent), cs-0ost,
sc-status, sc-substatus, sc-bytes, cs-bytes, time-taDen,
a##(
a##(
mul(3.44,to9int(to9string(to9localtime(to9timestamp(#ate,time)),E00E))),
mul(.4,to9int(to9string(to9localtime(to9timestamp(#ate,time)),EmmE)))
),
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EssE))
) as secs,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EyyE)) as yy,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EBBE)) as mo,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),E##E)) as ##,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),E00E)) as 00,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EmmE)) as mi,
to9int(to9string(to9localtime(to9timestamp(#ate,time)),EssE)) as ss,
to9lo<ercase(6@73AC792A71(cs-uri-stem)) as fpat0,
to9lo<ercase(6@73AC79(I'6:AB6(cs-uri-stem)) as fname,
to9lo<ercase(6@73AC796@76:SI8:(cs-uri-stem)) as fe"t
into S0p'og7able
from *.log
<0ere sc-status=!*4$
Referen&es
0og Parser 2"2 4htt?99go"!icrosoft"co!9fwlink9=0ink(dNJG9JLJ5