MainframePerformanceManagement

CanMaketheDinosaurDance

AUTHORS

SrinivasMurthyPotharaju,ETSMainframeCoE
HCLTechnologies,Hyderabad

DrUshaThakur,ATSTechnicalResearch
HCLTechnologies,Chennai

MainframePerformanceManagement:CanMaketheDinosaurDance
2010,HCLTechnologiesLtd.
October,2010

MainframePerformanceManagement

Contents
Introduction..................................................................................................................................................4
Purpose.........................................................................................................................................................4
SignificanceofMainframes...........................................................................................................................4
DriversforMainframePerformance.............................................................................................................5
MainframePerformanceServicesfromHCL.................................................................................................6
HCLApproach............................................................................................................................................7
Benefits.......................................................................................................................................................10
Endnotes.....................................................................................................................................................12

MainframePerformanceManagement

Introduction
TherewasatimewhenITfolkscouldshowvaluebyfocusingonlyonoperationalperformance.Today,
however, the focus has shifted more to showing the extent to which IT products and services create
value and contribute to their organizations business objectives. Businesses want to know how IT can
help in expanding their companys customer base, increasing products and services portfolios,
increasingtheprofitmargin,tappingintonewmarkets,identifyingactivitiesforoutsourcing,andmuch
more.ThealignmentofITactivitiesanddeliverablestobusinessgoalsgainsevenmoresignificancein
theworldofmainframes,giventhehighinvestmentsassociatedwiththem.
Itisnolongerenoughtoshowmainframeperformancemetricsonlyintermsofoperationalefficiency;
weneedmetricsthatcanreflecttheextenttowhichmainframeshelptheirorganizationfulfilbusiness
goals.ThatthisisnotjustafadisveryclearfromastudyundertakenrecentlybyGartnerandFinancial
ExecutivesInternational(FEI).AccordingtothatstudyafairlyhighnumberofCIOsarealreadyreporting
toCFOs,andthelatterhaveasignificantsayinallITinvestmentmatters. 1

Purpose
TheaudienceofthispaperareCIOs,CEOs,CFOs,andotherexecutiveswhosebusinessoperationsare
heavily dependent upon the performance of mainframes. It examines the main drivers for mainframe
performancemanagementbyorganizationsthathavedecidedtocontinueusingthem.Thispaperalso
highlightskeyfeaturesoftheperformancemanagementservicefromHCL. 2

SignificanceofMainframes
Throughout the history of the mainframe, IBM has retained its dominance, 3 though not without its
shareofcontroversy. 4 ItisestimatedthatIBMsmainframesaleswillreach$3.3billionin2010,which
wouldrepresentaboutthreequartersoftheoverallmainframemarketof$4.5billion. 5
Forsometimenow,therehasbeenapopularbeliefthatmainframesarelivingonborrowedtimeand
may not be around in the near future. Writing in 2003, John R. Phelps of Gartner pointed out that a
numberofdevelopmentsoverthepast15years 6 ledtostoriesaboutthedeathoftheIBMmainframe,
but the reality has been to the contrary. Currently, a great deal of attention is being given to the
decliningrevenuefrommainframesandserversingeneral, 7 andonceagainweareseeingstoriesabout
theendofmainframesintheverynearfuture.Whatisinteresting,however,isthatIBMhascontinued
to modernize and invest in a range of mainframes, improve pricing, and reach out to new customers,
especiallyinnewlyindustrializingeconomies, 8 whichindicatesthatIBMstillhasconfidenceincontinued
business from mainframes. In fact, IBM is solidifying its position in niche accounts. For instance, the

MainframePerformanceManagement

zEnterprisesystem(z196)latestmainframemodelfromIBMistargetedatthetop10%to20%ofits
customerswherethevolumeofMillionsofInstructionsPerSecond(MIPS)isveryhigh. 9 MIPSisoneof
themostimportantcriterionthatIBMusesforcalculatingthecostofmainframelicense. 10
Themainframestorywouldbeincompletewithoutitscustomers.Althoughsomecustomersareonthe
lookouttoreducedependencyonmainframesbymovingtosystemspoweredbynewertechnologies,
manystillcontinuetousethemforhistoricalreasons.Mainframecustomersarebigcompaniesindata
intensive industries such as distribution, financial services, general business, and public sector that
invested in mainframes several years ago. For these customers, reliability, security, and scalability are
toppriorities. 11 Thereisageneralconsensusthatanyimprovementintheendtoendperformanceof
mainframesisaveryimportantstepinbringingdownthecostofowningandrunningmainframes.

DriversforMainframePerformance
Ingeneral,themainbusinessdriversforimprovingtheperformanceofmainframesareasfollows:

ReducetheTotalCostofOwnership(TCO)ofmainframeservers

Increaseproductivity

Maketheonlineapplicationsavailable24/7toanincreasingnumberofcustomerswithoutany
delay 12

Improveenduserexperiencee.g.,comparingautoinsurancequotes 13 andtheliketransactions

Increasecustomerretention 14

EliminateSLAviolation

Assuming that IT is a means for achieving the business objectives then it is imperative for IT
managementandstafftobeonthesamepageasthebusinessmanagementandstaffintheircompany.
ThiswillnotonlygiveallITactivitiesanddecisionsabusinesscontextbutalsorevealtheirsignificance
andimpactonbusiness.
The changing business environment today is compelling IT management to look at ways of obtaining
maximumperformancefromthecoremainframeapplicationsandofaligningapplicationsdirectlytothe
overallbusinessobjectivesoftheirorganization.Thebulkofthemainframeapplicationsweredeveloped
yearsago,whichhavesincethenundergoneenhancementsorpatchwork.Consequently,theyhavelost
maintainabilityandtraceabilitytobusinessrequirements.
Today,more thaneverbefore,thereisanever changingworkloadbeingexecutedon themainframe.
Starting with traditional batch and online workloads, there is an increasing number of transactions

MainframePerformanceManagement

running on the mainframe which are invoked from the Web. Additionally, there are queries that are
usedforcomplexreportingandanalyticsinDataWarehouse&BusinessIntelligenceenvironments.
Poormainframeperformancemaybecausedbyalloracombinationofthefollowingproblems:

Environment,systemandsubsystemparametersarenotsetcorrectly

Aging and monolithic applications are large and complex, and therefore difficult to integrate
withmodernsystems,whicharedistributedandagile

Databaseparametersarenottunedforoptimalperformance

Applicationsanddatabasesarepoorlydesigned

ApplicationcodeandSQLsarenottunedforperformance

Onlineapplicationsmaynotbeavailableduetotheschedulingofjobsincriticalpathandend
ofdaybatchjobs

Fileblocksizeandbufferparametersarenotsetcorrectly,leadingtoprolongedandhighusage
ofCPUandI/O

TheabovementionedtechnicalproblemsneedtobeaddressedbyITteams,giventhatiftheyareleft
unattendedtheywillmostlikelyleadtothefollowingundesirableconsequences:

HigherlicensingcostofmainframesresultingfromahighconsumptionofMIPSbyapplications

Increasing customer dissatisfaction and decline in customer because of poor response time of
onlineapplications

Lower productivity of day time employees caused by extended batch windows cutting into
regularbusinesshours

SLAviolationsduetotheinabilityofapplicationstoscaleuptoincreasingdatavolumes

MainframePerformanceServicesfromHCL
This is a comprehensive service from HCL. Using proven methodologies and frameworks, it identifies
poorlydesignedprocesses,programs,databases,andenvironmentonthemainframe.Onthebasisofits
findings, HCL proposes a solution and remedial strategy. During implementation, HCL enhances the
configuration of the mainframe processes, programs, databases, and services such that the overall
response and execution time is faster than before. Customers who have availed of this service have
been able to reduce MIPS usage by their applications, lower TCO, and utilize the available processing
capacityfornewerworkloads.

MainframePerformanceManagement

HCLApproach
As illustrated in Figure 1, HCL uses its proprietary framework for workload optimization to help
customers analyze their workload and identify areas of application tuning. This approach takes into
account inputs on the architecture of business applications and databases, the design of online and
batch components, and also usage/runtime statistics (from SMF & RMF, performance management
tools,historyreports,andenvironmentaswellsystem/subsystemparameters).

Figure1:HCLProprietaryFramework

As highlighted below, the objective of the assessment approach adopted by HCL is to identify poorly
performing workloads on the mainframe so that appropriate recommendations could be made and a
remedialstrategydeveloped.
PERFORMANCEOPTIMIZATIONASSESSMENTAPPROACH
As shown in Figure 2, HCL first collects data on various parameters from the client systems and
subsystems.Itanalyzesthatdataandthenproposesasolution,togetherwitharemedialstrategy.

MainframePerformanceManagement

Analysis

Data
Collection

Analyzeusage
andruntime
datafromSMF
andRMFto
identifythe
mostexpensive
workloads

Identifyareasof
performance
improvement
using
performance
tools,history
reports,and
application
architecture
diagrams

Propose
Solutions
Make
recommendationsto
optimizeperformance
oftheworkload.

Estimatecost,effort
anddurationrequired
forimplementing
proposed
recommendations
aimedatoptimizing
performance.

PrepareGainVsEffort
matrixtoprioritize
eachrecommendation.

Remediate

Constructhighleveldesigns
foreachapproved
recommendation.
Conductbaseline
performanceofthecode
(productionversion)in
controlledtestregion.

Carryoutfunctionaland
performancetestsof
impactedcomponentsin
controlledtestenvironment.

Compareresultsfrom
performancetest,
implementagreedupon
recommendations,and
demonstratebenefits.

Providewarrantysupport
forimpactedcomponents.

Figure2:PerformanceOptimizationExecutionApproach

HCL follows a systematic approach in identifying areas of improvement in mainframe systems and
processes.Hereisaglimpseofthatapproach.
Inbatchjobs,HCL

Examinesthecriticalpathinthebatchcycleinordertoidentifyunduewaitingtimes
Identifiesthemostexpensivebatchjobs,programs,utilities,andrecurringjobfailures
Analyzesperformancedatareportsforselectedjobs

Inonlinetransactions,HCL

Identifiesthemostexpensiveonlinetransactionsandrecurringfailureevents
Analyzesperformancedatareportsforselectedtransactions

MainframePerformanceManagement


Foridentifyingdesignandarchitectureproblems,HCL

Understandsthepainpointsintheapplicationarchitecture
Locatesthepreciseproblemsinthedesignofdatabasesandexternalinterfaces
Analyzesperformancedatareportsforselectedjobsandtransactions

Foridentifyingspecificproblemsinthedatabase,HCL

Examinescontentiousissues
Considersorreconsiderdatacompression
ConsidersifappropriateIndexesarebeingusedbythequery
Takesintoaccountthepossibilityofreorganizingtablespaces
Exploresifpartitioningandsegmentationoftablespaceswouldbeappropriate

Forisolatingproblemsatthesystemandsubsystemlevels,HCL

Analyzesperformancedatareportsforz/OS,CICS,DB2andIMSsystems
Analyzesallsystemandsubsystemparameters
ExaminesBufferPool,SortPool,EDMPool,andDB2Catalog

PERFORMANCEMANAGEMENTBESTPRACTICES
HCLusesasetofperformancemanagementbestpracticesthatenableitto:

Identify early signs of performance problems by using performance tools and system
relatedperformancereports.

Automate realtime problem detection and resolution by putting in place performance

indicatorsandthresholds.

Prevent the occurrence of problems by using the SMF and RMF data reports to allocate
workload resources based on performance. Historically, the stellar reputation earned by
mainframes (for executing high performance operations continuously) has come, in part,
fromthefactthatengineershavebeenabletoallocatesystemresourcesproactively.This
proactive allocation was not a problem when mainframe systems were siloed, software
changeswererelativelyinfrequent,andworkloadswerepredictable.

MainframePerformanceManagement

Benefits
The overall business benefits that clients have enjoyed from an improvement in the performance of
mainframes arelowerTCOandincreasedproductivity.However,asillustratedinTable1and Table2,
thereareotherspecificadvantagesaswell.

USbasedAuto&HomeInsuranceCompany
AreaofEngagement

Solution

CustomerBenefits

PerformanceOptimizationof
applicationsinscope

SQLquerytuningoverall
60%gaininperformance

BusinessObjectives

Responsetime
improvementoverall
73%fasterviews

Improvetheperformanceofthe
productinproductionand
recommendimprovementsin
consumption,responsetime,
databaseimprovements,
architecture,WLM,etcbeforeits
Phase2development.

Newindexestoimprove
performance

Technology
MainframeTechnology(REXX,
DB2V7,ISPFServices,JCL,
OMEGAMONforDB2&other
technologies)

EngagementModel/Size
Onsite(30%)Offshore(70%)

ReducedTCO
Enhancedcost
effectiveness
Reducedtechnology
risk

Datamodelredesign
improvements

Optimizedbusiness
functionsandtheir
mappingtosystem
landscape

WLMgoalmodepolicy
recommendations
performancegainof
around20%

Improvedglobal
visibilityof
applicationsand
projects

Allsmalljobsnowrun
underaminuteas
comparedto560
minutesearlier

Reducedtimetotake
decisionsfornew
initiatives

ProjectDuration
4months
Table1:PerformanceOptimizationUSClient

MainframePerformanceManagement

10

GermanFinancialServicesProviderwithGlobalOperations
AreaofEngagement

PerformanceOptimizationof
applicationsinscope

Solution

CustomerBenefits

ExpectedOptimization

BusinessObjectives
Provideperformance
optimizationrecommendations
forapplications,SQLtuning,
relatedsubsystemandrelated
areas

Technology
MainframeTechnology(COBOL,
DB2,CICS,JCL,DB2PM,
OMEGAMONforDB2&other
technologies)

AddressedCPU
consumption~30,000
CPUHours
Expectedoptimizationfor
HCLTsrecommendations
Average~15%
OptimizationRecommendations
SQLquerytuning(100+
recommendations)
Newandmodified
Indexestoimprove
performance

EngagementModel/Size
Onsite(10%)Offshore(90%)

ProjectDuration
4months

Applicationcode
performance
improvements
Architecture&
applicationredesign

ReducedTCO
Easilymanageable
DefineaformalIT
governance
Enhancedcost
effectiveness
ReducedTechnology
risk
Optimizedbusiness
functionsandtheir
mappingtosystem
landscape
Improvedglobal
visibilityof
applicationsand
projects
Reducedtimetotake
decisionsfornew
initiatives

Table2:PerformanceOptimizationGermanClient

MainframePerformanceManagement

11

Endnotes

Oneofthe50questionsthattheresearchteamasked482seniorfinancialexecutiveswas:whodidITreporttoin
theirorganization?Theresponsewasasfollows:42%reportedtotheCFO,33%totheCEO,16%totheCOO,2%to
achiefadministrativeofficerand7%tootherofficers.Forfulldetailsofthesurveyresultsandanalysis,seeJohnE.
VanDecker,2010GartnerFEITechnologyStudy:TheCFOasTechnologyInfluencer"(9April2010IDNumber:
G00175773)p.2
http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&id=1338722&ref=clie
ntFriendlyUrl[October2010]>representswhenanURLwasaccessed.
2

Note:adiscussionontheprosandconsofmainframeliesoutsidethescopeofthispaper.

IBMsshareofthemarketformainframeisbelievedtobearound90%.Othermainframevendorsinclude
ComputerAssociates,BEA,Microsoft,HewlettPackard,FujitsuSiemens,Unisys,Sun,andMicroFocus.

ThelatestbeingantitrustcomplaintsbyTurboHercules,aFrenchmakerofopensourcesoftwareformainframe
computers,andT3Technologies,anAmericandistributorofFlexsoftwarethatrunsmainframesfiles.Onthebasis
ofthosecomplaints,theEuropeanCommissionhasopenedaninvestigationagainstIBM.Fordetails,seeKevinJ.
O'Brien,"EuropetoInvestigateAntitrustComplaintsoverI.B.M.Mainframes"(July26,2010)
http://www.nytimes.com/2010/07/27/business/27blue.html[October2010].Seealso"Antitrust:Commission
InitiatesFormalInvestigationsagainstIBMinTwoCasesofSuspectedAbuseofDominantMarketPosition"(July
26,2010;Ref.:IP/10/1006)
http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1006&format=HTML&aged=0&language=EN&gu
iLanguage=en[October2010].Seealso"NEONFilesMotionforPartialSummaryJudgmentinNEONv.IBM"
(September15,2010)http://www.marketwatch.com/story/neonfilesmotionforpartialsummaryjudgmentin
neonvibm20100915?reflink=MW_news_stmp[October2010].Similarconcernshavebeenexpressedina
comprehensivestudybytheIndianCouncilforResearchandInternationalEconomicRelations.Fordetails,see
RajatKathuria,etal.,"TheIssuesofCompetitioninMainframeandAssociatedServicesinIndia"(March11,2010)
http://openmainframe.org/featuredarticles/theissuesofcompetitioninmainframeandassociatedservic.html
[October2010].
5

KevinJ.O'Brien,Ibid.AccordingtoToniSacconaghiofBernsteinResearch,nearly40%ofIBMsprofitsare
mainframerelated.CitedinTheReturnofthemainframeBackinFashion(January14,2010)
http://www.economist.com/node/15276714?story_id=15276714&source=hptextfeature[October2010].

Heisreferringinparticulartothemovementtowardclient/serverplatformsandcomplementarymetaloxide
semiconductor(CMOS)technologycombinedwiththeavailability,reliability,security,scalability,and
manageabilityofmainframes,thegrowthoftheInternetandebusiness,andtheY2Kproblem,allofwhichled
manytobelievethatthemainframewaslivingonborrowedtime.Fordetails,seeJohnR.Phelps,TheFutureof
MainframesLooksSurprisinglyGood,(August/September2003issueofzJournal,nowapartofmainframezone)
http://www.mainframezone.com/itmanagement/gartnercioupdatefutureoftheibmmainframethelooks
surprisinglygood[October2010].

MainframePerformanceManagement

12

AccordingtoIDC,theworldwideserverrevenuein2009wassaidtohavedeclined18.9%to$43.2billionwhen
comparedto2008,andworldwideunitshipmentsdeclined18.6%to6.6millionunitsoverthesameperiod.For
details,seeIDC,"WorldwideServerMarketReboundsSharplyinFourthQuarterasDemandforBladesandx86
SystemsLeadstheWay"(24Feb2010)http://www.idc.com/getdoc.jsp?containerId=prUS22224510[October
2010].Accordingtooneestimate,IBMsmainframerevenuetookaplungeof39%inthesecondquarterof2009.
SeeSteveHamm,"IBMDefendsItsBigIron"(August4,2009)
http://www.businessweek.com/technology/content/aug2009/tc2009084_001429.htm[October2010].

Thereareseveralarticlesonthissubjects.See,forinstance,ZacksInvestmentResearch,"IBMsMainframeWins
Customer"(June30,2010)
http://www.zacks.com/stock/news/36260/Zacks+Analyst+Blog+Highlights%3A+Union+Pacific%2C+IBM%2C+Hewle
tt+Packard%2C+Vornado+Realty+and+KKR+Financial[October2010].SeealsoRobertL.Mitchell,""Morphingthe
mainframe"(February6,2006)http://features.techworld.com/operatingsystems/2229/morphingthemainframe/
[October2010].
9

ThezEnterprisesystemsissaidtostartatabout$1millionandaccordingtoLouMiscioscia,ananalystwithCollins
Stewart,theyareexpectedtogenerateaprofitmarginofabout70percent,vs.a46percentmarginforthe
companyasawhole.CitedinByKatieHoffman,"IBMMainframes:BoringbutProfitable"(July22,2010)
http://www.businessweek.com/magazine/content/10_31/b4189041885991.htm[October2010].SeealsoIBM,
"IBMzEnterpriseSystem,"http://www03.ibm.com/systems/z/hardware/zenterprise/andalso"SuccessStories,"
http://www03.ibm.com/systems/z/success/index.html[October2010].AccordingtoMartinKennedy,the
managingdirectorofCitigroup'senterprisesystemsinfrastructure,"wewillbeabletocollapsemultipleexisting
largesystemsintothenewwatercooledz196,whichisahugestepinthecompany'sattempttoshiftmoreofitsIT
dollarsawayfrominternaloperationsandmaintenanceandtowardcustomerfacingefforts.Formoredetails,see
BobEvans,"GlobalCIO:IBM'sBlazingNewMainframeWinsRavesfromCitigroup,"(September2,2010)
http://www.informationweek.com/news/global
cio/interviews/showArticle.jhtml?articleID=227200199&pgno=2&queryText=&isPrev=[October2010].Seealso
JohnR.PhelpsandMikeChuba,"IBMAddsIntegratedManagementRoleforMainframe,"July27,2010;ID:
G00205553)
http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&resId=1411013&ref=
QuickSearch&sthkw=mainframe[October2010].
10

MIPSreflectthemainframesfootprint,andoverthepastdecadeithaswitnessedasteadyrisewithinstalled
capacityreachingahighof11.1millionin2007.Fordetails,seeTimothyPrickettMorgan,"TheIBMMainframe
Base:AliveandKicking"(July10,2007)http://www.itjungle.com/big/big071007story01.html[October2007].

11

Foraquickoverviewonthisaspect,seeIBM,"TheIBMEnterpriseSystemAdvantageRunningtheWorld'sMost
SophisticatedBusinessTransactions,"http://www03.ibm.com/systems/migratetoibm/getthefacts/market.html
[October2010].SeealsoAnneRawlandGabriel,"DebateovertheFutureofMainframeComputingRagesOn"
(September16,2008)http://www.wallstreetandtech.com/itinfrastructure/210601591?pgno=1[October2010].

12

Forinstance,inthefuturebanksneedtoaccommodatefortheincreasingnumberofcustomersaccessing
mobilebankingapplication.AccordingtoastudybyBergInsight,"thenumberofactiveusersofmobilebanking
andrelatedfinancialservicesworldwideisforecastedtoincreasefrom20millionin2008to913millionin2014."

MainframePerformanceManagement

13


BergInsight,"MobileBankingandPayments,"p.3,http://www.berginsight.com/ReportPDF/ProductSheet/bi
mbpps.pdf[September2010].Anticipatingsuchhighvolume,IBMhasdevelopeda5.2GHzchipconsideredtobe
thehighestspeedratingtodateforitsfastestmainframecomputers.SeeBrookeCrothers,"IBMShips5.2GHz
Chip,ItsFastestYet"September1,2010)http://news.cnet.com/830113924_32001529764.html#ixzz10uOqdeX8
[September2010].
13

RyanArsenaultsobservationcapturestheviewsofmanyexpertsthus:[t]heproblemisnotthemainframe
platform,butthefactthatmanyinsurancecompaniesaremakinguseoflegacymainframeapplicationcodewhich
wasoriginallydesignedtoprovidequotationstorealpeople.Whatisneededisanimprovementinthe
performanceofhardwareandsoftwareapplicationsthatusecomputerresourcesforrunningatransactionsuchas
generatinginsurancequotes.Fordetails,seeRyanArsenault,"MainframesRunintoPerformanceProblemswith
OnlineInsuranceComparisonShopping,"http://itknowledgeexchange.techtarget.com/mainframe
blog/mainframesrunintoperformanceproblemswithonlineinsurancecomparisonshopping/[September2010.
14

Improvingperformanceofbusinessapplicationsisakeytodeliveringbusinessvaluetocustomersandtheirend
users.ArecentsurveybytheAberdeenGroupfoundthatcompaniesthatmonitoredandmeasuredapplication
performanceatthepointofconsumptionbyendusershadfewendusercomplaints.Forfulldetailsofthesurvey,
seeJeffreyHill,EndUserExperienceMonitoringandManagement(August2010)
http://v1.aberdeen.com/launch/report/benchmark/6778RAenduserexperiencemonitoring.asp[October2010].
Writingabouttheexcessivesignificancegiventooperationalperformancemetricsinthepost1990sera,Michael
Bittermannotedthat"ITcontinuestofocusonoperationsand...explainhowtheclockworkswhenusersare
asking,whattimeisit?Theavailabilityoftoolsformonitoringhardwareandnetworkperformance[hasgiven]IT
theabilitytooverwhelmuserswithstatisticsregardingITperformance.Users[become]baffledbythemetricsthat
showthatmainframeavailabilitywas99.99%yettheycouldntenterdataoraccesssystems.Forthecomplete
report,seeMichaelBitterman"ITMetricsfortheInformationAge"(July25,2004)http://www.performance
measurement.net/newsdetail.asp?nID=198[October2010].

MainframePerformanceManagement

14