Oracle Real Application Testing Tips from Bank of America

Presenter : Keerti Prasad, Vice President, Bank of America

Disclaimer

All the views expressed in this presentation are the opinion of the author and do not necessarily reflect the views of Bank of America

Speaker Info

Keerti Prasad
Working in Merrill Lynch/Bank of America for 6+ years on varied applications 18+ years experience with Oracle database and its related technologies

Agenda
! Overview ! Role of Real Application Testing in Migrations ! Database Replay details ! Issues encountered during Database Replay ! Summary ! Q&A

Overview What is Real Application Testing

! Real Application Testing ! Oracle tool to capture actual workload being performed on a database and then to replay it. ! Modules Capture, Preprocess, Instantiate, Replay ! Replay Components - DB Replay and SPA

Overview - Challenges
! Business often need new features which are available in latest software ! Infrastructure needs to be on supported version / cheaper alternative triggering migration ! Application or infrastructure changes need a rapid deployment for break fix or version changes. ! Performance of the newer systems need to be at par or better ! Any Changes Need to be tested. ! Testing methods need to be fast, reliable and consistent ! We need to simulate actual production data/volumes on the testing environment ! Real Application Testing can achieve all the above
6

Overview Factors Driving Changes

! Application : Break-fixes, increasing data volumes, new requirements ! OS Platforms : Server patching/upgrades, third party upgrades ! Database : Upgrades, patching, parameter changes ! Storage/Network : Datacenter Moves

Overview - Application/Database Info

! Database ! Database size is ~5Tb ! Workload can roughly be classified into 2 parts: ! OLTP transactions is done during business hours ! Batch processing is done after business hours with some overlap ! SLA is 3ms and Rate of Redo generated varies between 3-5Gb/hour ! Application ! Some are vendor supplied and some in house developed
8

Overview - Current Configuration

60+

End Users

Production Support

RAC: Online

Adhoc Queries

RAC: Reporting

OGG
Online Database Reporting Database

Role of Real Application Testing in Migration High Level

! Three major Migrations done within the span of 3years :

No. Date 1 2 3

RAT Testing Change Details Hardware, OS, Database, Application Sep 2010 Not Used Migration/Upgrade/Release OS, Database, Application Upgrade/ May 2011 Not Used Release OS Upgrade, Database migration to 3Aug 2012 Extensive Node RAC, Application Upgrade/Release

Issues Many Many None

Role of Real Application Testing in Migration 1

! Migration 1 (Oracle DB migration 10.2.0.4 on Solaris to 11.2.0.1 on Linux): ! Types of changes: ! Hardware changes ! OS migration Solaris to Linux ! Oracle Upgrade 10.2.0.4 to 11.2.0.1 ! Application Upgrade/Release ! Some problems we encountered after migration: ! Optimizer related issues ! Library cache and mutex issues ! High version counts ! One-off patches were applied and a few parameters were changed
11

Role of Real Application Testing in Migration 2

! Migration 2 (Oracle Upgrade 11.2.0.1 to 11.2.0.2): ! Types of changes: ! OS upgrade ! Oracle Upgrade 11.2.0.1 to 11.2.0.2 ! Application Upgrade/Release ! Some problems we encountered after migration: ! Memory resizing operations ! Issues caused due to cursor sharing set to force ! High version count issues ! One-off patches were applied and a few parameters were changed ! Real Application Testing was in a rollout phase

12

Role of Real Application Testing in Migration 3

! Migration 3 (Oracle migration Standalone to 3 Node RAC) : ! Types of changes: ! Hardware ! OS upgrade ! Oracle GI Upgrade 11.2.0.2 to 11.2.0.3 ! Database migration from standalone to 3 Node RAC without version change. ! Application Upgrade/Release ! Real Application Testing was used extensively. ! Migration was accomplished with minor issues and performance of the system was satisfactory.

13

Role of Real Application Testing in Migration Problems Exposed by DB Replay

! Problems occurred during migration which were exposed by DB Replay ! 11g Child Cursor issue replay was going into a hang situation. High version counts from child cursor was the cause ! Storage issue Replay was showing high i/o in some reports. Storage/ hardware changes were made which fixed the issue. ! Authentication issue Oracle database owner on OS is authenticated by third party tool. During replay, clients encountered errors. On investigation, it was found to be caused due to a bug in the authentication tool
14

DB Replay details - Typical Cycle

! Typical DB Replay cycle: ! Workload is captured on source(production) for one week daily between 9am-5pm ! Source database is restored to test system. ! DB Replay done for first day ! Batches are run by application team in the night. ! This is repeated for a week. ! In the end DB Replay reports, AWR comparison reports and batches load timings are analyzed

DB Replay details Impact on Production System during Capture

! Impact during Capture and steps taken to circumvent it ! Load on server is minimal ~ 5% CPU ! Only OLTP traffic is captured during business hours, since batches can be easily reproduced. ! Captures were done for a typical workload cycle Mon-Fri business hours. They were retaken ONLY when the database went through release.

16

DB Replay details Capture Tips

! We took a capture for a whole business day, but preprocessing ran into an issue since it creates objects in sysaux tablespace which grew a lot. So, a capture interval was set to create files of ~2Gb. Divergence increase minimal due to split capture. ! To facilitate the above, script was created to start/end capture and restart a new one. ! We needed to capture only the users that performed OLTP which is not easily reproducible, hence user filters were used.
17

DB Replay Details - Capture Report

Statistic Name Value % Total ---------------------------------------- ------------- --------DB time (secs) 48407.45 78.86 Average Active Sessions 13.44 User calls captured 13141577 89.62 User calls captured with Errors 43961 Session logins 726 9.45 Transactions 271724 90.02 ----------------------------------------------------------------

Major DB Time/ User Calls/ Transactions Captured

Top Service/Module Captured Service Module % Activity Action % Action -------------- ------------------------ ---------- ------------------ ---------APPCONN1.US.ML UNNAMED 55.22 UNNAMED 55.22 APPCONN2.US.ML UNNAMED 6.30 UNNAMED 6.30 APPCONN1.US.ML APP_SCHE.PKG_APPS_PFL.sp 5.62 Identify the Maste 5.62 PKG_APPS_PFL.SP_CALCULAT 1.62 Calculate EPI Alig 1.62 SQL*Plus 1.52 UNNAMED 1.52 -------------------------------------------------------------------------------Workload Filters DB: SOURCE Snaps: 3008-3011 # Filter Name Type Attribute Value --- ------------------------ ------- ------------ -------------------------1 APPSCHEM_USER_162 INCLUDE USER APPSCHEM_USER 2 APPSCHEMAA2_USER_162 INCLUDE USER APPSCHEMAA2_USER ----------------------------------------------------------------------------

Services that were captured

Filtered Users Captured

18

DB Replay Details - Replay Details/Tips

! SCN was used for synchronization parameter ! Calibration tool used to calculate number of clients ! Typically 15-25 clients were used for replay ! Remapping was achieved by defining database service name for the clients ! Think time auto correct was used. ! Script was used to run the replays. ! Scale up multiplier set to 1 as Real Application testing was used as a tool for performance testing and not stress testing.
19

DB Replay Details - Replay Report

Replay Options ================================================================= Option Name Value Synchronization SCN Connect Time 0% Think Time 0% Think Time Auto Correct TRUE Number of WRC Clients 15 (15 Completed, 0 Running ) Replay Statistics ================================================================= Statistic Replay Capture DB Time 32811.942 s 37622.648 s AvG Active Sessions 7.43 10.45 User calls 12824968 12824968 Network Time 2925862.226 s . Think Time 766.466 s . =================================================================== Replay Divergence Summary Divergence Type Count % Total Session Failures During Replay 0 0.00 Errors No Longer Seen During Replay 79 0.00 New Errors Seen During Replay 532 0.00 Errors Mutated During Replay 0 0.00 DMLs with Different Number of Rows Modified 3382 0.03 SELECTs with Different Number of Rows Fetched 514558 4.01

Replay errors were minor

20

DB Replay Details Capture and Replay Compare

Information About AWR and Time Periods
Capture End Time Duration Replay Start Time 2012-07-16 11:03:08 2012-07-16 17:45:34 2012-07-16 12:03:51 2012-07-16 18:57:42 1.01 hours 1.2 hours

Interconnect Usage
!!
average 8Kb ping time (ms) average fast grant time (ms) average fast block transfer time (ms) total GC blocks transferred total interconnect usage (G bits) interconnect throughput (Mb/s) Capture Not RAC Not RAC Not RAC Not RAC Not RAC Not RAC Replay 667.56 0.24 0.58 21228672 4001.231 926.56

Main Performance Statistics

!!
Database Time CPU Time User I/O Wait Time Cluster Wait Time Capture total time(s) 44009.17 10567.09 30913.41 N/A Replay total time(s) 34604 10623.24 23118.46 0

ADDM REPORT
!!
"User I/O" wait Class

!!
Capture Replay Capture Replay Capture Replay

impact (sec) 30608 22439 N/A 849 1615 2453

average active sessions 8.39 5.07 N/A 0.19 0.44 0.55

percent active sessions 69.31 63.49 N/A 2.4 3.66 6.94

I/O to Data and Temp Files

!!
average time of single block read (ms) total number of single block reads total physical read total physical write read rate write rate Capture 5.95 4537537 59.515 G 7.201 G 16.71M/sec 2.022M/sec Replay 3.8 5144026 72.303 G 9.861 G 16.743M/s 2.284M/s

Global Cache Messaging PL/SQL Execution

21

DB Replay Details Performance Comparison

AWR Performance Comparison for a Replay
1st (CAPTURE) Wait Class Event db file sequential read User I/O direct path read log file sync log file parallel write db file parallel write User I/O Commit System I/O System I/O Waits Avg Time Time (s) (ms) 2nd (REPLAY) Waits Time (s) Avg Time (ms)

Performance

4,263,441 37,343

8.76 2,868,731 11,541 32,702 75,012 357,429 809,049 213 498 688

4.02 Better 2.84 Better 1.39 Better 0.85 Better

78,285 3,551 45.41 205,154 3,090 15.06 291,138 1,547 257,180 1,081 5.32 4.2

613 18.75 Better

22

DB Replay Details Performance Comparison

AWR Performance Comparison for a Replay with I/O issues
1st (CAPTURE) Wait Class Waits Avg Time Time (s) (ms) 2nd (REPLAY) Waits Avg Time (s) Time Performance (ms) Avg.better but 1.54 Total Wait high 1.94 Better 0.4 Better 2.09 Better Avg.better but 0.66 Total Wait high

Event

db file sequential read User I/O Log file sync log file parallel write Commit System I/O

5,243,296 34,811 368,717 1,698 702,920 1,616 2,204

6.64 16,932,235 26,133 78,157 593 204,975 152 1 428 4.6 2.3

275,891 3,554 12.88

Read by other session User I/O db file scattered read User I/O

50 23.11 19,227,535 12,628

23

DB Replay Details Test Cycles

! Multiple cycles of OLTP captures and replays were done. ! Captures taken were minimal and retaken only when source had gone through a release cycle. ! Sometimes, after the restore, a guaranteed restore point was set on the target system, and the database was flash-backed to run multiple cycles with different settings to save restore time. ! One off patches applied were first tested with a DB Replay cycle

24

Issues encountered during DB Replay

Issue 1 : Pre-processing made system tablespace become full Fix : Run the capture in batches. Issue 2 : ORA-15590: encountered an incomplete workload capture file. Fix : As per oracle this could be ignored. Issue 3 : ORA-600 [17183] calling dbms_workload_replay process_capture Fix : Applied DB Replay patch

Issues encountered during DB Replay (Contd.)

Issue 4 : Replay seemed to be hanging Fix : There were some minor changes in workload. Replay ran fine after stats was gathered on the database. Issue 5 : Batches after replay complained of constraint violations. Fix : The violations were for sequences. Once these were bumped up, the replay ran successfully.

26

Issues encountered during DB Replay - Limitations

! Only selects can be ramped up. DMLs can only be replayed once. ! Some part of our application uses XA connections. They are not yet supported in Real Application testing. ! CTAS operations are not captured

27

Oracle ACS for Bank of America

Service Highlights
! ! ! Comprehensive support offering with Solution Support Center Focused engineering team Fast track solutions based on intimate knowledge of customer environment and business needs Proactive advice and knowledge transfer Performance and patch reviews Single point of contact for all technical issues - Technical Lead Single point of contact for all escalations Service Delivery Manager (SDM) Onsite assistance Proof of Concept and Test Case assistance

Value
!! Maximize Availability !! Mitigate

risks Performance costs

!! Optimize !! Reduced

! ! ! ! ! !

28

Summary
! Lessons Learned - Traditional testing methods limitations: ! They have to be setup each testing cycle ! They require a lot of resources and coordination is needed amongst all teams ! Baselines set can be questionable making it difficult to compare results, remediate or tune the system ! Last month our application in BofA migrated successfully to RAC platform without any issues unlike the previous migrations. Considering past migrations, this undoubtedly was due to a combination of traditional and Real Application Testing. ! Reports from DB Replay gave conclusive evidence of system performance which boosted DBA and Application teams confidence on the migration.

Q&A

Questions ?

30

Acknoledgements/Email
My Email: Keerti Prasad (Keerti.Prasad@bankofamerica.com) Acknowledgements: Sridhar Kakumanu (Sridhar.Kakumanu@bankofamerica.com) Mohammad Awwal, Ph.D. (Mohammad.Awwal@bankofamerica.com) Jeremy Schirer (Jeremy.Schirer@bankofamerica.com) Bryant Bisnar (Bryant.Bisnar@oracle.com) Stephen Ahn (Stephen.Ahn@oracle.com)

For general information on Advanced Customer Support Services, go to oracle.com/acs or e-mail acsdirect_us@oracle.com
31