1

<Insert Picture Here>

Maximizing Database Performance: Performance Tuning with DB

Time
Kurt Engeleiter, John Beresniewicz, Cecilia Gervasio
Oracle America

The following is intended to outline our general

product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracles
products remains at the sole discretion of Oracle.

Agenda

NEWS: DB Time is not news

DB Time Fundamentals
DB Time Method for Performance Tuning
DB Time Based Tools
ADDM
Enterprise Manager

<Insert Picture Here>

NEWS: DB Time is Not News

DB Time is Pervasive, Familiar, Evolutionary

DB Time is the fundamental measure of Oracle
performance
Arrived at progressively but designed-in from 10g

You are likely using DB Time methods already

AWR and ASH reports
Enterprise Manager

DB Time is performance analysis science matured

Wait event analysis, YAPP and tracing are all its ancestors

DB Time on the Database Home Page

Host CPU
Is host CPU-bound? Is this DB responsible?

Active Sessions
DB Time accumulation by CPU, User I/O, Wait

DB Time on the Performance Page

Area under curve is equal to DB time
Wait class breakdown indicates solution space

DB Time on the Top Activity Page

Plots DB Time from ASH data
Shows top DB Time consumers by SQL and Session

DB Time

DB Time in AWR Report

Time Model measures and

accumulates DB Time through
session instrumentation
DB Time and DB CPU most
important
Also shows DB Time expended
in various sub-operations

10

DB Time in AWR Report

If you tune by Wait

Events you are tuning by
DB Time

Top 5 Timed Events in

the AWR report shows
CPU time and non-idle
waits for foregrounds

11

DB Time in the Grid Control DB Loadmap

Maps DB Time across the enterprise
Cell size proportional to DB Time

12

<Insert Picture Here>

DB Time Fundamentals

13

Why Do We Care About Time?

Human time is critical to the enterprise

Systems performance affects business goals
Human time + technology resource time

Time is money
Performance improvement means doing things faster

Performance is always and only about time

14

DB Time Defined

Total time in database calls by foreground sessions

Includes CPU time, IO time and non-idle wait time
DB Time is not called response time on purpose
Common currency for Oracle performance analysis
Database time is total time spent by user
processes either actively working or actively
waiting in a database call.
15

Session-level Concepts
Database Time (DB Time) =
Total time session spends in all database calls

Active Session =
Session currently spending time in a database call
Average Activity of the Session (% Activity)
= Ratio of time active to total wall clock time
= Sum(Active Time) / Elapsed Time
Browse
books

Read reviews
for a book

Add to cart

Checkout

TIME

= time spent in database

16

Multiple Sessions
DB Time = Sum of DB Time over all sessions

Avg. Active Sessions = Sum of Avg. Activity over all sessions

= Sum(DB Time) / Elapsed Time
User 1
User 2
User 3

User n

TIME
= time spent in database

17

Visualizing DB Time

User 1
User 2
User 3
User n

t0

t1

4
3
2
1

Active Sessions over time

TIME

18

Enterprise Manager Performance Page

Active Sessions by wait class over time
Colored area is the amount of DB Time

DB Time on User I/O is always blue

DB Time on CPU is always green

19

<Insert Picture Here>

DB Time and
System Performance

20

System Load and DB Time

More users
More database calls
DB Time increases

Larger transactions
More SQLs per call
DB Time increases

DB time increases as system load increases.

21

System Performance and DB Time

I/O performance degrades
I/O wait time increases
DB Time increases

Application performance degrades

Longer elapsed time per SQL
DB Time increases

DB time increases as performance degrades.

22

Host Performance and DB Time

Host is CPU bound
Foregrounds accumulate active run-queue time
Wait event times are artificially inflated
DB Time increases

Tune for CPU before waits when CPU constrained

23

CPU Run-Queue and DB Time

Recorded wait time

Recorded wait time

User 1

Run-queue
Db file sequential read

Actual wait time

Run-queue
On CPU

Db file sequential read

On CPU

Actual wait time

DB time is inflated when host is CPU-bound

24

<Insert Picture Here>

The DB Time Method for

Performance Tuning

25

The DB Time Method: Short Course

or database
Determine where
time isjust
spent,
reduce it!
askand
ADDM

26

The DB Time Method: Process

1.
2.
3.
4.
5.

Identify performance issue

Scope the issue
Set goals
Data capture
AWR handles this
Investigate DB time distribution
Identify largest potential improvement

6. Modify system to tune for largest gain

7. Evaluate against goals
Performance tuning by removing excess DB time
27

Identify and Scope the Problem

Big spike = big problem!
Container (host) or database issue?

28

Investigate DB Time Distribution

Identify excess DB time
Look for large skews in time that may represent inefficiencies

System scope:
Resource limits is problem outside the DB?
Resource contention (latches)

Application scope:
Service, module, action
SQLID, rowsource
Resource contention (enqueues)

Session scope:
Long-running SQL
Resource contention (holding enqueues in uncommitted txns)

29

Investigate DB Time distribution

Which wait classes are dominant?
Top lists show skew in key dimensions (SQL, session, file, etc.)

30

Identify Time-saving Modifications

System scope:
Initialization parameter changes
Add resources or capacity

Application scope:

SQL Tuning Advisor: SQL profiles

Real-time SQL Monitoring
Access Advisor: Physical schema layout
Re-write SQL

Session scope:
Kill session for immediate contention relief
Application design changes often long-term solution

31

Critical Success Factors

Match solution scope to problem scope

Dont tweak optimizer parameters before tuning SQL

Address root causes, not symptoms

Proceed iteratively one fix at a time
Concurrent fixes should be orthogonal

Measure and validate results at each successive step

Stop when goals are met
32

Advantages of the DB Time Method

Tunes exactly what affects users:

Time wasted in database

Data capture not necessary:

Always on instrumentation framework (AWR, ASH)
No requirement to reproduce problem (unlike tracing)

Superior coverage of all scope levels

Works for concurrency problems such as locking
Tight integration with kernel-engineered tools

Maturation and extension of well-proven methodologies

33

<Insert Picture Here>

DB Time Based Tools:

ADDM
Enterprise Manager

34

Tools for Applying DB Time Method

Two use-cases, one method:

1. Tuning steady-state performance
Improve overall workload throughput or response time
Best practice: use ADDM

2. Diagnosing transient performance problems

Confirm and investigate reported performance issues
Best practice: use EM real-time screens

35

Example: Using ADDM

Users complain about bad performance
AWR report indicates some unusual issues on system
What to do?

36

What does the Internet advise?

Lets search on enq: HW contention
9,000 hits and LOTS of discussion

37

Example: Following the links

Many hits are other people asking the same question what do I
do about this wait event?

38

Solution: ADDM in Enterprise Manager

Performance Page shows a spike of unusual activity
Click on the ADDM analysis icon

39

Solution: Using ADDM

ADDM performance findings are DB Time based
Ranked by impact = percent of DB Time affected
Click on the issue with the highest impact

40

Solution: Using ADDM

ADDM recommendations describe solutions for findings
EM presents implement actions where possible

41

Best Practice: Enterprise Manager Real-time Interface

Transient (sub-hour) or immediate time scope

Requires interactivity of UI

Click on the big stuff

Data visualizations display skew directly

Takes some expertise to separate symptoms from

root causes

42

Click on the Big Stuff Follow the DB Time

Spike in DB Time signals possible performance issue
Click through to Top Activity for more details

43

Follow the DB Time Top Activity Drilldown

Top Activity Page confirms problem and shows highly skewed DB
Time by SQL ID. Click on top SQL ID for further details.

44

Click on the Big Stuff Follow the DB Time

SQL Details confirms this SQL produced the spike in DB Time and
shows parallel execution. Click on SQL Monitoring to see DB
Time by execution plan steps.

45

Follow the DB Time SQL Execution Details

46

DB Time Method Summary

DB time is the fundamental performance metric

The method allows DB time analysis at many scopes

Proper scoping of problems and solutions is critical to success

DB time based diagnosis removes value judgments

Scientific method, not sorcerers magic

Performance improvement means doing the same

work in less DB Time

47

Oracle Enterprise Manager 11g

Resource Center
Access Videos, Webcasts, White Papers, and More

Oracle.com/enterprisemanager11g

49

Additional Oracle Enterprise Manager Sessions

Wednesday, Sept. 22

Location

10:00 a.m. - Manage the Manager: Diagnosing and Tuning

Oracle Enterprise Manager

Moscone S Rm 102

11:30 a.m. - Maximizing Database Performance: Performance

Tuning with DB Time

Moscone S Rm 104

11:30 a.m. - Make Upgrades Uneventful Using Oracle

Enterprise Manager and My Oracle Support

Moscone S Rm 310

12:30pm Extracting Real Value from Your Data with Apache

Hadoop

Hilton Hotel, Plaza B

1:00 p.m.- Reducing the Risk of SOA Transactions

Marriott Marquis, Salon

6

1:00 p.m. - SQL Tuning Roundtable with Oracle Gurus

Moscone S Rm102

4:45 p.m.- Strategies for Monitoring Large Datacenters with

Oracle Enterprise Manager

Moscone S Rm102

4:45 p.m.- Oracle SOA Management Best Practices, Tips, and

Techniques

Moscone W L3, Rm
3018

4:45 p.m.- Oracle E-Business Suite Technology: Vision,

Release Overview, Product Roadmap

Moscone W L3, Rm
3002 / 3004

50

Additional Oracle Enterprise Manager Sessions

Thursday, Sept. 23

Location

9:00 a.m. - Oracle WebLogic Server Management for Oracle

DBAs

Marriott Marquis,
Salon 9

9:00 a.m. - Enabling Database as a Service Through Agile SelfService Provisioning

Moscone S. Room
102

9:00 a.m. - Reduce TCO with Oracle Application Management

Pack for Oracle E-Business Suite

Moscone W L2, Rm
2024

10:30 a.m. - Best Practices for Managing Your PeopleSoft

Applications

Marriott Hotel, Golden

Gate A

10:30 a.m. - Oracle Enterprise Manager Grid Control

Deployment Best Practices

Moscone S. Room
102

10:30 a.m. - Managing Sun SPARC Servers with Oracle

Enterprise Manager Ops Center

Moscone S. Room
252

10:30 a.m. - Heterogeneous Data Masking: Oracle, SQL Server,

and DB2 Database Best Practices

Moscone S. Room
306

12:00 p.m. - Scalable Enterprise Data Processing for the Cloud

with Oracle Grid Engine

Moscone S. Room
310

12:00 p.m. - Spot Problems Before Your Users Call: User

Experience Monitoring for Oracle Apps

Marriott Hotel, Golden

Gate A

12:00 p.m. - Reduce Problem Resolution Time with Oracle

Database 11g Diagnostic Framework

Moscone S. Room
102

51

Additional Oracle Enterprise Manager Sessions

Thursday, Sept. 23

Location

1:30 p.m. - Patching Enterprisewide Databases: Automation

Techniques and Real-World Insights

Moscone S. Room
310

1:30 p.m. - Managing User Experience: Lessons from eBay

Marriott Hotel, Golden

Gate A

1:30 p.m. - Deep Java Diagnostics and Performance Tuning:

Expert Tips and Techniques

Marriott Marquis,
Salon 9

1:30 p.m. - Oracle Enterprise Manager Configuration

Management Unleashed: Top 10 Expert Tips

Marriott Marquis,
Salon 6

1:30 p.m. - Oracle Enterprise Manager Security Best Practices

Moscone S. Room
102

3:00 p.m - The X-Files: Managing the Oracle Exadata and

Highly Available Oracle Databases

Moscone S. Room
102

3:00 p.m. - Monitoring and Diagnosing Oracle RAC

Performance with Oracle Enterprise Manager

Moscone S. Room
310

52

Oracle Enterprise Manager Hands On Labs

Monday September 20, 2010
3:30 p.m. - 4:30 p.m.

Database Performance Diagnostics and Tuning

Marriott Hotel,
Salon 12/13, YB Level

5:00 p.m. - 6:00 p.m.

Provisioning, Patch Automation, and Configuration Management Pack

Marriott Hotel,
Salon 12/13, YB Level

5:00 p.m. - 6:00 p.m.

Oracle Application Mgmt. Pack for Oracle E-Business Suite: Monitor/Clone

Marriott Marquis, Nob Hill

Tuesday September 21, 2010

11:00 a.m.-12:00 p.m.

Using Oracle Application Change Management Pack for Oracle E-Business Suite

Marriott Marquis, Nob Hill

12:30 p.m.-1:30 p.m.

Database and Application Testing

Marriott Hotel,
Salon 12/13, YB Level

2:00 p.m. - 3:00 p.m.

Oracle Fusion Middleware Management

Marriott Hotel,
Salon 12/13, YB Level

3:30 p.m. - -4:30 p.m.

Provisioning, Patch Automation, and Configuration Management Pack

Marriott Hotel,
Salon 12/13, YB Level

Wednesday September 22, 2010

4:45 p.m. - 5:45 p.m.

Database and Application Testing

Marriott Hotel,
Salon 12/13, YB Level

4:45 p.m. - 5:45 p.m.

Oracle Application Mgmt. Pack for Oracle E-Business Suite: Monitor/Clone

Marriott Marquis, Nob Hill

Thursday September 23, 2010

9:00 a.m. - 10:00 a.m.

Database Performance Diagnostics and Tuning

Marriott Hotel,
Salon 12/13, YB Level

10:30 a.m. - 11:30 a.m.

Oracle Fusion Middleware Management

Marriott Hotel,
Salon 12/13, YB Level

53

Oracle Enterprise Manager Demogrounds

DEMO TITLE

L O C AT I O N

Oracle Real Application Testing: Database Replay

Moscone West

Oracle Real Application Testing: SQL Performance Analyzer

Moscone West

Self-Managing Database: Automatic Performance Diagnostics

Moscone West

Self-Managing Database: Automatic Fault Diagnostics

Moscone West

Self-Managing Database: Automatic Application and SQL Tuning

Moscone West

Application Quality Management: Application Testing Suite

Moscone South - S022

Real User Monitoring with Oracle Enterprise Manager

Moscone South - S021

Siebel CRM Application Management

Moscone South - S024

Real User Monitoring with Oracle Enterprise Manager

Moscone West

Oracle WebLogic Server Management and Java Diagnostics

Moscone West

SOA Management with Oracle Enterprise Manager

Moscone West

Oracle Business Transaction Management

Moscone West

Push Button Provisioning and Patch Automation

Moscone West

Smart Configuration Management

Moscone West

Oracle Enterprise Manager Ops Center

Moscone West

Managing the Enterprise Private Cloud

Moscone West

System Management, My Oracle Support, and Oracle Enterprise Manager

Moscone West

Self Managing Database: Change Management for DBAs

Moscone West

Oracle Enterprise Manager: Complete Datacenter Management

Moscone West

Self-Managing Database: Data Masking for DBAs

Moscone West

54

55