Oracle On AIX - Best Practices

IBM Global Services
Oracle on AIX : Best Practices
Dr. Andreas Gruber

mySAP Technology Services
November 2003 © 2003 IBM Corporation

IBM Global Services
Agenda
1 Introduction
2 Design of Disk Layout for SAP/Oracle Databases
3 Performance Tuning for the Journaled Filesystem
4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)
5 Multiple SAP Databases on a Single System
2 Oracle on AIX : Best Practices © 2003 IBM Corporation

IBM Global Services
Agenda
1 Introduction

IBM Global Services
Introduction
"A best practice is a technique or methodology that, through

experience and research, has proven to reliably lead to a desired
result." http://searchVB.com
§ Knowledge transfer from SAP infrastructure projects based on

• IBM pSeries, Enterprise Storage Server, EMC Symmetrix,
• Storage Area Networks
• Tivoli Storage Manager and Data Protection for SAP
§ Performance tuning as a holistic approach that involves all

components of a system
§ Explanation and advertising of new features implemented by

hardware and software components

IBM Global Services
Introduction > Oracle Performance Tuning
Oracle Performance Tuning

List of Priorities
§ Business Rules
§ Data and Application Design
§ Logical Structure of the Database
§ Database Operations and Access Paths
§ Memory Allocation AIX / Hardware
§ I/O and Physical Structure AIX / Hardware
§ Resource Contention
§ Underlying Platform(s) AIX / Hardware

IBM Global Services
Agenda
1 Introduction

IBM Global Services
Disk Layout > Old Style
Old Style Disk Layout for SAP/Oracle Database
§ Use of single and separated disks

• SCSI-Attachment
• SSA-Loops
§ Placement of files according to the following criteria

• Type of Files (Redolog, Archives, Datafiles)
• Data / Index Tablespaces
• Master Data / Transaction Data
§ Elimination of arising hot spots via reorganization of tables into

separate tablespaces
§ Problem
• Wasting of disk resources (e.g. Redologs)
• Complex and difficult administration of scattered data files

IBM Global Services
Disk Layout > New Paradigm
New Paradigm for SAP/Oracle Database
§ Intelligent Disksubsystems
• Internal Read/Write Caches (0,5 - 64 GByte)
• Internal RAID-5 or RAID-10 (or similar) implementations
• SCSI or Fibre Channel-Attachment
§ Oracle's S.A.M.E Strategy (Stripe And Mirror Everything)

EMC Symmetrix / DMX IBM Enterprise Storage Server
Striping § Stripe almost everything § Stripe across ESS RAID Arrays

§ Don't subset by individual disks § Physical Partition striping (16/32 MB)
§ Hardware striping for Log Files
§ Use 1 MB stripe size
Mirroring § Mirror for data protection § Use RAID -5 or RAID -10
Placement Policy § Use "Outside Edge" of disks § Use "Middle" for ESS arrays
to minimize seek

IBM Global Services
Disk Layout > Example
Disk Layout Example for SAP/Oracle Database

IBM Global Services
Disk Layout > Golden Rules for ESS (Shark)
Golden Rules for ESS (Shark)
§ Number of Disksubsystems
• Should be based on throughput needs
• Should not be based on capacity needs (Terabytes)
§ Decision Support Systems (e.g. BW)

• 100-200 GByte Data per 2GBit Fibre Channel Port
• Moderate Cache Requirements
§ OLTP Systems
• 200-300 GB Data per 2GBit Fibre Channel Port
• Large Cache Requirements
§ Number of RAID Arrays according to throughput requirements

• 50 MB/s sustained sequential Read/Write Performance

IBM Global Services
Agenda
1 Introduction

IBM Global Services
Performance Tuning for JFS > JFS vs. Raw Devices
Journaled Filesystem vs. Raw Devices
§ "Raw Devices" correspond to Logical Volumes in AIX

• Disk devices (/dev/rhdisk10) are not allowed to be accessed directly
§ Filesystems are easier to manage for most administrators
§ Journaled Filesystems ensure the integrity of the filesystem

structure and data
• Locking of filesystem structures prevents simultaneous access
§ JFS access to data always uses the AIX Virtual Memory Manager
and the real memory as a buffer for files
• Improves performance when data is accessed multiple times
• Oracle data blocks are buffered twice and consume memory

IBM Global Services
Performance Tuning for JFS > Areas of Tuning
Areas of Tuning
ç Oracle Parameters init.ora
ç Asynchronous I/O
ç Read-Ahead
ç File Caching
ç Database Layout
ç Multipath and SAN
ç Cache and internal access
ç Number and speed of drives

IBM Global Services
Performance Tuning for JFS > File Caching
File Caching
§ Memory is categorised into two different types

• Computational memory: Consists of the pages that belong to
working-storage or executable files.
• File memory: Consists of the remaining pages. These are usually
pages from permanent data files in persistent storage.
§ AIX is using up all memory until only "minfree" pages are left
• The percentage of memory that is used for file cache (numperm) can
be seen with the command
/usr/samples/kernel/vmtune (AIX 4.3 + 5.1)
/usr/bin/vmstat -v (AIX 5.2)
• minperm If numperm falls below this level, the page-replacement
algorithm steals both file and computational pages, regardless of
repage rates.
• maxperm If numperm rises above this level, the page-replacement
algorithm steals only file pages.

IBM Global Services
Performance Tuning for JFS > VMM Tuning
Virtual Memory Manager (VMM) Tuning
§ Tuning of the file cache will only improve performance if the system
was memory constrained before
§ Tuning is meant to avoid duplicate buffering of Oracle data blocks
which leads to a waste of physical memory
§ If the SAP R/3 database server has more than 2 GB memory and is
only used for SAP services
• minperm should be set equivalent to ~ 150 MB
• maxperm should be set equivalent to ~ 300 MB
• In case of 3 GB memory this corresponds to
/usr/samples/kernel/vmtune -p 5 -P 10
§ The number of pages on the freelist

• minfree = 120 * Number of CPUs

IBM Global Services
Performance Tuning for JFS > Tuning Read-Ahead
Tuning Read-Ahead
§ VMM tries to anticipate the future need for pages of a sequential file
by observing the pattern a program uses to access the file.
• minpgahead
The number of pages read ahead when the VMM first detects the
sequential access pattern. If the program continues to access the file
sequentially, the next read ahead will be doubled.
• maxpgahead
The maximum number of pages the VMM will read ahead in a
sequential file.
§ Recommended values for SAP database servers

• minpgahead 2
maxpgahead 8 * #CPUs rounded to the next power of 2
maxfree minfree + maxpgahead
• In case of a 6 CPU System this corresponds to ( < AIX 5.2)
/usr/samples/kernel/vmtune -r 2 -R 64 -f 720 -F 784

IBM Global Services
Performance Tuning for JFS > Read-Ahead for backup
Read-Ahead for backup with TSM and TDP
§ Backup of Oracle databases with TSM and TDP heavily rely on

read-ahead
• Performance Improvements of 100% have been observed
§ Journaled Filesystem
• maxpgahead = 256
§ Enhanced Journal Filesystem (JFS2)

• j2_maxPageReadAhead = 256
§ High values of read-ahead might have negative effects on cache

efficiency of disk subsystems for SAP production
• Switch to these parameters only during backup

IBM Global Services
Performance Tuning for JFS > Asynchronous I/O
Asynchronous I/O
§ Asynchronous I/O is an AIX operating system feature consisting of
• Request Queue for I/O write requests
• Kernel processes (aioservers) that take requests from the queue
§ Programs must use the corresponding application programming
interface in order to use asynchronous I/O
• no wait for the completion of a write request
• no blocking of application execution
§ Parameters for tuning asynchronous I/O
• minservers The minimum number of aioservers that are started for
asynchronous disk I/O. The default value is 1.
• maxservers The maximum number of aioservers that are started for
asynchronous disk I/O. The default value is 10. Since each aioserver
uses memory, this number should not be much larger than the
expected amount of simultaneous asynchronous disk I/O requests.
• maxreqs Maximum number of asynchronous disk I/O requests that
can be stored in the queue. The default value is 4096.

IBM Global Services
Performance Tuning for JFS > AIO Tuning Recommendation
AIO Tuning Recommendations

§ Asynchronous I/O is treated like a device and can be managed via
smit aio
• autoconfig should be switched on, to make it "available" after reboot
§ Recommended starting values are
• minserver 2 or Number of CPUs -1, whatever is larger
• maxserver Two times the number of datafiles
• maxreqs 12288
§ The aioserver processes are started as oracle child processes
• for large SAP systems this number may become very large and the
maximum number of processes per user has to be adopted
chdev -l sys0 -a maxuproc=2000
§ The number of active aioservers can best be monitored with the

performance tool "nmon"
• http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/

IBM Global Services
Agenda
1 Introduction

IBM Global Services
AIX 5L and JFS2 > New features
New features with AIX 5L

§ Support for POWER4 processors and LPAR technology
§ Workload Management for CPU, Memory and I/O
§ Alternatively 32-Bit or 64-Bit AIX Kernel
• 64-Bit Kernel recommended for JFS2
§ Enhanced Journal Filesystem (JFS2)
• Max. Filesize 16TB (vs. 64 GB)
• Max. Filesystemsize 16 TB (vs. 1 TB)
• Dynamic number of i-nodes
• B-Tree directory structure
• Higher performance for creation and deletion of files
§ AIX 5.2 implements new commands for performance tuning
• Administration via smit
• Reboot safe

IBM Global Services
AIX 5L and JFS2 > Direct I/O
Direct I/O for JFS2 with AIX 5L

§ Reads under cached I/O - cache miss § Reads with direct I/O

IBM Global Services
AIX 5L and JFS2 > Implications of direct I/O
Implications and restrictions of direct I/O

§ In order to activate direct I/O, the filesystem must be mounted with
the option "mount -o dio"
§ Read-Ahead is disabled for these filesystem
§ The alignment of the read requests must be on 4K boundaries

• In AIX 5.2 ML01 this alignment must be according to "agblksize",
a parameter which is specified during file system creation
• If the alignment criteria is not met, normal cached access takes place
but the cache is discarded immediately
• Oracle Online-Redologs should be located in separate filesystems with
an allocation group blocksize (agblksize) of 512 Byte
§ Applications that see performance benefits when using raw logical

volumes for storage are likely to benefit from the use of Direct I/O
• e.g. Oracle

IBM Global Services
AIX 5L and JFS2 > Locking and serialization
Locking and serialization
§ Filesystem Inodes contain meta information about files or directories

• if a file is changed, also the inode is changed
• if the inode is changed the file might be unchanged
§ Inode locks prevent concurrent modifications of files that lead to
inconsistencies of data
• JFS2 uses a read-shared write-exclusive lock
• Read and write serialization granularity is only at file level
§ Database applications are more sophisticated and have a finer
granularity concerning locking and serializaton of access to data
• Table locking
• Row-level locking

IBM Global Services
AIX 5L and JFS2 > Concurrent I/O
Concurrent I/O
§ AIX 5.2 ML01 provides JFS2 with the concurrent I/O feature
• Multiple threads can simultaneously perform read and write operations
on a shared file
• The option can either be used to open files or a whole filesystem can
be mounted with
mount -o cio
§ If only a subset of files in a filesystem should be opened in

concurrent mode, the subdirectory can be mounted as namefs
• mount –v namefs –o cio /somefs/subsomefs /somefs
§ Direct I/O is implicitly used with concurrent I/O

§ Extension or truncation of files results in a change of the inode lock
from shared to exclusive during the time of modification

IBM Global Services
AIX 5L and JFS2 > Benchmark results
Benchmark results

IBM Global Services
AIX 5L and JFS2 > Tuning Parameters for JFS2
Tuning Parameters for JFS2
§ New tuning commands with AIX 5.2 replace vmtune and schedtune
• vmo, ioo, schedo
§ Values are preserved after reboot, if pre AIX 5.2 compatibility mode
is switched off
• chdev -l sys0 -a pre520tune=disable
§ JFS2 file cache does not use the minperm/maxperm values

• maxclient Hard limit of file pages for JFS2 and NFS and must be set
smaller than or equal to maxperm
§ Read-Ahead parameters must be set separately for JFS2 via ioo

• j2_minPageReadAhead
• j2_maxPageReadAhead

IBM Global Services
Agenda
1 Introduction

IBM Global Services
Multiple SAP Databases > Motivation
Motivation for consolidation of databases
§ Servers have become more and more powerful

• Provide more performance than can be used by one database
• Less operating system images to maintain
• Number of SAP systems and databases is exploding, budgets are
shrinking
§ Workload Management enables balancing of resources

• Better usage of computing resources, including I/O and network
• Peak usage can be satisfied
§ Dynamic Logical Partitioning enables on-demand resources

• CPUs and memory can be moved around freely
• Storage technology enables flexible disk assignment
• Hot-swapping of adapters

IBM Global Services
Multiple SAP Databases > Guidelines and hints
Guidelines and hints for Oracle

§ Naming and numbering conventions
• Enforce User-ID and Group-ID standards
• Use naming conventions for Networks, Volume Groups and Volumes
§ Oracle Installation Information

• Work with symbolic links for each SID for the installation directories
/oracle/jre, /oracle/oui, /oracle/inventory
(SAP OSS Note 350251)
§ AIX kernel and Oracle kernel extensions

• Oracle 8 is only supported as 32-Bit version on AIX 5L
• Oracle 8 kernel extensions only work with the 32-Bit AIX 5L kernel
• Oracle 8 Patch 2896876 (Metalink note 231901.1) provides extension
for 64-Bit AIX 5L Kernel
§ Oracle listeners
• Separate listeners should be configured for each instance with unique
port numbers

IBM Global Services
Multiple SAP Databases > Guidelines and hints SAP
Guidelines and hints for SAP
§ Transport Directory
• Each system environment should have a separate transport directory, e.g.
/usr/sap/trans<SID>
• The following variables should be set in the SAP system

DIR_TRANS=/usr/sap/trans<SID>
DIR_EPS_ROOT=/usr/sap/trans<SID>/EPS
§ Performance Collector saposcol

• Start the saposcol from an SID independent path such as /etc/sap/saposcol
• Create an entry in the inittab file
mkitab –i strload saposcol:2:once:su – sidadm –c /etc/sap/saposcol –l

IBM Global Services
Multiple SAP Databases > Workload Management
Workload Management
§ Workload Management included in AIX at no additional costs

• Easy setup for SAP and Oracle (30 Minutes)
• Passive mode for testing, performance monitoring and accounting
§ Management of CPU Resources

• Management of memory easier with SAP and Oracle parameters
§ Soft limits and hard limits possible

• No wasting of resources
§ Classification via User-IDs or Tags

• orasid and sidadm are perfectly suited for easy classification
• Separation of classification of Dialog/Background workprocesses through
direct assignment

IBM Global Services
Multiple SAP Databases > Workload Management Example
Workload Management Example

Superclasses for 4 SAP systems
Subclasses SAP, Oracle DB,
I1A,IPC, KPC and IPB
Live Cache for APO system I1A

IBM Global Services
Summary
§ Modern disk subsystems facilitate database layouts through large

caches and internal striping and mirroring
§ Correct configuration of file cache in AIX prevents waste of memory
§ Sufficient asynchronous I/O server processes are key to good

performance
§ JFS2 offers almost raw device performance with concurrent I/O
§ Workload Manager in passive mode ideal for monitoring

IBM Global Services
Questions ... ?
November 2003 © 2003 IBM Corporation

IBM Global Services
Agenda
1 Introduction
6 References

IBM Global Services
References
§ IBM Redbooks and Redpapers at www.redbook.ibm.com

• "A holistic approach to a reliable infrastructure for SAP R/3", SG24-5050
• "Consolidating multiple SAP systems on one pSeries", REDP-3626-00
§ SAP Market Place at service.sap.com

• "Database Layout for R/3 installation under Oracle"
§ Oracle Whitepapers at technet.oracle.com

• "Optimal Storage Configuration Made Easy"
§ Enterprise Storage Server white papers for Oracle and SAP

• http://www.storage.ibm.com/disk/ess/whitepaper.htm
§ AIX Whitepapers, e.g. on Concurrent I/O

• http://www-1.ibm.com/servers/eserver/pseries/library/wp_aix_lit.html

Oracle On AIX - Best Practices

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Oracle On AIX - Best Practices

Transféré par

Droits d'auteur :

Formats disponibles

IBM Global Services

Oracle on AIX : Best Practices

Dr. Andreas Gruber

November 2003 © 2003 IBM Corporation

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

2 Oracle on AIX : Best Practices © 2003 IBM Corporation

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

3 Oracle on AIX : Best Practices © 2003 IBM Corporation

"A best practice is a technique or methodology that, through

§ Knowledge transfer from SAP infrastructure projects based on

§ Performance tuning as a holistic approach that involves all

§ Explanation and advertising of new features implemented by

4 Oracle on AIX : Best Practices © 2003 IBM Corporation

Introduction > Oracle Performance Tuning

Oracle Performance Tuning

5 Oracle on AIX : Best Practices © 2003 IBM Corporation

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

6 Oracle on AIX : Best Practices © 2003 IBM Corporation

Disk Layout > Old Style

Old Style Disk Layout for SAP/Oracle Database

§ Use of single and separated disks

§ Placement of files according to the following criteria

§ Elimination of arising hot spots via reorganization of tables into

7 Oracle on AIX : Best Practices © 2003 IBM Corporation

Disk Layout > New Paradigm

New Paradigm for SAP/Oracle Database

§ Oracle's S.A.M.E Strategy (Stripe And Mirror Everything)

Striping § Stripe almost everything § Stripe across ESS RAID Arrays

8 Oracle on AIX : Best Practices © 2003 IBM Corporation

Disk Layout > Example

Disk Layout Example for SAP/Oracle Database

9 Oracle on AIX : Best Practices © 2003 IBM Corporation

Disk Layout > Golden Rules for ESS (Shark)

Golden Rules for ESS (Shark)

§ Decision Support Systems (e.g. BW)

§ Number of RAID Arrays according to throughput requirements

10 Oracle on AIX : Best Practices © 2003 IBM Corporation

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

11 Oracle on AIX : Best Practices © 2003 IBM Corporation

Performance Tuning for JFS > JFS vs. Raw Devices

Journaled Filesystem vs. Raw Devices

§ "Raw Devices" correspond to Logical Volumes in AIX

§ Filesystems are easier to manage for most administrators

§ Journaled Filesystems ensure the integrity of the filesystem

12 Oracle on AIX : Best Practices © 2003 IBM Corporation

Performance Tuning for JFS > Areas of Tuning

ç Oracle Parameters init.ora

ç Multipath and SAN

ç Cache and internal access

ç Number and speed of drives