Vous êtes sur la page 1sur 23

Data Guard and RAC: Maximum Availability at Verizon Wireless

Orlando, April 13, 2011 Session Id: 725

Syamal Bandyopadhyay Sharath Nasani

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Agenda
Agenda:
About The Presenter Objectives Maximum Availability Requirements at Verizon Wireless Database Deployment Architecture Why Data Guard ? Challenges with Data Guard Deployments Why RAC? Performance Challenges with RAC Flashback Technology Conclusion Q/A

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

About The Presenter


Name: Syamal Bandyopadhyay Expertise in Oracle Database Tuning, Scalability, High Availability, Deployment Architecture, and Capacity Planning Working at Verizon Wireless since 2002 Worked at: AT&T IBM Telcordia Technologies Bell Labs Contact Information: syamal.bandyopadhyay@verizonwireless.com

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

About The Presenter

Name: Sharath Nasani Expertise in Oracle Database Tuning, Scalability, High Availability and Deployment Architecture Working at Verizon Wireless since 2000 Contact Information: Sharath.nasani@verizonwireless.com

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Objectives

Sharing Maximum Availability Database Deployment Architecture at Verizon Wireless Sharing Lessons learned at Verizon Wireless Enabling you the attendees to be better positioned to design your Maximum Availability system

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Maximum Availability Requirements at Verizon Wireless

Maximize Availability Minimize unplanned outage System Maintenance without downtime Cost Efficient (e. g. Utilize Disaster Recovery Site for read only) Operationally manageable

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

HP-PARISC Oracle 10.2.0.4 Shareplex 6.x Veritas Filesystems


Data Center - 1

Database Deployment Architecture Provisioning/Billing


Data Center - 2 Data Center - 3

Oracle Data Guard

Prov

Prov DR
Shareplex replication

Shareplex replication

HP-Itanium Oracle 11.2.0.2 Shareplex 7.x ASM Storage Billing DB size: 50 TBs Compress Older partitions. Billing Reporting Application Accesses Active Data Guard Database

Prov Report ing

Billing DR/Re porting

Oracle

Billing

Active Data
Guard

Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Database Deployment Architecture: Customer Care

Data Guard

Shareplex Data Replication

CCARW CCARE DR

CCARW DR

CCARE

Data Guard

5 Node RAC cluster 2 sites Oracle 10.2.0.4 Veritas SF RAC 5.0 Data Guard for DR 2-way data replication using Shareplex Flashback DB size: 2.5 TBs Solaris 10 / M5000 servers Each cluster supports 2 databases (production + DR)

8
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Database Deployment Architecture Content Management

Global Switch System at Internet level (GSS)

Context Switch System at Web/App servers level (CSS)

DB Server:
Oracle 11.2.0.2 Oracle Flashback SUN M5000 Servers Symantec SFRAC 5.0 Hitachi/EMC Storage 4- Node RAC Cluster

Data Replication Shareplex


CMPEDB CMPWDB 9
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Why Data Guard?

Disaster Recovery for Site Failure Active/Active Data Guard: For Site Failure Read Access at DR Site Optimal utilization of H/W

10
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Challenges With Data Guard Deployment

Define the business purpose and SLA for DR site Redo Volume Distance Between Data Centers Available Network Bandwidth Network Tuning Selecting the appropriate mode to support SLA Select Appropriate Redo Transport mechanism Oracle 11g is better tuned for redo/archived log transportation synchronization

11
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Challenges With Data Guard Deployment (contd.)

Which Mode to use? Reduce Redo volume --- reduced by 80% for massive data load using DMExpress software from Syncsort Enforced logging for all DML With the exceptions of tables not required at the DR site Tuning Turned off Automated Failover Monitoring is extremely critical Very Reliable Oracle product

12
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Challenges With Data Guard - Tuning SQL*Net Tuning

TNS Names.ora
(SDU = 32767) ( Default is 2048 Bytes)

TNS & Listener.ora


(SEND_BUF_SIZE = 3145728) ( Default is 49152 Bytes) (RECV_BUF_SIZE = 3145728) log_archive_dest_2 service="rssbprddg", LGWR ASYNC max_connections=5 5 Arch processes are transmitting single file, and 5 RFS processes are receiving at Standby site. Log_archive_max_processes=30 recovery_parallelism = 8 ( In 11g this parameter is equivalent of Number of CPUs in the system). parallel_execution_message_size to 8192 ( default is 2152) db_cache_advice to OFF in standby only

Redo Transport Tuning

Redo Apply Tuning


13
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Challenges With Data Guard - Monitoring


Monitor MRP process in Standby database
SELECT PROCESS, STATUS, CLIENT_PROCESS,inst_id,THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM gV$MANAGED_STANDBY order by sequence#
PROCESS ARCH ARCH ARCH ARCH ARCH ARCH ARCH MRP0 RFS RFS RFS RFS RFS RFS STATUS CLOSING CLOSING CLOSING CLOSING CLOSING CLOSING CLOSING WAIT_FOR_LOG IDLE RECEIVING RECEIVING RECEIVING RECEIVING RECEIVING CLIENT_P ARCH ARCH ARCH ARCH ARCH ARCH ARCH N/A LGWR ARCH ARCH ARCH ARCH ARCH INST_ID 1 1 1 1 1 1 1 1 1 1 1 1 1 1 THREAD# 1 1 1 1 1 1 1 1 1 1 1 1 1 1 SEQUENCE# BLOCK# BLOCKS 32270 15360 32271 9216 32272 12288 32273 12288 32274 13312 32275 12288 32276 11264 32278 0 32278 944599 32277 812122 32277 212323 32277 232323 32277 43434 32277 2324213 125 709 42 591 940 477 443 0 1 800 323 534 542 663

Backlog Monitoring From Primary database


select name, sequence#, registrar,standby_dest,archived,applied from v$archived_log where name='rssbprddg and applied !='YES order by sequence#
NAME rssbprddg rssbprddg rssbprddg rssbprddg rssbprddg rssbprddg rssbprddg SEQUENCE# 32547 32548 32549 32550 32551 32552 32553 REGISTRAR ARCH ARCH LGWR ARCH ARCH ARCH ARCH STANDBY_DEST YES YES YES YES YES YES YES ARCHIVED YES YES YES YES YES YES YES APPLIED NO NO NO NO NO NO NO

14
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Why RAC?

Maximizing Database availability from server or instance failure in a single site Horizontal Scalability Reduced Cost of Ownership

15
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Challenges for RAC Deployment

Application Performance ---- faced severe challenges Database Object Tuning in achieving acceptable response time Process Isolation Data Isolation Massive testing including failure/failover scenarios Training

16
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

RAC Performance A Simple Concept

SQL Response Time Equation: Non RAC:

Response time = CPU time + Wait time (IO wait + Queue time)
RAC:

Response time = CPU time + Wait time (IO wait + Queue time + Cluster Wait Time)

17
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

RAC Performance SQL To Detect Cluster Wait Time


select parsing_schema_name owner, inst_id, sql_id, executions, elapsed_time/1000/executions elapt, cpu_time/1000/executions cput, cluster_wait_time/1000/executions cwt, (cluster_wait_time/1000/executions)/(elapsed_time/1000/executions) CWTtoELAPT , buffer_gets/executions lio, disk_reads/executions phyio , rows_processed/executions rowcnt from gv$sql where executions > 0 and (elapsed_time/1000)/(decode(executions,0,1,executions)) > 0 and parsing_schema_name in ( your schema name') and last_active_time > (sysdate - 1/1440) order by 7 desc
Per Execution OWNER INST_ID SQL_ID EXECUTIONS ELAPT CPUT CWT CWTTOELAPT LIO PHYIO ROWCNT

IVRPROD

C0MUSLA
ACSSPROD ACSSWEB ACSSWEB ACSSWEB

ACSSWEB
NEW_FOLLOWUP

4 1 4 2 1 2 1 2

7w6hskya9az3c 2,788 13,304.23 4,400.94 c0h16fgssh5uj 73 112.72 33.94 8phk00fbc0m7b 1,788 59.04 24.98 f29bk27cqtahw 408,199 144.54 14.79 f29bk27cqtahw 382,544 148.50 20.48 8mcgk5cs23jtj 311,375 91.79 12.12 8mcgk5cs23jtj 7,409,683 97.01 17.62 9rdxj8rpgt587 109,109 72.24 7.48

6,308.60 57.79 32.92 28.98 27.53 25.57 24.82 20.52

0.47 0.51 0.56 0.20 0.19 0.28 0.26 0.28

64,788.75 9,939.94 732.57 574.32 6.27 153.85 176.05 5.96 0.01 223.89 26.62 117.16 225.61 26.51 117.60 380.60 16.35 3.60 380.68 16.40 3.60 132.02 12.45 3.15

18
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

RAC Performance Techniques to Reduce Cluster Wait Time Identify SQLs and Database Objects having High Cluster Wait Time Cluster Wait Time must be reduced Use the Techniques to reduce Cluster Wait Time Define Services for data and process isolation Partition / Sub-partition tables Partition indexes (preferably Hash) Use Freelist Group Use Freelist Increase initrans Remove unnecessary Indexes Reduce Full Table Scans, Index Full Scans, Index Fastfull Scans, Skip Index scans Use Multiple block size (2K, 4K, 8K, etc.)
19
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

High Availability Using Flashback


Provides point-in-time quick recovery of database objects Facilitates Testing of planned Disaster Recovery twice a year Create a flashback point prior to the test Execute the test Execute flashback recovery to the pre test flashback point Continue applying logs Requires additional disk storage for flashback logs

Not a substitute for backup!!!

20
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Conclusion / Take Home for Attendees

Various available options used at Verizon Wireless Maximum Availability achieved using combinations of technology Selecting appropriate mode for Data Guard is critical Reduce Redo Generation by all means Understanding RAC Performance and Tuning is a big challenge

21
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Q/A

22
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Data Guard and RAC: Maximum Availability at Verizon Wireless

Orlando, April 13, 2011 Session Id: 725

Syamal Bandyopadhyay Sharath Nasani

23
Confidential and proprietary material for authorized Verizon Wireless personnel only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by an authorized representative of Verizon Wireless.

Vous aimerez peut-être aussi