EMC ITS ORACLE BACKUP AND

RECOVERY4X CHEAPER, 8X
FASTER, AND 10X BETTER
SYMMETRIX VMAX, DATA DOMAIN,
AND NETWORKER
DATA DOMAIN TRANSFORMS EMC ITS ORACLE
BACKUP AND RECOVERY INFRASTRUCTURE

ABSTRACT
Migrating from a legacy availability infrastructure for Backup and Recovery
creates challenges in terms of what are the best practices for a new Backup and
Recovery deployment with EMCs Oracle databases for Global Data Warehouse
and mission-critical Oracle applications. This white paper will illustrate the
transformation of EMC IT Oracle Backup and Recovery Infrastructure and
highlight how the Data Domain appliance transforms EMC IT Oracle Backup
infrastructure.
August 2012

WHITE PAPER

Copyright 2012 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change
without notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with
respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a
particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
EMC2, EMC, the EMC logo, and the RSA logo are registered trademarks or trademarks of EMC Corporation in the United States
and other countries. VMware is a registered trademark of VMware, Inc. in the United States and/or other jurisdictions. All other
trademarks used herein are the property of their respective owners. Copyright 2012 EMC Corporation. All rights reserved.
Published in the USA. 08/12 White Paper H10986

TABLE OF CONTENTS
EXECUTIVE SUMMARY ........................................................................................................................... 4
Audience ............................................................................................................................................ 4
EMC ITS BACKUP AND RECOVERY JOURNEY ......................................................................................... 5
EMC IT Overview ................................................................................................................................ 5
EMC ITs Backup and Recovery business drivers ...................................................................................... 5
EMC ITs Backup and Recovery Legacy Architecture................................................................................. 5
EMC Backup Profile Legacy EMC IT VTL infrastructure...................................................................... 5
Legacy Backup and Recovery Components........................................................................................ 5
Legacy Backup and Recovery Pain Points ............................................................................................... 8
EMC ITs New Backup and RecoveryData Domain Infrastructure ............................................................. 8
Components ................................................................................................................................. 8
EMC Data Domain Deployment Models ............................................................................................. 9
GDW/CRM BACKUP AND RECOVERY COMPONENTS.............................................................................. 10
Backup and Recovery enablers ............................................................................................................ 10
EMC ITS ORACLE BACKUP AND RECOVERY METHODS ......................................................................... 11
EMC ITs Offloaded Backup and Recovery Process for Oracle Databases .................................................... 11
Step 1- TimeFinder Clone .............................................................................................................. 11
Step 2- Clone Mounted on Proxy/Backup Host .................................................................................. 11
Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain .................................. 11
EMC ITs Regular Backup Process ......................................................................................................... 12
EMC ITS OFFLOADED BACKUP EXAMPLES ........................................................................................... 13
Global Data Warehouse (GDW) Backup Size .......................................................................................... 13
GDW Pain Points ................................................................................................................................ 13
Oracle CRM Backup and Recovery Problem Statement ............................................................................ 13
Advantages of Proxy Host Solution ....................................................................................................... 13
EMC TimeFinder Clone .................................................................................................................. 13
EMC TimeFinder Snapshot ............................................................................................................. 14
ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN DEPLOYMENT ............................... 15
Reuse of the legacy EMC IT Backup and Recovery process ...................................................................... 15
EMC Data Domain Deduplication technology ....................................................................................... 15
Deduplication Benefits................................................................................................................... 15
Full Backups versus Incremental .................................................................................................... 15
ROI/TCO ........................................................................................................................................... 15
4X Cheaper.................................................................................................................................. 15
10X Better ................................................................................................................................... 15
EMC IT LESSON LEARNED .................................................................................................................... 16
Bottlenecks ....................................................................................................................................... 16
Disks .......................................................................................................................................... 16
Proxy Server: 8X Faster ............................................................................................................... 16
EMC NetWorker Server .................................................................................................................. 16
Utilization versus Vulnerability ....................................................................................................... 16
CONCLUSION ....................................................................................................................................... 17
References ........................................................................................................................................ 18
Acknowledgments .............................................................................................................................. 18

EXECUTIVE SUMMARY
EMC IT has seen explosive growth over the last five years accelerating the need to move from its legacy, Virtual Tape Library
(VTL) Backup infrastructure to a new EMC Data Domain Backup infrastructure.
This creates challenges in terms of what are the best practices for a new Backup and Recovery deployment with EMCs Oracle
databases for the Global Data Warehouse and mission-critical Oracle applications.
EMC IT implemented a phased approach because of the size of the Oracle environments:

Over 670 Oracle databases in all environments:

o

Production

Test/Development

QA

Performance

Patch

Greater than 30 TB of Oracle Redo Logs generated per day

Mission-Critical environments (databases) backed up daily

Moving to the new EMC IT Data Domain infrastructure, for EMCs mission-critical Oracle Global Data Warehouse (GDW) and
Oracle CRM production environments has delivered the following advantages:

4X Cheaper - The Data Domain appliances are a quarter the cost of the legacy EDL/VTL

8X Faster Move from an incumbent EDL/VTL speed of 500 MB/hour to Data Domain speed of 4 TB/hour

10X Better Based on the following:

o

Reliability - More reliable than tapes guaranteed that backups can be restored.

Density - Stores more backups for a longer period of time, even old backups can be quickly restored.

Protection Production Data Domain appliances are replicated to a remote Data Domain appliance off-site.

Speed Both backups and restores are significantly faster. Backups are now completed within the Service Level
Agreement (SLA) with fewer support resources.

Complexity Standard full backups are much easier to execute and restore than incremental backups.

Cost Data Domain units are one-fourth the cost with up 10 times the capacity, after deduplication.

This white paper will illustrate the journey and benefits of EMC ITs Oracle Backup and Recovery Infrastructure and highlight how
the Data Domain appliance transformed EMC IT Oracle Backup architecture. It will also include lessons learned from EMC IT on
this migration from a legacy VTL infrastructure to a Data Domain infrastructure.

Audience
This white paper is intended for CIOs, Oracle architects, Backup and Recovery architects, storage architects, Oracle Database
Administrators (DBAs), and server and network administrators.

EMC ITS BACKUP AND RECOVERY JOURNEY

This section covers the drivers and the size of the journey, with respect to EMCs Backup and Recovery profile. It details the
legacy Backup and Recovery infrastructure, as well as the new EMC IT Backup and Recovery Data Domain infrastructure
deployment models.

EMC IT Overview
EMC is a company with over 53,000 users of IT services. It supports over 400,000 customers and partners in 5 Data Centers
with over 16 PB of storage. EMC IT has a portfolio with over 500 business applications and tools and over 8000 OS images with
more than 90 percent of all servers virtualized in 80 countries and 20 languages.

EMC ITs Backup and Recovery business drivers

The following are the business drivers for EMC ITs Backup and Recovery:

Inability to meet EMCs backup or restore SLAs with the legacy EMC VTL infrastructure
o

Erratic backup windows, some spanning over 24 hours

Daily hand-holding/fire fighting of the backup process by EMC IT Backup and Oracle SMEs

Limitations on the number of backup images stored on the legacy EMC VTL infrastructure

EMC ITs Backup and Recovery Legacy Architecture

This section describes EMCs Backup and Recovery profile as well as the legacy Backup and Recovery components.

EMC Backup Profile Legacy EMC IT VTL infrastructure

The following is the current EMC IT Oracle Backup and Recovery profile:

Over 670 Oracle databases

Over 100 Terabytes of Oracle Databases backed up per day

Mission-Critical production environments backed up daily FULL

Mission-Critical Dev and Test environments backed up every two days FULL

Non Mission Critical and dev/test environments backed up FULL 2X per week

Archive log backups are run daily and when triggered by space alerts

Legacy Backup and Recovery Components

The following are the baseline Backup and Recovery components used in the legacy (VTL) infrastructure:
EMC NetWorker
NetWorker delivers complete Oracle protection through the NetWorker Module for Oracle.
NetWorker integrates with Oracle Recovery Manager (RMAN)Oracles recommended Backup and Recovery interface for its
databases. Through this integration, NetWorker simplifies Backup and Recovery for Oracle and is able to integrate Oracle server
protection into several core features, including wizard-driven configuration and push installation.
The EMC NetWorker Module for Oracle has value-added features available to reduce backup impact and required storage. This
includes integrated deduplication and snapshot management functionality.
Additionally, an exclusive feature called event-based backup helps deliver more effective backup operations based on probing for
conditions rather than starting merely by time.
To gain the benefits of consolidation and flexible availability, the world is moving increasingly to operation in virtual server
environments. NetWorker, and its Oracle support, integrate with VMware, delivering in guest, application-consistent protection
and support for environments that leverage VMware technologies such as Distributed Resource Scheduler, vMotion, and High
Availability that move and migrate virtual machines in a VMware environment.
Another benefit of NetWorker is that it is tested, certified and supported in a broad range of operating systems and
environments.

Oracle Recovery Manager (RMAN)

Oracle RMAN is a built-in tool that allows the database administrator (DBA) to easily backup and recovery data in an Oracle
database. RMAN handles the coordination required to ensure that transaction integrity is preserved, and sufficient information is
maintained to recover the database to any appropriate point. RMAN can create backup sets that comprise as much or as little
recovery information as the DBA requires but usually include information from the database datafiles, control files, redo files and
archived log files.
EMC NetWorker/RMAN Integration
NetWorker delivers full integration and is fully compliant with Oracle RMAN. Through this integration, NetWorker delivers
complete Backup and Recovery for Oracle databases, including the ability to capture and protect databases, individual datafiles,
tablespaces, and logsall of the things that comprise an Oracle installation.
NetWorker, through the RMAN integration, is able to provide backups for Oracle whether the database is online or offline. Other
highlights include the ability to deliver block-level incremental backups, which helps to ensure backups are fast and capturing
the unique changes to the data. Other RMAN features include block corruption detection and surgical recovery of any detected
corrupted blocks to help ensure data is accurate and recoverable.
The solution also compresses any unused blocks to save time, space, and bandwidth. In addition, RMAN delivers restartable
backups from the point of failure for interrupted backupsvery helpful, especially for very large databases. Through Oracles
Data Recovery Advisor, any issues encountered and diagnosed are automatically repaired.
All of these capabilities help ensure recovery, which is the primary goal. By providing automatic log management, NetWorker is
able to bring the database back to a point-in-time, thereby restoring all data and replaying any needed logs to facilitate a
complete restore. Finally, not only can you restore back to where the data came from, but also in cases where there is other or
new hardware in place, NetWorker will restore the data to an alternate location.
EMC Disk Library (EDL)/Virtual Tape Library (VTL)
A virtual tape library is a disk based storage array front-ended by servers that emulate tape drive systems.

Disk based VTLs

have significant advantages over their tape counterparts as they dramatically improve throughput speeds. This is even more
significant when it comes time for a restore. Since the backups already exist on disk, restores do not suffer from the mechanical
limitations of either the tape drive streaming speeds or the robotic arms or even the fast forward and rewind actions associated
with finding the backup set.
The following is a high-level illustration of the legacy Backup and Recovery infrastructure:

NetWorker

Virtual Tape Library

Database Server

Figure 1. Legacy Backup and Recovery infrastructure with VTL

Proxy Host
A Proxy Host is a server used to mount a cloned copy of a database for backup purposes. It is primarily used to offload the
backup overhead from the production database. It can be used by many databases and has the advantage that the VTL can be
directly mounted to a single Proxy Host, giving significantly more throughput with SAN protocols, vs. backups over NFS. This is
much more important when performing partial restores, as the database can be rapidly restored to the Proxy Host and a partial,
or surgical restore can be performed very rapidly. Of course, if the data required is from the most recent backup, the database
does not even need to be restored from the VTL, as it still exists on the Proxy Host.

NetWorker

Proxy Host

Virtual Tape Library

Database Server

Figure 2. Legacy Backup and Recovery infrastructure with Proxy Host

The configuration of the proxy server does not require significant CPU resources. The focus of the server is strictly for IO
throughput. For this reason, EMC IT selects a server with a large number of expansion slots. As an example, for EMC ITs CRM
environment the proxy server configuration is shown in Table 1.

Quantity

Hardware

Cisco UCS C460 M2 - 128GB RAM - 2 x 73GB HDD

8GB DDR3-1333MHz

10 GB Network Card - Bonded

lpe1200E dual-port 8GB HBA 4 ports zoned for VMAX, 4 ports zoned to VTL device
SOFTWARE
Oracle RDBMS single instance
RedHat 5.6
ASM

Table 1. CRM Proxy Host configuration

TimeFinder Clone
EMC TimeFinder is an EMC replication technology that is used by EMC Symmetrix VMAX storage arrays to instantly create an
exact copy of a set of LUNS. For a database, these LUNS would represent the database files for building a standby or backup
copy and optionally include the redo files, for creating a reporting copy. This capability allows the backup or clone of a database
without putting the database in a HOT backup mode.

Legacy Backup and Recovery Pain Points

The legacy Backup and Recovery infrastructure, described above, created some pain points for EMC IT.
The following are the two that most impacted EMC IT:
Limited retention for old backups
Like all organizations, EMC IT has a need for retaining backups for extended periods. This generally consists of writing backups
off to tape and sending them to an offsite storage facility. Given the fragility of tape backup media, EMC IT could never be 100
percent sure that it was possible to restore these tapes and the cost of limited use media was expensive to maintain. By placing
these EDLs at a data center nearby, EMC IT was solving the offsite requirement, but at a cost of backup throughput.
Storage needed (large overhead)
With the introduction of EDL devices, EMCs version of a VTL, EMC IT quickly adopted them and began deploying the backups to
these, disk-backed, virtual tape libraries. It did not take long to outgrow these EDLs, as the amount of disk required to retain
EMC ITs long-term backups was massive. This required EMC IT to continue to rely on tapes for long-term backup retention.
Fortunately, the introduction of deduplication technology and devices like Data Domains VTL, Virtual Tape Library helped
address this challenge.

EMC ITs New Backup and RecoveryData Domain Infrastructure

This section describes the legacy Backup and Recovery tools that are re-used in the new Backup and Recovery Data Domain
infrastructure. This includes: the Data Domain appliance Backup and Recovery; the deployment model needed for the EMC ITs
shared GDW/CRM Backup and Recovery process; and components for their Oracle Database Proxy Host Backup and Recovery
process.

Components
The new Backup and Recovery infrastructure consists of EMC ITs current Backup and Recovery tools:

EMC NetWorker - 7.6.3

Oracle RMAN

EMC NetWorker Module for Oracle (NMO) 5.0

Proxy Hosts

EMC TimeFinder Clones

The major difference, between the new infrastructure and the legacy infrastructure, is the destination of the backups. In the new
infrastructure, EMC IT writes to the Data Domain appliances. The Data Domain appliances, though the deduplication feature,
allows EMC IT to store many more backups in a fraction of the space previously required. This capability has eliminated the
need for the EDL/VTL and more importantly the need for any fragile tapes.
For the offsite requirements, EMC IT uses Data Domain replication to replicate the backups to a secondary datacenter 600 miles
away, completely eliminating the need for any tapes.
Data Domain
EMC Data Domain deduplication storage systems are designed and optimized, specifically for backup and archive of data.

Support for any conventional backup or archive application through generalized support for network-attached storage (NAS)
interfaces over Ethernet; a virtual tape library (VTL) interface option over Fibre Channel; and product-specific interfaces such
as NetBackup OpenStorage and EMC Data Domain Boost

High-speed, inline deduplication using small, variable-sized sequences to identify and eliminate redundant data segments
before storing to disk

Integrated data protection technologies such as RAID 6, post-backup data verification, and periodic validation checks of
existing data sets

Automated replication of backup data for disaster recovery (DR) using cost-effective, low-bandwidth WAN links, which
enables faster time-to-DR readiness

EMC Data Domain Deployment Models

EMC Data Domain systems can be deployed as a NAS device or as a VTL:
NAS deployment
When the Data Domain system is deployed as NAS, RMAN can write backups directly to it as a disk backup. This can be
extremely useful for environments that have only a few databases to backup. As the number of databases grows, the
management all of these disparate mounts can be overwhelming. For larger organizations, integration with an enterprise
backup solution, such as EMC NetWorker, is recommended.

This integration provides many benefits, such as a centralized

location for managing and cataloging backups. EMC NetWorker, along with many of the other solutions, also includes
technology that can significantly improve backup throughputs.

Data Domain

Database Server

Figure 3. Data Domain deployed as NFS mount to Host

VTL deployment
When the Data Domain system is deployed as a VTL, RMAN must be integrated with an enterprise backup application like EMC
NetWorker. This is because, like all VTLs, Data Domain systems operate with VTL or Open Storage protocols and RMAN is not
capable of interfacing with them directly.
With a VTL deployment, Data Domain systems can be connected though a 1 GB or 10 GB network typically dedicated to backup
traffic. They can also be connected directly to database server HBA ports achieving much higher rates of throughput.

NetWorker

Data Domain

Database Server

Figure 4. Data Domain deployed as a VTL device

GDW/CRM BACKUP AND RECOVERY COMPONENTS

The following illustrate the new EMC IT Backup and Recovery infrastructure components, when offloading backups to a
Proxy Host:

Figure 5. Oracle Database Proxy Host Backup and Recovery Process

As the diagram illustrates, EMC IT deploys a shared Backup and Recovery infrastructure using the following components and
best practices:

VMAX TimeFinder Clones

Deployment of a Proxy Host or EMC NetWorker Storage node

Oracle database single instance (backup license)

EMC NetWorker Module for Oracle

RedHat AS 5.6

Data Domain appliance deployed as a VTL device DD890

Backup and Recovery enablers

The following technologies enable EMC ITs Oracle Backup and Recovery today:

10

8 Gigabit Storage Area Network

10 Gigabit dedicated Local Area Network

EMC Data Domain 890 Appliances

EMC NetWorker Version 7.6.3

EMC NetWorker Module for Oracle (NMO) Version 5.0

Oracle RMAN 11.2

EMC VMAX TimeFinder Clone

Proxy Clone Hosts

EMC ITS ORACLE BACKUP AND RECOVERY METHODS

EMC ITs Offloaded Backup and Recovery Process for Oracle Databases
The following are the high-level steps for the offloaded Backup and Recovery Proxy Host backup process:
1.

Create a TimeFinder Clone

2.

Include the redo logs so that the database can be verified prior to the backup. This is not necessary, but adds an extra
level of confidence that the database backup is good. It also allows the database to be opened, read-only for
investigation, reporting or extract purposes.

3.

Mount the Clone on the Proxy Host server

4.

Invoke the NetWorker/RMAN scrip to backup the TimeFinder Clone to Data Domain infrastructure

Step 1- TimeFinder Clone

EMC IT uses EMC TimeFinder/Clone Technology as the first step in the backup process. The Clone is a full size point-in-time
replica of the source database.

Once the Clone is originally created, subsequent recreate operations only apply the

incremental changes made to the source since the last recreate of the Clone. This reduces the first step of the backup
process. EMC ITs daily clone typically takes 40 minutes for a 20TB database.
After 40 minutes, the database has a viable Backup and Recovery solution. This backup, is now available for surgical or
complete restores.

Step 2- Clone Mounted on Proxy/Backup Host

EMC IT uses a Proxy Host, or backup host, to offload the RMAN backup to a virtual tape that is the Data Domain appliance. With
a Proxy Host, a separate host is used, configured with Oracle Binaries and ASM installed. The ASM diskgroups on the Proxy Host
are made up of the source databases cloned devices.
Deploying a backup host has many benefits:

Allows full backup to run on the Proxy Host producing no performance impact to the production database and production
user community

Enables surgical restores from the Proxy Host from the most recent backup or from a restored backup from any point in
time. Having a point-in-time restore on a separate host allows individual tables or tablespaces to be restored using
export/import or transportable tablespace or even database links for very specific requirements.

Enables ad-hoc querying for end-users. Queries can run from the clone for ad-hoc reporting allowing full autonomy for
reports running from the clone. This can stop less efficient or resource intensive reports from interfering with the source
database (Production).

Reverse Clone For databases that require a quick and efficient RTO (Recovery Time Objective) the Reverse Clone allows
super quick restores back to the source.

Offloaded Stats Generation before the next backup run, the database can be opened and stats generated on the cloned
version of the database. This allows for complete level of generation, with no impact to the production system.

Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain

Once the Clone operation is complete, the Clone is opened as a read only database, thus verifying a valid copy, and backed up
to Data Domain using RMAN/NetWorker. EMC IT re-used the same legacy scripts that were already in place when EMC IT
moved from the EDL to the Data Domain appliance deployed as a VTL.
The only additional consideration was to change the filesperset setting, which was changed to 1. This contributes to the most
favorable compression ratio and efficiently stores subsequent full backs to the Data Domain with a very small storage footprint.
Using the Data Domain appliance to store backups has enabled many advantages including:

Virtually no change to existing backup scripts, RMAN catalogs or pre-determined scheduling the only consideration or
potential change being a change to the filesperset parameter.

11

Integrates easily with EMC NetWorker infrastructure.

The deduplication benefits of Data Domain allow many more images of full backups to be stored with minimal storage
overhead. This is especially significant for databases that carry a lot of unchanged history as the bulk of the data, which is
the case in all data warehouses and most OLTP databases.

Data Domain enables EMC IT to eliminate the need for a traditional Oracle incremental backup strategy. This simplifies the
restore steps and shortens the recovery time window.

RMAN Script Example

Here is an RMAN script excerpt from EMC ITs backup script used to backup EMC ITs offloaded databases.
The script uses standard Oracle RMAN commands:

connect rcvcat DatabaseA/xxxxxx@rmancatolog

connect target xxxxxxx/xxxxxx@DatabaseA
run {
set command id to 'rman database';
allocate channel t1 type 'SBT_TAPE' format '%d_full_%U';
allocate channel t2 type 'SBT_TAPE' format '%d_full_%U';
allocate channel t3 type 'SBT_TAPE' format '%d_full_%U';
.
(EMC IT uses 24 channels)
..
send 'NSR_ENV=(NSR_DATA_VOLUME_POOL=DBORAP)';
backup tag = DatabaseAbkp_hot_0209_214224' filesperset 1 database;
backup current controlfile channel t1 format '%d_cf_%U';
release channel t1;
release channel t1;
release channel t1;

Table 2. EMC IT legacy RMAN script example

EMC ITs Regular Backup Process

For the majority of EMCs databases offloading to a proxy server is not required.
For those databases, where speed is not an absolute necessity and the overhead of an RMAN backup directly on the database
server does cause performance issues, EMC IT will not offload the backup to a proxy server.
In this case, EMC IT executes the backup directly from the database server. Backups are scheduled every other day as full
database backups to the same Data Domain appliances as the offloaded backups. Since the deduplication benefits are the
same, as offloaded backups, there is no need to do incremental backups, which are impacted by the extra complexity, and
restore times associated with incremental backups.
For EMC ITs larger databases, using this regular backup process EMC IT installs a dedicated network card to the database
server. This additional network card in the database server reduces the overhead to the public network that a large backup
would add, and assists with overall throughput.
In the future, EMC IT is looking at Data Domains Boost technology to both alleviate that overhead as well as reduce the backup
window from deduplication at the source.

12

EMC ITS OFFLOADED BACKUP EXAMPLES

Global Data Warehouse (GDW) Backup Size
EMCs Global Data Warehouse (GDW) is a central source of relational data that provides business intelligence to many key
business groups within EMC including Sales, Marketing, Finance, Manufacturing, Customer Service, Professional Services, Human
Resources and Legal departments.
It is a critical component of intellectual property and its supported and maintained accordingly.

One key and highly important

support requirement is the ability to backup and provide realistic and flexible recovery options to the business groups that rely
on this data.
GDW runs on a six-node Linux-based Oracle Real Application (RAC) Cluster that went live in November of 2009, as a 10TB
database. Since 2009, GDW has significantly grown and as of May of 2012 GDW has doubled to 20 TB with archive log
generation totaling from 500 to over 1500 GB daily.

GDW Pain Points

In the summer of 2011, the time to complete an online RMAN/NetWorker backup of the GDW to an EMC Disk Library (EDL)
degraded. The throughput was erratic and at times would exceed 24 hours, wrapping over the scheduled start time of the
daily TimeFinder Clone used as part of the backup solution.
This undermined EMC ITs ability to successfully provide a backup of the database.
There were periods where it took a week to successfully backup the database to EDL, as the scheduled clone would cause the
online backup to fail. This resulted in daily handholding of backups and undermined the ability of IT to protect this valuable
investment.
The backup/recovery solution was in need of remediation and some redesign as: backups ran too long and would sometimes fail
leaving data protection capability limited and vulnerable. Additional staff-hours were needed to fire-fight daily challenges.
A strict limitation to the number of full backup images that could be stored on the Disk Library, due to the large storage footprint
each backup imposed. Archived log backups became increasingly important and the number of logs needed to satisfy recovery
points became suspect.

Oracle CRM Backup and Recovery Problem Statement

EMC ITs Oracle 11i eBusiness Suite Database grew from 8 TB to 10 TB over a period of two years. During this two-year period,
the backup time increased from 6 hours to over 22 hours, causing significant challenges in getting a full, successful, daily
backup. This also meant that restores, to this environment, would take longer than 22 hours. These times were well beyond
EMC ITs SLA for the databases Recovery Time Objective (RTO).
This, over 3X increase in the backup window, was due to more than just growth in database size. It also was due to the
increase in the number of databases that needed to backup to this shared infrastructure, and was guaranteed not to be solved
on its own. As this was EMC ITs mission-critical database, with a zero data loss policy, 11i Database had to be backed up daily.
The increase in backup time, violated EMC ITs restore SLAs as well.
To resolve this challenge, EMC IT embarked on a new solution with EMC VMAX Clone and Data Domain.

Advantages of Proxy Host Solution

EMC IT uses a Proxy Host to run backups for its Oracle 11i eBusiness Suite database. The copy that is mounted to the Proxy
Host is built using EMC TimeFinder clone technology.

EMC TimeFinder Clone

TimeFinder Clone provides single or multiple point-in-time copies of the full database within the same storage array.
The cloned database is mounted on the Proxy Host for backup. A database clone requires the same disk space as the source
database. Clones are extremely fast, as the data never leaves the storage array.
The initial clone for a 20 TB database takes about 4 hours, with a low quality of service, QOS. EMC IT uses the low QOS to
eliminate the impact to production. After the initial clone, all subsequent clones use a differential clone, only copying the
changed tracks to the cloned devices. For example, EMCs GDW 20 TB database clone takes 40 minutes to complete. From the
Proxy Host, backup of this 20 TB database to Data Domain appliance takes 4 hours.

13

EMC TimeFinder Snapshot

Another option is to use EMC TimeFinder snapshots.
A snapshot is a point-in time "virtual" copy that occupies very little disk space. A snap requires only 10 to 20 percent of source
database space for storing only changed tracks.
In this case, backup will actually read most of the data from production disks. This is a very efficient method of offloading the
backups, but does incur a small overhead to the production database, since the database and backup are competing for the
same physical disks. A good way to minimize this is to use virtual pools in the storage array. This provides many more disks to
spread out the I/O load.
The full clone solution offloads the entire backup load from the production database server and doesnt impact performance for
EMC ITs OLTP online users. This is especially useful when backup runs for a long time.
Proxy Host copy also provides much better Recovery Time Objectives (RTO). In the event you need to restore a production
database, restore from backup media will take hours. Since proxy copy is already on disk, it takes significantly less time to
restore. You can do Reverse Clone to restore a production database and then apply missing archive logs.
This method also provides some protection against accidental data corruption/loss due to human errors. For example, a few
critical tables in the production database were dropped one hour ago due to human error. Application will work fine if you can
restore only these tables. You cannot afford to restore complete production database and suffer 1-hour data loss. Instead, you
can restore proxy database point in time just before table drop, and then copy tables back to production manually. Proxy copy is
extremely useful in such surgical repair scenarios.

14

ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN

DEPLOYMENT
The following are the advantages the EMC IT gained when deploying the new Backup and Recovery Data Domain infrastructure

Reuse of the legacy EMC IT Backup and Recovery process

The first gain was virtually no change to existing EMC IT Backup and Recovery processes such as backup scripts and RMAN
catalogs. There was one consideration potential change to filesperset as defined in the EMC white paper, Oracle RMAN Design
Best Practices with EMC Data Domain.
Another advantage was that the Data Domain infrastructure integrated seamless with the legacy EMC IT backup software, EMC
NetWorker.

EMC Data Domain Deduplication technology

Deduplication technology allowed EMC IT to move its long-term backups to the Data Domain infrastructure with complete
confidence that EMC IT would be able recover those backups, if the need ever arose.
It also solved the issue with off-site storage as these Data Domain devices have the ability to replicate all backups to a Data
Domain unit at a remote site. For EMC IT, the replicated Data Domain unit is located at EMC ITs DR site, giving us additional
protection.

Deduplication Benefits
Data Domains deduplication technology has enabled EMC IT is to retain many more backup images with minimal storage
overhead. EMC IT can now restore backup images from several months pastwithout having to go through the difficult process
of locating off-site tapes and hoping that they are still valid.

Full Backups versus Incremental

Due to Data Domains ability to deduplicate the backups, EMC IT was able to use daily full backups versus having to do
incremental backups. This allows a much quicker recovery process since incremental backups require restoring the full backup
then restoring the incrementals on top of that. This not only reduces the complexity of the restore process, but it also reduces
EMC ITs recovery window.

ROI/TCO
4X Cheaper
Typically, it is difficult to calculate actual dollars saved when utilizing shared infrastructure and the costs to administer an
environment that is not meeting SLAs. In EMC ITs case, the actual cost of the Data Domain units EMC IT replacing its EDL
with are one fourth the cost and between double and quadruple the capacitydepending on the actual de-dupe rates.
In addition, to these savings, Data Domain technology is also increasing EMC ITs ability to retain backups longer, and
completely eliminate the need for tapes. When factoring in all of these additional cost reductions, it is clear that overall costs
greatly exceed the 4x cheaper claim.

10X Better
The following is a mix of quantitative/qualitative measurements of Better that include people, process, and technology:
1) Faster8X faster with Proxy server deployment with 8GB FC deployment
2) ReliabilityMore reliable than tapes guaranteed backup can be restored
3) CostCosts less for hardware and environments (power and cooling)
4) DensityStores more backups creating a greater retention period
5) ProtectionReplicated remote Data Domain appliance can be deployed off-site

15

EMC IT LESSON LEARNED

As with all functions in IT, following is a set of best practices that will yield better performance, scalability and stability.

Bottlenecks
Disks
When configuring for high speed backups, there are many potential locations for bottlenecks. The most obvious of these is the
disk layout, in the storage array. When it comes to I/O, the capability of the disks is most often the bottleneck.
To give some idea, a 15K fibre channel drive is capable of driving between 47 and 53 GB/hour. This means that to support an
8TB/hour backup, you would need 155 of these disk drives.
For the Data Domain Backup and Recovery environments, EMC IT does not need 8 TB/hour. This means EMC IT configured its
disk pool with only 76 disks and is achieving 4 TB/hour.

Proxy Server: 8X Faster

Next up, in the stack, is the connectivity between the storage array and the proxy server. This was initially overlooked when
EMC IT migrated from the EDL to the Data Domain. The Backup and Recovery infrastructure had 8 GB SAN switches, HBAs and
ports on the SAN; however, the Backup and Recovery infrastructure initially connected it to the 2 GB SAN switch.
Even after re-cabling to the 8 GB SAN, EMC IT still found the ports on the storage array had been configured for 4 GB. After
fixing these issues, the new Backup and Recovery infrastructure effectively doubled the initial throughput of 2 TB/hour to 4
TB/hour. When compared to EMC ITs legacy EDL backup speeds, EMC IT improved its backup speeds by a factor of 8 from
about 500 MB/hour on the legacy EMC DL4206 EDLs to about 4 TB/hour on the new Data Domain VTL

EMC NetWorker Server

After relieving that bottleneck, the new Backup and Recovery infrastructure began to have a problem with the NetWorker server.
The NetWorker server that was deployed was already overloaded with backup requests and was not able to keep up the large
number of backup sets that were now being requested to be processed. Keep in mind that EMC IT set files per set to 1, for best
results in deduplication. Since this node was part of the original shared infrastructure and was clearly overloaded, configuring
and deploying another NetWorker server relieved the bottleneck and improved overall backup performance throughout the data
center.

Utilization versus Vulnerability

One of the constant challenges EMC IT faces with their backup infrastructure is how to maximize the utilization without creating
vulnerability.
In order to maximize the investment in the new backup infrastructure EMC IT moved from a dedicated model to shared model.
This has allowed EMC ITs Backup and Recovery team to become much more efficient. However, there was no mechanism to
measure the effect of consolidation on individual backup and restore times. This is what had lead EMC IT into the problem of
backups not completing. It is extremely important to monitor backup times and investigate individual backups that start taking
longer than was originally configured. An obvious example of this would be to use the proxy backup server for more than just
one environmentbut without coordinated backup windows, both backups could suddenly fall out of the configured times.

16

CONCLUSION
EMC IT implemented a Backup and Recovery solution to deal with the growing size of its Oracle environments:

Over 670 Oracle databases in all environments (Test/Development/QA/Performance/Patch/Production)

Over 30 TB of Oracle Redo Logs per day

Mission-critical environments (databases) backed up daily

Migrating from a legacy EMC EDL/VTL solution to Data Domain brought the following advantages to EMC ITs Backup and
Recovery deployment:

Moving away from limited Backup and Recovery backup retention to enabling of simple full backup to reduce the complexity
of incremental restore process

Accommodates months of full Backup and Recovery backups enabled by the Data Domain appliance

Deployment of a Proxy host architect in the Backup and Recovery architect delivers the following:
o

Off load backup process off of production enabling EMC business users experience no disruption or performance issues
on the CRM production server

Ability to reuse legacy Backup and Recovery processes/scripts

Implementation of a shared infrastructure for EMC ITs mission critical CRM and GDW environments

EMC TimeFinder technology, which was able to create database clones in minutes (18 TB in 40 minutes) off the production
server and mount on the proxy server and backup in less than four hours

EMC IT also learned valuable lessons from the EMC EDL/VTL to Data Domain Backup and Recovery journey:

DisksTo meet the throughput in a Data Domain the source database Array needs the correct number of drives. To drive to
8 TB/hour the array would need 155 drives

Proxy ServerConfirm the infrastructure cabled correctly and enough HBAs

NetWorker ServerIdentify a near capacity NetWorker server and plan for enough nodes to meet your total Backup and
Recovery community needs

Utilization versus VulnerabilityMonitor the backup and restore times in a shared deployment to increase the Backup
and Recovery infrastructure utilization while diminishing vulnerability of long running backups , failed backed or long running
restores or failed restores

Moving to the new EMC IT Data Domain infrastructure for EMCs mission-critical Oracle Global Data Warehouse (GDW) and
Oracle CRM production environments has enabled the following:

4X CheaperThe Data Domain appliances are a quarter the cost

8X FasterMove from a legacy EDL speed of 500 MB/hour to an EMC Data Domain speed of 4 TB/hour

10X BetterBased on the following:

o

ReliabilityMore reliable than tapes; guaranteed backup restoration

DensityStores more backups for a longer period of time; even restores for old backups can restored quickly

ProtectionProduction Data Domain appliances are replicated to a remote Data Domain appliance offsite

SpeedBoth backups and restores are significantly faster; backups are completed within the SLA with fewer support
resources

17

ComplexityStandard full backups are much easier to execute and restore than incrementals

CostEMC Data Domain appliances are one-fourth of the cost with up 10 times the capacity, after deduplication

References
EMC Symmetrix VMAX
http://www.emc.com/storage/symmetrix-vmax/symmetrix-vmax.htm
EMC TimeFinder
http://www.emc.com/storage/symmetrix-vmax/timefinder.htm
EMC Data Domain
http://www.emc.com/backup-and-recovery/data-domain/data-domain.htm
EMC NetWorker
http://www.emc.com/backup-and-recovery/networker/networker.htm
IDC Study Worldwide Purpose Built Backup Appliances
http://www.emc.com/collateral/analyst-reports/11530-idc-ww-pbba-2011-2015-forecast.pdf
EMC IT
http://www.emc.com/microsites/emc-it-proven/index.htm

Acknowledgments
The author would like to thank the EMC IT teams for assistance in the creation of this white paper.

CONTACT US
To learn more about how
EMC products, services, and
solutions can help solve your
business and IT challenges,
contact your local
representative or authorized
reselleror visit us at
www.EMC.com.

EMC2, EMC, the EMC logo are registered trademarks or trademarks of EMC Corporation in the
United States and other countries. VMware [add additional per above, if required] are registered
trademarks or trademarks of VMware, Inc., in the United States and other jurisdictions.
Copyright 2012 EMC Corporation. All rights reserved. Published in the USA. 08/12
EMC White Paper H10986
EMC believes the information in this document is accurate as of its publication date.
The information is subject to change without notice.

EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 In North America 1-866-464-7381
www.EMC.com

18