Tips and Tricks for Managing and Administering Oracle Exadata Session 12961

Joo Paulo Lunardelli

LAD Solution Center Senior Manager, Oracle
1

Exadata Management Quiz

A) Your patching days are over. Exadata doesnt need to be patched.

B) All management for Exadata can be done through a single console.

C) None of the above

Agenda

Exadata Patching Exadata Manageability

Exadata Architecture Review

Database Grid
Database Servers Firmware / OS Oracle GI / RDBMS

Storage Grid
Exadata Storage Servers

Other Components InfiniBand Network

Switches Ethernet Switch Power Distribution PDU Keyboard, Video, Mouse KVM

Patch Planning and Testing

Patching Improvements Timeline

Patching Improvements
Reduce Downtime Simplify Install Reduce Risk Transfer Knowledge
BP DG Standby-First install Minimal Pack Linux kernel update BP RAC rolling install Exadata bundle patch (BP) Out-of-place patching step-by-step Minimal Pack firmware update Cell rolling patch apply BP EM install DB+GI BP merge Cell firmware update BP OPatch Auto install BP OPlan IB Switch update w/ ILOM Linux upgrade step-by-step 11.2.0.2 upgrade step-by-step Oracle Advisor webcasts IOUG Exadata SIG webcast Exadata Technical Newsletter

2009

2010

2011

Patch Planning
Latest News and Current Releases (October 2011)

MOS 888828.1
The Patching Note Critical issues and additional required patches Guidelines for related products (OEM, DBFS, ResMgr)

Exadata Technical Newsletter

blogs.oracle.com/MAA
Component Exadata Storage Server Database Server Latest Software 11.2.2.4.0 11.2.0.2: BP11 11.2.0.1: DB_BP12, GI_BP6 Recommended Minimum 11.2.2.3.5 11.2.0.2: BP6 11.2.0.1: DB_BP10, GI_BP5 1.3.3-2
7

InfiniBand Switch 1.3.3-2

Patch Planning
Release Frequency

Component Database Server GI/DB BP

Patch Release Frequency Current patch set: Monthly Previous patch set: Quarterly Ends when Error Correction Support ends

Exadata Storage Server Quarterly

InfiniBand Switch

Semi-annually to annually
Subject to change without notice

Error Correction (bug fix) Policy

Standard database patch set policy determines when BPs stop MOS 209768.1

Patch Planning
New Release Adoption

Planning
README and referenced Support Notes BP required overlays for critical issues and one-off conflicts Bug fix list 11.2.0.2 - MOS 1314319.1 11.2.0.1 - MOS 1316026.1

Stage

General Recommendation

Production and late pre-production Every mandatory patch Wait for field experience to grow Early pre-production or evaluation Keep current with latest release

Patch Testing Reduces Risk

Output of testing
Application functionality Application performance Installation method Timings Rehearsal, documentation, automation HA stages during patching Failure contingency

Patching Guidelines MOS 1262380.1

10

Test Environment
Test DBM Env Comments Identical Equivalent to production environment
Primary database Standby database Middle tier

Full data set w/ identical schema stats Workload framework to mimic production Smaller or Older None No production scale performance test No firmware patching test DANGER

+
11

RISK

Exadata Patching
Planning Recommendations

Current knowledge, current software

Exadata Technical Newsletter and MOS 888828.1 Dont fall far behind

Know your risks

Plan to test Test your plan

12

Patch Installation

13

Patch Methods
Patching Tools Component Software Patch Method patchmgr Linux - Minimal Pack Solaris - PKG OPatch OUI and DBUA ILOM

Exadata Storage Server All OS and firmware Database Server

GI/DB BP GI/DB Patch set

InfiniBand Switch

All

System readiness
Exachk (MOS 1070954.1)

14

Patch Methods
Rolling versus Non-Rolling

+
Non-Rolling

Database Downtime

+ +
Rolling

Patch Install Time Complexity

Patch non-rolling if maintenance window allows Rolling patching considerations

Patch during low workload Cell Rolling Patching Reduce disk failure risk w/ high redundancy DB Rolling Patching - Use Automatic Workload Management
15

Exadata Storage Server Patching

Patch contains all cell software and firmware
Install nothing but this patch on cells Unsupported configuration changes will be lost

patchmgr controls install and rollback

One invocation patches all cells Rollback to most recent successfully installed version
Method Rolling Non-Rolling Downtime None Up to 2 hours total Patch install time Up to 2 hours per cell Up to 2 hours total

16

Database Server Patching

Operating System and Firmware

Oracle Linux
Minimal Pack install.sh Kernel (Linux), OFED, Firmware, OSW, minor configuration 5.3 to 5.5 MOS 1284070.1
Stop Instances and CRS

Patch

Start Instances and CRS

Solaris 11 Express
Support Repository Updates (SRU) - MOS 1275533.1 Image Packaging System (IPS) pkg update

Patch new BE

Reboot
17

Database Server Patching

Database and Grid Infrastructure Bundle Patches

Use OPlan to generate patching plan

Patch using Out-of-Place plan Rolling or Non-rolling switch to patched home Do not hard code ORACLE_HOME in user scripts

18

Standby System
Data Guard

Patch replicated copy and switch

Reduce risk and downtime

Patching on standby without impact to primary

Exadata Storage Server Database Server Operating System and Firmware (Minimal pack) Grid Infrastructure home InfiniBand switch

Data Guard Standby-First Installable qualified BPs only

Database home

19

Database Server Patching

Database and Grid Infrastructure Patch Set Upgrade

Wait until Exadata validated

11.2.0.2 MOS 1315926.1 11.2.0.3 in progress Storage Server compatibility
Database 11.2.0.1 11.2.0.2 11.2.0.3 Required Storage 11.2.1 or later 11.2.2 or later 11.2.3 or later

GI upgrade is rolling Database upgrade requires downtime Reduce downtime and risk (Transient) Logical standby Golden Gate

20

Exadata Patching
Patching Recommendations

Only patch healthy system

Run Exachk before (and after)

Patch copy, switch to copy

Out-of-place Standby-First

Patch non-rolling (if possible)

Test and rehearse lowers downtime

Patch rolling (when necessary)

During low workload Reduce disk failure risk w/ high redundancy Use Automatic Workload Management

21

Agenda

Exadata Patching Exadata Manageability

22

Monitoring using EM Grid Control 11g

Monitor different components separately:
Monitor the database grid Monitor non-database components using EM Plug-Ins

Monitor components as one

Service Dashboards

Additional Monitoring Tools

ASR OCM

23

Enterprise Manager Architecture

Oracle Management Agent (and Plugins)

Oracle Management Service(OMS) Grid Control Console

Oracle Management Repository (OMR)

24

Exadata Management Setup: Manageability Note MOS 1110675.1

1.Deploy Oracle Management Agent to all database servers 2.Deploy the system monitoring plug-ins for Database Machine
a. b. c. d. e. f. Oracle Exadata Storage Server Oracle Exadata ILOM Oracle Exadata InfiniBand Switch Oracle Exadata Cisco Switch Oracle Exadata Avocent MergePoint Unity (KVM) Switch Oracle Exadata Power Distribution Unit

3.Configure ASM and Oracle Database targets 4.Configure user-defined metrics for additional network monitoring 5.Configure the plug-ins for high availability 6.Configure an Exadata Database Machine aggregate service and dashboard

25

OEM Agent and Exadata Plug-ins

1. OEM Agent and Plug-Ins are deployed under Oracle Home Agent incorporates additional functionality in Plug-In Agent and Plug-in communicates with Storage Server Plug-Ins are available for monitoring all Exadata hardware components OEM Agent communicates with the OEM
26

DB Server on Exadata DBM

2.

3.

4.

5.

High Availability for Plug-Ins

Normally a plug-in target is bound to a specific agent
If the agent is down the target cannot be monitored

A procedure exists to facilitate target failover to a secondary agent

Again My Oracle Support note 1110675.1 emcli relocate target

Oracle Exadata Database Machine

Database Server A
Primary Management Agent
To Targets To Targets

Database Server B
Secondary Management Agent

Plug-in Plug-in Plug-in Plug-in

Plug-in Plug-in Plug-in Plug-in

Oracle Management Service

27

Configuring Grid Control :Automation

Theres hope you dont have to do this alone! Agent automation download from Patch 11852882 Use Cases: Install and configure the management agent and Exadata plug-ins on: A newly established Grid Control installation An existing Grid Control environment with existing targets The approach is integrated with the Database Machine installation and initial configuration processes Linux and Solaris database server OS options are both supported

28

Databases, ASM, Listeners, Hosts

(Oh my!)

What we can see at the target

? Generally quite a bit!

managing the Oracle infrastructure for a long time

With the plugins, we expanded it

Information and alerts through plugins

..

But, whats important to monitor?

MOS note 1110675.1

How do I know its fast enough?

System baselines

29

A view across the Database Machine EM Systems and Services

Service view of Exadata components Detailed charts of key performance and usage information Service model and topology displaying relationships between components Tests can validate service availability from service endpoints

30

Service Topology
Aggregate Service

Service

Service

Service

System

System

System

Components

Components

Components

Components

Components

Components

METRICS

METRICS

METRICS

METRICS

METRICS

METRICS

31

Exadata Service Dashboards

Monitor many as one

Service information can be published for executives and business owners Many components reporting up to one Exadata Service

32

Auto Service Request

Automatically and securely opens service requests with Oracle for common hardware faults Minimal data is collected and transmitted No IP addresses are communicated One-way (customer to Oracle) SSL encrypted communications Can be integrated with existing monitoring tools ASR manager can send SR notifications via SNMP traps to existing monitoring tools ASR Manager version 3.3 or higher Supported using OEL, Solaris or OVM Oracle Exadata Quick Installation Guide for ASR oracle.com/asr
33

Maintain
Fault Diagnostics: ASR Flow
Customer Data Center
FRU replaced by Field Engineer

Oracle Support Services

Oracle Field Engineer

!
Fault occurs
Customer

FRU dispatched by Support Engineer

Comprehensive Fault Coverage For Compute Nodes, Storage Cells CPU Disk controllers Disks Flash Cards Flash modules Cards Memory System Board Power supplies Fans

SR creation email notification to customer

SR routed to Support Engineer

ASR Manager

Fault telemetry securely transmitted to Oracle

ASR Service Service Request created

Products auto-diagnosis facility sends SNMP trap to ASR Manager

Oracle Configuration Manager

Oracle Configuration Manager collects configuration information for propagation to Oracle Support. Benefits include: Faster service request reporting More efficient problem diagnosis Configuration information can be uploaded to Oracle either automatically or manually Configuration information remains confidential

35

Exadata
Oracle Configuration Manager & Auto Service Request
Storage Grid
Same As Below

Database Grid
InifiniBand Network

Two CSIs ASR OCM

Database Grid Storage Grid

Support HUB ASR Manager

My Oracle Support

Database Grid 8 1U Compute Servers Storage Grid 14 2U Storage Servers InifiniBand Network Redundant 40Gb/s switches

Server

Supported by ASR Supported by OCM

36

Exadata Management in GC12 Exadata as GC Target

Integrated view of hardware and software Hardware view
Schematic of cells, compute nodes and switches Hardware components alerts

Software/system view:
Performance, availability, usage by databases, services, clusters Software alerts db, cluster, ASM) Topology view of database systems/clusters

Manageability presentations this week

Session ID: 38121 Session Title: Eliminate Oracle Exadata Monitoring Anarchy with Oracle Enterprise Manager (Genentech) Tomorrow Moscone South, Room 304 Session ID: 14648 Session Title: Oracle Exadata Management Fundamentals: New Features for Monitoring, Tuning, and More Session ID: 38120 Session Title: Complete Oracle Exadata Monitoring with Oracle Enterprise Manager (Epsilon) Cloud Control 12c Exadata demo Moscone North, Keynote area (Outside Hall D)

38

Reference Notes
(a little homework)

General
Database Machine Owners Guide http://blogs.oracle.com/MAA

Patching
The Patching Note MOS 888828.1

Manageability
The Manageability Note MOS 1110675.1 Agent installation kit Patch 11852882 ASR setup MOS 1185493.1 (oracle.com/asr)

39

Patching Presentations
Recent Recordings

IOUG Exadata SIG presentation

Exadata Patching: Overview and Guidelines for Success

Oracle Support Advisor Webcasts MOS 740966.1

Exadata Patching Strategy Exadata Storage Software Patch demo Exadata Bundle Patch demo

40

2011 OOW Exadata Track Sessions

Day
Wed

Session Title
Atualizao de Desenvolvimento Oracle/SAP

Speakers
Tiago Barroso, Diretor de Infra-estrutura, EDP Bandeirante; Claudio Uemura, Principal Sales Consultant, Oracle Everson Forte, Advanced Customer Services Sr. Specialist, Oracle

Schedule
12:45 13:45 Mezanino Sala 5 16:45 - 17:45 Auditrio 3

Wed

Dicas, Truques e Melhores Prticas para o Oracle Exadata: Realizao e Recuperao de Backup do Oracle Exadata Consolidao no Oracle Exadata: Experincias de Implantao e Lies Aprendidas Uma viso tcnica aprofundada do Oracle Exadata: Arquitetura e Caractersticas Intrnsecas Princpios Bsicos do Oracle Exadata Management: Novos Recursos de Monitoramento, Tuning e Muito Mais Melhores Prticas para upgrade do Oracle Database

Thu

Daniele Nascimento, Solution Architect, Oracle; Fernando Silva, Solution Specialist, Oracle Fernando Melo, Principal Sales Consultant, Oracle Deba Chatterjee, Senior Product Manager, Database Manageability, Oracle

13:00 - 14:00 Auditorio 1 15:30 - 16:30 Auditorio 1 16:45 - 17:45 Auditorio 2

Thu

Thu

Thu

Juan Mikalef, Solution Architect Director, Oracle

16:45 - 17:45 Auditorio 3

41

Obrigado!
Joao Paulo Lunardelli LAD Solution Center Senior Manager
joao.lunardelli@oracle.com

42