Académique Documents
Professionnel Documents
Culture Documents
Disaster-Tolerant solution for HP AppSystems for SAP HANA (ScaleOut) based on HP Continuous Access EVA with HP X9300 Network Storage System and SAN boot
Table of Contents Executive summary ................................................................................................................................................. 2 Business needs ........................................................................................................................................................ 2 Using Continuous Access EVA .............................................................................. Error! Bookmark not defined. Explaining DT objectives ..................................................................................................................................... 5 Synchronous versus asynchronous replication mode ........................................................................................ 6 Solution overview .................................................................................................................................................... 6 Value added by X9300 NAS system .................................................................................................................... 8 Execution ................................................................................................................................................................. 8 Site failover ......................................................................................................................................................... 8 Site failback ........................................................................................................................................................ 9 Solution qualification ............................................................................................................................................ 10 More on the loss of inter-site links ................................................................................................................... 11 Conclusion ............................................................................................................................................................. 11 Appendix: Installing and setting up the DT solution ............................................................................................. 12 Qualified hardware and software ..................................................................................................................... 12 Configuring SAN switches for a Continuous Access environment .................................................................... 13 EVA & X9300 storage map and SAP HANA layout ............................................................................................. 13 Multi-pathing .................................................................................................................................................... 13 Enabling replication for the HANA database .................................................................................................... 14 Deploying X9300 on SAN boot .......................................................................................................................... 15 Using the DT solution ........................................................................................................................................ 15 ............................................................................................................................................................................... 16 For more information ............................................................................................................................................ 17
Executive summary
As SAP HANA (High Performance Analytic Appliance) becomes increasingly available as an enhanced replacement for SAP Business Warehouse (BW) installations, a broad range of options for implementing high availability and disaster tolerance is being evaluated. Disaster recovery capabilities become crucial in the event that an entire computing site is lost and the organization cannot afford the time required to rebuild HANA Database1. When considering a disaster recovery solution for HANA, factors such as cost, recovery time objective (RTO), and recovery point objective (RPO) are important. In response to these factors, HP has developed a disaster-tolerant (DT) solution for HP AppSystems for SAP HANA Scale Out (the DT solution) based on HP Continuous Access EVA and HP X9300 Network Storage System with SAN Boot. This solution allows shared storage to be connected to the SAP HANA server blades via Network File System (NFS) of HP IBRIX fusion file system using SAN booted IBRIX segment servers. This test plan describes an active/passive HP-validated DT solution wherein human decision initiates the failover of a production site to an alternate site following the failure of the production site. Depending on the particular needs and the distance between sites, the DT solution allows for deployment of a HANA non-production instance at the alternate site until the site is required to become active in the event of a disaster. Target audience: This paper is intended for HANA administrators wishing to learn more about deploying a DT solution. In general, however, the reader does not require experience with SAP HANA software. This white paper describes the design, systems integration, and validation performed by HP in December 2011, and the Certification received from SAP on [tbd]. (see SAP PAM at [insert URL here]
Business needs
When data security and availability are critical to the success of their businesses, SAP customers require a computing solution that protects their information systems from disasters such as power outages, earthquakes, fires, floods or acts of vandalism. The effects of a disaster can range from temporary loss of availability to the outright physical destruction of a facility and its assets. In the event of a disaster, the HANA environment must allow customers to shift their information-processing activities to another site as quickly as possible. Thus, procedures for disaster recovery must be predictable, well-defined, documented in advance, and executed by qualified Systems Administrators and IT decision-makers. Figure 1 outlines the Configuration requirements for a DT solution for HANA.
The time taken depends on log and data area volume size in the HANA system.
More than 25 years of experience implementing hundreds of disaster tolerant and continuous computing solutions around the world More than 80 percent of the worlds stock exchanges and worldwide banking transactions run on HP solutions Integrated Appliance solution with built-together not puttogether storage replication technology, server clustering, testing and ongoing management services HP Disaster Tolerant solutions are rated #1 among UNIX vendors in Gartner Group studies HP is a pioneer in disaster tolerant computing, including technologies such as clustering and data mirroring HP AppSystems for SAP HANA provide bulletproof availability and application up time
IBRIX Native HA IBRIX Cluster nodes FC Link IBRIX Cluster Nodes FC Link
IBRIX Native HA
SAN
Making the applications replicated data accessible at the destination site Starting HANA instance and X9300 segment servers at the destination site to restore application availability
Explaining DT objectives
There are two key objectives for a DT solution:
Recovery point objective (RPO) RPO refers to the point-in-time up to which data can be recovered following a disaster;
in general, RPO specifies the amount of data loss you can tolerate.
Recovery time objective (RTO) RTO refers to the maximum length of time taken for the recovery site to be up-andrunning following a disaster. More information on these objectives is provided below. RPO Some customers require an RPO of zero. For a HANA environment this means that in the event of a failure causing the loss of the storage service, you cannot lose a single committed I/O transaction; you must be able to recover data up to the exact time the disaster occurred to avoid any inconsistency in HANA database. One of the implications of implementing a DT solution with an RPO of zero is the requirement for synchronous replication, described below. If, however, an RPO of greater than zero is appropriate that is, you can tolerate some data loss asynchronous replication may be a viable option. The DT solution described in this white paper uses synchronous replication. RTO From the perspective of the DT solution described in this white paper, RTO includes the time taken to achieve the following:
Fail over replicated logical unit numbers (LUNs) to the remote EVA Recover and bring X9300 segment servers online at the remote site Recover and bring HANA instance online at the remote site
System administrator time including customer configuration- and usedependent number and sequence of failover invocation and startup steps 2 Synchronous versus asynchronous replication mode
In general, Continuous Access EVA supports dynamic switching between synchronous and asynchronous replication modes. These modes can be characterized as follows:
Synchronous mode
Data on the remote site is always consistent with the local site Data is mirrored in real-time Synchronous mode is appropriate when data consistency is highly critical to the business Depending on the distance between sites, synchronous mode can affect overall performance by increasing the time required for writes
Asynchronous mode
I/O completion status is returned to the host when local writes complete; remote writes are deferred until a later time The concurrency of data at the remote site can lag behind the local site by a number of write I/O Asynchronous mode supports longer distances between local and remote sites In-order delivery of new or modified data blocks at the remote site is guaranteed
Note Currently, the HANA DT solution supports synchronous replication mode only
Solution overview
The DT solution for HANA provides a mechanism for configuring a disaster-tolerant HANA configuration that is distributed between distant computer sites. This solution combines the following components:
HP Continuous Access EVA HP X9300 Network Storage system with enabled SAN boot
A HANA Disaster Tolerant solution configuration using Continuous Access EVA requires mirrored appliance configurations that is, a complete replica of configuration hardware and at both the primary and secondary sites, as shown in Figure 2. The solution also requires same virtual host names at primary and secondary sites 3. The HANA production instance deployed in the primary site is connected to the production system and runs the external transactions at this site. Unless a disaster occurs, all I/Os take place on the storage subsystem at the primary site. Meanwhile, Continuous Access EVA has exclusive access to the storage LUN at the alternate site (target/secondary site), to which it synchronously replicates write I/Os performed on storage at the primary site on HANA. If a significant failure were to occur at the primary site, data processing could be resumed at the alternate site where data would be intact and consistent.
Dual-purposing note: RTO also depends upon the purposing of the second instance. If dedicated exclusively to DR, no additional impact. If the second site is dual-purposed for non-production environment, additional manual steps to remove the non-production instances are required 3 Alternative: scripts that change iLO IP and/or MAC translation addresses.
Figure 2: HP Storage Works DT solution for HANA on EVA with X9300 NAS System.
The DT solution can span the distances typically encountered in a commercial or college campus up to a metropolitan distance (estimated at 50 km / 30 miles), depending on the type of inter-site link (ISL) implemented on the SAN used for replication. The initiation of a site failover requires human intervention and some customized scripts which run at X9300 NAS system layer. Using the CV EVA and iLO management console, you can initiate Storage system failover and bring X9300 segment servers online, resulting in the stable state start-up of HANA at the alternate site as shown in Figure 3.
Note The scripts referred to above are part of the DT solution and are delivered when the solution is implemented.
Execution
Site failover
After deciding to perform a planned or unplanned failover, you should follow the appropriate Steps to initiate a failover to the alternate site. The steps are described in Table 1. Failover execution for the DT solution
Step 1 Primary (failed) site Stop the HANA production instance Alternate site
2 3
Stop (Shutdown) the X9300 segment servers Initiate Storage (EVA) DR Group failover for CA using Command View EVA Bring up the X9300 segment servers using iLO console Bring up the HANA instance Restart the NFS service on HANA node and remount the X9300 NFS share Start the HANA instance
4 5 6
. If the failover is in response to an unplanned failover, the steps listed at the failed site may not get executed. In this case, ensure that the failed site has been powered down or is in some other way isolated from the IP network and FC SAN before bringing up the secondary site.
Table 1. Failover execution for the DT solution
Step 1 2 3
Primary (failed) site Stop the HANA production instance Stop (Shutdown) the X9300 segment servers Initiate Storage (EVA) DR Group failover for CA using Command View EVA
Alternate site
4 5 6
Bring up the X9300 segment servers using iLO console Bring up the HANA instance Restart the NFS service on HANA node and remount the X9300 NFS share Start the HANA instance
Site failback
After failover to perform a failback, you need to follow the appropriate steps to initiate a failback to the earlier source site. The steps to initiate a failback are described in Table 2. If the failback is in response to an unplanned failover where the source site would have been completely destroyed, before failback is initiated, the source site in the failed state has to be powered on with working IP and FC networks. Full resync or full data copy between the source site EVA DR Group and the failed over site EVA DR Group has to be initiated.
Note Full data copy can be initiated by pressing the force full copy button in Data Replication/Members context, using VC EVA. In the Connections tab, switch replication status from Suspended to Resumed.
Step 1 2 3
Alternate site Stop the HANA production instance Stop (Shutdown) the X9300 segment servers Initiate Storage (EVA) DR Group failover for CA using Command View EVA
4 5 6
Bring up the X9300 segment servers using iLO console Bring up the HANA instance Restart the NFS Service on HANA node and remount the X9300 NFS share Start the HANA instance
Solution qualification
After setting up the solution as outlined in Appendix: Installing and setting up the DT solution, HP simulated a broad range of unplanned failure events and tested the solutions ability to fail over to the alternate site. Table 3. Recommended responses to unplanned failure events
Type of failure Recommended action Average time taken for failover 10 minutes
Total loss of the primary site Loss of one fabric at the primary site Loss of the EVAs controller-pair at the primary site Loss of all inter-site links
Manually initiate failover Do not initiate failover Manually initiate failover Decide which site should continue processing, then either: - Disable Failsafe mode* to continue processing at the primary site, or - Manually initiate failover
10 minutes
Total loss of the alternate site Loss of one fabric at the alternate site Loss of the EVAs controller-pair at the alternate site Loss of a single storage controller at the primary site Loss of a single network switch at the primary site Extended power outage at the primary site Loss of a single host bus adapter at the primary site
10
Loss of a single disk in redundant storage at the primary site Loss of single NFS cluster node at the primary site Total loss of the primary site
above lists common unplanned failover events and recommends the response you should take to each. However, you should note the following provisions before initiating a site failover:
It is important to verify that all components at the alternate site are operational It may be preferable to continue processing at the primary site rather than initiating a failover if you determine that the
failure of a single component can be resolved within an acceptable timeframe Table 3. Recommended responses to unplanned failure events
Type of failure Recommended action Average time taken for failover 10 minutes
Total loss of the primary site Loss of one fabric at the primary site Loss of the EVAs controller-pair at the primary site Loss of all inter-site links
Manually initiate failover Do not initiate failover Manually initiate failover Decide which site should continue processing, then either: - Disable Failsafe mode* to continue processing at the primary site, or - Manually initiate failover
10 minutes
Total loss of the alternate site Loss of one fabric at the alternate site Loss of the EVAs controller-pair at the alternate site Loss of a single storage controller at the primary site Loss of a single network switch at the primary site Extended power outage at the primary site Loss of a single host bus adapter at the primary site Loss of a single disk in redundant storage at the primary site Loss of single NFS cluster node at the primary site Total loss of the primary site
11
Important The table below gives a general failover estimate. These are estimates and there are many customer-specific details that can impact the failover. As a matter of best practice for all deployments, HP recommends implementing a proof-of-concept using a test environment that matches as closely as possible the planned production environment. In this way, appropriate data can be obtained. For help with a proof-of-concept, contact an HP Services representative (http://www.hp.com/hps/contacts/index.html) or your HP partner.
Type of failure
Recommended action
Total loss of the primary site Loss of one fabric at the primary site Loss of the EVAs controller-pair at the primary site Loss of all inter-site links
Manually initiate failover Do not initiate failover Manually initiate failover Decide which site should continue processing, then either: - Disable Failsafe mode* to continue processing at the primary site, or - Manually initiate failover
10 minutes
Total loss of the alternate site Loss of one fabric at the alternate site Loss of the EVAs controller-pair at the alternate site Loss of a single storage controller at the primary site Loss of a single network switch at the primary site Extended power outage at the primary site Loss of a single host bus adapter at the primary site Loss of a single disk in redundant storage at the primary site Loss of single NFS cluster node at the primary site Total loss of the primary site
* Refer to More on the loss of inter-site links (below) for more information on setting Failsafe mode for a Continuous Access EVA -enabled application.
12
Type of failure
Recommended action
Total loss of the primary site Loss of one fabric at the primary site Loss of the EVAs controller-pair at the primary site Loss of all inter-site links
Manually initiate failover Do not initiate failover Manually initiate failover Decide which site should continue processing, then either: - Disable Failsafe mode* to continue processing at the primary site, or - Manually initiate failover
10 minutes
Total loss of the alternate site Loss of one fabric at the alternate site Loss of the EVAs controller-pair at the alternate site Loss of a single storage controller at the primary site Loss of a single network switch at the primary site Extended power outage at the primary site Loss of a single host bus adapter at the primary site Loss of a single disk in redundant storage at the primary site Loss of single NFS cluster node at the primary site Total loss of the primary site
, initiating a failover when all inter-site links are lost is not recommended. Without these inter-site links, it is impossible to maintain a complete copy of the data at the secondary site. Each DR group has an associated write history log (WHL) that is used to store data when replication to a secondary DR group has been stopped because either this group is unavailable or suspended or a network failure has occurred.
Note If Failsafe mode was disabled, this process (known as logging) took place automatically when the loss of inter-site links was simulated.
When replication resumes, WHL contents are replicated to the secondary DR group a synchronization process known as merging. Because the data is written to the secondary group in the order that it was written to the log, merging results in a consistent copy of the virtual disk at the alternate site. However, in the event of a catastrophic disaster such as a fire, replication never resumes; thus, the contents of the WHL are never applied at the secondary site. In this scenario, it is probable that data at the secondary site would be consistent but incomplete.
13
To ensure that data remains complete as well as consistent, you are recommended to enable Failsafe on linkdown/powerup for the DT solution as shown in Figure A-3 in the Appendix of this document. These setting stalls write I/Os in the event that all inter-site links are lost, ensuring that data remains synchronized. Write I/Os are resumed in the following circumstances:
The links are restored, or One site is allowed to continue processing by disabling Failsafe mode in anticipation of the eventual, full
re-synchronization of both sites
Conclusion
Many organizations are considering DT configurations for their SAP HANA instances; however, a DT implementation can create challenges in the areas of storage utilization, performance, management, and availability. HP addresses these challenges through a robust, easy-to-use solution that combines the following technologies:
14
Server-side Hardware HANA production blades 16 HP ProLiant BL460c server blades, each configured with: 32 GB RAM Two 15,000 rpm SAS hard drives One two-port 10 GbE network mezzanine card
Shared storage
One HP Storage Works P6500 storage array, configured with: 4 GB cache 16 high-performance FC hard drives (450 GB, 15,000 rpm)
Storage management server One HP ProLiant server acting as a Command View EVA server
Table A-2: Software components at each site for the DT solution validated by HP and SAP
Server-side Software
Shared storage
HANA production blades 16 HP ProLiant BL460c server blades, each configured with: SUSE Linux SLES 11 SP1 SAP HANA revision 1.0 SPS4
One HP Storage Works EVA4400 storage array, configured with: XCS 10 Continuous Access EVA
15
HSV controllers in the EVA4400 storage arrays FC adapters (FCAs) in the HANA blades and, if appropriate, X9300 cluster nodes Management zones for storage management server appliances at primary and alternate sites
Fabric settings The following settings for Brocade SAN switches and Continuous Access EVA are based on rules detailed in the HP SAN Design Reference Guide:
aptpolicy 1 sets port-based routing policy iodset guarantees in-order delivery dlsreset disables dynamic path selection
For more information on configuring switch settings for a particular environment, see the HP SAN Design Reference Guide.
Multi-pathing
Since only deploying a single fabric would create a single-pointof-failure, maintaining two separate fabrics is a prerequisite for Continuous Access EVA in an SAP production environment. SUSE Linux Enterprise Server 10 (SLES 10) SP3 & RHEL5.6 (used in IBRIX segment servers) provides multipath I/O (MPIO) support, allowing you to maintain two or more separate paths to the storage. This MPIO support, which is based on the Linux kernels Device Mapper (DM) multipath module, provides the functionality needed to switch to an alternate path should one path be unable to complete application I/Os. Figure shows that there are four paths to the 100 GB LUN used in the DT solution qualified by HP.
Figure A-1. DM multipathing
16
Creating the DR group starts the initial, full replication to the target EVA storage. In general, you should not initiate such replication during periods of high I/O activity due to the additional write I/O load imposed. Figure A-3 shows the HANA DR group under normal conditions.
Figure A-3: General view of the HANA DR group
After the full copy has finished, the Group host access state in DR Group Properties becomes Normal, indicating that primary and alternate sites are synchronized and ready to toggle their Source and Destination roles. Figure A-4 shows connection settings for the DR group; in this example, Write mode has been set to Synchronous to support the synchronous replication used in this DT solution.
17
Figure A-4: Specifying settings for the connection between EVA storage arrays in the DR group
Refer to IBRIX Boot from SAN Guide to Configure X9300 for SAN boot Refer to HP X9300 IBRIX V6.1.1 with P6500 for SAP HANA guide for X9300 (IBRIX6.1) Installation on SAN volumes
Copy the script developed by HP to the X9300 segment servers at both the sites. The location where the scripts are copied
should be identical in both X9300 cluster nodes
Copy the DT-01.rules file provided to /etc/udev/rules.d directory, and comment out all the rules described in
60-net.rules file under/etc/udev/rules.d directory
Edit the ibrixaps_ha.sh file, by setting the HOME_DIR to the directory where you have copied the scripts Edit /etc/init.d/network with/<HOME_DIR>/ibrix_nic.sh , preferably at the 30th line of the file and before
the network is started at boot time by the init network script
Edit the /etc/rc.local file with /<HOME_DIR>/ibrixaps_ha.sh 1 Edit the /etc/hosts file in both the IBRIX cluster nodes, specifying the IP addess with the hostname, at both the Sites
a) b) c) d) Edit the ibrixha.cfg file with the following details : IBRIX_USER = iLO user IBRIX_PASSWD = password for iLO user IBRIX_POLL = Default is 10 (For 2 IBRIX Nodes at each site), can be changed depending on the number of IBRIX nodes we have at each site (Should be changed by HP TS) REMOTE_RETRY = Default is 10 (For 2 IBRIX Nodes at each site), can be changed depending on the number of IBRIX nodes we have at each site (Should be changed by HP TS)
18
A site failover is initiated via steps described below; Figure A-5 shows how to trigger a failover:
Initiate Storage (EVA) DR Group failover for CA, using Command View EVA from the destination (secondary site EVA) Bring up the X9300 segment servers using iLO console
Figure A-5: Initiating a site failover
19
Other solution whitepapers, forums, and webinars by Customer Focused Testing Linux
http://www.hp.com/go/ /linux
Get connected
hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. AMD is a trademark of Advanced Micro Devices, Inc. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates.
20