Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

An Oracle White Paper March, 2011

Oracle Exalogic Elastic Cloud: X2-2 Hardware Overview

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

Disclaimer
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

Introduction
Balancing cost, performance, complexity in engineering the most optimal hardware and software system is non-trivial. Regardless of approach, the end result may not yield the desired outcome. As a result, IT has started the inevitable shift to tightly integrated engineered systems as a cost effective simple way of obtaining an optimized purposefully built system that considers all of the traditional variables. In addition, engineered systems offered by systems companies offer improved quality and efficiency over custom systems built and assembled ad-hoc within IT departments. Software vendors, by and large, write software to function on a variety of platforms varied by device drivers, operating systems, chip sets, network and storage devices. Java made this considerably simpler by abstracting the hardware layer away from the developer offering a development platform well suited to building portable enterprise class applications regardless of hardware and operating system. However, this portability has come at a cost, specifically performance. While many applications built on Java perform extremely well, tuning applications is often an expensive and time consuming process. Exalogic Elastic Cloud takes performance of Java and Java EE based applications to the next level by offering a platform optimally suited for deployment of Java based applications with a particular focus on WebLogic server and the rest of Oracle Fusion Middleware. This paper focuses on the hardware aspects of Exalogic Elastic Cloud, including the selection process and rationale, as well as technical insights into what systems modifications have been engineered to offer a system optimally suited for middleware workloads.

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

Hardware: Foundation for Acceleration

Hardware Design Choices
Specifying and selecting the proper hardware is a time consuming and labor intensive process, assuming that testing of the various configurations is performed. The Exalogic engineering staff has conducted extensive testing on a wide range of hardware configurations to arrive at the optimal configuration for middleware type deployments. Rationales for these decisions are outlined below.
Design Goals

The following design goals guided the system architecture of Exalogic Elastic Cloud X2-2: Density: A key driver in the decision making process was to architect a system that provided a high level of compute density which is necessary for both high performance applications and for the use case of consolidating disparate middleware and application deployments. State of the art- Since Exalogic was to represent the best general purpose middleware deployment platform, it was necessary to pursue the most advanced technologies available in the Sun/Oracle hardware portfolio. This is evidenced by the choices in all component areas: compute, network and storage. Balance: Architecting a high performance system is not necessarily a difficult proposition. However, designing a high performance system that is balanced in terms of power consumption, storage, compute, cost, serviceability etc. is non-trivial. Exalogic puts forth a balanced system design that considers all of the relevant system design factors suited for middleware type workloads. Serviceability: The ability to field service the system is extremely important when it comes to providing a fault tolerant system. The ability to perform reliable field upgrades1 as well as the swapping of failed components is important in maintaining uptime in the overall system. Serviceability is not only evident in the mundane (but important) aspect of how the cable design has manifested itself in Exalogic, but also the choice for individual hardware components such as

For example, the ability to upgrade a quarter rack to a half rack or from a half rack to a full rack configuration is intended to be performed in a reliable predictable fashion.
1

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

redundant InfiniBand switches and storage heads that can be replaced without requiring any downtime. High performance networking: At the center of any clustered super computer sits a high performance network; preferably a scalable fat-tree network such as InfiniBand, enabling high speed communication between constituent applications and components. As described below, InfiniBand is a key technology behind the success of Exalogic as it provides a reliable high performance network interconnect while at the same time offloading compute cycles from host CPUs further increasing performance of the overall system. Consolidation: A primary goal was also to provide a platform optimally suited for the consolidation of heterogeneous applications. This implies the ability to support multiple application stacks running on various operating systems.
Compute Node

Selecting a compute system optimally suited for high performance Java workloads leads to selecting a processor that has the ability to efficiently support multiprocessing. Complimentary to this, is the ability to efficiently be able to support multiple hardware threads utilized by the executing Java software stack which tends to be highly threaded. The compute node within Exalogic is comprised of multiple 1U X4170 M22 compute nodes which hosts dual socket 5600 Westmere series rack optimized Intel Xeon processor. Each processor consists of 6 cores for a total core count of 12 cores per compute node with 24 hyper-threads allowing for highly concurrent workloads to execute efficiently. The specific configuration of X4170 M2 includes twelve 8 GB DIMMs (Dual in-line memory modules) and two 32 GB SATA high performance high MTBF solid state disk drives configured in RAID 1 with redundant fans and redundant power supplies. The end result is a high performance, fault tolerant compute node with no moving parts besides the dual redundant fans. Besides high performance processors, Java benefits from high performing RAM to efficiently handle frequent memory allocation and garbage collection. For this reason, the X4170M2 is equipped with exactly 96GB of RAM; the maximum amount of RAM enabling the higher memory bus speed of 1333 MHz. Although the X4170M2 has a max capacity of 144GB, increasing memory beyond the 96Gb barrier requires a 40% drop in the memory bus speed to 800Mhz which is sub-optimal for Java based applications. The end result is 8GB of RAM per

Sun Fire X4170 Data Sheet: http://www.oracle.com/us/products/serversstorage/servers/x86/sun-fire-x4170-m2-ds-079875.pdf

2

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

core, which is more than enough for the most memory intensive applications. Figure 1 and figure 2 illustrate the front and back view of the physical X4170M2 unit.

FIGURE 1: COMPUTE NODE FRONT VIEW3

FIGURE 2: COMPUTE NODE REAR VIEW

Storage

In order to facilitate use cases that involve high levels of elasticity and high availability, Exalogic is outfitted with the 7320 ZFS storage appliance4. As individual compute nodes are intended to only host operating system binaries, the 7320 storage appliance stores all other application binaries, log files and content necessary for the applications to execute. The compute nodes

The X4170M2 Exalogic Elastic Cloud configuration does not include a DVD drive See 7320 Data Sheet: http://www.oracle.com/us/products/servers-storage/zfs-storage-family-ds173093.pdf
3 4

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

mount configured partitions of the 7320 to access necessary data over the InfiniBand network. The notion of mounting the 7320 networked attached storage device eliminates the need to back up individual compute nodes and in the event of failure of a compute node, the partition can simply be mounted from another compute node while the failed node can be replaced. The 7320 supports multiple protocols (NFS v2/v3/v4, CIFS, HTTP, WebDAV, FTP/SFTP/FTPS) and are all made available for use. However, NFS on Internet Protocol over InfiniBand (IPoIB) is the recommended configuration as a result of extensive reliability and performance testing. More information on this topic can be found in the Oracle Exalogic Enterprise Deployment Guide5. The 7320 appliance in Exalogic contains 20 2TB drives which can be configured in numerous ways depending on high availability needs and desire for replication. Typically, two 2TB drives are set aside as hot spares in case of drive failure. Any drive failures are automatically handled by the storage system through re-partitioning with the spare drive being put into service. Figure 3 shows an illustration of the 7320 ZFS storage appliance. As shown, the two redundant storage heads (Processor 1 and Processor 2) contain 2TB of high performance read cache, often referred to as readzillas. These caches store data retrieved through previous reads performed on the spinning disks contained in the JBOD6 tray. The cache significantly enhances performance of reads as access to slower spinning drives are not necessary. Complimentary to the readzilla caches is the logzilla cache co-located in the storage tray. This cache is a highly performant write cache which allows writes to the appliance to be done with minimal latency. Data that is written to the logzilla cache is asynchronously written to the spinning drives.

The Exalogic Enterprise Deployment Guide can be found here: http://download.oracle.com/docs/cd/E18476_01/doc.220/e18479/toc.htm 6 Just a Bunch of Disks
5

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

FIGURE 3: 7320 STORAGE APPLIANCE FRONT VIEW

Networking

During the last several years, chips have not become significantly faster in terms of clock speed. However, chips are able to deliver on much higher workloads through denser core counts. This trend is likely to continue based on current roadmaps from microprocessor leading vendors7. The implication of denser core counts is that each processor is able to perform more work. Therefore each compute node is likely to demand escalating network resources in the form of remote method invocations, file transfers and database interactions. The network can easily become a bottleneck from both a throughput and latency perspective. In a traditional IP over Ethernet network configuration, the compute node requesting network resources is involved in the processing of the top layers in the TCP stack, requiring additional compute resources for all network interactions. Providing a high performance network as the foundation for a high performance system is fundamental in constructing a highly performant system with todays high core count processors. It is for this reason that both Exalogic and Exadata are built from the ground up using InfiniBand.

See Intels microprocessor roadmap: http://download.intel.com/products/roadmap/roadmap.pdf and Oracles SPARC roadmap: http://www.oracle.com/ocom/groups/public/@ocom/documents/webcontent/044518.pdf
7

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

The InfiniBand Trade Association (IBTA) was founded in 1999 is led today by Oracle, Intel, IBM, Mellanox, QLogic and Voltaire. The IBTA is responsible for the advancement of the InfiniBand specification. The goal of the IBTA is to put forth a communication link suitable for use in high performance computing systems such as Oracle Exadata and Oracle Exalogic.

Comparing state of the art 10Gb/s Ethernet networks with InfiniBand Quad Data Rate (QDR) shows us the advantage of InfiniBand as a high performance network infrastructure. In addition to raw network performance and reduced latency, the compute node does not need to be concerned with TCP management resulting in more cycles spent executing the actual application. While InfiniBand is a high performance network, all applications are not able to take advantage of the native features offered by its infrastructure, namely the sockets direct protocol (SDP), a lower level protocol enabling writing directly to the wire. This requires an InfiniBand aware software stack such as Oracle Fusion Middleware or Oracle Database. Although most applications cannot take advantage of the native InfiniBand SDP protocol, the Open Fabrics Enterprise Distribution (OFED) stack that ships with Exalogic provides an IP compatible implementation known as Internet Protocol over InfiniBand (IPoIB). IPoIB is an IP network interface implementation operating over InfiniBand allowing all InfiniBand unaware applications to communicate unchanged over InfiniBand. Applications utilizing this compatibility network stack see significant benefit of the InfiniBand network through higher throughput and lower latency compared to a 10Gb/sec Ethernet network. Applications running on compute nodes communicate with applications on other compute nodes using this InfiniBand network. Interconnected Exalogic and Exadata Machines communicate via IPoIB. Exalogic machines can be connected to an external network, including a standard database hosted on a machine outside of the Exalogic machine, via the InfiniBand-to-10 Gb Ethernet gateways using Ethernet over InfiniBand (EoIB). Each Exalogic machine configuration includes at least 2 such gateways, which also act as InfiniBand switches connecting all compute nodes and the Sun ZFS Storage 7320 appliance within the Exalogic machine.
InfiniBand Switches

Within Exalogic, the InfininBand gateway switches provide a shared, high speed network for application clusters that are comprised of compute nodes and the shared storage appliance. The

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

gateway switch features a standard 1u data center form factor, thirty two 40Gb InfiniBand ports, eight 10Gb Ethernet ports, and the Sun ILOM8 management interface. Exalogics converged fabric leverages the properties of the InfiniBand architecture to enable high-performance in demanding, clustered data center environments. The fabric supports the creation of logically isolated network partitions, as well as advanced features for traffic isolation and quality of service (QoS) management. QDR InfiniBand delivers 40 Gigabits of raw bandwidth. The InfiniBand base rate is 10 Gigabits per second. QDR capable products deliver four times the base rate. Deploying a converged fabric minimizes the cost and complexity of building and operating scalable systems by reducing the number of adapters, cables, and switches to install, manage, and maintain. LAN traffic carried over Oracle's converged data center fabric uses familiar Ethernet semantics. Furthermore, the LAN ports on the Sun Network QDR InfiniBand Gateway Switch appear as endpoints to the LAN, thus avoiding potential interoperability issues with network management tools and methods. Within the Sun Network QDR InfiniBand Gateway Switch, the embedded fabric manager is enabled to support active/standby dual manager configurations, ensuring a seamless migration of the fabric management services in the event of a module failure. The switch is also provisioned with redundant power and cooling for high availability. The front panel of the Sun Network QDR InfiniBand Gateway Switch has 36 QSFP connectors (See Figure 4). The connectors are each dedicated to InfiniBand or Gateway functions as follows:

The left-most thirty-two ports support QDR InfiniBand connectivity. The two upper right-most ports support Ethernet connectivity. The two lower right-most ports are covered with a plug marked Do Not Remove. No connections should be made to these ports.

See http://www.sun.com/systemmanagement/ilom.jsp for more information.

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

FIGURE 4: INFINIBAND SWITCH

The switch has two physical ports dedicated to Ethernet functionality. These ports can support up to four 1/10 Gb Ethernet connections each. Oracle provides splitter cables which fan out a single QSFP (Quad Small Form Pluggable) switch port to four optical fiber LC connectors. A QSFP transceiver is available to connect the splitter cable to the switch.
Network Extensibility

All X4170 compute nodes and the 7320 storage appliance are connected via the fully switched InfiniBand backplane. If desired, additional Exalogic and Exadata can be added to the network fabric to facilitate high performance computing across eight full racks without requiring additional networking hardware such as a data center switch. A common use case is the integration of Exalogic and Exadata to form a high performance compute system combining raw compute power with Online Transaction Processing (OLTP).
Operating System

Exalogic provides users with a choice of Oracle Linux or Oracle Solaris operating systems. Solaris is the number one UNIX operating system in the market and Oracle continues to invest in Oracle Linux. Exalogic is 100% compatible with all standard Oracle Linux 5 and Solaris 11 applications. To this end, no special certification for Exalogic is required. As a result, all Oracle applications that are certified for the appropriate releases of Oracle Linux and Solaris are supported on Exalogic. When Oracle released Exadata and Exalogic, Oracle tested those servers with the old Red Hat compatible kernel and found that the OS based on the 2.6.18 kernel was not able to drive the

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

system given the high performance networking and storage infrastructure. In order for Exalogic to leverage the hardware, SDP and IPoIB, a modern OFED stack was needed. This was not available with RHEL and therefore set out to evolve the RHEL kernel. To produce UEK (Unbreakable Enterprise Kernel), Oracle began with a stable release of the 2.6.32 kernel, representing a major leap forward from the RHEL5 2.6.18 kernel. This kernel included enhancements focused on performance and stability. Considerable work went into ensruing that heavy network and IO loads could be spread across all CPUs in order to optimize RDS and InfiniBand performance. UEK represents the best Linux performance Oracle can deliver today. By moving to 2.6.32 as base for the kernel many features in the areas of hardware fault management, data integrity and diagnostics are advanced, such as Detecting and logging hardware errors before any affect to OS or application Automatic isolation of defective CPUs and memory Avoiding system crashes Improved application uptime

Taking advantage of these enhancements, require no changes to existing applications. The optimizations provide up to 12x InfiniBand performance, 60% higher workload, 50% reduced latency, and 50% improved IPoIB performance9. Oracle Solaris 11 Express10 raises the bar on the functionality introduced in Oracle Solaris 10, continuing Oracle's leadership for providing the best choice in mission critical operating systems for enterprise environments. As a result of a host of new features such as:

Network based package management tools to greatly decrease planned system downtime and provide for a complete safe system upgrade Built-in network virtualization and delegated administration for an unprecedented level of flexibility for application consolidation Continued leadership for providing the highest levels of security

Oracle Solaris 11 Express is simply put, the most exciting release of the Oracle Solaris platform to date.

See Oracle Exalogic Elastic Cloud: A Brief Introduction for optimization details http://www.oracle.com/us/products/middleware/exalogic-wp-173449.pdf 10 For detailed documentation on Oracle Solaris 11 Express see http://www.oracle.com/technetwork/server-storage/solaris11/overview/index.html
9

10

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

Oracle Solaris Zones is built-in OS virtualization which continues to build on its long and distinguished pedigree. One of the most highly adopted, highly used, mature virtualization technologies, Oracle Solaris Zones was first introduced as a core part of Oracle Solaris 10. With Oracle Solaris 11 Express, Oracle Solaris Zones11 has become even more central to both the application and the end user, and continues to introduce enhancements with new improvements and features including: Integration into the new packaging system Improved observability Increased control over administration Tight integration with ZFS Support for Oracle Solaris 10 Zones Integration with the new Oracle Solaris 11 network stack architecture

Best in class manageability

Oracle Enterprise Manager provides application-to-disk management through Enterprise Manager Grid Control and Enterprise Manager OpsCenter. Enterprise Manager allows every individual hardware component within an Exalogic deployment to be monitored in real time and, at the customers option, have system status automatically reported to Oracle Support for proactive system maintenance. Through integration with Oracle Support, Enterprise Manager can apply tested patch bundles tailored for Exalogic that cover every layer of the system, from device firmware and operating system to JVM, application server, upper-stack Fusion Middleware, and Oracle applications.

See Oracle Solaris Virtualization: http://www.oracle.com/technetwork/articles/servers-storage-admin/sol11evirt-186209.pdf

11

11

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

FIGURE 5: EXALOGIC ELASTIC CLOUD MANAGEMENT INFRATRUCTURE

Enterprise Manager Grid Control (EMGC)

Enterprise Manager Grid Control focuses on managing the software assets deployed on Exalogic, with particular attention to Fusion middleware and Oracle applications. Exalogic specific functionality allows representation, management and monitoring of Exalogic deployments.
Enterprise Manager Operations Center (EMOC)

Enterprise Manager Operations Center is Oracle's single, integrated solution for managing all aspects of the data center. It allows for the discovery, provisioning, update, management, and monitoring of the physical and virtual assets in multiple data centers from a single console. Enterprise Manager Ops Center is an enterprise management solution that sits off the servers and communicates to the hardware assets through the ILOM (Integrated Lights Out Manager). Operating System agents can also be deployed for insight into Operating System level metrics. The remote management capabilities are designed to help increase availability and utilization and minimize downtime. Although Oracle Enterprise Manager is optional in the Exalogic machine environment, Oracle recommends the use of EMGC to monitor deployed software assets and

12

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

EMOC to monitor Exalogic hardware components. Oracle Enterprise Manager 11g Ops Center supports the following key features in the Exalogic machine environment: Hardware lifecycle management InfiniBand fabric views and Ethernet network view Console access to launch the browser user interface for managing the Sun ZFS Storage 7320 appliance and the InfiniBand switches Serial console access to the service processors of compute nodes, switches, and storage appliance Integration with Oracle Services such as My Oracle Support Problem management for the Sun ZFS 7320 Storage appliance

In addition, Oracle Enterprise Manager Ops Center supports bare metal provisioning, discovery of hardware assets, patch automation, import of firmware images, creation policies, and firmware upgrade for the hardware components of an Exalogic machine.
ILOM (Integrated Lights Out Manager):

Oracle Integrated Lights Out Manager (ILOM) provides advanced service processor (SP) hardware and software that can be uses to manage and monitor Exalogic components, such as compute nodes, gateway switches, storage appliance, InfiniBand switches and power distribution units. ILOM's dedicated hardware and software is preinstalled on all Exalogic components. ILOM enables the ability to actively manage and monitor compute nodes in the Exalogic machine independently of the operating system state, providing a reliable Lights Out Management (LOM) system. The ILOM service processor (SP) runs its own embedded operating system and has a dedicated Ethernet port, which together provides out-of-band management capability. In addition, ILOM can be accessed from the compute node's operating system. Using ILOM, compute node can be remotely managed, effectively providing a virtual KVM switch. The ILOM management interface is integrated with Oracle Enterprise Manager Ops Center

Summary
The need for enterprise IT organizations to provide next-generation cloud features such as elastic capacity while meeting ever more demanding performance and reliability requirements is driving demand for a new approach to infrastructure. Whether workloads are Web-based or thick-client, whether data-intensive or processing-intensive, whether homogeneous or highly heterogeneous, the key to success is hardware and software engineered together for performance, reliability, and scale. Building or using custom, special purpose systems for different applications is wasteful and expensive. Oracle Exalogic Elastic Cloud, the worlds first and only integrated middleware

13

Oracle White PaperOracle Exalogic Elastic Cloud X-2 Hardware Overview

machine, dramatically surpasses alternatives and provides enterprises the best possible foundation for running applications. By consolidating applications to Oracle Exalogic Elastic Cloud, enterprises will:

Accelerate the performance of Java applications by as much as 10X Improve reliability and scalability beyond even the most mission-critical requirements Reduce deployment effort by up to 95% and reduce costs by as much as 60%

We invite you to begin your datacenter transformation with Exalogic today.

14

White Paper Title February 2011 Author: Erik Bergenholtz Contributing Authors: Michael Palmeter Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com 0109 Copyright 2009, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.