Académique Documents
Professionnel Documents
Culture Documents
Cloud Computing
2
Optimizing Hadoop Deployments
74
32W savings Intel Server Board S5520UR
106
Load (%)
up to 30% wall
power savings!
248
100%
42W savings
290
3
Optimizing Hadoop Deployments
0.9 1x 1x 1x
0.8 .87x
0.7
.74x Intel® Xeon® processor 5400 series
0.6
Job .64x Intel Xeon processor 5500 series
0.5
Running
Time 0.4
0.3
0.2 4 System Software Selection
0.1 and Configuration
0 4.1 Selecting the Operating System
Sort WordCount TeraSort
and JVM
Hadoop Benchmarks Using a Linux* distribution based on kernel
version 2.6.30 or later is recommended
Figure 2. The Intel® Xeon® processor 5500 series improves on the performance of its when deploying Hadoop on current-
predecessors across a range of Hadoop* benchmarks.3 generation servers because of the
optimizations included for energy and
threading efficiency. For example, Intel has
observed that energy consumption can
3.6 Choosing Processors Intel Xeon processor 5500 series provides be up to 60 percent (42 watts) higher at
up to 86 percent more throughput than idle for each server using older versions of
The processor plays an important role
the previous generation processors, as Linux.6 Such power inefficiency, multiplied
in determining the speed, throughput,
shown in Figure 3. This characteristic over a large Hadoop cluster, could amount
and efficiency of Hadoop clusters.
allows Hadoop clusters to handle far larger to significant additional energy costs.
The Intel Xeon processor 5500 series
provides excellent performance for highly datasets and more operations in a given For better performance, the local
distributed workloads such as those amount of time.4 file systems (for example, ext3 or
associated with Hadoop applications.2 This testing helps to support Intel’s xfs) are usually mounted with the
recommendation of the Intel Xeon noatime attribute. In addition, Sun
Lab testing was performed to establish
processor 5500 series as the server Java* 6 is required to run Hadoop, and
the performance benefits of the Intel
engine-of-choice for Hadoop clusters. the latest version (Java 6u14 or later)
Xeon processor 5500 series relative to
is recommended to take advantage
previous-generation Intel processors It is important to note that each Intel of optimizations such as compressed
(Figure 2). The results show the Intel Xeon Xeon processor 5500 series typically ordinary object pointers.
processor 5500 series can provide up to has four cores, each of which can handle
36 percent faster performance for some two threads when Intel® Hyper-Threading The default Linux open file descriptor limit
key Hadoop workloads.3 Technology (Intel® HT Technology) is is set to 1024, which is usually too low
enabled. Some Hadoop workloads, such for Hadoop daemons. This setting should
The latest Intel Xeon processor is not only
as JavaSort*, can improve in performance be increased to approximately 64,000
faster at Hadoop tasks; it can also handle
by as much as 25 percent when running using the /etc/security/limits.conf file
more throughput, defined here as the
with Intel HT Technology enabled in the or alternate means. If the Linux kernel
number of tasks completed per minute
processor, versus having the capability 2.6.28 is used, the default open epoll
when the Hadoop cluster is at 100 percent
turned off.5 It is recommended that file descriptor limit is 128, which is too
utilization processing multiple Hadoop jobs.
customers running Hadoop clusters turn low for Hadoop and should be increased
Intel lab tests have demonstrated that the
on Intel HT Technology. to approximately 4096 using the
/etc/sysctl.conf file or alternate means.
4
Optimizing Hadoop Deployments
4.2 Choosing Hadoop Versions The primary source for securing the 5.1 General Configurations
and Distributions latest distribution is the Apache Software • The numbers of NameNode and
When selecting a version of Hadoop Foundation Web site (www.apache.org). JobTracker server threads that handle
for the implementation, organizations For companies planning Hadoop remote procedure calls (RPCs), specified
must seek a balance between the installations, it may be worthwhile to by dfs.namenode.handler.count and
enhancements available from the most evaluate the Cloudera distribution, which mapred.job.tracker.handler.count,
recent available release and the stability includes RPM and Debian* packaging and respectively, both default to 10 and
available from more mature versions. tools for configuration. Intel has used should be set to a larger number (for
For example, at the time of this writing, Cloudera’s distribution on some of its lab example, 64) for large clusters.
the most recent stable version of Hadoop systems for performance testing.
• The number of DataNode server threads
is 0.18.3, while the latest release of
5 Hadoop Configurations that handle RPCs, as specified by
Hadoop, version 0.20.0, contains important
and Tuning dfs.datanode.handler.count, defaults
enhancements, including pluggable
to three and should be set to a larger
scheduling API, capacity scheduler, fair To achieve maximum results from
number (for example, eight) if there are a
scheduler, and multiple task assignment. Hadoop implementations, Intel lab testing
large number of HDFS clients. (Note: Every
has identified some key considerations
One other potential advantage of additional thread consumes more memory.)
for configuring the Hadoop environment
using Hadoop 0.20.0 is in the area of
itself, which are described in this section. • The number of work threads on
performance. Intel’s lab testing shows
Similar to the system hardware and the HTTP server that runs on each
that some workloads within Hadoop can
software recommendations given above, TaskTracker to handle the output of map
benefit from the multi-task assignment
these settings must be tailored to the tasks on that server, as specified by
features in 0.20.0. Although the Map
needs of the individual implementation, tasktracker.http.threads, should be set in
stage in v0.20.0 is slower and uses more
and users are encouraged to experiment the range of 40 to 50 for large clusters.
memory than v0.19.1, the overall job
with their own systems and environment.
runs at about the same speed or up to 8 5.2 HDFS-Specific Configurations
Nevertheless, there are factors in
percent faster in v0.20.0 in the case of
common among Hadoop deployments • The replication factor for each block
Hadoop Sort.7
that provide general guidance. of an HDFS file, as specified by
dfs.replication, is typically set to three
for fault-tolerance; setting it to a smaller
Hadoop Benchmark Comparison value is not recommended.
Between Two Generations of Intel® Xeon® Processors
Throughput Test (Higher Values are Better) • The default HDFS block size, as specified
by dfs.block.size, is 64 MB in HDFS, and it is
usually desirable to use a larger block size
2
(such as 128 MB) for large file systems.
1.5
0.5
0
Hadoop
Hadoop Sort
WordCount
Figure 3. The Intel® Xeon® processor 5500 series provides higher throughput than
its predecessors on selected Hadoop* benchmarks.4
5
Optimizing Hadoop Deployments
5.4 Map Task-Specific Configurations Once the preliminary system configurations are complete, the tuning advice given in this
• The total size of result and metadata paper enables implementing organizations to improve their Hadoop environments further.
buffers associated with a map task,
as specified by io.sort.mb, defaults to
100 MB and can be set to a higher level, Join the conversation in The Server Room:
such as 200 MB. communities.intel.com/community/server
• The percentage of total buffer size that
is dedicated to the metadata buffer, as
specified by io.sort.record.percent, which
defaults to 0.05, should be adjusted
according to the key-value pair size of the
particular Hadoop job.
6
Optimizing Hadoop Deployments
7
Optimizing Hadoop* Deployments
1 PSU.
2 For more information on the Intel® Xeon® processor 5500 series, see http://download.intel.com/products/processor/xeon/dc55kprodbrief.pdf.
3 Source: Intel internal measurement on September 1, 2009 running Hadoop* Sort, Word Count, and TeraSort benchmarks. Raw score of Intel ® Xeon® processor
54xx-based cluster versus cluster based on Intel Xeon processor 55xx series. Cluster architecture includes 5 nodes per server processor architecture under
comparison (Intel Xeon processor 54xx series and Intel Xeon processor 55xx series). Each has 1 NameNode/JobTracker and 4 DataNodes/TaskTrackers. 1xGbE
connectivity to a single GbE switch. Intel Xeon processor 54xx series servers: 2 x Intel Xeon processor X5460 3.16 GHz (8 cores per node), 16 GB DDR2 RAM,
1 SAS (for system and log files) and 4 SATA disks (for Hadoop Distributed File System (HDFS) and intermediate results) per node.
Intel Xeon processor 55xx series servers: 2 x Xeon processor X5570 2.93 GHz (8 cores per node), 16 GB DDR3 RAM, 5 SATA disks per node (1 for system and log
files, and the other 4 for HDFS and intermediate results). Both EIST (Enhanced Intel® SpeedStep Technology) and Turbo mode disabled. Both hardware prefetcher
and adjacent cache-line prefetch enabled. SMT (Simultaneous Multi-Threading) enabled.
Software: Red Hat Enterprise Linux* 5.2 (kernel 2.6.18-92.el5 SMP x86_64). Ext3 filesystem (mounted with noatime enabled). Sun JVM 1.6 (Java* version 1.6.0_02
Java SE Runtime Environment Java HotSpot* 32-bit server virtual machine). Hadoop version 0.19.1 with patch JIRA Hadoop-5191.
4 Source: Intel internal measurement on September 1, 2009 running Hadoop* Sort and Word Count benchmarks. Raw score of Intel ® Xeon® processor 54xx-based
cluster versus cluster based on Intel Xeon processor 55xx series. Cluster architecture includes 5 nodes per server processor architecture under comparison (Intel
Xeon processor 54xx series and Intel Xeon processor 55xx series). Each has 1 NameNode/JobTracker, 4 DataNodes/TaskTrackers.
1 x GbE connectivity to a single GbE switch. Intel Xeon processor 54xx series servers: 2 x Intel Xeon processor X5460 3.16 GHz (8 cores per node), 16 GB DDR2
RAM, 1 SAS (for system and log files) and 4 SATA disks (for Hadoop Distributed File System (HDFS) and intermediate results) per node, and Red Hat Enterprise
Linux* 5.2 (kernel 2.6.18-92.el5 SMP x86_64).
Intel Xeon processor 55xx series servers: 2 x Intel Xeon processor X5570 2.93 GHz (8 cores per node), 16 GB DDR3 RAM, 5 SATA disks per node (1 for system
and log files, and the other 4 for HDFS and intermediate results). Both EIST (Enhanced Intel ® SpeedStep Technology) and Turbo mode disabled. Both hardware
prefetcher and adjacent cache-line prefetch enabled. SMT (Simultaneous Multi-Threading) enabled. Red Hat Enterprise Linux 5.2 (kernel 2.6.30 x86_64).
Software: Ext3 filesystem (mounted with noatime enabled). Sun JVM 1.6 (Java version 1.6.0_02 Java SE Runtime Environment Java HotSpot* 32-Bit Server VM).
Hadoop version 0.20.0.
5 Source: Intel internal measurement based on the following cluster and server configuration: 7 nodes in each, configured with 2GbE connectivity to each server. Intel®
Xeon® processor 54xx series server configuration:Intel® whitebox server based on the Intel® Server System SR1560SF, 2 x Intel Xeon processor X5482 3.2 GHz, 16
GB DDR2, 1 x Seagate Barracuda* ES 250 GB SATA drives. Intel Xeon processor 55xx series server configuration: 2 x Intel Xeon processor X5560 2.8 GHz, 18 GB
DDR3, 1 x 500 GB. Intel® Hyper-Threading Technology (Intel® HT Technology) requires a computer system with an Intel® processor supporting Intel HT Technology
and an Intel HT Technology-enabled chipset, BIOS, and operating system. Performance will vary depending on the specific hardware and software you use.
See www.intel.com/products/ht/hyperthreading_more.htm for more information including details on which processors support Intel HT Technology.
6 Source: Intel internal measurement as of September 14, 2009 based on running the same server with two different Linux* distributions: CentOS* 5.2 and Fedora* 11.
Power (W) at idle was measured at 110W when running CentOS 5.2 and 68W when running Fedora 11. Server configuration was an Intel® whitebox server based on
the Intel® Server Board SB5000WB, 2 x Intel® Xeon® processor L5520, 16 GB RAM, 1 HDD.
7
Sources: Intel internal measurement on September 14, 2009 and September 1, 2009 running Hadoop* Sort benchmark. Raw score of cluster based on
Intel ® Xeon ® processor 55xx series. Cluster architecture includes 5 nodes of Intel Xeon processor 55xx series. Each has 1 NameNode/JobTracker,
4 DataNodes/TaskTrackers. 1 x GbE connectivity to a single GbE switch. Intel Xeon processor 55xx series servers: 2 x Intel Xeon processor X5570 2.93
GHz (8 cores per node), 16 GB DDR3 RAM, 5 SATA disks per node (1 for system and log files, and the other 4 for HDFS and intermediate results). Both EIST
(Enhanced Intel® SpeedStep Technology) and Turbo mode disabled. Both hardware prefetcher and adjacent cache-line prefetch enabled. SMT (Simultaneous
Multi-Threading) enabled.
Software: Red Hat Enterprise Linux* 5.2 (kernel 2.6.30 x86_64 x86_64). Ext3 filesystem (mounted with noatime enabled). Sun JVM 1.6 (Java* version 1.6.0_02 Java
SE Runtime Environment Java HotSpot* 32-bit server virtual machine). Hadoop version 0.19.1 with patch JIRA Hadoop-5191vs. Hadoop 0.20.0.