Optimizing data centers for high-density computing

technology brief, 2nd edition

Abstract.............................................................................................................................................. 2
Introduction......................................................................................................................................... 2
Power consumption and heat load ......................................................................................................... 2
Power consumption .......................................................................................................................... 2
Heat load........................................................................................................................................ 5
The Need for Planning ......................................................................................................................... 7
Optimizing the effectiveness of cooling resources .................................................................................... 8
Raised floors.................................................................................................................................... 8
Perforated tiles ............................................................................................................................. 8
Air supply plenum ........................................................................................................................ 9
Racks.............................................................................................................................................. 9
Cooling footprint ........................................................................................................................ 10
Internal airflow........................................................................................................................... 10
Hot and cold aisles..................................................................................................................... 11
Rack geometry ........................................................................................................................... 11
Computer room air conditioners....................................................................................................... 12
Capacity of CRAC units .............................................................................................................. 12
Placement of CRAC units ............................................................................................................. 12
Discharge velocity ...................................................................................................................... 13
Airflow distribution for high-density data centers.................................................................................... 14
Ceiling return air plenum ................................................................................................................ 15
Dual supply air plenums.................................................................................................................. 15
Ceiling-mounted heat exchangers .................................................................................................... 16
Advanced thermal management techniques .......................................................................................... 16
Static Smart Cooling....................................................................................................................... 17
Dynamic Smart Cooling .................................................................................................................. 18
Conclusion........................................................................................................................................ 19
For more information.......................................................................................................................... 20
Abstract
This paper describes factors causing the increase in power consumption and heat generation of
computing hardware. It identifies methods to optimize the effectiveness of cooling resources in data
centers that are deploying high-density equipment or that are already fully populated with high-density
equipment. The intended audience for this paper includes IT managers, IT administrators, facility
planners, and operations staff.

Introduction
From generation to generation, the power consumption and heat loads of computing, storage, and
networking hardware in the data center have drastically increased. The ability of data centers to meet
increasing power and cooling demands is constrained by their designs. Most data centers were
designed using average (per unit area) or "rule of thumb" criteria, which assume that power and
cooling requirements are uniform across the facility. In actuality, power and heat load within data
centers are asymmetric due to the heterogeneous mix of hardware and the varying workload on
computing, storage, and networking hardware. These factors can create "hot spots" that cause
problems related to overheating (equipment failures, reduced performance, and shortened equipment
life) and drastically increase operating costs.
The dynamic nature of data center infrastructures creates air distribution problems that cannot always
be solved by installing more cooling capacity or by using localized cooling technologies. A more
sophisticated scientific method can help to find the most effective solutions. Research at HP
Laboratories has found that proper data center layout and improved Computer Room Air Conditioner
(CRAC) 1 utilization can prevent hot spots and yield substantial energy savings.
This paper is intended to raise awareness of present and future challenges facing data centers
beginning to deploy or already fully populated with high-density equipment. This paper describes
power consumption and heat load, recommends methods to optimize the effectiveness of data center
cooling resources, and introduces thermal management methods for high-density data centers.

Power consumption and heat load

In the past when data centers mainly housed large mainframe computers, power and cooling design
criteria were designated in average wattage per unit area (W/ft2 or W/m2) and British Thermal Units
per hour (BTU/hr), respectively. These design criteria were based on the assumption that power and
cooling requirements were uniform across the entire data center. Today, IT managers are populating
data centers with a heterogeneous mix of high-density hardware as they try to extend the life of their
existing space. This high-density hardware requires enormous amounts of electricity and produces
previously unimaginable amounts of heat.
For example, IT infrastructures are now using 1U dual-processor and 4U quad-processor ProLiant
blade servers that can be installed together in a rack-mounted enclosure, interconnected, and easily
managed. This high-density server technology lowers the operating cost per CPU by reducing
management expenses and the requirements for floor space. Despite speculation that high-density
server technology has the opposite effect on power consumption and heat load, a closer server-to-
server comparison reveals that HP p-Class blades consume less power and generate less heat load.

Power consumption
HP provides online power calculators to estimate power consumption for each ProLiant server. The
power calculators provide information based on actual system measurements, which are more

1
CRAC units are sometimes referred to as air handlers.

2
accurate than using nameplate ratings. Figure 1 displays an HP power calculator, which is a macro-
driven Microsoft Excel spreadsheet. Power calculators for all current HP ProLiant servers can be found
at http://www.hp.com/configurator/calc/Power Calculator Catalog.xls.

Figure 1. Screen shot of ProLiant DL380 G4 server power calculator

3
From generation to generation, the power consumption of high-density servers is increasing due to the
extra power needed for faster and higher capacity internal components. For example, the power
required by a ProLiant DL360 G3 featuring a 3.0-GHz Intel® Xeon™ processor is 58 percent higher
than its predecessor with a Pentium III 1.4 GHz processor (see Table 1).

Table 1. Increase in power consumption from generation to generation of ProLiant DL and BL servers

Rack Unit Power Supply Rating per Server Generation Power Increase
(CPUs/Memory/
Drives/Adapters)

1U DL360 G2 DL360 G3 DL360 G4

(2P, 4 GB, 246 W 389 W 460 W
2 HDD, 1 PCI)
1.2 A @ 208V 1.6 A @ 208V 1.9 A @208V
840 BTU/hr 1328 BTU/hr 1570 BTU/hr

2U DL380 G2 DL380 G3 DL380 G4

(2P, 4 GB, 362 W 452 W 605 W
6 HDD, 2 PCI)
1.8 A @ 208V 2.2 A @ 208V 3.0 A @ 208V
1233 BTU/hr 1540 BTU/hr 2065 BTU/hr

4U DL580 G1 DL580 G2 DL580 G3

(4P, 8 GB, 456 W 754 W 910 W
4 HDD, 3 PCI)
2.3 A @ 208V 2.7 A @ 208V 4.0 A @ 208V
1556 BTU/hr 2573 BTU/hr 3106 BTU/hr

The table above compares the power consumption of individual servers; however, standard racks can
house several of these servers. Estimating the power consumption of a rack of servers is more difficult
because several variables (number of servers per rack, type and number of components in each
server, etc.) contribute to the amount of power consumed. For racks, a very useful metric is power
density, or power consumption per rack unit (W/U). Power density captures all of the key variables
that contribute to rack densification. Figure 2 illustrates the overall power density trend from
generation to generation of HP ProLiant BL and DL servers.

Figure 2. Power density of HP ProLiant BL and DL servers (Watts/U)

4
Heat load
Virtually all power consumed by a computer is converted to heat. The heat generated by the computer
is typically expressed in BTU/hr, where 1 W equals 3.413 BTU/hr. Therefore, once the power
consumption of a computer or a rack of computers is known, its heat load can be calculated as
follows:
Heat Load = Power [W] × 3.413 BTU/hr per watt
For example, the heat load for a DL360 G4 server is 460 W × 3.413 BTU/hr/W =1,570 mBTU/hr.
The heat load of a 42U rack of DL 360 G4 servers is almost 65,939 BTU/hr, which is more than that
of a typical one-story house. Table 2 lists the power requirements and heat loads of racks of density-
optimized ProLiant DL and BL class servers. The table shows the trend toward higher power and heat
load with rack densification.

Table 2. Power and heat loads of fully-configured, density-optimized ProLiant servers*

DL 580 G3 DL380 G4 DL360 G4 BL20p G2

ProLiant
Server

Servers per 10 21 42 48
Rack

Power 10 × 900W = 21 × 575W = 42 × 460W = 6 × 2458W =

9.0 kW 12.08 kW 19.32 kW 14.75 kW

Heat load 30,717 BTU/hr 41,212 BTU/hr 65,939 BTU/hr 50,374 BTU/hr
* These calculations are based on the product nameplate values for fully configured racks and therefore may be higher than
the actual power consumption and heat load.

IT equipment manufacturers typically provide power and heat load information in their product
specifications. HP provides a Rack/Site Installation Preparation Utility to assist customers in
approximating the power and heat load per rack for facilities planning (Figure 3). The Site Installation
Preparation Utility uses the power calculators for individual platforms so that customers can calculate
the full environmental impact of racks with varying configurations and loads. This utility can be
downloaded from
http://h30099.www3.hp.com/configurator/calc/Site%20Preparation%20Utility.xls.

5
Figure 3. The ProLiant Class, Rack/Site Installation Preparation Utility available on the HP website

6
The Need for Planning
The server densification trend is being driven by customers’ need to maximize the use of valuable
data center floor space. Because concentrated heat generation is an inevitable byproduct of
concentrated computing power, data centers must ensure adequate localized cooling capacity to
match non-uniformly distributed heat loads. Most data centers have sufficient cooling resources
already in place; their main challenge is directing cooling to racks that generate high heat loads.
For data centers that cannot afford to upgrade cooling capacity to handle concentrated heat loads,
HP’s small form factor servers offer the flexibility to limit rack power density (kW/rack) based on the
capacity of nearby cooling resources. For example, Figure 4 shows several rack configurations using
six generations of ProLiant DL360 servers, each limited to 10 kW by controlling the number of servers
per rack.
Some data center personnel believe that limiting rack power density neutralizes the space-saving
benefits of densification because the racks are not full. However, from generation to generation the
compute power per server is increasing faster than the corresponding increase in power consumption,
resulting in higher efficiency. The ability to limit rack power density while increasing computing power
can prolong the lifecycle of space-constrained infrastructures. Data centers that have sufficient
capacity or that can afford to add cooling capacity can use HP small form factor servers to maximize
rack computing power so that different facilities can be consolidated to reduce overall operating
costs.

Figure 4. Limiting rack power density based on power consumption results in lower rack utilization.

The goal of all data centers is to optimize the effectiveness of existing cooling resources. The
following section describes proven methods to achieve better airflow distribution. Given that power
consumption and cooling demands will continue to increase, future data center designs will have to
take a more holistic approach that examines cooling from the chip level to the facility level. The
section titled “Advanced thermal management techniques” outlines breakthrough research by HP
Laboratories to create intelligent data centers that dynamically redistribute compute workloads and
provision cooling resources for optimum operating efficiency.

7
Optimizing the effectiveness of cooling resources
This section recommends methods to optimize the effectiveness of cooling resources in raised floor
infrastructures, a common configuration used in today's data centers.

Raised floors
Most data centers use a down draft airflow pattern in which air currents are cooled and heated in a
continuous convection cycle. The down draft airflow pattern requires a raised floor configuration that
forms an air supply plenum beneath the raised floor (Figure 5). The CRAC unit draws in warm air
from the top, cools the air, and discharges it into the supply plenum beneath the floor. Raised floors
typically measure 18 inches (46 cm) to 36 inches (91 cm) from the building floor to the top of the
floor tiles, which are supported by a grounded grid structure. The static pressure in the supply plenum
pushes the air up through perforated floor tiles to cool the racks. Most equipment draws in the cold
supply air and exhausts warm air out the rear of the racks. Ideally, the warm exhaust air rises to the
ceiling and returns along the ceiling back to the top of the CRAC units to repeat the cycle. Many
traditional data centers arrange rows of racks in the front-to-back layout shown in Figure 5. The
mixing of cold and hot air in the aisles is very inefficient and wastes valuable cooling resources and
energy. While this layout can work with lower power densities and heat loads, as the power density
and heat load increase, the equipment inlet temperatures will begin to rise (shown in the figure) and
overheat critical resources.

Figure 5. Traditional raised floor configuration with high-density racks arranged front to back

Perforated tiles
Floor tiles range from 18 inches (46 cm) to 24 inches (61 cm) square. The percentage and placement
of perforated floor tiles are major factors in maintaining static pressure. Perforated tiles should be
placed in front of at least every other rack. In higher density environments, perforated tiles may be
necessary in the front of each rack. Perforated tiles are classified by their open area, which may vary
from 25 percent (the most common) to 56 percent (for high airflow). A 25 percent perforated tile
provides approximately 500 cubic feet per minute (cfm) at a 5 percent static pressure drop, while a
56 percent perforated tile provides approximately 2000 cfm.

8
Air supply plenum
The air supply plenum must be a totally enclosed space to achieve pressurization for efficient air
distribution. The integrity of the subfloor perimeter (walls) is critical to prevent moisture retention and
to maintain supply plenum pressure. This means that openings in the plenum perimeter and raised
floor must be filled or sealed. Subfloor plenum dividers should be constructed in areas with large
openings or with no subfloor perimeter walls.
The plenum is also used to route piping, conduit, and cables that bring power and network
connections to the racks. In some data centers, cables are simply laid on the floor in the plenum
where they can become badly tangled (Figure 6). This can result in cable dams that block airflow or
cause turbulence that minimizes airflow and creates hot spots above the floor. U-shaped “basket”
cable trays or cable hangers can be used to manage cable paths, prevent blockage of airflow, and
provide a path for future cable additions. Another option is to use overhead cable trays to route
network and data cables so that only power cables remain in the floor plenum.
Electrical and network cables from devices in the racks pass through cutouts in the tile floor to
wireways and cable trays beneath the floor. Oversized or unsealed cable cutouts allow supply air to
escape from the plenum, thereby reducing the static pressure. Self-sealing cable cutouts are required
to maintain the static pressure in the plenum (Figure 7).

Figure 6. Unorganized cables (left) and organized cables (right) beneath a raised floor.

Figure 7. Self-sealing cable cutout in raised floor

Racks
Racks (cabinets) are a critical part of the overall cooling infrastructure. HP enterprise-class cabinets
provide 65 percent open ventilation using perforated front and rear door assemblies (Figure 8). To
support the newer high-performance equipment, glass doors must be removed from older HP racks
and from any third-party racks.

9
Figure 8. HP enterprise-class cabinets

Cooling footprint
The floor area that each rack requires must include an unobstructed area to draw in and discharge
air. Almost all HP equipment cools from front to rear so that it can be placed in racks positioned side-
by-side. The cooling footprint (Figure 9) includes width and depth of the rack plus the area in front for
drawing in cool air and the area in back for exhausting hot air.
Equipment that draws in air from the bottom or side or that exhausts air from the side or top will have
a different cooling footprint. The total physical space required for the data center includes the cooling
footprint of all the racks plus free space for aisles, ramps, and air distribution. Typically, a width of
two floor tiles is needed in front of the rack, and a width of at least one unobstructed floor tile is
needed behind the rack to facilitate cable routing.

Figure 9. Cooling footprint

Internal airflow
Front and rear cabinet doors that are 65 percent open to incoming airflow also present a 35 percent
restriction to air discharged by the equipment in the rack. Servers will intake air from the path of least
resistance. Therefore, they will access the higher-pressure discharge air flowing inside the cabinet
easier than they will access cooling air coming through the front of the cabinet. Some configurations
such as those with extreme cable or server density may create a backpressure situation forcing heated
exhaust air around the side of a server and back into its inlet. In addition, air from the cold isle or hot

10
isle can flow straight through a rack with open "U" spaces. Gaskets or blanking panels must be
installed in any open spaces in the front of the rack to support the front-to-back airflow design and
prevent these negative effects (Figure 10).

Figure 10. Airflow in rack without blanking panels (top) and with blanking panels (bottom)

Hot and cold aisles

The front-to-rear airflow through HP equipment allows racks to be arranged in rows front-to-front and
back-to-back to form alternating hot and cold aisles. The equipment draws in the cold supply air and
exhausts warm air out the rear of the rack into hot aisles (Figure 11). The amount of space between
rows of racks is determined as follows.
• Cold aisle spacing should be 48 inches, two full tiles, and hot isle spacing should be at least one
full tile, 24 inches minimum. This spacing is required for equipment installation and removal and for
access beneath the floor.
• Cold aisles should be a minimum of 14 feet apart center-to-center, or seven full tiles.

Figure 11. Airflow pattern for raised floor configuration with hot aisles and cold aisles

Rack geometry
Designing the data center layout to form hot and cold aisles is one step in the cooling optimization
process. Also critical is the geometry of the rack layout. Research by HP Laboratories has revealed
that minor changes in rack placement can change the fluid mechanics inside a data center and lead
to inefficient utilization of CRAC units. See the "Static Smart Cooling" section for more information.

11
Computer room air conditioners
A common question with respect to cooling resources is how many kilowatts a particular CRAC unit
can cool. Assuming a fixed heat load from the equipment in its airflow pattern, the answer depends
largely on the capacity of the CRAC unit, its placement in the facility, and its discharge velocity.
Capacity of CRAC units
The heat load of equipment is normally specified in BTU/hr. However, in the U.S., CRAC unit
capacity is often expressed in "tons" of refrigeration, where one ton corresponds to a heat absorption
rate of 12,000 BTU/hr. The "tons" capacity rating is measured at 80˚F; however, the recommended
operating conditions for CRAC units are 70˚ to 72˚F and 50 percent relative humidity (RH). At 72˚F,
the CRAC unit output capacity is considerably reduced. Furthermore, the tons rating is very subjective
because it is based on total cooling, which is comprised of "sensible cooling" and "latent cooling."
Computer equipment produces sensible heat only; therefore, the sensible cooling capacity of a CRAC
unit is the most useful value. For this reason, CRAC unit manufacturers typically provide cooling
capacities as "total BTU/Hr" and "sensible BTU/Hr" at various temperatures and RH values.
Customers should review the manufacturer's specifications and then divide the sensible cooling
capacity (at the desired operating temperature and humidity) by 12,000 BTU/Hr per ton to calculate
the useable capacity of a given CRAC unit, expressed in tons of cooling.
Cooling capacity is also expressed in volume as cubic feet per minute (cfm). The volume of air
required is related to the moisture content of the air and the temperature difference between the
supply air and return air (ΔT):
Cubic feet per minute = BTU/hr ÷ (1.08 × ΔT)
The cooling capacity calculations presented here are theoretical, so other factors must be considered
to determine the effective range of a particular CRAC unit. The effective cooling range is determined
by the capacity of the CRAC unit and the heat load of the equipment in its airflow pattern. Typically,
the most effective cooling begins about 8 feet (2.4 m) from the CRAC unit. Although units with
capacities greater than 20 tons are available, the increased heat density of today's servers limits the
cooling range to approximately 30 feet or 9.1 m (Figure 12).

Figure 12. Cooling ranges of CRAC units

Placement of CRAC units

The geometry of the room and the heat load distribution of the equipment determine the best
placement of the CRAC units. CRAC units can be placed inside or outside the data center walls.
Customers should consider placing liquid-cooled units outside the data center to avoid damage to
electrical equipment that could be caused by coolant leaks.
CRAC units should be placed perpendicular to the rows of equipment and aligned with the hot aisles,
discharging air into the supply plenum in same direction (Figure 13). This configuration provides the

12
shortest possible distance for the hot air to return to the CRAC units. Discharging in the same direction
eliminates dead zones that can occur beneath the floor when blowers oppose each other. Rooms that
are long and narrow may be cooled effectively by placing CRAC units around the perimeter. Large,
square rooms may require CRAC units to be placed around the perimeter and through the center of
the room.

Figure 13. CRAC units should be placed perpendicular to hot aisles so that they discharge cool air beneath the floor in the
same direction.

Discharge velocity
To force air from beneath the raised floor through the perforated tiles, the static pressure in the supply
air plenum must be greater than the pressure above the raised floor. The velocity of the cooled air is
highest near the CRAC unit because the entire flow is delivered through this area. The air velocity
decreases as air flows through the perforated tiles away from the CRAC unit. The decrease in velocity
is accompanied by an increase in static pressure with distance from the CRAC unit.
Excessive discharge velocity from the CRAC unit reduces the static pressure through perforated tiles
nearest the unit, causing inadequate airflow (Figure 14). The static pressure increases as the high-
velocity discharge moves away from the unit, thereby increasing the airflow through the perforated
tiles. To counter this situation, airfoils under the raised floor can be used to divert air through the
perforated tiles. 2 Another option is to use a fan-assisted perforated tile to increase the supply air
circulation to a particular rack or hot spot. Fan-assisted tiles can provide 200 to 1500 cfm of supply
air.

2
From Changing Cooling Requirements Leave Many Data Centers at Risk. W. Pitt Turner IV, P.E. and Edward C. Koplin, P.E. ComputerSite
Engineering, Inc.

13
Figure 14. Plenum static pressure greater than pressure above the floor (left). High-velocity discharge reduces static pressure
closest to the unit (right).

Airflow distribution for high-density data centers

To achieve an optimum down draft airflow pattern, warm exhaust air must be returned to the CRAC
unit with minimal obstruction or redirection. Ideally, the warm exhaust air will rise to the ceiling and
return to the CRAC unit intake. In reality, only the warm air closest to the intake may be captured; the
rest may mix with the supply air. Mixing occurs if exhaust air goes into the cold aisles, if cold air goes
into the hot aisles, or if there is insufficient ceiling height to allow for separation of the cold and warm
air zones (Figure 15). When warm exhaust air mixes with supply air, two things can happen:·
• The temperature of the exhaust air decreases, thereby lowering the useable capacity of the CRAC
unit.
• The temperature of the supply increases, which causes warmer air to be recirculated through
computer equipment.

Figure 15. Mixing of supply air and exhaust air

14
Ceiling return air plenum
In recent years, raised floor computer rooms with very high heat density loads have begun to use a
ceiling return air plenum to direct exhaust air back to the CRAC intake. As shown on the right of
Figure 16, the ceiling return air plenum removes heat while abating the mixing of cold air and
exhaust air. Once the heated air is in the return air plenum, it can travel to the nearest CRAC unit
intake. The return air grilles in the ceiling can be relocated if the layout of computer equipment
changes.

Figure 16. Ceiling return air plenum

Dual supply air plenums

As power and heat densities climb, a single supply air plenum under the raised floor may be
insufficient to remove the heat that will be generated. High-density solutions may require dual supply
air plenums, one above and one below (see Figure 17). In this configuration, additional supply air is
forced downward in the cold aisle.

Figure 17. Dual air supply plenum configuration for high-density solutions

15
Ceiling-mounted heat exchangers
The drive to maximize the compute density of data centers has prompted research and development
of alternative cooling methods that do not require floor space. 3 Figure 18 shows a representation of
an alternate cooling method using modular air-to-liquid heat exchangers in the ceiling. The heat
exchanger units collect the hot air exhaust from the racks and cool it using circulated chilled water.
The units eject the cool air downward by using fan trays located over on the intake side of each rack.
The advantage of this approach is the proximity of the heat exchangers to the racks. With this
scheme, rack cooling can be localized. Unique mechanical design ideas, such as the ability to move
the intake and exhaust sections, have been implemented in the heat exchangers to direct airflow to
and from the racks. Additionally, modular heat exchangers offer the flexibility to scale the cooling as
needed. More importantly, ceiling-mounted heat exchangers save revenue-generating floor space and
allow the raised floor to be mainly used for cable distribution.

Figure 18. Ceiling-mounted air-to-liquid heat exchangers

Advanced thermal management techniques

Heat loads vary throughout a data center due to the heterogeneous mix of hardware types and
models, changing compute workloads, and the addition or removal of racks over time. The variation
in heat load is too complex to predict intuitively or to address by adding cooling capacity.
HP Laboratories has devised two thermal analysis approaches—Static Smart Cooling 4 and Dynamic
Smart Cooling—that model heat distribution throughout a data center using computational fluid
dynamics (CFD). Static Smart Cooling uses CFD modeling to aid planners in designing the physical
layout of the data center for optimum distribution of cooling resources and heat loads. Static Smart
Cooling can also predict the changes in heat extraction of each CRAC unit when the rack layout and
equipment heat load are varied.
Dynamic Smart Cooling offers a higher level of automated facility management. It enables intelligent
data centers that dynamically provision cooling resources to match the changing heat dissipation of
computing, networking, and storage equipment. It also redistributes compute workloads based on the
most efficient use of cooling resources within a data center or a global network of data centers.

3
For more information, please read “Computational Fluid Dynamics Modeling of High Compute Density Data Centers to Assure System Inlet Air
Specifications” at http://www.hpl.hp.com/research/papers/power.pdf.
4
For more information, please read “Thermal Considerations in Cooling Large Scale High Compute Density Data Centers” at
http://www.hpl.hp.com/research/papers/2002/thermal_may02.pdf.

16
Static Smart Cooling
Static Smart Cooling uses CFD modeling to determine the best layout and provisioning of cooling
resources based on fixed heat loads from data center equipment. The heat extraction of each CRAC
unit is compared to its rated capacity to determine how efficiently (or inefficiently) the CRAC unit is
being used, or "provisioned."
The provisioning of each unit in the data center is presented as a positive or negative percentage as
follows:
• An under-provisioned CRAC unit (positive percentage) indicates that the cooling load is higher than
the capacity of the unit.
• A closely provisioned CRAC unit (small negative percentage) signifies that the cooling load is less
than but reasonably close to the capacity of the unit, leading to efficient use of energy resources.
• An over-provisioned CRAC unit (large negative percentage) operates significantly below the
capacity of the unit. This results in wasted energy if operation of the unit cannot be adjusted to
match the lower cooling load.

For example, Figure 19 shows the row-wise distribution of heat loads (41 kW to 182 kW) for a
combination of compute, storage, and networking equipment in a typical raised floor data center with
four CRAC units. The CFD model shows that the provisioning of the CRAC units is completely out of
balance.

Figure 19. Poorly provisioned CRAC units

17
In Figure 19, the 102-kW row and the 182-kW row have been repositioned to better distribute the
heat load. This CFD model shows that the CRAC units are now provisioned within 15 percent of their
capacity.

Figure 19. Statically provisioned CRAC units

Dynamic Smart Cooling

A “smart” data center requires a distributed monitoring system and a feedback control system that
continually provisions the cooling resources based on the workload distribution. 5 Computing
resources must be pooled and virtualized—rather than dedicated to a particular user or application—
so that workloads can be allocated dynamically. The control system schedules compute workloads
across racks of servers in a way that minimizes energy use and maximizes cooling efficiency. The
computing resources not in use are put on standby to improve operating efficiency.
Due to the high airflow rates in data centers, thermal management is achievable only if hot and cold
air mixing is minimal. The hot air must return to the CRAC units with minimal infiltration into the cold
zones because such mixing increases the inlet temperatures at the racks. Researchers have developed
dimensionless parameters, known as Return Heat Index (RHI) and Supply Heat Index (SHI) that can be
used as control points to allocate compute workloads and cooling to minimize energy use. 6 RHI
denotes the degree of mixing of the cold air with the hot return air to the CRAC units. SHI is a
measure of heat infiltration into cold aisles. SHI is the primary control parameter used to minimize the
energy required to meet inlet air specifications.
Dynamic Smart Cooling is possible for a data center with the following features:
• Distributed sensors such as
- temperature sensors on the racks measuring supply and exhaust air temperature of the systems
- temperature sensors in the aisles measuring three dimensional temperature distribution in the
data center
- temperature sensors at the CRAC return and supply
- pressure sensors in the air distribution plenum

5
Patel, C.D., Sharma, R.K, Bash, C.E., Beitelmal, A, Friedrich, R., “Smart Cooling of Data Centers,” July 2003, IPACK2003-35059, Proceedings
of IPACK03- International Electronics Packaging Technical Conference and Exhibition, Maui, Hawaii.
6
Sharma R, Bash, C.E, Patel, C.D, June 2002, “Dimensionless Parameters for Evaluation of Thermal Design and Performance of Large Scale Data
Centers,” AIAA-2002-3091, American Institute of Aeronautics and Astronautics Conference, St. Louis, MO.

18
 - sensors that measure power drawn by machines
• Variable air flow devices to modulate flow work, and variable cooling coil temperature in the
CRACs to modulate thermodynamic work
• Data aggregation system that
- collects sensed data from all locations
- visually presents the real time power draw
- calculates data center control parameters: RHI and SHI
• Control system that modulates the variable air conditioning resources through a control algorithm
for a given distribution of workloads (heat loads)
• Data center manager (computerized system) that uses thermal policies to distribute workloads in the
data center
At the time of this writing, HP Laboratories is developing the control system and the data center
manager, and plans to report its progress in future papers.

Conclusion
The growing power consumption of computing components requires modular data center designs with
sufficient headroom to handle increasing power and cooling requirements. To determine actual
requirements, facilities planners must consider several factors, including room geometry and the
capacity and placement of the CRAC units. Planners must also give special attention to factors that
affect airflow distribution, such as supply plenum static pressure, airflow blockages beneath raised
floors, and configurations that result in airflow mixing in the data center.
HP is a leader in the thermal modeling of data centers. HP Professional Services can work directly
with customers to optimize existing data centers for more efficient cooling and energy consumption.
The modeling services can also be used to confirm new data center designs or predict what will
happen in a room when certain equipment fails. As long as the data center has the power and
cooling resources to support the expected loads, Static Smart Cooling can rectify cooling problems as
well as enhance the overall efficiency of air conditioning resources. In most cases, the energy savings
alone may pay for the cost of the service in a relatively short period.

19
 For more information
For additional information, refer to the resources detailed below.

Resource description Web address

Thermal Considerations in http://www.hpl.hp.com/research/papers/2002/thermal_may02.pdf

Cooling Large Scale High
Compute Density Data
Centers white paper

HP Rack/Site Installation http://h30099.www3.hp.com/configurator/calc/Site%20Preparation%20Utility.xls

Preparation Utility

Power calculators http://h30099.www3.hp.com/configurator/calc/Power%20Calculator%20Catalog.xls

© Copyright 2004, 2005 Hewlett-Packard Development Company, L.P. The

information contained herein is subject to change without notice. The only
warranties for HP products and services are set forth in the express warranty
statements accompanying such products and services. Nothing herein should
be construed as constituting an additional warranty. HP shall not be liable for
technical or editorial errors or omissions contained herein.
Intel and Xeon are trademarks or registered trademarks of Intel Corporation or
its subsidiaries in the United States and other countries.
TC050901TB, 9/2005