Data Center Cooling Study on Liquid Cooling

DirectNET Confidential © 2007 DirectNET, Inc. All rights reserved.

2007 Data Center Cooling Study:

Comparing

Conventional Raised Floors with Close Coupled Cooling Technology

November 14, 2007

Presented By:


Speakers and Sponsor

• Daniel Kennedy, Solution Center Coordinator for Rittal Corp.– Degreed Electrical Engineer with a

concentration in power systems– Over 4 Years Experience with Design and

Implementation of data center facilities– Project Engineer for Rittal’s closed-coupled

liquid cooling systems in North America

• Jennifer Osborn, Data Center Infrastructure Consultant for DirectNET – 5 Years Experience in data center planning and

management – Professional portfolio includes over 250 clients,

including Fortune 100 organizations– Successfully managed over 25 enterprise

implementations, including deployment, training, and support


Agenda

• Power: The Change Agent of Today’s Data Center • A Review of Your Cooling Options • An In-Depth Assessment of Close Coupled Cooling:

2007 Data Center Cooling Study • 7 Things to Consider before Making a Cooling

Investment• Q&A


Data Center Development Trends

AutomationAutomation

AssetmanagementAssetmanagement

Energy-EfficiencyEnergy-Efficiency

VirtualisationVirtualisation

Physical / logical SecurityPhysical / logical Security

Adaptive InfrastructureAdaptive Infrastructure

Remote ManagementRemote Management

Fuel cell technologyFuel cell technology

Wireless SensorsWireless Sensors

Cooling / PowerCooling / Power

Co

mp

on

e nt s

So

luti

on

s Cost intensiveInfrastructure:

- None adaptive- High energy consumption- People-intensive- Island-character- Local managed- Lights-on- Wired

Cost intensiveInfrastructure:

- None adaptive- High energy consumption- People-intensive- Island-character- Local managed- Lights-on- Wired

Co

mp

on

e nt s

So

luti

on

s

Cost-effectiveInfrastructure:

- Adaptive- Low energy consumption- Automated- Shared IT- Remote managed- Lights-off- Wireless

Cost-effectiveInfrastructure:

- Adaptive- Low energy consumption- Automated- Shared IT- Remote managed- Lights-off- Wireless

Today Future


Inside the Data Center

• Floor Systems• Cable Plant

– Structured Wiring• Cable Routing

– Ladder Rack– Cable Tray

• Climate Control– CRAC, CRAH

• Security– Access Control – Video & Biometrics

• Power – Direct, UPS, EDG‘s

• Outside Service – OC-XXX, Telco

• Enclosures


The Issue

• Optimized air circulation cannot be achieved without significant facility changes• The high costs for energy and climate management increase TCO. • New methods and technologies must be defined for the increased climate control requirements.


Cooling Trends: Where is it Headed?


Space Comparison: White Space vs. Mechanical Space

Floor Area Required in Data Center

- Rising load density of IT areas

- Solutions for High Density Areas needed

- Cooling solutions with air/water heat exchangers

TODAYTODAY TOMORROWTOMORROW

Mechanical Space 40%

White Space 60%

Mechanical Space 60%

White Space 40%

Source: Internal estimation Rittal 2007


Cooling Trends: The Way Forward


Cooling Options: A Tutorial

Close Coupled Cooling

Rear Door

Air Cooled

Supplemental In Row Solutions

Active Air

0 2 10 15 20 28 35 40 kW

Chip + Enclosure Cooling


Cooling Options: Air Cooled

Ways to Cool – Where they are Used

Air Cooled

0 2 10 15 20 28 35 40 kW

• Air cooled Cabinet in a traditional hot aisle / cold aisle arrangement


Cooling Options: Rear Door


0 2 10 15 20 28 35 40 kW

• Rear Door type units are a lower capacity supplemental cooling system

• Allow user to bring air return temperatures from dense cabinets down to inlet temperatures (70F)

• Can be deployed along with CRAC units to bring the hot aisle temperatures down on struggling units

Rear Door


Cooling Options: Active Air


0 2 10 15 20 28 35 40 kW

• Active Air solutions

• Rely on current cooling capacity to handle the load, but more effectively return it to the CRAC units

• Provides no actual cooling capacity, utilizes drop ceiling if present as a return path to CRAC unit

Active Air


Cooling Options: Supplemental In Row


0 2 10 15 20 28 35 40 kW

• Supplemental In Row Solutions / Above Row

• Provide cold air and heat removal from typical hot aisle / cold aisle arrangement

• Supplements the CRAC unit much like rear door units, typically, but not always of greater capacity

Supplemental In Row Solutions


Cooling Options: Close Coupled Cooling


0 2 10 15 20 28 35 40 kW

• Close Coupled Cooling – Closed Loop

• Provides cooling regardless of room conditions

• Performs all the functions of the CRAC unit, but brings the cooling directly to the cabinet itself

• Creates microclimate that exist only inside the rack

Close Coupled Cooling


Cooling Options: Chip-Level


0 2 10 15 20 28 35 40 kW

• Chip + Enclosure cooling

• Deployed in ultra dense environments, using customized servers

• Typically not for commercial deployment

• Aimed at research environments that can benefit from ultra dense clusters

Chip + Enclosure Cooling


Cooling Options: Visuals

• Open Systsem (supplemental cooling)

• Closed System(less than 10% Heat to the room)

• CPU (Chip) Cooling • Combined Solution


Determining an Approach

• Hot/Cold Aisle Configuration?– Is it possible in your data center

• If not rear door solutions maybe ideal

• What is the Current Density of your Rack? – < 15kW - Supplemental System

• Supplemental systems can provide cooling were needed on a hotspot basis

– > 15kW – Close Coupled System • Can remove all heat being produced by the system, leaving

nothing to chance in the ambient environment


The Concept: LCP Plus

Touch screen


Differentiating Closed Couple Cooling

+ =

LCP Server Cabinet High-Performance Solution

HEX


Differentiating Closed Couple Cooling

HEX

fan

alternative 2


Cooling: Redundancy Considerations

LCP

M1

M2

M3

LCP

LCP

LCP


Cooling: LCP Extend

For pre-deployed racks


Cooling: Supplemental Solutions

Cold aisle

Cold aisle


TCO: Real Estate Cost Analysis

Real Estate Savings $129,617$195, 946


TCO: Real Estate Cost Analysis

common

40KW

10 rack sp.

4KW 4KW 4KW 4KW 4KW 4KW 4KW 4KW 4KW 4KW4KW

Available

40KW, 3 rack space

with LCP

LCP

LCP

20KW 20KW

30 – 40 KW

Available

40+KW, 2 rack space

LCP

LCP

30 – 40 KW


TCO: Energy Cost Analysis

Energy Savings Excellent - $69KAverage - $128KPoor - $302K


Cooling: Total Cost of Ownership

$295,507.68$541,764.08

$1,280,533.27

Cumulative Annual Savings

Excellent ($/Year)Average ($/Year)

Poor ($/Year)

• Allowing for warmer water temperatures, while still maintaining high density loads could save a great deal of money using conventional chillers

• Depending on the operational condition of the chiller plant, the numbers below could be realized annually on an installation of less than 70 close couple cooled cabinets!



• Real Estate savings– Having the ability to cool dense loads allows for

server consolidation into a single rack, or adoption of denser server technologies such as blades

• Running costs– Closed coupled cooling allows for multiple savings in

regards to energy • Lower fan energy costs• Lower lighting cost for smaller, more dense data center• Lower chilled water plant costs• Evaporative and dry free coolers



Freecooler

Low noise Chiller

PumpHeatrecovery

Pump-station

BuffertankEmergency

water

buildingairconditioning

Chiller options


Evaporative or Dry Coolers vs. Chiller Plant

• Typical Chiller plants provide chilled water at <45F, which is then past to the CRAC units in the data center

• Raising this water temperature lowers operating cost, but the cooling solution must be able to handle the warmer water while providing the same amount of cooling

• Close coupled cooling allows for high density installations, while using water as warm as 70F!

• This warm water temperature allows for more hours each year where evaporative or dry air side economizers / coolers can be used lower operating cost, by reduced electricity usage


Chiller Example – California

• With a load of 28kW per cabinet, and with an ASHRAE allowable intake temperature to the servers of 77F, we can use water as warm as 70F, which can be realized whenever the Wet Bulb temperature is below 63F.

• In Oakland this is the case over 97% of the year!


Chiller Example: Major Metropolitan Areas

Water Side Economizer Potential

0102030405060708090

100

Atlant

a

Chica

go

Miam

i

New Y

ork

Oaklan

d

Phoen

ix

Pe

rce

nta

ge

of

Ye

arl

y H

ou

rs

72F Air -65F Water77F Air -70F Water

82F Air -75F Water


Comparison with Traditional Methods - PUE

• Typical data centers range from 3.0-1.6, the lower the number the better– PUE = (Total input Power to Data

Center / Total IT Load)• Example:

– Total Mechanical Load including chillers, UPS units, etc. ~ 216 kW

– Total IT Load at 108 kW– PUE = 216 kW / 108 kW = 2.0

PUE

• Even small LCP+ systems, without economizers, result in a PUE of 1.54!

WattsTotal IT Load 216024

Chiller 87752UPS Input 229296LCP Fan Load 16640

Total Support Load 333688

Resultant PUE 1.54


The Green Impact

The EPA estimates that at the current data center growth rate that we will need 10 new power plants before 2012 just to support IT growth!

http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency_study

The use of close coupled cooling results in less energy loss in moving the air through the data center, saving fan energy, as well as reduces the cost to produce chilled water. This can result in significant energy usage reduction to cool the IT load, the place where the biggest impact can be seen today.


A Look Back…

NowThen


Drawbacks of Closed Couple

• If you are using water cooled CRAH units, then its already there!

• The technology has been around for better than 50 years.

• As densities rose in the 70s with TTL logic, water was required, with the advent of CMOS, it went away, but our densities are back where water is needed.

• Bringing the cooling to the rack offers major advantages!

Mainframes in these data centers were cooled via water, direct to the

chips!


7 Top Cooling Considerations

• Focus on complete data center cooling designs• Provide complete cooling redundancy

– Include a comprehensive monitoring and alarm system at all component levels

• Understand The Complete Facility– Hot/cold aisle – common aisle for exhaust– Placement of vented or cutout floor tiles– Air flow paths

• Consider alternate cabinet configurations– Supplemental Cooling Systems– Close Couple Cooling– Energy-efficient, cost –effective way

• Group common products together• Develop component installation standards• Plan for new cooling solutions



Q&A

To Receive a Copy of Today’s Presentation:

[email protected]