Not Rocket Science

A Fresh Look at Datacenter Power, Cooling, and Cabling

By Al Riske

11.Oct.07- Dean Nelson leads a seven-person team that has been consolidating Sun datacenters and labs around the world and saving the company millions of dollars.

"This is not rocket science," he says.

But, by using one simple concept, along with several best-of-breed products, Nelson and team have not only saved Sun a boatload of cash, they have made the company more agile.

And they've been showing customers how they can do the same.

Director of Global Lab and Datacenter Design Services, Nelson points out that Sun must be part of the solution because it is also part of the problem.

While our servers are more compact and energy efficient than ever — addressing the space and power concerns of datacenter operators — their high-density design makes it harder to keep them cool.

"You get more compute per watt but you also have more watts per rack, and those numbers are going to continue to increase," he says. "So we're actually building the products that cause an increase in heat. If we can't solve that in our own house, how can we expect our customers solve it?"

One look at the products we're building (and plan to build) and Nelson knew the old way of designing datacenters was on its way out.

"The legacy thinking of power, cooling, cabling — how we used to do it — just doesn't work anymore. A raised floor and forced air? You can't properly cool stuff that way, because as you get higher in load — 4 kilowatts, 8 kilowatts, 10 kilowatts, and greater per rack — you just can't get the cooling where you need it," he says. "Air is coming up the vents, but you can't specify that this rack is 2 kilowatts and that one is 4 kilowatts So you have hot spots all over the place."

And that's just one of the challenges. How do you get the right number of watts to the right rack without rewiring the whole datacenter? And how do you deal with all the cables — thousands of them going into and out of all those servers, switches, and storage systems?

"There's no single answer to the problem," Nelson says. "But there's a single approach."

The approach is modular. No surprise there. "This is not rocket science," Nelson repeats. It is, however, thoughtful and practical.

The basic building block that Nelson and his team came up with is called a pod.

Think of it a room inside a room.

"In 3000 square feet, you have space for about six pods. A pod may have 20 racks," Nelson explains. "You have the ability to put a 2 kilowatt rack next to a 9 kilowatt rack, so as your hardware refresh program happens, your infrastructure stays the same. That includes power, cooling, and cabling."

The power comes through an overhead busway system with modules that allow you to snap in the type of outlet you need. "Need to convert a regular 110 plug to an L630? No problem. Take out the 110 and snap in the L630," Nelson says.

The cooling can be handled with intelligent in-row units that sense the temperature and speed up or slow down their fans as needed. But if floor space is at a premium, overhead fans can be used to target hot spots. "The overhead units save square footage but don't have intelligence. They're just on," he says. "But you are localizing the cooling so it's still very efficient."

Liquid cooling is also an option for super dense environments, but is still a few years out, Nelson says. "Frankly, the majority of the datacenters don't need direct liquid yet. The loads will increase, but you should not be forced to liquid racks until your averages break 20 kilowatts per cabinet. Consider the analogy of a skyscraper as a high-density cabinet. You have a small number of these skyscrapers surrounded by hundreds of smaller buildings in a city. Datacenters are no different. You need to accommodate both types, but smaller racks will still be the majority of the space. That is the reality."

Cabling, meanwhile, was becoming a nightmare with as many as 300 cables per 40-system rack — a 1300 percent increase in just the past few years. In a pod with 20 racks, that's 6000 cables.

"Instead of going up to a patch panel and all the way back to an MDF, or main distribution frame, the 6000 cables remain within the pod," Nelson says. "We collapse them into an IDF, or intermediate distribution frame, and have just eight cables coming out of the pod."

The new approach saves material, labor, and of course money.

Breaking with design convention, Sun has constructed new energy-efficient datacenters in the United Kingdom, India, and the United States, where the company recently compressed 152 datacenters (202,000 square feet) at various California locations into 14 new centers (76,000 square feet) in Santa Clara. The first phase of this consolidation was compression in place. Hardware replacement.

That translates into an 88 percent reduction in real estate -- and a 60 percent reduction in overall power consumption.

Which cut utility bills by more than $860,000 in the first nine months.

"Moreover, it reduced the new construction requirement by 20,000 square feet!," Nelson says. "That was a $9 million cost avoidance."

At the same time, the new Santa Clara centers are delivering a 450 percent improvement in compute performance and a 244 percent increase in storage capacity.

The hardware replacement portion of the project, which built on lessons learned in the UK and India, was completed in three months and is expected to pay for itself in just three years.

It will also reduce carbon emissions by 3,227 metric tons annually.

"The hardware replacement program is a perfect example of how investments in Sun's newer technologies can produce massive cost savings and productivity improvements," Nelson says. "But it's even bigger than that. It helps the environment. We're applying our eco strategy in our own backyard."

Though the pod system is simple and modular, it represents dramatic changes for folks in both IT and facilities.

Nelson and his team live in both worlds and help bridge the two.

"Facilities — they're electrical, they're cooling, that type of thing. Previously, they weren't required to understand the compute end of it and frankly were skeptical of the projected heat loads," he says.

"The IT side doesn't understand what facilities has to have to make it work. They'll order equipment, roll it into the datacenter, and tell facilities that they need to turn it on. But on the facilities side, it's ... 'Oh, my god, this is 8000 watts in a cabinet. How am I supposed to deal with it? To compound the problem, facilities pays the power bill. So, not only are they having to react to these huge heat load increases, they have to absorb the increase in costs.'"

To solve the problem, Nelson and his team have been educating construction crews — and working with Sun's engineering groups, who, in some cases, have modified their designs because of how the product would be implemented in the datacenter.

"That is the loop we've got to have going forward. Period. Because our products are going to keep getting hotter, while they are getting better. Our customers want the compute capacity, they want the speed, the efficiencies, but they also have to grapple with the heat loads," Nelson says.

"We've been working with the System Group on their next-generation products. They've reached a point where it's crossed that line. We can't just sell this incredibly powerful compute box anymore. It's so dense, so hot, that we have to guide the customer on how to design for it in their datacenter architecture. If we can go out to customers and say, 'Here's the product and here's how to design your datacenter to accommodate it' ... well, there you go!"

Nelson says Sun spent between 10 percent and 15 percent extra to be sure the datacenters have room to grow, easily, without the need for new construction or new wiring — just new equipment.

This despite questions from construction teams, 'Why do we need to build it like this? We can build this cheaper and save Sun money immediately.'

"Absolutely you can do it cheaper, but that would limit Sun's ability to seize business opportunities in an ever-changing environment. In the past we had fixed environments. Lab spaces that you couldn't change easily. By doing this, we trap our own growth by limiting the flexibility. There are immediate short-term savings, but you lose much more in terms of agility. These new designs allow our customers to easily adapt," Nelson says.

"We are basically an internal architecture firm with IT expertise. Knowing the physical. Knowing the technical. When you can balance and translate between both of these areas, you can achieve the objectives of both. I believe every company should have a group doing this function. If you don't, you're wasting tons of money, and you're stunting the ability of your company to turn on a dime."