Peeking at the TACC Stampede supercomputer

6400 Xeons, 6400 Phis, and one Linux instance

Intel Xeon Phi LogoTo brief journalists on the new Knights Corner/MIC/Xeon Phi cards, Intel brought us to TACC to see a supercomputer based on it. The Stampede supercomputer housed there is an interesting device in and of itself, lets take a look at it.

The Texas Advanced Computing Center has several supercomputers, the latest is called Stampede. Stampede has 6400 Xeon E5 nodes, each will eventually have a Phi card added to the Dell blade like servers. The supercomputer is still being built out, during our visit there were only about 2000 Phis in place, but the servers were up and running. The racks looked like this.

An aisle at TACC stampede

A look down a TACC row

There are 182 racks of servers in ten rows and four racks of switches in the center. The interconnects are a 56Gbps Melanox Infiniband links, run over fiber of course, and 128 Nvidia K20s for visualization work. Backing it all up is 150PB of disk space located on the far side of TACC from Stampede, all connected via a 150Gbps link. This should be more than enough to load that pesky Crysis level without much lag. Yes we know that Stampede runs Linux and Crysis doesn’t work on that OS, it is a joke guys, chill.

A Stampede node

The server blades

The nodes are each in a very unusual tray form factor that can fit a Xeon CPU, a full size PCIe card, eight hard drives, and likely quite a bit more. We would love to go in to more details, but Dell seems to have a penchant for pulling us off their press mailing lists despite the effort of a few individuals there. The end result is that we can’t get anything more than a cursory release or two out of the company, a real briefing is right out. This precludes us writing about anything technical even if they make some amazing form factors. Twits.

A Stampede rack with a cooler

A rack and a cooler, blue dots are Phi, green Xeons

Once completed, Stampede will have 2+ PF from the CPUs, 7+ PF from the MICs, will pull 6MW of power, and is cooled by chilled water from a separate plant a few buildings away. Stampede has a hot aisle/cold aisle configuration, but since construction was still ongoing, the roof of the hot aisle was not in place yet. In case you were wondering, it is quite loud, the servers themselves are bad, and the CRAC coolers don’t lessen the din at all. If you stand by a rack, you can have your hair blown out by the coolers and sucked in on the other side of your head by the servers as this Intel engineer helpfully demonstrates.

It is much louder than it sounds here

By early January, Stampede will be fully up and running. 6400 Xeon nodes plus 6400 Phi cards is a pretty solid start, but future CPU and GPU-like upgrades are planned to keep it current, not to mention Windows 8 for touch screen supercomputing goodness. Just kidding. In weeks, the Supercomputer will be able to process over a million concurrent threads without breaking much of a sweat, and do it with very low latency. Best of all, the programming is going to be much easier than before, so the machine will be much more accessible than predecessors to the average scientist. Other than the noise, what’s not to like?S|A

The following two tabs change content below.

Charlie Demerjian

Roving engine of chaos and snide remarks at SemiAccurate
Charlie Demerjian is the founder of Stone Arch Networking Services and is a technology news site; addressing hardware design, software selection, customization, securing and maintenance, with over one million views per month. He is a technologist and analyst specializing in semiconductors, system and network architecture. As head writer of, he regularly advises writers, analysts, and industry executives on technical matters and long lead industry trends. Charlie is also available through Guidepoint and Mosaic. FullyAccurate