To brief journalists on the new Knights Corner/MIC/Xeon Phi cards, Intel brought us to TACC to see a supercomputer based on it. The Stampede supercomputer housed there is an interesting device in and of itself, lets take a look at it.
The Texas Advanced Computing Center has several supercomputers, the latest is called Stampede. Stampede has 6400 Xeon E5 nodes, each will eventually have a Phi card added to the Dell blade like servers. The supercomputer is still being built out, during our visit there were only about 2000 Phis in place, but the servers were up and running. The racks looked like this.
A look down a TACC row
There are 182 racks of servers in ten rows and four racks of switches in the center. The interconnects are a 56Gbps Melanox Infiniband links, run over fiber of course, and 128 Nvidia K20s for visualization work. Backing it all up is 150PB of disk space located on the far side of TACC from Stampede, all connected via a 150Gbps link. This should be more than enough to load that pesky Crysis level without much lag. Yes we know that Stampede runs Linux and Crysis doesn’t work on that OS, it is a joke guys, chill.
The server blades
The nodes are each in a very unusual tray form factor that can fit a Xeon CPU, a full size PCIe card, eight hard drives, and likely quite a bit more. We would love to go in to more details, but Dell seems to have a penchant for pulling us off their press mailing lists despite the effort of a few individuals there. The end result is that we can’t get anything more than a cursory release or two out of the company, a real briefing is right out. This precludes us writing about anything technical even if they make some amazing form factors. Twits.
A rack and a cooler, blue dots are Phi, green Xeons
Once completed, Stampede will have 2+ PF from the CPUs, 7+ PF from the MICs, will pull 6MW of power, and is cooled by chilled water from a separate plant a few buildings away. Stampede has a hot aisle/cold aisle configuration, but since construction was still ongoing, the roof of the hot aisle was not in place yet. In case you were wondering, it is quite loud, the servers themselves are bad, and the CRAC coolers don’t lessen the din at all. If you stand by a rack, you can have your hair blown out by the coolers and sucked in on the other side of your head by the servers as this Intel engineer helpfully demonstrates.
It is much louder than it sounds here
By early January, Stampede will be fully up and running. 6400 Xeon nodes plus 6400 Phi cards is a pretty solid start, but future CPU and GPU-like upgrades are planned to keep it current, not to mention Windows 8 for touch screen supercomputing goodness. Just kidding. In weeks, the Supercomputer will be able to process over a million concurrent threads without breaking much of a sweat, and do it with very low latency. Best of all, the programming is going to be much easier than before, so the machine will be much more accessible than predecessors to the average scientist. Other than the noise, what’s not to like?S|A
Latest posts by Charlie Demerjian (see all)
- HyperX ships it’s 60 millionth enthusiast memory module - Oct 15, 2018
- Bittware/Nallatech water cools 300W of Xilinx FPGA - Oct 12, 2018
- More on Intel’s 10nm process problems - Sep 17, 2018
- Intel puts out another 14nm 2020 server platform - Sep 11, 2018
- Why Can’t Intel Supply Enough 14nm Xeons? - Sep 10, 2018