Energy Department kicks off its giant Titan supercomputer with thousands of Nvidia and AMD chips

The Oak Ridge National Laboratory has completed work on the world’s fastest “open science” supercomputer, dubbed Titan. The system is powered by thousands of microprocessors from Advanced Micro Devices and graphics chips from Nvidia. It can perform more than 20 petaflops (20 million billion floating-point math operations per second), and will be used to solve all sorts of tough computing problems such as calculating the effects of climate change on the Earth.

Made by Cray, Titan, also known as the Cray XK7 supercomputer, has some gargantuan specifications. The machine has the computing power that is equal to 500,000 laptop computers, or 20,000 trillion calculations per second. It has 18,688 compute nodes, or servers that are part of the larger supercomputer. It has 299,008 AMD Opteron cores, or microprocessor brains. It has 18,688 Nvidia Tesla K20 “Kepler” graphics accelerators. It also has 710 terabytes of system memory, and total peak system performance of 20 petaflops. The system has 200 server cabinets and a sophisticated cooling system. Each computing node has an AMD Opteron 6274 processor and an Nvidia GPU.

Nvidia says about 90 percent of the mathematical floating-point capability comes from its Tesla K20 graphics processing unit (GPU). The power efficiency of GPUs, which process lots of tasks in parallel, enables much more efficient computing to happen on Titan.

“We can augment the CPUs to provide power efficiency that is far better,” said Sumit Gupta, general manager of Nvidia’s Tesla GPU business. “This is like putting 500,000 laptops in a space the size of a basketball court.”

The supercomputer will attack problems related to climate change, biofuels, materials science, astrophysics, combustion, and nuclear energy. Scientists using the supercomputers will use it to make more efficient engines, higher capacity and lighter weight batteries, and search for cures for diseases.

Operated by the U.S. Department of Energy‘s Oak Ridge National Laboratory in Tennessee, Titan is another step toward an exascale computer, which targets performance of 1,000 petaflops by 2020. The supercomputer is part of an “open science” program that allows researchers from academia, industry and government to use it to solve all sorts of problems.

The machine is ten times faster and five times more energy efficient than its predecessor, the 2.3-petaflop Jaguar system, while occupying the same floor space. If Oak Ridge had upgrade the CPU-only Jaguar system by adding more microprocessors, the system would have been four times larger and consume more than 30 megawatts of power. That’s enough to power tens of thousands of homes. Jaguar itself required 7 megawatts, or enough to power 7,000 homes.

“Basing Titan on Tesla GPUs allows Oak Ridge to run phenomenally complex applications at scale, and validates the use of accelerated computing to address our most pressing scientific problems,” said Steve Scott, chief technology officer of the GPU Accelerated Computing business at Nvidia. “You simply can’t get these levels of performance, power- and cost-efficiency with conventional CPU-based architectures. Accelerated computing is the best and most realistic approach to enable exascale performance levels within the next decade.”

Work on Titan began three years ago with Oak Ridge’s plant to upgrade Jaguar, which was one of the former most powerful supercomputers in the world. Titan has three times the performance per watt of power used and three times higher performance per unit server.

“Science and technology have always been our primary goal, and Titan is a groundbreaking tool that will allow researchers worldwide to leverage GPU-accelerated computing to make unparalleled breakthroughs,” said Jeff Nichols, associate laboratory director for computing and computational sciences at Oak Ridge National Laboratory. “The new Tesla GPU accelerators offer the performance and energy efficiency that enable Titan to scale to unprecedented performance levels without consuming the energy equivalent of a small city.”

The Energy Department has awarded 4.7 billion hours of computing time on Titan and other supercomputers to 61 science and engineering projects with high potential for accelerating discovery and innovation through its Incite program. For instance, the department gave Boris Kozinsky of the Robert Bosch Corp. a grant 48 million core hours to discover and optimize new classes battery technology.