Michigan State University is leading pioneering research on the world’s fastest supercomputer, thanks to a new grant from the U.S. Department of Energy.

The DOE has awarded an MSU-led team 1.3 million node hours of computation time on the Frontier supercomputer (Frontier is made up of 9,400 computing nodes, and one hour of computing on a single node is equal to one “node hour”).

Michigan State University Professor Brian O’Shea

Frontier is capable of performing up to 2 quintillion calculations per second and is currently the fastest supercomputer in the world.

With the grant, MSU researchers and their colleagues will harness the power of Frontier to better understand galaxies. Specifically, the team will be exploring the relationships between large galaxies, the seas of hot matter known as plasma that surround them and the supermassive black holes that exist at their centers.

In doing so, the team also will provide valuable leadership and learning experiences for the larger research community in how to work with this unprecedented computational technology. That community includes not only researchers interested in the cosmos, but also scientists and engineers studying plasma for fusion energy applications.

“The DOE has always supported computation, but they’ve also really supported open science and they’re doing that with the best supercomputers we have,” said Brian O’Shea, the grant’s leader.

O’Shea is a professor in MSU’s Department of Physics and Astronomy, the Facility for Rare Isotope Beams and the Department of Computational Mathematics, Science and Engineering, or CMSE, which is administered by both the College of Natural Science and the College of Engineering.

“The DOE wants researchers to push the boundaries computationally,” O’Shea said. “We’ll be studying galaxies and the physics of plasmas around galaxies, but we’ll also be pushing this new computer in ways it hasn’t been pushed before. The things we learn, the code we develop, these are tools that the community can adopt.”

In surpassing 1 quintillion calculations per second, Frontier established itself as what’s known as an exascale computer. With the DOE making its services available to researchers around the country through a competitive grant process, Frontier is the first public exascale computer.

Although the computer is new, the grant is part of an existing program called INCITE, which stands for Innovative and Novel Computational Impact on Theory and Experiment. The DOE launched INCITE in 2003 to accelerate science by awarding researchers access to supercomputers to help address large, challenging and computationally intensive problems.

“This is an exceptionally important year for us,” said Gina Tourassi in a recent announcement. Tourassi is the director of the National Center for Computational Sciences at Oak Ridge National Laboratory, the DOE lab that’s home to Frontier.

“Users now have access to a machine nearly ten times as powerful as our previous flagship system, opening up avenues throughout the whole of science that were until now completely inaccessible,” she said.

Postdoctoral researcher Benjamin Wibking and Professor G. Mark Voit in the Department of Physics and Astronomy round out the grant project’s contingent at MSU. Also joining the team are Forrest Glines at Los Alamos National Laboratory, Philipp Grete at the University of Hamburg in Germany and Deovrat Prasad at Cardiff University in Wales.

Scaling past the summit

O’Shea knows all of the off-site collaborators well from their time at MSU. Glines earned his doctoral degree at Michigan State. Both Grete and Prasad worked at the university as postdoctoral scholars.

This snapshot of a simulated galaxy gives a preview of the studies Michigan State University researchers and their colleagues will launch on Frontier, the world’s fastest supercomputer. These new studies will be performed at higher resolution and include more of the physics at work in galaxy formation and evolution. Credit: Deovrat Prasad, Cardiff University

In fact, the team has already worked together, developing and validating code for its new project using a smaller supercomputer known as Summit. But don’t go thinking Summit is diminutive.

It’s currently ranked as the world’s fifth fastest supercomputer, according to the website top500.org, which keeps track of the planet’s most powerful computers. Summit topped that list just four years ago.

Today, Frontier is the new leader, clocking in at roughly eight times the speed of Summit.

For another point of comparison, consider MSU’s own supercomputer at the Institute for Cyber-Enabled Research, or ICER, of which O’Shea is the director.

ICER has a good-sized supercomputer to support a university like MSU, where about 1,500 researchers across campus rely on the machine to help with their projects, O’Shea said. Yet, in terms of raw power, it still offers just a fraction of what Frontier does.

“With our grant, we’ve been given about 0.75% of Frontier’s available time for the two years of our allocation. That’s equal to all of ICER’s capacity for the same period of time,” O’Shea said. “I would have to use MSU’s supercomputer by myself for two years to get the same results.”

Philipp Grete, a postdoctoral fellow at the University of Hamburg

O’Shea and his team understand the responsibility that comes with the position they’re in, earning time easily worth upward of $2 million on an unparalleled research tool. They’re part of a select group of researchers who are now tasked with helping Frontier live up to its name.

“The whole point of this is enabling new science,” O’Shea said.

The DOE also is well aware of the value of Frontier’s computation time. That’s why the agency made it a requirement that grant applicants had already demonstrated the feasibility of their projects on other, less powerful supercomputers.

For O’Shea and his team, their previous efforts have made getting the grant that much sweeter while preparing them for the challenges ahead.

“It is very rewarding because we have effectively been preparing for this for many years,” said Grete, who is currently a Marie Skłodowska-Curie Actions Postdoctoral Fellow at the University of Hamburg.

“I think the most challenging component will be the unknown issues that typically start to show up when one pushes the simulation limits,” Grete said. “Based on past experience, it’ll just be a matter of time until something breaks, but we are prepared for this.”

Numbers beyond what you can dial

Although the average person probably hasn’t spent too much time contemplating the complexities of the team’s research, a lot of the challenges the team will face are connected to a central issue that we can all appreciate: Outer space is really, really big.

Using the Frontier supercomputer, a team led by Michigan State University researchers will be working to understand how galaxies like Cygnus A use their plasma atmosphere (blue) and high-energy jets (red) to regulate themselves as they form and evolve. Credit: NASA/CXC/SAO, NASA/STSCl, NSF/NRAO/AUI/VLA

More than that, though, things in the cosmos can be big in different ways at the same time. That’s, in part, why O’Shea needs exceedingly powerful supercomputers to study the galaxies he’s interested in. They need to be able to simulate what’s happening across a large range of size scales over very long periods of times.

Through these studies, the team is aiming to better understand the cosmic rules governing the lifecycles of galaxies, as well as clusters of galaxies. In doing so, they also stand to create tools that will help advance nuclear fusion and sustainable energy studies here on Earth.

“When it comes to doing simulations of galaxies, we’ve hit the point where the computers are powerful enough and the code is good enough that we have to think very hard about how we treat the physics of plasmas,” O’Shea said. “That’s something the DOE supports because, if you want to do fusion energy, you need to have a broad and deep understanding of plasma physics.”

In fact, O’Shea said his project shares a lot of similarities with the new $15 million DOE project for fusion energy led by Andrew Christlieb, a professor and colleague in the Department of Computational Mathematics, Science and Engineering. O’Shea is also a co-investigator in this DOE fusion energy project.

Deovrat Prasad, a postdoctoral fellow at Cardiff University

Compared with a fusion reactor, though, a galaxy’s plasma is spread out over much larger areas. O’Shea and his team also are interested in situations in which that plasma is being fed by a supermassive black hole at the center of a galaxy.

These black holes gobble up matter from the galaxy and, in doing so, also spew out extremely energetic jets of light and particles.

The jets deposit a lot of their contents in the sea or atmosphere of plasma that envelopes the galaxy.

“That atmosphere, though, is also the thing that’s feeding the black hole which is making these jets,” O’Shea said.

There’s thus a feedback loop that allows these galaxies — through the plasma atmosphere, the black hole and its jets — to regulate their mass and energy distributions, steering how they form and evolve.

The process is analogous to setting a thermostat to regulate a room’s temperature, O’Shea said, but the specifics of how the galaxy’s thermostat works are still murky.

“One of the unsolved puzzles in the formation and evolution of galaxies, groups and clusters has been the supermassive black hole feedback process,” said Prasad, who is currently a postdoctoral fellow at Cardiff University.

The problem is complicated by the relative sizes of everything. Galaxies are enormous and, in comparison, black holes are tiny, even the supermassive ones. Similarly, a black hole’s jets can stretch for hundreds of thousands of light-years, yet they are incredibly narrow.

With Frontier, the team will attempt to take account for the physics playing out at all these relevant size scales.

Forrest Glines, a postdoctoral fellow at Los Alamos National Laboratory

“Our planned simulations will be the highest resolutions with length scales spanning six orders of magnitude, exploring length scales from a few parsecs to several megaparsecs,” Prasad said.

A parsec is a unit of measure that’s a little more than 3 light-years — roughly 19 trillion miles — that comes in handy in astrophysical studies.

In addition to resolving important physics at these different but interconnected size scales, the team also will simulate how events play out over galactic timescales, which is to say millions of years.

“To our knowledge, these simulations will be the largest scale models of galaxies and galaxy clusters to date,” said Glines, now a Metropolis Postdoctoral Fellow at Los Alamos National Laboratory.

“The sheer volume of data that will be produced by these simulations will provide its own challenge once the simulations are finished,” Glines said. “We will have to develop new analysis pipelines to explore the simulations with data at that scale.”

As it brings its project to Frontier, the team knows it will face a host of challenges. But the researchers seem excited to face them. Though the problems may be new and different from what they’ve experienced, the team is confident in its collaborative approach.

“Together, we form a complete group with the needed expertise for the project,” Prasad said.