Fluid flow simulation on Frontier earns Gordon Bell finalist selection

Using a new computational technique called information geometric regularization (IGR) researchers from the Georgia Institute of Technology and the Courant Institute of Mathematical Sciences at New York University conducted the largest-ever computational fluid dynamics (CFD) simulation of fluid flow on the Frontier supercomputer at the Department of Energy's Oak Ridge National Laboratory.

Their undertaking - and the methods behind it - earned the team a finalist selection for the Association for Computing Machinery's 2025 Gordon Bell Prize for outstanding achievement in high-performance computing (HPC).

"It's exciting to link a grand challenge scientific problem to an interesting new method, with an implementation tailored for the latest supercomputer architectures," said Spencer Bryngelson, an assistant professor in Georgia Tech's College of Computing who led the project with Florian Schäfer, an assistant professor at the Courant Institute.

CFD simulations are often used to predict the behaviors of new aircraft designs, showing the potential interactions of proposed rockets and airplanes - and their engines - with the atmosphere. In this CFD study, Bryngelson and his team used their open-source Multicomponent Flow Code (available under the MIT license on GitHub) to examine rocket designs that feature clusters of engines. Predicting how all those engines' exhaust plumes may interact upon launch will help rocket designers avoid mishaps - especially with the scale and speed afforded by the Georgia Tech team's method.

The team used Frontier to simulate a 33-engine configuration, like the one used by the SpaceX Starship Super Heavy Booster, reflecting the aerospace industry's move toward first-stage multi-engine layouts in rocket design. The flow from the individual engines was modeled at 10 times the speed of sound, a regime at which gases behave violently and unpredictably due to extreme pressure and temperature shifts. This simulation achieved a resolution of over 200 trillion grid points, or 1 quadrillion degrees of freedom (variables that must be solved).

The methodology was optimized to use the unified CPU-GPU memory on Frontier and achieve a step-change increase in resolution and scale over previous record-holding CFD simulations. At the same time, the team achieved a 4 times faster time to solution and increased energy efficiency by 5.7 times over current state-of-the-art numerical methods.

"I'm very excited. I view myself more on the math side than really on the HPC side, so for me it's just very gratifying to see that people in the HPC world get interested enough in these methods to make them run at scale. It's quite rare. Having our method implemented at that scale is very gratifying. I'm thankful that it worked out," Schäfer said.

IGR was originally developed by Schäfer and his former student Ruijia Cao. The key to IGR's ability to increase the scope and performance of CFD simulations is that it uses a new strategy to represent shock dynamics. Flowing fluids can sometimes form "shocks" when the flow speed exceeds the speed of sound. The shocks appear as discontinuous changes in fluid properties, such as pressure, temperature and density. These discontinuities must be accounted for in CFD simulations to avoid numerical instabilities and to properly simulate the flow.

Computational scientists have long attacked this issue by using various flavors of shock-capturing methods, such as introducing artificial viscosity or using numerical methods that average over the discontinuity to smooth out its effect on the simulation. IGR replaces discontinuities with near discontinuities that preserve the flow features, including post shock behavior.

"IGR is something I originated two years ago, and it's really a blend of ideas coming from optimization and statistics that found a slightly unusual application in fluid dynamics. This idea is really about changing the notion of a straight line," Schäfer said. "It had been marinating in my mind for a long time, and I always thought maybe there's a way to apply it to fluids. And then I heard Spencer and some other people talk about shocks. For the first time, it clicked that, really, this notion of changing a straight line could become useful in the treatment of shock dynamics."

The team used Bryngelson's OLCF Director's Discretionary and ASCR Leadership Computing Challenge allocations on Frontier to test and demonstrate the method. This work ultimately enabled them to prove IGR's suitability for practical CFD and its ability to scale CFD simulations to new heights. ACM's recognition of their work also reflects their persistence in pursuit of an unorthodox approach.

The winners of the Gordon Bell Prize will be announced at this year's International Conference for High Performance Computing, Networking, Storage, and Analysis (SC25), held in St. Louis, Missouri, from Nov. 16 to 21.

Frontier, the world's most powerful supercomputer for open science, is managed by the Oak Ridge Leadership Computing Facility, a DOE Office of Science user facility at ORNL.

UT-Battelle manages ORNL for DOE's Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE's Office of Science is working to address some of the most pressing challenges of our time. For more information, visit energy.gov/science.