Overview -- Numerical Relativity Using CUDA C/C++ Is Easier Than You Think!

Simulation results of a binary black hole head-on collision run on a GPU based home gaming computer

Not too long ago, the only way to numerically solve the Einstein-Hilbert field equations required the use of a supercomputer that ran a hundred or more CPUs in parallel for days at a time. In 2005, Frans Pretorius used a cluster of 125 CPUs at the University of Texas to run the first every numerical simulation of the merger of an orbiting binary pair of black holes. His simulation took 30 days to complete.

Univ. of Texas Advanced Computer Center - the Stampede Cluster

link to Frans Pretorius Home Page

For massive simulations, like cosmology models, large clusters and packaged routines are still the way to go. Bu, for smaller simulations, like head-on binary black hole simulations, this is no longer the case.

Since the wide-spread use the Graphics Processor Units (GPU) in home computers, the average Joe has access to systems that run at TFLOP (Trillions of Floating-Point Operations per second) rates.

HP Omen Home Gaming Computer w/ 11-TFLOPS rated NVIDIA GeForce 1080-Ti

NVIDIA GeForce 1080Ti GPU

The problem is that many physicists I have spoken with who do numerical relativity for a living are not at all familiar with programming CUDA C/C++. This is a real bummer, because it means one is limiting oneself to either using canned routines or only clusters that use CPUs.

Every serious researcher interested in the best simulation is going to want to go beyond canned routines. Better yet, I  would hope that those  folks who maintain canned numerical relativity routines would be interested in porting their code to CUDA to exploit the massive speed ups one can get on GPU clusters.

And so, I've developed this tutorial to help people who want to utilize the power of GPUs in their own work.

My view is, if I can do it, certainly the people who have made a career from numerical relativity can learn to program in CUDA C/C++ as well.

Methods abound, but for the purposes of this example, I will focus on use of the BSSN moving puncture method with dynamic gauge variables in Cartesian coordinates using a 3D second order finite difference method for spatial derivatives and a Euler finite difference for evolution in time. 

The simulation results presented in this video were completed on a HP Omen gaming computer that is equipped with a NVIDIA GeForce 1080-Ti GPU card, which is rated at 11 TFLOPS. The language used in the simulation is CUDA C/C++. The total simulation time was about 8 hours or so. And, the code is hardly optimized.

Yes, I chose simplicity for the purpose of this example. The point is that if one were to learn how to program a simple example, then hopefully one can take those learnings and apply them to programming large clusters of GPUs.

No doubt these methods can be improved, and it is the hope of the author that the reader will do so in his or her own version. However, the purpose here is to demonstrate how to utilize the power of the GPU to perform a realistic numerical relativity example. In that sense, the simpler the method the better. In this case, we will examine how to use CUDA C/C++ to simulate a head-on collision of two black holes.

The video below is the result of one such simulation.Again, it took about 8-hours to run to completion. The graphs are of the W variable used in the moving puncture BSSN formulation, which can be thought of as the strength of the gravitational field.

References

[1] Shibata, Masaru. Numerical Relativity :100 Years of General Relativity. World Scientific Publishing Company (November 5, 2015).

 [2] CUDA C Programming Guide, https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

[3] Duane Stori and Mete Yurtoglu. CUDA for Engineers: An Introduction to High-Performance Parallel Computing. Addison-Wesley Professional; 1st edition (November 12, 2015).

Next: Background Material on General Relativity -->

Table of Contents
Overview
Background Material on General Relativity
Cracking the Code
Moving Puncture BSSN Evolution Equations
Dynamical Gauge Equations
Initial Conditions for Head-On Collision of Two Black Holes
Kreiss-Oliger Spatial Filtering
Structures Used to Implement on GPU Machine
Implementation on CPU example
Implementation on GPU Example
Finite Differences on a GPU Machine
Variable Glossary