GEAR-RT: Towards Exa-Scale Moment Based Radiative Transfer For Cosmological Simulations Using Task-Based Parallelism And Dynamic Sub-Cycling with SWIFT
Numerical simulations have become an indispensable tool in astrophysics and cosmology. The constant need for higher accuracy, higher resolutions, and models of
ever-increasing sophistication and complexity drives the development of modern tools
which target largest computing systems and employ state-of-the-art numerical methods and algorithms. Hence modern tools need to be developed while keeping optimization and parallelization strategies in mind from the start.
In this work, the development and implementation of Gear-RT, a radiative transfer
solver using the M1 closure in the open source code Swift, is presented, and validated
using standard tests for radiative transfer. Gear-RT is modeled after Ramses-RT (Rosdahl et al. (2013)) with some key differences. Firstly, while Ramses-RT uses finite
volume methods and an adaptive mesh refinement (AMR) strategy, Gear-RT employs
particles as discretization elements and solves the equations using a finite volume particle method (FVPM). Secondly, Gear-RT makes use of the task-based parallelization
strategy of Swift, which allows for optimized load balancing, increased cache efficiency, asynchronous communications, and a domain decomposition based on work
rather than on data.
Gear-RT is able to perform sub-cycles of radiative transfer steps w.r.t. a single hydrodynamics step. Radiation requires much smaller time step sizes than hydrodynamics,
and sub-cycling permits calculations which are not strictly necessary to be skipped.
Indeed, in a test case with gravity, hydrodynamics, and radiative transfer, the subcycling is able to reduce the runtime of a simulation by over 90%. Allowing only a
part of the involved physics to be sub-cycled is a contrived matter when task-based
parallelism is involved, and it required the development of a secondary time stepping
scheme parallel to the one employed for other physics. It is an entirely novel feature
in Swift.
Since Gear-RT uses a FVPM, a detailed introduction into finite volume methods and
finite volume particle methods is presented. In astrophysical literature, two FVPM
methods are written about: Hopkins (2015) have implemented one in their Gizmo
code, while the one mentioned in Ivanova et al. (2013) isn't used to date. In this work,
I test an implementation of the Ivanova et al. (2013) version, and conclude that in its
current form, it is not suitable for use with particles which are co-moving with the
fluid, which in turn is an essential feature for cosmological simulations.
Finally, the implementation of Acacia, a new algorithm to generate dark matter halo
merger trees with the AMR code Ramses, is presented. As opposed to most available
merger tree tools, it works on the fly during the course of the N-body simulation. It
can track dark matter substructures individually using the index of the most bound
particles in the clump. Once a halo (or a sub-halo) merges into another one, the algorithm still tracks it through the last identified most bound particle in the clump,
allowing to check at later snapshots whether the merging event was definitive. another one. The performance of the method is compared using standard validation
diagnostics, demonstrating that it reaches a quality similar to the best available and
commonly used merger tree tools. As proof of concept, Acacia is used together with
a parametrized stellar-mass-to-halo-mass relation to generate a mock galaxy catalogue
that shows good agreement with observational data.
EPFL_TH9973.pdf
N/a
openaccess
copyright
45.08 MB
Adobe PDF
9e2115775c3ff0b758c32c5d09ae5482