Add Particle I/O: restart & restart_checkpoint

[JBlaschke]

Adding particle I/O. Working on proper tests now.

This will also read FHDeX particles.

@ax3l let me know if you have early feedback

[ax3l]

Looks great so far! 👍

[BenWibking]

Does a separate particle container need to be created for each app? We have this (+more to come):
https://github.com/quokka-astro/quokka/blob/c37cfb21266fb793d2ff39d865a3f2174a3912e9/src/particles/CICParticles.hpp#L25

[BenWibking]

The real problem is that we will have some ParticleContainers with a variable number of real components for each simulation...

[ax3l]

Does a separate particle container need to be created for each app? We have this (+more to come):
https://github.com/quokka-astro/quokka/blob/c37cfb21266fb793d2ff39d865a3f2174a3912e9/src/particles/CICParticles.hpp#L25
The real problem is that we will have some ParticleContainers with a variable number of real components for each simulation...

Yes, so far.
I see two ways to solve this:
a) expose the header for PC creation, so you can specialize for your code downstream or
b) more forward looking, codes could try to use a minimal set of compile-time components (e.g., ND positions only in pure SoA format) and instead use runtime components for all other attributes. That way, we only have to specialize one base class that everyone ca use. I doubt there is a significant performance impact, but we need to evaluate that in applications.

Another change I will put into BLAST codes is to use the Polymorphic particle container, where Arena is not a template parameter. On the PC level, having the Arena a C++ template appears unnecessary to me and should not have any performance benefits (in symmetry, the MultiFabs also have it only as a runtime parameter).

[JBlaschke]

I'm currently stalled on a head-scratcher: when I run test_aos.py on its own, it works fine. If I run it in pytest I get:

tests/test_aos.py::test_aos_init amrex::Error::0::ParmParse::Initialize(): already initialized! !!!

Most strange ... any advice?

[ax3l]

The issue is a double init. You can remove this:
#342 (comment)

We init and finalize amrex in a fixture for all tests here:
https://github.com/AMReX-Codes/pyamrex/blob/development/tests/conftest.py
so this is not needed again in the test file.

To run only your latest test while you develop, run this:

python3 -m pytest tests/test_plotfileparticledata.py

the tests are not intended to be run without the pytest module, but can be sub-selected for productivity to feel like a standalone script.

[ax3l]

It looks like when building in debug mode (with more checks), the test still fails.

[JBlaschke]

Any idea why the tests fail? Seems to depend on compiler version... Which is never a good sign...

[ax3l]

Please see comments from yesterday. It fails in debug mode, where AMReX does more checks for validity. Likely because the added positions do not line up with the domain decomposition.

[ax3l]

@JBlaschke This fixed the pc.OK() issue that you saw when running AMReX in Debug mode. Please double check (maybe with print debugging) if this does what you expect to do or if you geometry needs adjustment, if the pc.redistribute removes particles.

[ax3l]

Thank you, Johannes! 🚀 ✨

[ax3l]

@JBlaschke do you need this for 1D, 2D and 3D AMReX builds or only for 3D?