Documentation

CODE_OF_CONDUCT.rst

@@ -0,0 +1,61 @@
+.. _coc:
+
+Code of Conduct
+===============
+
+Our Pledge
+----------
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+Our Standards
+-------------
+
+Examples of behavior that contributes to creating a positive environment include:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+- The use of sexualized language or imagery and unwelcome sexual attention or advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic address, without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+Our Responsibilities
+--------------------
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+Scope
+-----
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.
+Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+Representation of a project may be further defined and clarified by project maintainers.
+
+Enforcement
+-----------
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at amrex@lbl.gov.
+All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances.
+The project team is obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+Attribution
+-----------
+
+This Code of Conduct is adapted from the `Contributor Covenant <https://www.contributor-covenant.org>`_, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

docs/source/acknowledge_us.rst

@@ -0,0 +1,46 @@
+Acknowledge pyAMReX
+===================
+
+Please acknowledge the role that pyAMReX played in your research.
+
+
+In Publications
+***************
+
+If a project using pyAMReX leads to a scientific publication, please consider citing it.
+This helps to keep in touch with the community, shows its use and supports the project.
+
+.. code-block:: latex
+
+  \usepackage{hyperref}
+  This research used the open-source code pyAMReX \url{https://github.com/AMReX-Codes/pyamrex}.
+  We acknowledge all AMReX contributors.
+
+
+- Huebl A, Ananthan S, Grote D P, Sandberg R T, Zoni E, Lehe R, Jambunathan R, Myers A, Zhang W.
+  **pyAMReX**.
+  *software*, 2023.
+  `github.com/AMReX-Codes/pyamrex <https://github.com/AMReX-Codes/pyamrex>`__
+
+You can also add an acknowledgement, e.g.,
+
+  This research used the open-source code pyAMReX [citation].
+  We acknowledge all AMReX contributors.
+
+
+Further pyAMReX References
+**************************
+
+Works using pyAMReX:
+
+- Sandberg R T, Lehe R, Mitchell C E, Garten M, Qiang J, Vay J-L and Huebl A.
+  **Hybrid Beamline Element ML-Training for Surrogates in the ImpactX Beam-Dynamics Code**.
+  14th International Particle Accelerator Conference (IPAC'23), WEPA101, *in print*, 2023.
+  `preprint <https://www.ipac23.org/preproc/pdf/WEPA101.pdf>`__,
+  `DOI:10.18429/JACoW-IPAC-23-WEPA101 <https://doi.org/10.18429/JACoW-IPAC-23-WEPA101>`__
+
+- Huebl A, Lehe R, Mitchell C E, Qiang J, Ryne R D, Sandberg R T, Vay JL.
+  **Next Generation Computational Tools for the Modeling and Design of Particle Accelerators at Exascale**.
+  2022 North American Particle Accelerator Conference (NAPAC'22), TUYE2, pp. 302-306, 2022.
+  `arXiv:2208.02382 <https://arxiv.org/abs/2208.02382>`__,
+  `DOI:10.18429/JACoW-NAPAC2022-TUYE2 <https://doi.org/10.18429/JACoW-NAPAC2022-TUYE2>`__

docs/source/acknowledgements.rst

@@ -0,0 +1,4 @@
+Funding and Acknowledgements
+============================
+
+This work was supported by the Laboratory Directed Research and Development Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02-05CH11231.

docs/source/coc.rst

@@ -0,0 +1 @@
+../../CODE_OF_CONDUCT.rst

docs/source/developers/debugging.rst

@@ -0,0 +1,53 @@
+.. _debugging-pyamrex:
+
+Debugging the code
+==================
+
+Sometimes, the code does not give you the result that you are expecting.
+This can be due to a variety of reasons, from misunderstandings or changes, system specific quirks, or bugs.
+You might also want to debug your code that uses pyAMReX or pyAMReX itself, as you implement new features during development.
+
+This section gives a step-by-step guidance on how to systematically check what might be going wrong.
+
+Debugging Workflow
+------------------
+
+Debugging Python processes works the same way as debugging regular applications.
+For investigating bugs on the Python extension side (e.g., in AMReX/pyAMReX/your application), the Python intepreter itself does *not* need to be compiled in debug mode.
+
+Try the following steps to debug a simulation:
+
+#. Check the output text file, often called ``output.txt``: are there warnings or errors present?
+#. On an HPC system, look for the job output and error files, usually called ``<name>.e...`` and ``<name>.o...``.
+   Read long messages from the top and follow potential guidance.
+#. If your simulation already created output data files:
+   Check if they look reasonable before the problem occurred; are the initial conditions of the simulation as you expected?
+   Do you spot numerical artifacts or instabilities that could point to missing resolution or unexpected/incompatible numerical parameters?
+#. Did the job output files indicate a crash? Check the ``Backtrace.<mpirank>`` files for the location of the code that triggered the crash.
+   Backtraces are read from bottom (high-level) to top (most specific line that crashed).
+#. In case of a crash, Backtraces can be more detailed if you :ref:`re-compile <install-developers>` with debug flags: for example, try compiling with ``-DCMAKE_BUILD_TYPE=RelWithDebInfo`` (some slowdown) or even ``-DCMAKE_BUILD_TYPE=Debug`` (this will make the simulation way slower) and rerun.
+#. If debug builds are too costly, try instead compiling with ``-DAMReX_ASSERTIONS=ON`` to activate more checks and rerun.
+#. If the problem looks like a memory violation, this could be from an invalid field or particle index access.
+   Try compiling with ``-DAMReX_BOUND_CHECK=ON`` (this will make the simulation very slow), and rerun.
+#. If the problem looks like a random memory might be used, try initializing memory with signaling Not-a-Number (NaN) values through the runtime option ``fab.init_snan = 1``.
+   Further useful runtime options are ``amrex.fpe_trap_invalid``, ``amrex.fpe_trap_zero`` and ``amrex.fpe_trap_overflow`` (see details in the AMReX link below).
+#. On Nvidia GPUs, if you suspect the problem might be a race condition due to a missing host / device synchronization, set the environment variable ``export CUDA_LAUNCH_BLOCKING=1`` and rerun.
+#. Consider simplifying your input options and re-adding more options after having found a working baseline.
+
+Fore more information, see also the `AMReX Debugging Manual <https://amrex-codes.github.io/amrex/docs_html/Basics.html#debugging>`__.
+
+Last but not least: the community of AMReX developers and users can help if you get stuck.
+Collect your above findings, describe where and what you are running and how you installed the code, describe the issue you are seeing with details and input files used and what you already tried.
+Can you reproduce the problem with a smaller setup (less parallelism and/or less resolution)?
+Report these details in a :ref:`pyAMReX or AMReX GitHub issue <contact>`.
+
+Debuggers
+---------
+
+See the `AMReX debugger section <https://amrex-codes.github.io/amrex/docs_html/Basics.html#breaking-into-debuggers>`__ on additional runtime parameters to
+
+* disable backtraces
+* rethrow exceptions
+* avoid AMReX-level signal handling
+
+You will need to set those runtime options to work directly with debuggers.

docs/source/developers/documentation.rst

@@ -0,0 +1,85 @@
+.. _developers-docs:
+
+Documentation
+=============
+
+Python API documentation
+------------------------
+
+to document:
+
+- Python doc style
+- ``.pyi`` stub files generation & usage
+- Sphinx autodocs/module used
+
+
+Doxygen documentation (C++)
+---------------------------
+
+pyAMReX uses a `Doxygen documentation <https://www.doxygen.nl/manual/docblocks.html>`__ for its C++ part.
+Whenever you create a new class, please document it where it is declared (typically in the header file):
+
+.. code-block:: cpp
+
+   /** A brief title
+    *
+    * few-line description explaining the purpose of my_class.
+    *
+    * If you are kind enough, also quickly explain how things in my_class work.
+    * (typically a few more lines)
+    */
+   class my_class
+   { ... }
+
+Doxygen reads this docstring, so please be accurate with the syntax! See `Doxygen manual <http://www.doxygen.nl/manual/docblocks.html>`__ for more information. Similarly, please document functions when you declare them (typically in a header file) like:
+
+.. code-block:: cpp
+
+  /** A brief title
+   *
+   * few-line description explaining the purpose of my_function.
+   *
+   * \param[in,out] my_int a pointer to an integer variable on which
+   *                       my_function will operate.
+   * \return what is the meaning and value range of the returned value
+   */
+  int my_class::my_function(int* my_int);
+
+An online version of this documentation is :ref:`linked here <development-doxygen>`.
+
+
+Breathe documentation
+---------------------
+
+Your Doxygen documentation is not only useful for people looking into the C++ side of the pyAMReX code, it is also part of the `pyAMReX online documentation <https://pyamrex.readthedocs.io>`_ based on `Sphinx <http://www.sphinx-doc.org>`_!
+This is done using the Python module `Breathe <http://breathe.readthedocs.org>`_, that allows you to read Doxygen C++ documentation dorectly in the source and include it in your Sphinx documentation, by calling Breathe functions.
+For instance, the following line will get the Doxygen documentation for ``make_Vector`` in ``src/Base/Vector.H`` and include it to the html page generated by Sphinx:
+
+.. doxygenfunction:: make_Vector
+
+.. .. doxygenfunction:: make_ParticleContainer_and_Iterators
+
+
+Building the documentation
+--------------------------
+
+To build the documentation on your local computer, you will need to install Doxygen as well as the Python module ``breathe``.
+First, change into ``docs/`` and install the Python requirements:
+
+.. code-block:: sh
+
+    cd docs/
+    pip install -r requirements.txt
+
+You will also need Doxygen (macOS: ``brew install doxygen``; Ubuntu: ``sudo apt install doxygen``).
+
+Then, to compile the documentation, use
+
+.. code-block:: sh
+
+    make html
+    # This will first compile the Doxygen documentation (execute doxygen)
+    # and then build html pages from rst files using sphinx and breathe.
+
+Open the created ``build/html/index.html`` file with your favorite browser.
+Rebuild and refresh as needed.

docs/source/developers/doxygen.rst

@@ -0,0 +1,8 @@
+.. _development-doxygen:
+
+C++ Objects & Functions
+=======================
+
+We generate the documentation of C++ objects and functions *from our C++ source code* by adding :ref:`Doxygen strings <developers-docs>`.
+
+Our Doxygen C++ API documentation in classic formatting `is located here <../_static/doxyhtml/index.html>`_.

docs/source/developers/implementation.rst

@@ -0,0 +1,9 @@
+.. _developers-implementation:
+
+Implementation Details
+======================
+
+.. note::
+
+   TODO :-)
+   Link Dec 2022 slides from Axel or similar...

docs/source/developers/repo_organization.rst

@@ -0,0 +1,115 @@
+.. _developers-repo-structure:
+
+pyAMReX Structure
+=================
+
+Repo Organization
+-----------------
+
+The pyAMReX source code is located in ``src/`` and contains pybind11 C++ code, exposing C++ objects and functions to Python.
+All sub-directories have a pretty straightforward name, mirroring the `AMReX Src/ structure <https://github.com/AMReX-Codes/amrex/tree/development/Src>`__.
+
+Additionally, the ``src/amrex/`` directory contains Python import modules and extensions written in pyre Python.
+
+
+Code organization
+-----------------
+
+The main pyAMReX import module(s) are ``amrex.space1d``, ``amrex.space2d`` and ``amrex.space3d``, implemented in ``src/pyAMReX.cpp``.
+This import calls helper functions to initialize all other Python objects on import.
+
+Build System
+------------
+
+pyAMReX uses the :ref:`CMake build system generator <building-cmake>`.
+Each sub-folder contains a file ``CMakeLists.txt`` with the names of the source files (``.cpp``) that are added to the build.
+Do not list header files (``.H``) here.
+
+C++ Includes
+------------
+
+All pyAMReX header files need to be specified relative to the ``src/`` directory.
+
+- e.g. ``#include "pyAMReX.H"``
+- files in the same directory as the including header-file can be included with ``#include "FileName.H"``
+
+By default, in a ``MyName.cpp`` source file we do not include headers already included in ``MyName.H``. Besides this exception, if a function or a class
+is used in a source file, the header file containing its declaration must be included, unless the inclusion of a facade header is more appropriate. This is
+sometimes the case for AMReX headers. For instance ``AMReX_GpuLaunch.H`` is a façade header for ``AMReX_GpuLaunchFunctsC.H`` and ``AMReX_GpuLaunchFunctsG.H``, which
+contain respectively the CPU and the GPU implemetation of some methods, and which should not be included directly.
+Whenever possible, forward declarations headers are included instead of the actual headers, in order to save compilation time (see dedicated section below). In pyAMReX forward
+declaration headers have the suffix ``*_fwd.H``, while in AMReX they have the suffix ``*Fwd.H``.
+The include order (see `PR #874 <https://github.com/ECP-WarpX/WarpX/pull/874#issuecomment-607038803>`__ and `PR #1947 <https://github.com/ECP-WarpX/WarpX/pull/1947>`__) and `proper quotation marks <https://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html>`__ are:
+
+In a ``MyName.cpp`` file:
+
+1. ``#include "MyName.H"`` (its header) then
+2. (further) pyAMReX header files ``#include "..."`` then
+3. pyAMReX forward declaration header files ``#include "..._fwd.H"``
+4. AMReX header files ``#include <...>`` then
+5. AMReX forward declaration header files ``#include <...Fwd.H>`` then
+6. other third party includes ``#include <...>`` then
+7. standard library includes, e.g. ``#include <vector>``
+
+In a ``MyName.H`` file:
+
+1. ``#include "MyName_fwd.H"`` (the corresponding forward declaration header, if it exists) then
+2. pyAMReX header files ``#include "..."`` then
+3. pyAMReX forward declaration header files ``#include "..._fwd.H"``
+4. AMReX header files ``#include <...>`` then
+5. AMReX forward declaration header files ``#include <...Fwd.H>`` then
+6. other third party includes ``#include <...>`` then
+7. standard library includes, e.g. ``#include <vector>``
+
+Each of these groups of header files should ideally be sorted alphabetically, and a blank line should be placed between the groups.
+
+For details why this is needed, please see `PR #874 <https://github.com/ECP-WarpX/WarpX/pull/874#issuecomment-607038803>`_, `PR #1947 <https://github.com/ECP-WarpX/WarpX/pull/1947>`_, the `LLVM guidelines <https://llvm.org/docs/CodingStandards.html#include-style>`_, and `include-what-you-use <https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYU.md>`_.
+
+Forward Declaration Headers
+---------------------------
+Forward declarations can be used when a header file needs only to know that a given class exists, without any further detail (e.g., when only a pointer to an instance of
+that class is used). Forward declaration headers are a convenient way to organize forward declarations. If a forward declaration is needed for a given class ``MyClass``, declared in ``MyClass.H``,
+the forward declaration should appear in a header file named ``MyClass_fwd.H``, placed in the same folder containing ``MyClass.H``. As for regular header files, forward declaration headers must have
+include guards. Below we provide a simple example:
+
+``MyClass_fwd.H``:
+
+.. code-block:: cpp
+
+   #ifndef MY_CLASS_FWD_H
+   #define MY_CLASS_FWD_H
+
+   class MyClass;
+
+   #endif //MY_CLASS_FWD_H
+
+``MyClass.H``:
+
+.. code-block:: cpp
+
+   #ifndef MY_CLASS_H
+   #define MY_CLASS_H
+
+   #include "MyClass_fwd.H"
+   #include "someHeader.H"
+   class MyClass{/* stuff */};
+
+   #endif //MY_CLASS_H
+
+``MyClass.cpp``:
+
+.. code-block:: cpp
+
+   #include "MyClass.H"
+   class MyClass{/* stuff */};
+
+Usage: in ``SimpleUsage.H``
+
+.. code-block:: cpp
+
+   #include "MyClass_fwd.H"
+   #include <memory>
+
+   /* stuff */
+   std::unique_ptr<MyClass> p_my_class;
+   /* stuff */