[ 222] Developer Docs: Using the testing environment

docs/developers/testing.md

@@ -14,7 +14,7 @@ Additionally, there is a set of tests having to do specifically with federated z
 
 ### How to run the tests
 
-The iRODS Testing Environment is the primary tool for verifying the iRODS server software. If you do not want to use the iRODS Testing Environment, you must install the iRODS server packages on some host and run the following command **as the iRODS service account user**:
+The [iRODS Testing Environment](#irods-testing-environment) is the primary tool for verifying the iRODS server software. If you do not want to use the iRODS Testing Environment, you must install the iRODS server packages on some host and run the following command **as the iRODS service account user**:
 ```
 python /var/lib/irods/scripts/run_tests.py
 ```
@@ -59,7 +59,7 @@ The list of tests can be found here: [https://github.com/irods/irods/blob/main/u
 
 ### How to run the tests
 
-The iRODS Testing Environment is the primary tool for verifying the iRODS server software. If you do not want to use the iRODS Testing Environment, you must build the iRODS server packages with the `IRODS_ENABLE_ALL_TESTS` CMake option enabled, install the iRODS server packages on some host, and run the following command **as the iRODS service account user** for each `TEST_NAME` in the list of tests:
+The [iRODS Testing Environment](#irods-testing-environment) is the primary tool for verifying the iRODS server software. If you do not want to use the iRODS Testing Environment, you must build the iRODS server packages with the `IRODS_ENABLE_ALL_TESTS` CMake option enabled, install the iRODS server packages on some host, and run the following command **as the iRODS service account user** for each `TEST_NAME` in the list of tests:
 ```
 /var/lib/irods/unit_tests/TEST_NAME
 ```
@@ -111,3 +111,118 @@ If you added a unit test file, you must add your new test to the `unit_tests_lis
     // ... other tests ...
 ]
 ```
+
+## iRODS Testing Environment
+
+The iRODS Testing Environment is a Docker Compose-based project written in Python that is used by the Consortium to test the iRODS server and its plugins. The Testing Environment is driven by a series of scripts to run the tests on a number of different platforms with various database backends and other configuration options. This is the primary tool used for day-to-day testing for iRODS developers as well as for verification in releasing the iRODS server. In addition to running tests, the Compose projects can be useful for manually testing things and quickly deploying specific environments for reproducing issues or developing a proof of concept.
+
+This documentation will briefly describe a few useful ways to employ the iRODS Testing Environment. The README in the project's GitHub repository provides extensive details on the purpose and usage for the various scripts: [https://github.com/irods/irods_testing_environment/blob/main/README.md](https://github.com/irods/irods_testing_environment/blob/main/README.md) Each script available in the project also has help text available with the `--help` option which describes the available options and their purposes.
+
+### Getting started
+
+In order to configure your environment to use the iRODS Testing Environment, using a virtual environment is highly recommended. Here's how to set up a virtual environment and install the required Python packages:
+```
+virtualenv -p python3 ~/irods_testing_environment
+source ~/irods_testing_environment/bin/activate
+pip install docker-compose GitPython
+pip freeze
+```
+
+#### If you cannot install `docker-compose` ...
+
+The iRODS Testing Environment is built on top of the now-defunct Python implementation of Docker Compose called `docker-compose`: [https://github.com/docker/compose/tree/1.29.2](https://github.com/docker/compose/tree/1.29.2) `docker-compose` is being phased out by Docker and you may experience problems installing it via `pip`.
+
+If this happens to you, try doing the following:
+
+Clone the Docker Compose Git repository:
+```bash
+git clone https://github.com/docker/compose
+```
+
+Check out the latest tag which was still using the Python implementation:
+```bash
+cd compose
+git checkout 1.29.2
+```
+
+At this point, you can make the modifications needed to fix any problems you may encounter.
+
+Once done, back out and pip install the local directory:
+```bash
+cd -
+pip install ./compose
+```
+
+### How to run core tests: `run_core_tests.py`
+
+`run_core_tests.py` is the main script for running the iRODS "core" test suite (see [Core test suite](#core-test-suite)).
+
+Here is the minimal form for running the test suite for a particular `OS`, `OS_VERSION`, `DATABASE`, and `DATABASE_VERSION`:
+```
+python run_core_tests.py --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION
+```
+
+This will serially run the entire test suite using the latest release of the iRODS server. This will create a container for the iRODS server configured as the sole catalog service provider, a container for the database where the catalog is stored, and a network so that they can communicate as if it were a distributed environment.
+
+#### Using locally built iRODS packages
+
+The `--irods-package-directory` option allows the user to provide iRODS packages as files on the host machine. This is the standard way for iRODS developers to test modifications to the iRODS server software:
+```
+python run_core_tests.py \
+    --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION \
+    --irods-package-directory /path/to/directory/containing/irods/package/files
+```
+
+#### Using concurrency to reduce test time
+
+The `--concurrent-test-executors` option allows the user to specify how many iRODS zones to deploy and use for splitting up the list of tests to run. Note that this option can quickly increase the number of containers created by the Compose project (2 containers per test executor), so make sure to choose a number appropriate to the host system's capabilities. Here is an example of how to use the option:
+```
+python run_core_tests.py \
+    --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION \
+    --concurrent-test-executors 4
+```
+This will deploy 4 iRODS zones (4 * (1 catalog service provider + 1 database) = 8 containers) and will run tests from the list of tests by taking them from a queue and concurrently executing on the catalog service providers in each zone.
+
+The test suite takes 8-10 hours to complete when run serially. With 4 concurrent executors on an 8-core host system, the test suite takes about 2 hours to complete.
+
+#### Running specific tests
+
+The `--tests` option allows the user to specify a space-delimited list of specific tests to run. Here is an example of how to use the option:
+```
+python run_core_tests.py \
+    --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION \
+    --tests test_ils test_icp test_ihelp
+```
+This will deploy a single iRODS zone and execute the following tests in order on the catalog service provider: `test_ils`, `test_icp`, and `test_ihelp`. This can be used in conjunction with the `--concurrent-test-executors` option, but the tests will not be run in a deterministic order as the concurrent executors will take tests from the queue as they complete them.
+
+### How to run unit tests: `run_unit_tests.py`
+
+`run_unit_tests.py` is the main script for running the iRODS "unit" test suite (see [Unit test suite](#unit-test-suite-catch2)).
+
+Here is the minimal form for running the test suite for a particular `OS`, `OS_VERSION`, `DATABASE`, and `DATABASE_VERSION`:
+```
+python run_unit_tests.py --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION
+```
+
+This will serially run the unit test suite using the latest release of the iRODS server. This will create a container for the iRODS server configured as the sole catalog service provider, a container for the database where the catalog is stored, and a network so that they can communicate as if it were a distributed environment. A running iRODS server is required because the Catch2 tests are not actually unit tests. Most of the same options available for `run_core_tests.py` are available for this script as well.
+
+The longest unit test takes about 2 minutes to complete.
+
+### How to stand up an iRODS zone: `stand_it_up.py`
+
+`stand_it_up.py` simply stands up an iRODS zone as specified. The minimal iRODS zone is a single catalog service provider and the database with which is communicates to maintain the catalog. This is the default behavior for this script.
+
+Here is the minimal form for creating a minimal iRODS zone using the latest release of the iRODS server for a particular `OS`, `OS_VERSION`, `DATABASE`, and `DATABASE_VERSION`:
+```
+python stand_it_up.py --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION
+```
+
+#### Adding catalog service consumers
+
+The `--consumer-instance-count` option allows the user to specify how many catalog service consumers to deploy alongside the catalog service provider (of which there can only be one at this time). Here is an example of how to use the option:
+```
+python run_core_tests.py \
+    --project-directory projects/OS-OS_VERSION/OS-OS_VERSION-DATABASE-DATABASE_VERSION \
+    --consumer-instane-count 3
+```
+This will deploy a single iRODS zone with 1 catalog service provider, 1 database instance, and 3 catalog service consumers (5 containers).