Skip to content

Unit Testing

Jump to bottom
pierce314159 edited this page Jun 7, 2022 · 7 revisions

Arkouda Unit/Functional Testing

We have a number of different kinds of tests in Arkouda both stylistically and language-based. In general we have:

  • Chapel based unit tests
  • Python based unit and end-to-end tests
  • Python based benchmark testing

Invoking the various test components can be a little tricky so this wiki entry is going to give a brief overview of how to run some of the tests and add some of your own!

Python tests

This is probably the easiest segment by far to read, write, and run tests. We use the pytest testing framework to run tests located in ${ARKOUDA_HOME}/tests (note the plural).

To run the tests you have two options

# Option 1
make test

# Option 2
pytest

How it works

pytest is configured by the pytest.ini file in the root of ${ARKOUDA_HOME}. You can look at this file to see which tests are configured. The python unit & end-to-end tests are located in the tests directory.

make test

The target for make test is linked to make test-python and executes python3 -m pytest -c pytest.ini along with any options included in $ARKOUDA_PYTEST_OPTIONS.

Run a test manually

To run tests in a test class or even a single test within a test class you can invoke pytest directly. Here are a few examples

# Run all tests in the CategoricalTest class (-v will print out the test name)
python3 -m pytest tests/categorical_test.py::CategoricalTest -v

# Run a single test from CategoricalTest named foo_test
python3 -m pytest tests/categorical_test.py::CategoricalTest::foo_test

Pytest vs. Python unittest

In the tests directory python test files you'll notice there is a bit of a mis-match between pure pytest and python's unittest. In short this is where we're at in the evolution of arkouda testing. At some point you may see a migration away from python unittest towards pytest and fixture based design, but pytest includes general support for working with and running unittest.

Adding your own test

There are two basic choices

  1. Add your unit test to an existing file using the same pattern of def test_.... inside the class extending ArkoudaTest. In general this takes care of standing up an arkouda_server instance for the lifetime of the test class (see setUpClass and setUp functions in most tests). If you need to test end-to-end functionality this is the way to go (just make sure you re-compiled your server-side code to pick up any Chapel-based changes). You should have a ready made ak client available for use along with the unittest based assertion functions.
  2. Create a new test file. For now you should follow the pattern of using unittest inside of your new test class if you're planning on testing end-to-end client-to-server functionality. See one of the existing tests as an example; extending ArkoudaTest will handle the process of standing up and connecting to an arkouda_server instance for you. IMPORTANT, to make sure your new test file is added to the pytest test runner, make sure you add it to the pytest.ini file!

In the future we may look into migrating or offering pytest based fixtures for use in unit & end-to-end testing in python.

Chapel Unit Tests

There are a number of server-side tests implemented in Chapel but they don't run automatically as part of the normal make, make test process. While make test-chapel and make test-all targets exist, they currently only compile the Chapel unit tests, they do not run them and compare to expected output.

Where

Chapel based unit tests currently exist at location ${ARKOUDA_HOME}/test (singular test).

How to run Chapel tests

The start_test utility is included in your build/install of Chapel under ${CHPL_HOME}/util/start_test. This is the Chapel test harness for compiling, running, and comparing test output for Chapel-based unit tests; make sure it is included on your PATH variable via which start_test on most *nix based systems.

To run the Chapel-based unit tests

start_test test

Again, this will compile the tests under the test/ directory, run the test, and compare the output to the *Test.good files which contain the expected output of the test. If you are making a new test file you should also make a .good file to accompany it. It may be blank (sometimes no news is good news ;) or contain some type of expected output from your test.

This runs test in a serial fashion, we have added a simple wrapper script which runs separate start_test processes in parallel. The wrapper is located in util/test/parallel_start_test.py, you can invoke the help message to see various usage

python3 util/test/parallel_start_test.py --help

The start_test utility has a number of supplemental files beyond the .good files which govern the execution of Chapel-side tests as well as a number of environment variables. These are outlined in Chapel's sub_test.py source code; see chapel-lang/chapel/util/test/sub_test.py. One example is that you may want to set a specific logging level for a specific unit test which is normally passed as a command line option to the arkouda_server. Here we would use a TestFileName.execopts file containing the additional command line options we'd like to pass for running all tests in that corresponding test file.

Writing a Chapel test

There are two main conventions for writing Chapel unit tests.

  1. Use the Chapel UnitTest module which gives you access to assertion procedures see UnitTest docs
  2. Create your own output to be compared to a .good file

Both methods of creating tests involve creating a proc main() {...} and calling your test.

Chapel UnitTest Module

A few key points to using the Chapel UnitTest module. You will need to add a use UnitTest statement to your import/use section which then gives you access to create a var t = new Test(); variable. This can be passed to your test procedure via something like proc myTest(test: borrowed Test)... and then you can use assertion statements along the lines of test.assertTrue(myThing == anotherThing); Take a look at the Chapel documentation on how to use UnitTest. Note: We are not currently compatible with the mason testing infrastructure.

Rolling your own

As stated previously you can have your test generate output to stdout and compare it to a .good file which should be named the same as your test file but swapping in the .good extension in place of .chpl. Here you would use a simple writeln statement to output the test value.

Example

You can combine both styles, this should give you a rough example of the basic usage, consider file named ExampleTest.chpl

use UnitTest;
use TestBase;
use IO;
use <your new module>;

proc testMeWithUnitTest(test: borrowed Test) throws {
    ... run your proc you want to test ...
    test.assertTrue( testSomeCondition );
}

proc anotherTest() {
    ... run the proc to test ...
    writeln("anotherTest");
    writeln("Expected >>>> 5, Actual >>>> %t".format(myVal));
}

proc main () {
    var t = new Test();
    testMeWithUnitTest(t);
    anotherTest();
}

The contents of your ExampleTest.good file should be:

anotherTest
Expected >>>> 5, Actual >>>> 5
Clone this wiki locally