Dev guide darktable testing tools

This page provides guidance on how to use:

• an integration test suite focused on regression testing for changes to IOP modules
• how to use the tool darktable-cli to help with further ad-hoc testing
• the CMocka unit testing framework

Integration tests

The base suite of integration tests can be found in src/tests/integration. There is a detailed README.txt included.

The tests use an industry measure of colour distance, created by the CIE, called Delta E (dE.) The threshold in use by Darktable developers is currently dE < 2. A limit of zero is not practical as the complex colour space computations used by darktable WILL vary slightly depending on libraries, versions, floating point rounding differences, factoring across multiple threads, and underlying platform and OS.

The integration test scripts require some additional tools in order to be able to carry out this measurement. See below for a list of these pre-requisites; once they are installed you simply change to your src/tests/integration directory and run (for example)

./run 0001-exposure

To use these tests you require an environment which supports the following:

• ImageMagick's "compare" tool - note this is NOT the "compare" script shipped elsewhere in the current source tree
• a python3 environment, which must include:
  • opencv
  • numpy
  • colour (the package is called "colour-science" in most distros; note English spelling)

The intention is to include a test for every IOP. If you are testing changes to a module and there is no integration test please create one by following the instructions in README.txt or raise an issue.

• The PNG format is used when comparing images in these tests because it is lossless and should be pixel-accurate to the intended output of darktable's pipeline.
• The "expected" PNG result is managed within the source tree and was originally generated in @turbogit's development environment.
• The tests force LCMS2 off (otherwise they would test the accuracy of the upstream LCMS2, not darktable!)
• The test runs once with OpenCL disabled and once with it enabled, to allow comparison of results from these two codepaths in addition to the comparison with the known-good baseline image provided by the maintainers.
• The output png shows red pixels where significant differences between the images are found.

History

• These tests were added in PR #4818
• Issue 5203 added further explicit options to ensure test stability: memory limit, thread limit, disable LCMS2.

darktable-cli

Darktable includes a tool for running a set of operations repeatably from the command line. This is found in the bin subdirectory of the installed location. It is useful for carrying out repeatable, automated operations during testing. It is the basis for the included suite of pipeline tests discussed in the following section.

Usage: darktable-cli <input file> [<xmp file>] <output file> [options] [--core <darktable options>]

You can use this tool to:

• re-run existing XMP files on the same input, repeatably, in order to test how code or configuration changes affect the output;
• apply styles repeatably
• create consistent batch operations for use in testing performance

The --core flag will pass the remainder of the command line through to darktable itself, which is useful for setting debugging options.

Debug options useful for testing include:

• -d perf - adds profiling (timing) output to the console as darktable runs through the steps in a pipeline.
• --bench-module <modname>[,<modname2>...] - runs the named module(s) 50 times before continuing with pipeline processing and reports the overall time for each. <modname> is the basename of the shared library for the iop, which is the same as the base of its source file's name in src/iop (e.g. White Balance is 'temperature').
• -t <threads> sets the maximum number of threads used by OpenMP. This is useful for reducing to single-threaded if you suspect multithreading is causing problems, and also useful for determining whether more threads is actually helping with performance.

Notes on performance testing

-d perf provides figures for some specific parts of back-end processing within darktable only. There are multiple further stages (and caches) between this pure computation, all the way up to how the UI renders and displays the photo, so sometimes measuring front-end and back-end performance or measuring performance across different tools can give wildly varying results. For an example see issue #2724 where the original report of "sluggish" UI behaviour did not show up in the opencl_profiling debug figures at all, and was fixed by changing the relationship of the front-end and the back-end even though both were performing OK.

The default configuration for OpenCL is set to prioritise reponsiveness, which means that it will set priorities for all available OpenCL devices, but these are not absolute, and may also lead to situations where a slower OpenCL GPU picks up processing that would be better forced onto a faster GPU. There are also conditions in the default config where one of the four pipelines may execute on CPU rather than stopping and waiting for an available GPU if they are busy on tasks from the other three pipelines. Consider use of the + modifier (see OpenCL section of user manual) to force operations onto particular GPUs if that is important to your testing.

Further reading

• darktable-cli section of the user manual

Unit testing

In PR 4154 a framework for unit testing using the CMocka toolset was created. Pre-reqs:

• darktable must be built with -DBUILD_TESTING=On
• apt install libcmocka-dev or pacman -S mingw-w64-x86_64-cmocka