THIS IS THE XSLT 3.0 TEST SUITE

This text can also be viewed online in the Public, freely accessible mail archive: https://lists.w3.org/Archives/Public/public-xsl-wg/2016Feb/0001.html

GETTING THE REPO

The current version of the XSLT 3.0 test suite is in GitHub at https://github.com/w3c/xslt30-test

Prior to 31 March 2019 it was in a Mercurial (Hg) repository: https://dvcs.w3.org/hg/xslt30-test/

If you don't feel like using a DVCS, you can download a ZIP archive from the above link. Changes can then be mailed to me and we will add them to the repository. However, GitHub pull requests are preferred.

For committing changes, unless you have write access, you can submit a pull request with your proposed changes. New tests are always welcome. Changes to existing tests require more care, and should always carry some explanation or justification, especially if the test is long-established and/or is not your own. If the change is because you believe the existing expected results are incorrect, you should raise a bug report and achieve consensus for the change before submitting it. Changes to test metadata, for example adding a previously unnoticed dependency, do not require the same level of formality.

GETTING AROUND

At first the test suite may look a bit daunting. A lot of documentation can be found in the catalog-schema.xsd in the /admin directory. When editing or adding tests, you should validate the XML against this schema (oXygen can do this automatically and will also give you tooltip and other context-sensitive help).

The tests themselves are structured as follows:

Catalog.xml: this contains the catalog of all tests. It is a list of elements that themselves point to named test-sets in other XML files. Typically you will not need to edit this file unless you are going to add a new category or group of tests. In oXygen you can hit Ctrl-Enter on any of these test-set @file attributes to go directory to the test-set XML file.

/tests/: this directory contains all the tests. On top of each subdirectory is a test-set XML file, say _accept-test-set.xml which typically has a name that matches the category. When working on a category, this is the file to edit. Test sets are organized in main groups, like "attr" for testing attributes of XSLT instructions and "decl" for XSLT declarations. Under each of these is a further specialization, like "import-schema" directory for xsl:import-schema tests.

_xxxx-test-set.xml: files on top of each test-set directory start with an underscore to easily find them in the directory. The structure of test-sets has a number of on top which describe some metadata of the test, mainly the XSLT and XML source files, collections or other dependencies. The element sets whether tests are dependent on XSLT 3.0 or 2.0 or a certain other feature, like higher-order-functions. The bulk of each of these files consists of a long list of elements. These contain the meta-information for the actual tests. The easy thing to do is to take a test that closely resembles the metadata you are going to need for your next test, copy the element, update the name (it must be unique, the numbering is up to you) and update the XSLT source. If necessary, update other dependencies as well.

WRITING TESTS

First, I [Abel Braaksma] set up oXygen to have two Transformation Scenarios for each processor I want to use: one that takes as input an XSLT source and an XML source by the same name as the XSLT source (with only the extension different) and one that runs an XSLT source without any input, using the "xsl:initial-template" feature of XSLT 3.0.

Here's the way I do it then:

1. Open the _xxxx_test-set.xml in oXygen
2. Locate a test that is close to what you want to do
3. Create or copy the XSLT file, number it closely to the test, say accept-010.xsl
4. Add a source file with the same name, accept-010.xml (even if such file already exists, in which case the in the test set does not need to change, but this is just easier one-click testing)
5. Write the test using XSLT features that you know should work
6. Run it successfully
7. Update the test with features you are unsure should work (i.e., the part you want to actually test)
8. Run again, whether fail or success is now irrelevant, but from step (5) you know your XSLT is probably valid
9. Copy a bunch of with similar parameters that test variants of your current scenario

With all this, I usually use the new XSLT feature for static parameters heavily, because that makes it easier to write many tests without requiring a new XSLT stylesheet for each test. This is also the way Michael Kay often does it.

COMMITTING TESTS

Once you are done, commit your changes. Always write a comment when committing so that others know what you added and why. If there is a bug entry that is relevant to the tests, mention that.

Do not add redundant files. You can right-click and ignore them with TortoiseHg so that you do not accidentally commit them, or just don't select them in the commit window of your Mercurial client.

Thanks and happy testing!