jQAssistant Test Impact Analysis Plugin

Table of Contents

• How It Works
• Prerequisites
• Setup
• Creating And Running A Test Suite
• Selecting Specific Tests
  • Integration Tests
  • Impact On Artifacts
• Test Gap
• Configuration
• Feedback
• Acknowledgements

NOTE

This project has been archived and is no longer actively developed. Please refer to the existing releases and documentation.

The project provides a jQAssistant plugin for generating test suites based on changes in the version control system (i.e. Git). This allows to selectively run expensive tests to provide fast feedback. Furthermore an experimental feature identifies potential test gap, i.e. changed or added Java classes without unit tests.

How It Works

The test impact analysis covers the following steps:

1. Determine source files that have been changed in the Git repository by evaluating

   • the last commit in the current branch or

   • all commits in the current branch that are not contained in a given base (i.e. reference) branch

2. Match the changed source files to their corresponding Java types (classes, interfaces, enumerations or annotations)

3. Find JUnit test classes that have a dependency to these types and write their file names to test suite files.

The plugin provides concepts (i.e. queries) and a report implementation for jQAssistant to perform these steps.

The test suites are represented by files which can be passed as parameters to Maven Surefire Plugin or Maven Failsafe Plugin via the includesFile option.

Prerequisites

• Java 8 or higher

• Maven 3.2.5 or higher

• jQAssistant 1.6.0 or higher (see below)

Setup

The plugin can be enabled in a Maven based project by adding it as a dependency to the jQAssistant Maven plugin:

pom.xml

<build>
    <plugins>
        <plugin>
            <groupId>com.buschmais.jqassistant</groupId>
            <artifactId>jqassistant-maven-plugin</artifactId>
            <version>1.10.0</version>
            <executions>
                <execution>
                    <id>default-cli</id>
                    <goals>
                        <goal>scan</goal>
                        <goal>analyze</goal>
                    </goals>
                    <configuration>
                        <failOnSeverity>MAJOR</failOnSeverity>
                        <warnOnSeverity>MINOR</warnOnSeverity>
                        <useExecutionRootAsProjectRoot>true</useExecutionRootAsProjectRoot>
                        <scanIncludes>
                            <scanInclude>
                                <path>${project.basedir}/.git</path>
                            </scanInclude>
                        </scanIncludes>
                        <reportProperties>                                               <!--(1)-->
                            <testImpactAnalysis.report.directory>${session.executionRootDirectory}/target/testimpactanalysis</testImpactAnalysis.report.directory>
                        </reportProperties>
                    </configuration>
                </execution>
            </executions>
            <dependencies>
                <dependency>                                                             <!--(2)-->
                    <groupId>org.jqassistant.contrib.plugin</groupId>
                    <artifactId>jqassistant-test-impact-analysis-plugin</artifactId>
                    <version>1.10.0</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

1. Configures the directory where the jQAssistant test impact analysis plugin will create the files (one per artifact)

2. Declares the plugin as dependency for jQAssistant

Furthermore profiles should be defined for making execution easier:

pom.xml

<profiles>
    <profile>
        <id>create-commit-testsuite</id>                                                                     <!--(1)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForLastGitCommit</concept>            <!--(2)-->
                        </concepts>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>create-branch-testsuite</id>                                                                     <!--(3)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <concepts>
                            <concept>test-impact-analysis:SurefireSuiteForCurrentBranch</concept>            <!--(4)-->
                        </concepts>
                        <ruleParameters>
                            <testImpactAnalysisGitBaseBranch>heads/master</testImpactAnalysisGitBaseBranch>  <!--(5)-->
                        </ruleParameters>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>

    <profile>
        <id>run-testsuite</id>                                                                               <!--(6)-->
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <includesFile>${session.executionRootDirectory}/target/testimpactanalysis/${project.artifactId}</includesFile>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>com.buschmais.jqassistant</groupId>
                    <artifactId>jqassistant-maven-plugin</artifactId>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profile>

1. Defines the profile to be used for creating a test suite based on the last Git commit

2. Activates the concept test-impact-analysis:SurefireSuiteForLastGitCommit

3. Defines the profile to be used for creating a test suite based on the commits within the current Git branch

4. Activates the concept test-impact-analysis:SurefireSuiteForCurrentBranch

5. Defines the base branch to use for determining changes in the current branch (optional, default: "heads/master")

6. Defines the profile for running the tests defined in the generated test suite

Creating And Running A Test Suite

The profiles create-commit-testsuite and create-branch-testsuite are used to create the required test suite files. Either one of the following commands needs to be executed:

mvn clean verify -Pcreate-commit-testsuite

mvn clean verify -Pcreate-branch-testsuite

The test suite files (one per artifact) are available in the 'target/testimpactanalysis' folder of the module where Maven has been executed, e.g.

target/testimpactanalysis/spring-petclinic

org/springframework/samples/petclinic/web/OwnerControllerTests.java
org/springframework/samples/petclinic/web/PetControllerTests.java
org/springframework/samples/petclinic/web/PetTypeFormatterTests.java
org/springframework/samples/petclinic/web/VetControllerTests.java
org/springframework/samples/petclinic/web/VisitControllerTests.java

The following command triggers a build which only executes the tests which are defined in the generated test suite files:

mvn verify -Prun-testsuite

Selecting Specific Tests

It is possible to select only specific tests to be included in the test suite files. Therefore specific rules can be created using Cypher (Neo4j query language) which perform required filtering. The rules must be located in a file that can be interpreted by jQAssistant. Allowed formats are Asciidoc or XML, the files must be located in the folder "jqassistant" in the root module of the Maven project:

Example project structure

my-project/
          /jqassistant/my-rules.adoc
          /module1/pom.xml
          /module2/pom.xml
          /pom.xml

Integration Tests

A case might be that only integration tests shall be selected for execution by the the Maven Failsafe Plugin. The following two rules select test classes by checking their name for the suffix "IT":

jqassistant/my-rules.adoc

= My Rules

[[my-rules:IntegrationTestsForLastGitCommit]]                                       // (1)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByLastGitCommit",reportType="surefire-suite"]
.Reports all integration test classes affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"                                                          // (2)
RETURN
  artifact as Artifact, collect(test) as Tests
----

[[my-rules:IntegrationTestsForCurrentGitBranch]]                                    // (3)
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TestsAffectedByCurrentGitBranch",reportType="surefire-suite"]
.Reports all integration test classes affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)
OPTIONAL MATCH
  (artifact)-[:CONTAINS]->(test:Affected:Test)
WHERE
  test.name ends with "IT"
RETURN
  artifact as Artifact, collect(test) as Tests
----

1. Declares the concept for integration tests affected by the last Git commit

2. The filter clause for test classes having a name with the suffix "IT"

3. Declares the concept for integration tests affected by commits within the current Git branch

The result of the concepts are reported as "surefire-suite". This report type requires each row to provide two columns:

Artifact

The artifact where affected test classes are located.

Tests

The collection of affected tests for the artifact.

For activating the rules the above described Maven profiles need to be adopted accordingly.

Impact On Artifacts

For projects consisting of multiple Maven modules it might be interesting to select all tests in Maven modules that depend on modules where Java types have been changed. The according rules would be as follows:

jqassistant/my-rules.adoc

= My Rules

[[my-rules:TestsInAffectedArtifactsForLastGitCommit]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByLastGitCommit,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by the last Git commit.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

[[my-rules:TestsInAffectedArtifactsForCurrentGitBranch]]
[source,cypher,role=concept,requiresConcepts="test-impact-analysis:TypesChangedByCurrentGitBranch,junit4:TestClass",reportType="surefire-suite"]
.Reports all test classes of artifacts affected by commits in the current Git branch.
----
MATCH
  (:Maven:Project)-[:CREATES]->(artifact:Artifact)-[:CONTAINS]->(:Type:Changed),
  (:Maven:Project)-[:CREATES]->(affectedArtifact:Artifact),
  shortestPath((affectedArtifact)-[:DEPENDS_ON*0..]->(artifact))
WITH DISTINCT
  affectedArtifact
MATCH
  (affectedArtifact)-[:CONTAINS]->(test:Type:Test)
RETURN
  affectedArtifact as Artifact, collect(test) as Tests
----

Test Gap

Note	This feature is considered experimental, Feedback is highly appreciated.

The plugin provides two additional constraints for test gap analysis. Both determine public methods of changed Java types that are not invoked by JUnit test methods:

test-impact-analysis:TestGapForLastGitCommit

Determines changes from the last Git commit

test-impact-analysis:TestGapForCurrentGitBranch

Determines changes within the current Git branch

The constraints may be verified from the command line:

mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForLastGitCommit

mvn verify -Djqassistant.constraints=test-impact-analysis:TestGapForCurrentGitBranch

If jQAssistant is already used in the Maven project it is recommended to include the desired constraint in a group that is executed:

jqassistant/my-rules.adoc

= My Rules

[[default]]
[role=group,includesConstraints="test-impact-analysis:TestGapForCurrentGitBranch"]
== Default Rules

Configuration

The Surefire Report plugin accepts several options that might be passed as reportProperties in the configuration section of the jQAssistant Maven plugin:

Property	Description	Default
testImpactAnalysis.report.directory	Specifies the directory where the test suite files will be written	jqassistant/report
testImpactAnalysis.surefire.file	If provided all affected test names will be written to one file with that name (relative to the directory)
testImpactAnalysis.surefire.artifactColumn	The name of the column providing the artifact containing a test	Artifact
testImpactAnalysis.surefire.testsColumn	The name of the column providing the collection of tests for a specific artifact	Tests

Note	The properties specifying the artifactColumn and testColumn must reflect the columns used in the RETURN clause of the query, e.g. by default 'RETURN artifact as Artifact, collect(test) as Tests'.

Feedback

Please report any issues here.

Acknowledgements

The plugin could not provide its functionality without the support of the following open source projects:

• jQAssistant

• Git Plugin for jQAssistant

• Neo4j

• Apache Maven