Skip to content

ECJ Testing

Jump to bottom
Stephan Herrmann edited this page Nov 1, 2024 · 4 revisions

< JDT Core Programmer Guide | ECJ

For testing the compiler it is not necessary to ever run a maven build.

Running org.eclipse.jdt.core.tests.RunCompilerTests (or any of its constituents) as a JUnit (3) Plug-in Test suffices. For faster launching, open the "Main" page of the run dialog and choose "Headless Mode" as the application.

For selectively running only a single test method, or a group of test methods, see that each test class has a static block with a commented assignment to TEST_NAMES and similar. Uncomment and insert the name of the desired test and run the class. Note, that the given name will be used for prefix-matching, so tests testMethod1, ... testMethod13 can all be run by specifying "testMethod". Please try not to commit this change of TEST_NAMES.

Parameters of the compiler test suite

compliance

Specify at which compliance levels a test should be run by adding a VM argument like this (using a comma separated list of values):

-Dcompliance=1.8,14

Some compiler tests not only compile but also execute the compiled class files. For those ensure that the test is executed on a JRE that is the same or newer as the newest compliance level specified.

Not specifying a compliance level causes the tests to be executed for each supported compliance level compatible with the current running JRE, which can be very time consuming.

run.javac

This special test mode is used to compare the behaviors of ecj and javac.

As of https://github.com/eclipse-jdt/eclipse.jdt.core/issues/386 we have two levels of enabling:

  • -Drun.javac=enabled will enable this mode for the entire compiler test suite
  • -Drun.javac=optin enables this mode only for "blessed" classes that opt in to the mode. (this is done by setting AbstractCompilerTest.runJavacOptIn to true in a setUp() method).

If nothing else is specified, setting run.javac will pick up javac from the JDK running the tests.

You may optionally specify a different JDK by saying

  • -Djdk.root=<path-to-jdk> -- to specify exactly one JDK
  • -Djdk.roots=<paths-separated-by-pathseparator> -- to specify a list of JDKs

In any case, any version of javac will only be run when the current test compliance matches the version of javac. So, for effective run.javac mode, also -Dcompliance=... is relevant.

Jenkins jobs

Dedicated Jenkins jobs exist to test in this mode for different compliance levels:

The active job is triggered once per week (on weekends).

Issue tracking

Catalog of known differences

Known differences are documented using constants of type org.eclipse.jdt.core.tests.compiler.regression.AbstractRegressionTest.JavacTestOptions.Excuse during test invocation. For conveniently passing the correct "Excuse" in each affected test, it is recommended to use class "Runner" as shown below, and its field javacTestOptions.

jdt.flow.test.extra

Flow analysis has separate implementations for the first 64 variables and subsequent ("extra") variables. Since most tests only exercise the first implementation, a mode has been added to explicitly test the second implementation (by faking 64 additional variables).

-Djdt.flow.test.extra=true

For better test coverage, this mode is enabled for one of the time-scheduled jobs: eclipse.jdt.core-run.javac-10

Since that job is known to fail, watching for regressions of this job is particularly relevant.

jdt.performance.asserts

A few JDT tests include some sort of performance assertions, ideally not tied to specific wall time thresholds, but testing the characteristics at which complexity grows (linearly vs. O(2^n) etc).

Since these tests frequently fail on the fluctuating jenkins infrastructure of gerrit jobs, those assertions are disabled for gerrit:

-Djdt.performance.asserts=disabled

Other test runs still keep them enabled (the default).

When new performance-related tests are added to the suite, they should guard their performance assertions under the same flag, in code accessible as org.eclipse.jdt.core.tests.util.AbstractCompilerTest.PERFORMANCE_ASSERTS.

jdt.test.output_directory

This property will tell the test suite to create test files in a non-standard location (otherwise files will be created below the default temp dir).

Authoring / editing tests

different styles of test invocation

Inside each test method of the compiler test suite, an invocation like runConformTest, runNegativeTest passes the source file and additional parameters to the actual testing infra structure.

Traditionally, a myriad of overloads of these methods has been created to accommodate the needs of different tests, which makes picking the appropriate method tedious and still uninteresting parameters have to be provided in many cases.

More recently, class org.eclipse.jdt.core.tests.compiler.regression.AbstractRegressionTest.Runner has been created to enable providing exactly the relevant test parameters before invoking one of the run*Test() methods.

Clone this wiki locally