Support new standard for reporting test results in XML format

[sebastianbergmann]

ota4j-team/opentest4j#9 (comment)

[nfalco79]

The JUnit5 XSD file referenced in #2964 is not by junit. It is a test resource get from jenkins xunit plugin.

[keradus]

ages ago, we found sth like JUnit v10:
https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/2.12/doc/junit-10.xsd
I believe it was found in Travis repos or somewhere like that

(yeah, no official xsd so far, sadly)

---

also, after issue title got change, you may be interested in checkstyle (don't worry, it's not about only style, but any tool reporting), also widely used on different CI runners:
https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/2.12/doc/checkstyle.xsd

[ColinLaws]

I hope this input is in the right spot or is appropriate for this thread. I have been having issues with using the test output of PHPUnit with systems that support JUnit. Proper JUnit support would be ideal (at least in the scope of my own personal implementation), as I have seen JUnit as a widely accepted test format.

[keradus]

Proper JUnit support would be ideal

issue is that nobody knows how proper support would look alike, as there is no official data scheme for JUnit format

[nfalco79]

Proper JUnit support would be ideal

issue is that nobody knows how proper support would look alike, as there is no official data scheme for JUnit format

yes and no.
JUnit java library has not define an official schema for XML report. You can choose to follow one of the most supported schemas (maven surefire or junit ant task). Or define you own schema (away from junit) for php (to cover your needs) and ask to support it to reporting CI tools.

[ColinLaws]

@keradus Forgive me, I'm definitely not the expert when it comes to this. I'm just an end user who consumes PHPUnit and would like to use it in Azure DevOps (and they don't support nested test suites). I wish everyone could agree on an open testing standard, but I don't know if there'd be a way to make it fit everyone's needs given there may be language specific data.

[mindplay-dk]

JUnit 5 does come with an XSD:

https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/resources/jenkins-junit.xsd

This appears to match most of the third-party XSD's I've been able to find.

Nested testsuite elements? I'm surprised it works - does anything besides PHPUnit emit those?

(Kind of crazy how the entire OS world is using JUnit XML as the de-facto standard, while literally nobody seems to know exactly what it is they supposedly agree on - and we all just go along with it...)

[epdenouden]

This appears to match most of the third-party XSD's I've been able to find.

Lovely, that sounds just like the kind of well-described standard to suit a testig framework. 😨
Refactoring the event and logging subsystem is one of my chew toys and I've added the JUnit 5 reference to my reading list. Thanks!

[nfalco79]

JUnit 5 does come with an XSD:

https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/resources/jenkins-junit.xsd

This appears to match most of the third-party XSD's I've been able to find.

Nested testsuite elements? I'm surprised it works - does anything besides PHPUnit emit those?

(Kind of crazy how the entire OS world is using JUnit XML as the de-facto standard, while literally nobody seems to know exactly what it is they supposedly agree on - and we all just go along with it...)

JUnit does not come with an XSD. That XSD was copy for test purpoise (platform-tests project) by the xunit plugin (read the header of XSD). That file was created by Gregory Boissinot, the original author of jenkins xunit plugin making a sort of mixup of all cases found.

[mindplay-dk]

JUnit does not come with an XSD

Well, it's part of the test-suite - so, in my optic, it's part of the specification.

Regardless of where it originated, this is the spec to which the JUnit codebase itself adheres.

Or not?

[epdenouden]

My main concern is before allocating effort: is the standard (XSD or not) stable, at least for now? I'm happy to implement it in PHPUnit if so.

[epdenouden]

Well, it's part of the test-suite - so, in my optic, it's part of the specification.

I would agree if and only if the XSD has been included for this purpose. Many test suites also include tests for scenarios that cannot be rolled back because the development community adopted another 'forever beta' thing.

In PHPUnit's own collection there are a few guards for old logger behaviour, which frankly isn't great.

[nfalco79]

Well, it's part of the test-suite - so, in my optic, it's part of the specification.

Regardless of where it originated, this is the spec to which the JUnit codebase itself adheres.

Or not?

To be considered as a specific it should be explicited in the documentation. Since you had find a test resource this does not mean that it's the official schema documentation. The name of the file jenkins-junit.xsd should be another signal that has nothing to do with it JUnit official schema. You are free to ask to them about if that is an official schema for the JUnit (I would rememeber that till JUnit4 the library does not produce any report in any format JSON or XML or other)

[mindplay-dk]

Right, I don't disagree, it obviously isn't the official schema, but... there isn't one.

This appears to be the closest thing to be found in their source-repository or anywhere.

[nfalco79]

but... there isn't one.

The point is that JUnit it's just a library to run test. It does not to produce report so there is not reason to define any standard. Who use JUnit an engine and than produce a report than it can define a standard (that could be also in other format like JSON). For now there are two historical executor widely used: Maven and Ant. Both of them defined an own schema for XML report and they are completely different from one another. This is the reason of this issue, define a PHPUnit standard (without any restriction to JUnit) to fit the PHP needs. But till now I did not see any movement.

[sebastianbergmann]

Maybe we should create our own logging format for PHPUnit. (Yes, I know, https://xkcd.com/927/.)

To some degree this already exists, thanks to @theseer, with the logfiles that are produced using --coverage-xml. The generator for that would need to be refactored (and moved, at least partially, from php-code-coverage to PHPUnit itself) to only optionally include code coverage data.

Or we wait and hope that ota4j-team/opentest4j#9 (comment) leads to something we can adopt.

[mindplay-dk]

The generator for that would need to be refactored (and moved, at least partially, from php-code-coverage to PHPUnit itself) to only optionally include code coverage data.

What would happen to those of use who use php-code-coverage independently of php-unit?

[sebastianbergmann]

Thank you for raising this point, @mindplay-dk. This would need to be taken into account.

[staabm]

ages ago, we found sth like JUnit v10:
https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/2.12/doc/junit-10.xsd
I believe it was found in Travis repos or somewhere like that

(yeah, no official xsd so far, sadly)

also, after issue title got change, you may be interested in checkstyle (don't worry, it's not about only style, but any tool reporting), also widely used on different CI runners:
https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/2.12/doc/checkstyle.xsd

Having checkstyle support would be great because with that we also easily get github checks annotations via angeneric script which is already compatible with phpstan, psalm and php-cs-fixer (and every other tool able to generate a checkstyle-formatted report)

https://github.com/staabm/annotate-pull-request-from-checkstyle/

[staabm]

@sebastianbergmann would you accept a PR adding checkstyle xml reports?

as you can see from https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/2.16/src/Report/CheckstyleReporter.php#L35 the implementation would not be complex

[sebastianbergmann]

No. I do not believe that Checkstyle XML is a good fit for test results.

[staabm]

did you see the checks api result-printer https://github.com/mheap/phpunit-github-actions-printer ?

we could support this kind of feature easily when the report would be generated in checkstyle format (with a general purpose checkstyle to github annotations converter like https://github.com/staabm/annotate-pull-request-from-checkstyle)

[sebastianbergmann]

Superseded by #4321.