Twister: test hierarchy, nomenclature and output

[PerMac]

Hi. As it was already seen by users (#30100), the nomenclature used in testing with Twister is confusing. The same "object" can be referred to by different names and there still exists some wording issues in Twister output. I created a small presentation (Zephyr test hierarchy and nomenclature.pdf ) showing the structure of tests and the names I suggest we can start using to avoid confusion. Most of the naming is already used in zephyr, however, not consistently. One of the currently most confusing is "test case" https://docs.zephyrproject.org/latest/guides/test/twister.html#test-cases. The presentation also explains the current output of Twister, what and how is counted, and what errors are still present there.
The goal of this discussion is to define/agree on the testing nomenclature used at different levels. Once it is done we can start fixing the documentation so the naming is consistent.
I think that we can also have here a good place to discuss what and how detailed we would like to have in the twister output.
In the presentation, I put in square brackets optional expansions of names. I think that in the documentation it is better to use the full name to reduce confusion. I tried to sort it out in a way that short names are unique for each "object".

[nashif]

@PerMac maybe attach the PDF in here instead of the external link

[PerMac]

I didn't know how to do this. It seems it can be done by just dragging and dropping a file into comment :)

[PerMac]

Zephyr test hierarchy and nomenclature.pdf

[nashif]

seems reasonable in general with the following comments:

• some yaml configuration describe tests, not configurations or scenarios, most in samples:

common:
    tags: introduction
    integration_platforms:
      - native_posix
    harness: console
    harness_config:
      type: one_line
      regex:
        - "Hello World! (.*)"
tests:
  sample.basic.helloworld:
    tags: introduction

So we need a better way to describe those and keep things consistent. There are also tests that are not ztest based which follow the same model.

• We need to reflect this in the generated junit files, so for example, for one test on 1 single platform: : twister -p native_posix -T tests/kernel/semaphore/ --platform-reports currently we get:

<?xml version="1.0"?>
<testsuites>
  <testsuite name="native_posix" time="2.017428" tests="23" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="zephyr-v2.4.0-3165-ge4f95863f98e"/>
    </properties>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_define" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_init" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_thread2thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_thread2isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_reset" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_count_get" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_from_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_from_thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_no_wait" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_no_wait_fails" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_fails" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_forever" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_multiple" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_take_from_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_correct_count_limit" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multiple_threads_wait" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_measure_timeouts" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_measure_timeout_from_thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multiple_take_and_timeouts" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multi_take_timeout_diff_sem" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_queue_mutual_exclusion" time="2.017428"/>
  </testsuite>
</testsuites>

testsuite, classname and name need to reflect whatever we have in the proposal, so testsuite would be the actual testsuite for example. When you run multiple testsuite on a platform, we need to end up with multiple testsuite in the XML file.

testsuite could be whatever we have in the source file, for example here:

        ztest_test_suite(test_semaphore,
                         ztest_user_unit_test(test_k_sem_define),
                         ztest_user_unit_test(test_k_sem_init),

so, test_semaphore. or semaphore or some other name we can come up with.

[LeiW000]

<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_queue_mutual_exclusion" time="2.017428"/>

I still want to suggest to add "test_" back to the test case name in junit file. For example, kernel.semaphore.**test_**sem_queue_mutual_exclusion.

[erwango]

The proposal seems fine, but I don't think the unit_test (or test case as called in the pdf), should be reported at high level.
If a configuration instance (a build) as a test case failed, it should be reported as failed. We can provide ways to get more details using additional commands, but I don't think twister should report this detail of failure. So on that particular aspect, the current way of working is fine.

I agree on other parts of the proposal.

[PerMac]

I saw @nashif using the name "test app" in the testing channel on Slack. Maybe it is a better name for "[test] configuration [instance]" from my proposal? IMO "test app" feels more intuitive, as something that can be executed on a device/qemu?

[erwango]

What about "test build" or "test binary" ? That's prosaic but unambiguous.

[nashif]

to move forward with this:

• extract all of the definitions discussed here and the in the PDF and submit those to the documentation, preferably in a new section. Basically describe the hierarchy with new terminology cleanly. I suggest to do this in a forward looking way, meaning that we should not be forced to accommodate things that are suboptimal and not well-defined.
• Map the current definitions as we have them in twister, testcase.yaml, ztest and junit and identify where changes need to be made. I know for a fact that TestSuite, TestCase and TestInstance in twister do NOT map correctly to any of the other areas. TestSuite in twister is actually not a Test Suite as used in junit or ztest for example, same for TestCase.. So we need to fix this across the board and eliminate confusion.
• Start from ztest (and probably look at Unity as well, we might start using this at some point) and go up: ztest -> testcase.yaml -> twister -> junit (reporting in general)

[nashif]

@PerMac was toying with output from twister based on recent proposed changes to terminology can have the following output for

twister -c --platform-report -p native_posix -T tests/kernel/sched/schedule_api/ -T tests/kernel/fifo/

Please give feedback.

report.xml.txt

[nashif]

another optimization:

report.xml.txt

[PerMac]

I like the new output. The thing I caught is that the structure for samples don't match (BTW how it ended in the report???):
testsuite name="samples/hello_world/sample.basic.helloworld"
as compared to:
testsuite name="test_fifo_timeout"
I am in the middle of going through the PR with the changes. Haven't run anything by myself yet. Let me know if the PR is in a runnable state and I will set our internal CI to checkout on it so I can see how Jenkins is handling this format.

In addition, I would opt for adding .default to the scenario name if there is only module.submodule in the name. I believe it would make it cleaner, that the cases like below are on the same level, just differ by the scenario (for me it looks like .no_timeslicing is at lower level due to parent.child structure in objective languages)
<testcase classname="kernel.scheduler" name="test_bad_priorities" time="0"/>
and
<testcase classname="kernel.scheduler.no_timeslicing" name="test_bad_priorities" time="0"/>

[nashif]

(BTW how it ended in the report???):

oh, I added that to test the sample case.

The thing I caught is that the structure for samples don't match

we need to find a better way, right now I am just filling the gaps. One possibility is to autogenerate that from the sample name, i.e. test_hello_world or have the sample name as is. Another options is to add this is a key/value in the yaml file.

I am in the middle of going through the PR with the changes. Haven't run anything by myself yet. Let me know if the PR is in a runnable state and I will set our internal CI to checkout on it so I can see how Jenkins is handling this format.

it should be, however the changes leading to this output are still in my tree only.

In addition, I would opt for adding .default to the scenario name if there is only module.submodule in the name. I

fine with me.

[PerMac]

The above relates already to the second file

[PerMac]

@nashif I think there still exists differentiation on sample.yaml and tests.yaml in you PR, right? Any reasons why keeping both of them? Maybe we can unify both names to tests.yaml? Those are treated the same in twister, no?

[nashif]

This will follow, I did not spend much time on this yet, should be easy and mostly involve renaming the files from sample.yaml to tests.yaml