Board Classifications

[nashif]

Abstract

Add new board class property for boards (in their yaml file) which signifies the class of those platforms or boards. For example we could have the following classes:

• Development Boards / Reference: Boards available through various silicon vendors that are generally available and expose all features of those boards with documentation, debug interfaces and support.
• Emulation and Testing: Qemu or other simulators/emulators, native_posix etc.
• Products: Derivative of an existing SoC or Platform with limited or modified functionality of the baseline SoC and additional custom features.
• Hobbyist and Maker Boards: Derivative with customisations
• ....

Using this classification we should be able to cover majority of testing with the baseline boards (Reference Boards) and limit testing and building for derivatives to only the additional features not covered by the reference boards.

For example a maker board or a product based on STM32F769I (note: randomly selected) from ST Micro will be covered for the most part by the stm32f769i_disco boards available in Zephyr. If a product or a derivative of this board is available for example with Wifi, only the Wifi functionality is tested and verified on the derivative and we will not try to build every single available test or sample with the derivative.

Why?

To allow for better test coverage and improve targeted testing in CI and to avoid duplication and time consuming unnecessary builds both in CI activity related to PRs and nightly builds

How?

Add class to the board yaml file and enhance twister to do the needed filtering. We can select a few general tests or samples that would have to run on any class as a way to make sure the basic board integration is correct.

[galak]

I like the general idea, would be good to flush out a bit of the logic that twister would need to implement around use of this board classifier field.

[erwango]

Definitely an improvement. We're already doing something similar on our bench (by generating list of dev boards at each run)

[gregshue]

It seems to me that emulation vs. real is independent of the other values. What is the distinction between Product vs Hobbyist/Maker?

[nashif]

It seems to me that emulation vs. real is independent of the other values.

not sure what you mean here.

What is the distinction between Product vs Hobbyist/Maker?

• bbc_microbit is a hobbyist/maker board based on nrf51. Generally available and can be used for developing different things.
• Legend boards (boards: Add support for Legend boards #35853) are part of a product, they are not generally available standalone and are not used for development etc. They are based on some STM SoC (STM32F070cb)

[gregshue]

not sure what you mean here.

Just observing that emulation versions of actual boards could exist along with the real versions. We ought to consider how they would be distinguished. :-)

• bbc_microbit is a hobbyist/maker board based on nrf51. Generally available and can be used for developing different things.

• Legend boards (boards: Add support for Legend boards #35853) are part of a product, they are not generally available standalone and are not used for development etc. They are based on some STM SoC (STM32F070cb)

Also consider that Raspberry PIs (hobbyist/maker board) have been used in products and that R&D labs may use product boards to develop different things.

I haven't noticed any actual benefit to distinguishing Product vs Hobbyist/Maker. It seems like either the CI system has the HW to run it on or doesn't. (This is true whether Zephyr Project is running the CI on commercially available HW or I am on my own prototypes.) What am I missing?

[nashif]

I haven't noticed any actual benefit to distinguishing Product vs Hobbyist/Maker.

the former is not something you can go to digikey and get, it probably will have no debug interfaces and will be deeply embedded in some product, so, we can't for example have bugs against it block a release, because nobody will be able to reproduce and fix. The latter, is something you can have on your desk and you can use for different things and for development, it is however going to be covered by some other reference board from a vendor for the most part.

[gregshue]

For those of us reusing the Zephyr ecosystem for product development it would be helpful for the category names and policies to still make sense as we add our proprietary development boards/devices into the list of targets.

[Hobbyist/Maker] is however going to be covered by some other reference board from a vendor for the most part.

It seems that Zephyr Project shouldn't accept bugs on HW that it can't reproduce and verify the fix for (whether Product or Hobbyist/Maker) - which seems wise. :-)

I'd like to see an example of where a bug on a non-Development/Reference/emulation/testing board would be accepted. Have you come up with one?

[PerMac]

I like this idea. As I understand the goal of such differentiation is to provide better scope (improved transparency and traceability), therefore QA, for on-target testing by introducing more descriptive "hierarchy" for platforms available in zephyr. E.g. there are ~27 nrf platforms defined in https://github.com/zephyrproject-rtos/zephyr/tree/main/boards/arm. However, they have different values from the testing perspective.
In the on-target setup I am maintaining only the main representants of our products families: 4 dk variants, which counts as 7 platforms in zephyr (due to non-secure options and a multi-core platform). We find all of them "representative" as they should cover all the major features and running tests on all existing nrf platforms is not possible in practice. Results from those platforms are then published to the upstream. Other platforms refer to dks e.g with some minor variations like having more external memory, some variation in peripherals etc.
@greg As I understand the discussion is not whether we should accept bugs reported on some very specific platforms but about assigning the proper priority/severity to the detected issues based on their presence on the selected platforms and their type.

My thoughts for the proposed naming and definition in such context:

• "Development Boards / Reference" - "Reference" seems to better express the idea behind the name. Development Board is too general. There are a lot of various dks in nrf line but only several are selected as representative in the testing context we consider such a board to be mature and supported enough. It also should cover the majority of Zephyr features. The maintainers of hw test setups will be obliged to regularly provide on-target test results for the board. Any failure detected/reported on such a board will be given high priority/severity by default. Priority/severity can be lowered if the failure is vendor-specific (e.g. issue with vendor-specific drivers).
• "Hobbyist and Maker Boards" - According to Anas explanation the difference with "reference boards" is the fact that "hobbists boards" are based on components that are common to the "reference" ones, with some possible variation in available peripherals etc. I don't see sense in differentiating those in terms of priority/severity of reported issues. E.g. bbc_microbit or reel_board might have a very broad audience and it should be in our interest to treat them with high priority. However, I think the major difference is in the "owner" of such board: is it a major company developing zephyr or not. It might affect the interests of major contributors in running regular hw tests on such boards.
• "Product": We assume that the scope provided by such platform should already be covered by the "reference" boards, hence there is no need for regular hw test on them. Nevertheless, it doesn't mean the reported issues (if reproducible) will be ignored. If I understood Anas's differentiation correctly I think a name like "offspring" might be more intuitive.
• We might add a category like "feature-specific" for platforms focused on a certain feature, e.g. Bluetooth.

From my point of view, having in mind discussions during the testing wg meetings, it is important to first define the "functional differentiation", which will be used to give a "tier" for a platform. I propose to evaluate boards in three categories:

• regularity of hw tests performed and publishing the results to the upstream
• priority/severity given by default to the issues reported for a given platform
• functional coverage provided by a platform

In my opinion, the difference between "reference" and "maker" boards is not precise enough for our purpose. I think boards like "reel_board" and "bbc_microbit" could end up in the same "top tier" category (I think "reference" is a good name) as e.g major dks from Intel or Nordic but it would require major contributors taking them under their hood, running regular hw tests on them and publishing the results (which is already the case for "reel_board").

[PerMac]

I realized my answer was not exactly addressing "Why? To allow for better test coverage and improve targeted testing in CI and to avoid duplication and time consuming unnecessary builds both in CI activity related to PRs and nightly builds". However, it addresses the on-target testing and reporting which is also currently under our evaluation and is still related to boards classification. In the upstream CI we only execute emulator/simulator -based tests and configurations for physical boards are only built, hence the need for results from on-target test setups maintained by contributors.

[PerMac]

@carlescufi I think you might be interested in this discussion.

[mbolivar-nordic]

There seem to be two different concerns here, which don't overlap completely:

1. testing (maintainer concern)
2. support level (user concern)

I fully support the goals of

• decreasing test burden by not repeating useless SoC-level testing on "product" boards that has already been tested on "reference" boards
• still testing any board-specific features (wifi, sensors, etc.) that aren't covered by generic SoC-level testing on reference platforms

However, I think that as a user, I don't care about that so much. As a user, I want to know which boards are supported and actively maintained by Zephyr. I have seen several users ask "what is a good board to get started with Zephyr"? The answer to this doesn't have to be a "reference" board sold by the SoC vendor -- in fact, a "hobbyist and maker" board which is maintained by an active company may be better supported, if the maintainer is active on Discord, etc.

I think it would be good if this proposal can untangle these separate concerns.