Proposal to add intermediate tutorial on integration testing

[abaeyens]

Current situation

The ROS 2 doc introduction to testing page makes clear that automated testing constitutes part of effective development. The subsequent two pages discuss unit testing for C++ and Python. Next to unit testing, it is also common to do integration testing, where several nodes are run simultaneously. ROS 2 provides several packages well-suited for this such as launch_testing_ament_cmake and ament_cmake_ros. Unfortunately, a quick search on these two package names doesn't turn up any hits in the ROS 2 doc except for changelogs, and the important topic of integration testing currently isn't discussed in the ROS 2 doc.

Related issues

ROS 2 Testing documentation #1441 from three years back.

Proposed changes

• Add an additional testing tutorial in a style similar to Writing Basic Tests with C++ with GTest and Writing Basic Tests with Python that shows how to develop simple integration tests.
• Content-wise based on this recent blog article I wrote and code in this repo. I'm open to all feedback, if this initial proposal gets approved I'll also ask the maintainers of the key dependencies for review to ensure the tutorial aligns with these packages' intended usage. If desired, I can add a discussion of launch_pytest, which the above article doesn't discuss.
• Subjects to touch in this tutorial:
  • how to launch several nodes simultaneously
  • how to ensure different ROS_DOMAIN_IDs for each launch (to avoid crosstalk between parallel-running tests)
  • the purpose of (and distinction between) active tests and post-shutdown tests
  • how to get usable JUnit reports out
  • how to visualize an overview of the passed and failed tests with the produced JUnit tests

Questions

Before I start writing and create a PR, I'd appreciate your feedback on this proposal. Is a tutorial about integration testing indeed desired? And is the linked blog article above suitable as a base for the ROS 2 doc tutorial, or would major content changes be necessary?

This is my first contribution to the ROS project, please let know in case I missed anything related to the contribution guidelines.

[fujitatomoya]

IMO proposal sounds reasonable and helpful for user documentation.

we can have a general integration test tutorial with introducing some of the contents from https://github.com/ros2/launch/tree/rolling/launch_testing, but not much details. so that we can guide the user to https://github.com/ros2/launch/tree/rolling/launch_testing for more information.

thank you very much for considering the 1st contribution for ROS community. let's keep this open for a while to get more feedback from other developers.

[abaeyens]

general integration test tutorial with introducing some of the contents ...

Good idea to put more light on and guiding to that package, thanks!

If anyone else has suggestions, please share.

[kscottz]

@abaeyens this sounds wonderful and we would greatly appreciate it!

Your blog post is wonderfully helpful! I was going to share it with the wider community tomorrow. You might want to consider making a post on ROS Discourse about your recent work and asking for some community feedback. Integration testing is still a bit of a black art in ROS and I would be interested in hearing how some of the larger companies in the ROS ecosystem are approaching it.

[abaeyens]

@kscottz your post was a really nice surprise, thanks, greatly appreciated!

You might want to consider making a post on ROS Discourse about your recent work and asking for some community feedback.

Excellent idea, I'll do so later this week.

[ottojo]

Hi @abaeyens , great blog post! I was down the same rabbit hole six months ago, my (somewhat abandoned since then) notes are here. I think the one part i have to add is the add_ros_test cmake function, so one more way of registering tests in cmake (nr. 4, i think? 🙃 ). I havent yet compared it with the other methods though.

I will have to look at xunit-viewer as well, i think the default colcon test experience is terrible, maybe the page on Running Tests in ROS 2 from the Command Line could even list this, i think it might be useful outside the context of launch testing as well!

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/towards-a-rosdoc-integation-testing-tutorial/40701/1