Skip to content

Latest commit

 

History

History
542 lines (452 loc) · 21.5 KB

rep-0141.rst

File metadata and controls

542 lines (452 loc) · 21.5 KB
Raw

REP: 141 Title: ROS distribution files Author: Dirk Thomas Status: Final Type: Standards Track Content-Type: text/x-rst Created: 18-Dec-2013 Post-History: 18-Dec-2013

Abstract

This REP specifies a set of files which define ROS distributions and facilitate the building, packaging, testing and documenting process. The intention is to formalize the existing infrastructure and simplify hosting of (potentially customized) ROS buildfarms. This REP is a revised version of REP 137 [6].

Motivation

A ROS distribution consists of numerous software packages maintained by various contributors forming a coherent set. Until now there was only one buildfarm responsible for running unit tests and generating Debian packages for various Ubuntu versions and no formal description of the semantics of the files used in that process was given so far. Expanding the current capabilities to support other platforms and architectures will be important extensions.

This REP aims to specify all information and file formats necessary to release and package ROS distributions for various platforms and architectures. By formalizing these data structures it should be enabled to:

  • release packages using bloom [1] for all targeted platforms
  • build binary packages for selected platforms and architectures
  • run unit tests on selected platforms and architectures
  • generation documentation

The use cases this REP is designed to address are:

  1. simplify the setup of a buildfarm which builds, tests and documents a ROS distribution, i.e. building, testing and documenting individual ROS distributions on separate buildfarms
  2. enable building binary packages for (experimental) platforms and architectures
  3. enable building binary packages of custom packages on top of the existing ROS distributions for publicly or privately hosted projects

Design requirements

The information formalized in this REP is used in five separate processes:

  • releasing a package (This is the process of exporting upstream source and generating platform specific build files specific meta-information)
  • building binary packages for a specific platform and architecture
  • running automated tests
  • running documentation jobs
  • checking out packages/repositories and their recursive dependencies from release repositories as well as source repositories

There should be a single configuration which contains (or references) all information required to run the above processes. The configuration must be easily readable and editable by both humans as well as machines and be extendable with future functionality.

The set of packages for which to build binary packages must be configurable for each platform and architecture.

The release process should stay independent of the build process. The testing can be performed on either the upstream source or on the released tarball (depending on if the upstream source contains enough information to run the test suite). The build process can only work on released packages.

Each configuration file must contain a label which identifies the semantics of the file as well as a version number to enable future format updates.

It must be possible to unambiguously correlate source and release repositories of the same packages.

Rationale

A single configuration containing or referencing all required information allows for a single entry point for automated processing in the above mentioned processes.

Specifying individual subsets of packages per platform and architecture is necessary since not all released packages can and should be build on all platforms and architectures.

The information for the various processes could be separated from each other to ease customizing individual processes. But then it would not be reliably possible to correlate entries from different processes. E.g. a released repo can not be unambiguously correlated to a source repository. Especially for tools like rosinstall_generator this would be valuable in order to enable certain features.

Specification

This REP gives a formal specification for the files that are used for that. Currently the information about the repositories is stored in three different files: * release.yaml * source.yaml * doc.yaml

In order to correlate them reliably this information will be stored in a single distribution.yaml file. For each repository the content from the separate files is nested under a common repository entry in the new distribution file.

The subfolder information for each package is being dropped. Since the location within the repository might be different for release/source/doc it would need to be maintained separately for each of them. Currently the package.xml is only fetched for released packages and then it is retrieved from the known location in the release repository.

The status information is not specific to the release repository anymore but applies to the source and doc repository as well. This allows for annotating the status of repositories and packages even if they are not released.

All current information can be converted into the new format using a script in the reference implementation.

File format

Files use YAML 1.1 for the sake of human and machine readability and write-ability. As a good practice, the files should contain a header such as:

%YAML 1.1
# ROS index|distribution|release-build|source-build|doc-build file
# see REP 141: https://ros.org/reps/rep-0141.html
---

Index file

The index file acts as a single configuration entrypoint which lists all available ROS distributions, along with references pointing to further process specific information. The references can be either relative to the index file or absolute.

The information stored in the index file is:

  • distributions: a list of ROS distributions

    For each distribution further information are referenced:

    • distribution: reference to the distribution file
    • release_builds: list of references to the release-build files used to build the binary packages
    • release_cache: reference to a release cache. Whether this field is a dictionary, a list or a scalar is left as an implementation detail. The following examples will assume that the implementation necessitates an url.
    • source_builds: list of references to the source-build files used to run the tests
    • doc_builds: list of references to the doc-build files used to run the documentation

  • type: must be 'index'

  • version: version number, this REP describes version 2 (instead of version 1 described in REP 137 [6])

Example

An index file referencing multiple distribution with their release, source and doc files and their corresponding build files.

%YAML 1.1
# ROS index file
# see REP 141: https://ros.org/reps/rep-0141.html
---
distributions:
  groovy:
    distribution: groovy/distribution.yaml
    release_builds: [groovy/release-build-ubuntu.yaml, releases/release-build-arm.yaml]
    release_cache: http://www.example.com/groovy-cache
    source_builds: [groovy/source-build.yaml]
    doc_builds: [groovy/doc-build.yaml]
  hydro:
    ...
type: index
version: 2

Distribution file

A distribution is identified by the code name of the ROS distribution in the index file. Each distribution file contains the following information:

  • repositories: a list of repositories which are identified by unique names

    • release: all release repository related information are nested under this key

      • url: the URL of the release git repository The URL must be anonymously readable and (with the appropriate credentials) writable.

      • version: version number for which packages are released. This can be a tag, branch or hash.

      • packages: an optional list of package names. If no package is specified, one package with the name of the repository is assumed.

      • tags: a dict of tags which can be used by tools like bloom and the rosinstall generator, preventing the need for those tools to guess about tags in release repositories, e.g. 'release/package/1.2.3' vs 'release/groovy/package/1.2.3-4'. This provides a useful future proofing mechanism for tools which use tags in the release repository, rather than guessing the format of the tags they can infer them directly from the tag templates.

        • tag_name {release, debian, etc...}: Format of tags are strings with {template_variables} e.g. 'release/groovy/{package}/{version}' A non-exhaustive list of possible template tags:

          • package - name of the package which this release tag corresponds
          • version - full version of the package being released, e.g. 1.2.3-4
          • upstream_version - upstream version of package being released, e.g. 1.2.3
          • debian_distro - target debian distro codename
          • debian_package_name - name of package, with any prefix and sanitized for debian

          The only required tag_name is 'release', others like 'debian' are optional.

    • source: all source repository related information are nested under this key

      • type: the type of SCM in use. Typically 'git', 'svn'...
      • url: the URL of the source repository The URL must be anonymously readable.
      • version: For git and hg this is the tag, branch or hash to be checked out. For svn the version should not be set since the branch/tag is encoded in the url.

    • doc: all doc repository related information are nested under this key

      • type: the type of SCM in use. Typically 'git', 'svn'...
      • url: the URL of the source repository The URL must be anonymously readable.
      • version: For git and hg this is the tag, branch or hash to be checked out. For svn the version should not be set since the branch/tag is encoded in the url.
      • blacklist_packages: an optional list of package names which should not be documented. (default: [])
      • depends: list of repository names. Other repositories to perform cross referencing in the documentation. This is only necessary if the packages from the dependent repositories are not released. (default: [])

    • status: defines the status of the packages of the repository. Can be one of the following: developed, maintained, unmaintained, end-of-life

    • status_description: an optional field describing in a short sentence the current status of the repository. For example detailing the reason for EOL and the recommended upgrade path.

    • status_per_package: an optional list of package names. For each package name a dictionary with the following optional keys allows to override the status information:

      • status: overrides the repository-wide status
      • status_description: overrides the repository-wide status description

  • release_platforms: a dict of target platforms for which packages are released. Each key contains the OS name while the value is a list of OS code names. These OS names and OS code names are determined by rospkg.os_detect [4]. Each target platform will result in a different bloom release.

  • type: must be 'distribution'

  • version: version number, this REP describes version 1

Example: A distribution file listing repositories and the release platforms. All changes to REP 137 are commented on inline.

%YAML 1.1
# ROS distribution file
# see REP 141: https://ros.org/reps/rep-0141.html
---
# prefixed with 'release_' since these are the platforms we release for
release_platforms:
  debian: [wheezy]
  ubuntu: [precise, quantal, raring]
repositories:
  catkin:
    # each repo contains the (optional) sections for 'doc', 'release' and 'source'
    doc:
      type: git
      url: https://github.com/ros/catkin.git
      version: groovy-devel
    release:
      tags:
        release: release/hydro/{package}/{version}
      url: https://github.com/ros-gbp/catkin-release.git
      version: 0.5.77-0
    source:
      type: git
      url: https://github.com/ros/catkin.git
      version: groovy-devel
  genmsg:
    doc:
      type: git
      url: https://github.com/ros/genmsg.git
      # specified separately from 'source' since it might use different version
      version: 0.4.23
    release:
      tags:
        release: release/hydro/{package}/{version}
      url: https://github.com/ros-gbp/genmsg-release.git
      version: 0.4.23-0
    source:
      type: git
      url: https://github.com/ros/genmsg.git
      version: hydro-devel
  roscpp_core:
    doc:
      # optional blacklist packages to avoid documentation of these
      blacklist_packages: [test_rostime]
      depends: [genmsg]
      type: git
      url: https://github.com/ros/roscpp_core.git
      version: hydro-devel
    release:
      # the packages are populated by bloom
      # the subfolder is not stored anymore
      packages: [cpp_common, roscpp_core, roscpp_serialization, roscpp_traits, rostime]
      tags:
        release: release/hydro/{package}/{version}
      url: https://github.com/ros-gbp/roscpp_core-release.git
      version: 0.3.16-0
    source:
      # blacklisting packages should be done with CATKIN_IGNORE files
      # rather than in this file to also cover users checking out the repo
      type: git
      url: https://github.com/ros/roscpp_core.git
      version: hydro-devel
    # status also covers 'doc' and 'source' (not only release, e.g. if not even released)
    status: maintained
    status_description: 'Very actively maintained'
    # status can be overridden per package
    status_per_package:
      roscpp_core:
        status: end-of-life
        status_description: 'Metapackage is not necessary anymore'
      rostime:
        status: unmaintained
type: distribution
version: 1

Release build file

A release build file contains the information necessary to build packages of the packages specified in the release file:

  • package_whitelist: a list of package names to build. If this is omitted all packages specified in the release part of the distribution file are built. Any upstream packages are implicitly included.
  • package_blacklist: a list of package names excluded from build. If this is omitted no packages are excluded. Any downstream package are implicitly excluded. The blacklist overrides the whitelist.
  • notifications: An optional section to configure email notifications.
    • emails: a list of email addresses to which to send all notification emails. (default: empty)
    • maintainers: a boolean flag used to enable email notification to listed maintainers. (default: false)
    • committers: a boolean flag used to enable email notification to committers. (default: false)
  • targets: a nested dict of targets for which packages are built. The first level key contains the OS name. The second level key contains the OS code name. The third level key contains the CPU architecture. The OS names and OS code names specified must be listed as a release platform in the corresponding distribution file. Each level can contain a key _config which can contain a dict with arbitrary data. These configurations can be used to specify target specific information (e.g. apt_target_repository or yum_repository).
  • jenkins_url: the url to the associated Jenkins master
  • jenkins_sourcedeb_job_timeout: the timeout in minutes for the sourcedeb jobs (optional).
  • jenkins_binarydeb_job_timeout: the timeout in minutes for the binarydeb jobs (optional).
  • sync: specify the criteria that need to be fulfilled for packages to be synced. Any of the following options can be set and all of them must be fulfilled:
    • package_count: integer. Specifies how many packages need to be successfully built to perform a sync. (default: 0)
    • packages: list of package names. All the packages of this list must be successfully built to perform a sync. (default: [])
  • type: must be 'release-build'
  • version: version number, this REP describes version 1, since the specification is the same as in REP 137

Example: A build file selecting a subset of packages from the release part of a distribution file and specifying the platforms and architectures.

%YAML 1.1
# ROS release-build file
# see REP 141: https://ros.org/reps/rep-0141.html
---
package_whitelist: [ros_tutorials, common_tutorials]
package_blacklist: [turtlesim]
notifications:
  emails: [buildfarm-admin@example.com]
  maintainers: true
  committers: false
targets:
  _config:
    apt_target_repository: http://archives.example.com/ros
    apt_mirrors: [http://archives.example.com/ros, http://packages.foo.org/repos/example]
  ubuntu:
    oneiric:
      amd64:
      i386:
    precise:
      amd64:
      i386:
      armel:
jenkins_url: http://jenkins.example.com:8080
sync:
  packages: [ros_tutorials]
type: release-build
version: 1

Release cache file

Collection of all meta information of the ROS distribution, including all the information from the package.xml files. The cache must reference the release file and store a hash of the release file it was build from to be able to detect if the cache is invalid. The format of that cache is considered an implementation detail and is not specified in this REP.

Source build file

The source build file uses a specification similar to the release build file, but does not have sync information. Also the white- and blacklist is on a repository level and does not consider any kind of dependencies.

  • repository_whitelist: a list of repository names to build. If this is omitted all repositories specified in the source part of the distribution file are built.
  • repository_blacklist: a list of repository names excluded from build. If this is omitted no repositories are excluded. The blacklist overrides the whitelist.
  • notifications: as specified for the release build file.
  • targets: as specified for the release build file.
  • jenkins_url: as specified for the release build file.
  • jenkins_job_timeout: the timeout in minutes for each job (optional).
  • type: must be 'source-build'
  • version: version number, this REP describes version 1, since the specification is the same as in REP 137

Documentation build file

The documentation build file uses a specification similar to the source build file.

  • repository_whitelist: a list of repository names to build. If this is omitted all repositories specified in the doc part of the distribution file are built.
  • repository_blacklist: a list of repository names excluded from build. If this is omitted no repositories are excluded. The blacklist overrides the whitelist.
  • notifications: as specified for the source build file.
  • targets: as specified for the source build file. But the list of targets must only have one entry.
  • jenkins_url: as specified for the source build file.
  • jenkins_job_timeout: the timeout in minutes for each job (optional).
  • doc_tag_index_repository: a repository storing the tag index
    • type: the type of SCM in use. Must be 'git'
    • url: the URL of the release git repository The URL must be writable (with the appropriate credentials).
    • version: This is the branch to be checked out and committed to.
  • type: must be 'doc-build'
  • version: version number, this REP describes version 1, since the specification is the same as in REP 137

Reference implementation

This REP is to be implemented in version 0.3 of the Python module rosdistro. This module will serve as a reference implementation for this REP. Any ROS tool requesting information defined in this REP should use this reference API implementation.

Compatibility issues

To leverage the new file format additional API will be introduced (e.g. 'rosdistro.get_distribution_file()'). The existing API to access the previously separated release/source/doc repositories will still be available and return the information from the new distribution file.

As soon as the new version of 'rosdistro' is released the 'ros/rosdistro' repository [7] is being updated to the new format (for Groovy and higher). Any client accessing trying to access the data with an old 'rosdistro' version will get an error message like this:

Unable to handle 'index' format version '1', please update rosdistro

Affected tools

For all tools it must be ensured that the used API of rosdistro (even if deprecated) is still working flawlessly:

  • bloom [1]
  • buildfarm
  • jenkins_scripts
  • jenkins_tools
  • prerelease_website
  • rosdep [2]
  • rosinstall_gen
  • rosinstall_generator
  • roslocate [3]

After this REP has been deployed 'rosinstall_generator' can be modified to enable not only checking out released packages but also source repositories.

The other tools should be updated in the future to use the newly introduced rather then the deprecated API. After some migration period the deprecated API might be removed from 'rosdistro'.

Use case examples

A complete example for a ROS distribution called foo with all the above specified files can be found at [5].

References

[1](1, 2) Bloom: a release automation tool (http://www.ros.org/wiki/bloom)
[2]rosdep2: https://ros.org/reps/rep-0125.html
[3]roslocate: http://www.ros.org/reps/rep-0115.html
[4]rospkg.os_detect: http://www.ros.org/reps/rep-0114.html
[5]Example files specifying a ROS distro: https://github.com/ros-infrastructure/rosdistro/tree/master/test/files
[6](1, 2) REP 137: http://www.ros.org/reps/rep-0137.html
[7]rosdistro repository: https://github.com/ros/rosdistro

Copyright

This document has been placed in the public domain.