Red Hat ISV Operator Certification Pipelines

Red Hat OpenShift pipelines for certifying ISV Operator Bundles.

Getting Started

Refer to the developer guide.

Note: This documentation is intended for pipeline developers/maintainers only.

Partners/users should refer to this documentation instead.

Usage

Operator CI Pipeline

The Operator CI pipeline is a Tekton pipeline that can be triggered by a partner using on-premise infrastructure. The pipeline validates an Operator Bundle, builds it and installs it to an OpenShift environment. After installation, pre-flight tests are executed which validate that the Operator meets minimum requirements for Red Hat OpenShift Certification. If all preceding tasks pass, the CI pipeline optionally uploads results and submits a pull request to trigger the next stages of the operator certification workflow.

Note: Execution of the CI pipeline is NOT required in the overall certification workflow.

If using the default internal registry, the CI pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-ci-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_branch=main \
  --param bundle_path=operators/kogito-operator/1.6.0-ok \
  --param env=prod \
  --workspace name=pipeline,volumeClaimTemplateFile=templates/workspace-template.yml \
  --showlog

If using an external registry, the CI pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-ci-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_branch=main \
  --param bundle_path=operators/kogito-operator/1.6.0-ok \
  --param env=prod \
  --param registry=quay.io \
  --param image_namespace=redhat-isv \
  --workspace name=pipeline,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=registry-credentials,secret=registry-dockerconfig-secret \
  --showlog

If using an kind cluster with registry, the CI pipeline can be triggered using the tkn CLI like so:

Warning: This mode is currently in development and it might not work yet.

Note: kind cluster with registry setup is documented here

tkn pipeline start operator-ci-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_branch=main \
  --param bundle_path=operators/kogito-operator/1.6.0-ok \
  --param env=prod \
  --param gitInitImage=quay.io/operator_testing/pipelines-git-init-rhel8:latest \
  --param builder_image=quay.io/operator_testing/buildah:latest \
  --param registry=$(hostname):5000 \
  --workspace name=pipeline,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=registry-cacert,config=registry-ca-cert
  --showlog

A subset of tasks in the pipeline requires privilege escalation which is no longer supported with OpenShift Pipelines 1.9. Thus a new SCC needs to be created and linked with pipeline service account. Creating SCC requires user with cluster-admin privileges.

# Create a new SCC
oc apply -f ansible/roles/operator-pipeline/templates/openshift/openshift-pipelines-custom-scc.yml
# Add SCC to a pipeline service account
oc adm policy add-scc-to-user pipelines-custom-scc -z pipeline

To enable opening the PR and uploading the pipeline logs (visible to the certification project owner in Red Hat Connect), pass the following argument:

--param submit=true

To open the PR with submission, upstream repository name must be supplied (eg. test-org/test-repo):

--param upstream_repo_name=<repository_name>

To enable digest pinning, pass the following arguments:

--param pin_digests=true \
--param git_repo_url=<github_repo_ssh_url> \
--param git_username=<github_user_name> \
--param git_email=<github_email> \
--workspace name=ssh-dir,secret=github-ssh-credentials

Note: The git_repo_url param needs an SSH URL to commit the pinned digests.

If any of bundle's related images are stored in a private registry the user needs to provide tokens to pull from those registries. See more details about how to provide registry tokens in the pipeline environment setup documentation.

Operator Hosted Pipeline

The hosted pipeline is used to certify the Operator bundles. It’s an additional (to CI pipeline) layer of validation that has to run within the Red Hat infrastructure. It is intended to be triggered upon the creation of a bundle pull request and successfully completes with merging it (configurable).

Note: Execution of the hosted pipeline is ALWAYS required in the overall certification workflow. Prior execution of the CI pipeline may influence its behavior if results were submitted. Preflight testing may be skipped in such a case.

The hosted pipeline can be triggered using the tkn CLI like so:

tkn pipeline start operator-hosted-pipeline \
  --use-param-defaults \
  --param git_pr_branch=test-PR-ok \
  --param git_pr_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test/pull/31 \
  --param git_fork_url=https://github.com/MarcinGinszt/operator-pipelines-test.git \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_username=foo@redhat.com \
  --param git_commit=0aeff5f71e4fc2d4990474780b56d9312554da5a \
  --param git_base_branch=main \
  --param env=prod \
  --workspace name=repository,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=results,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=registry-credentials-all,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=registry-credentials,secret=hosted-pipeline-registry-auth-secret \
  --showlog

To ignore the results of the publishing checklist, pass the following argument:

  --param ignore_publishing_checklist=true

⚠️ Only quay-based registries are supported by the hosted pipeline. There are some quay specific tasks for configuring the repositories where the bundle and index images are pushed.

Operator Release Pipeline

The release pipeline is responsible for releasing a bundle image which has passed certification. It's intended to be triggered by the merge of a bundle pull request by the hosted pipeline. It successfully completes once the bundle has been distributed to all relevant Operator catalogs and appears in the Red Hat Ecosystem Catalog.

Note: Execution of the release pipeline is ALWAYS required in the overall certification workflow.

tkn pipeline start operator-release-pipeline \
  --use-param-defaults \
  --param git_repo_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test.git \
  --param git_commit=3ffff387caac0a5b475f44c4a54fb45eebb8dd8e \
  --param git_pr_url=https://github.com/redhat-openshift-ecosystem/operator-pipelines-test/pull/31 \
  --param dest_image_namespace=redhat-isv-operators \
  --workspace name=repository,volumeClaimTemplateFile=templates/workspace-template.yml \
  --workspace name=results,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=image-data,volumeClaimTemplateFile=templates/workspace-template-small.yml \
  --workspace name=registry-pull-credentials,secret=release-pipeline-registry-auth-pull-secret \
  --workspace name=registry-push-credentials,secret=release-pipeline-registry-auth-push-secret \
  --workspace name=registry-serve-credentials,secret=release-pipeline-registry-auth-serve-secret \
  --showlog

Using a Custom Pipeline Image

All the pipelines share a common pipeline image for many of the steps. This image can be overridden by passing the following to any tkn pipeline start command.

--param pipeline_image=<image-pull-spec>

Testing

The repository comes with a default configuration for integration tests. In order to execute tests locally a user needs an access to OCP cluster with operator-pipelines pre-installed.

The tests are orchestrated using Ansible and contain a several steps:

• Configuration of OCP cluster
• Preparation of test data (new version of operator bundle)
• Execution of operator pipelines
• Evaluation of pipelines statuses

To run test locally running the Makefile is recommended.

# Set a version of a new operator based on the latest successful test run in
# Github action
export OPERATOR_VERSION_RELEASE="402-1"

make build-and-test-isv

# or
make build-and-test-community
# or
make build-and-test-isv-fbc-bundle
# or
make build-and-test-isv-fbc-catalog

Running test in PR workflow

A repository is configured to automatically run all integration tests on pull request. This makes sure a code change doesn't break existing workflow. Since a tests are running OCP cluster and requires some resources a tests are execute only when ready-for-testing label is added to a PR. Adding this label executes additional workflow.

All PRs should pass a tests before merging.

Additional Documentation

• OpenShift cluster configuration
• Index signature verification