Openshift CI

Folder openshift-ci in this repo contains files, required for OpenShift CI. It allows us to verify, that released Camel Quarkus works on a new version of Openshift.

How it works

The tests are run regularly (every Monday at the time of writing). Test execution is based on configs: [1],[2].

During the run CI takes the Dockerfile from openshift-ci folder, builds an image from it and runs tests from Camel Quarkus TS modules defined by $PROJECTS variable.

Results are posted into quarkus-qe channel in Slack and new issue in QQE Jira is created automatically for every failure. Job history can be accessed from the Prow Dashboard or via dashboard.

Requirements

Released version of Camel Quarkus

Due to the nature of tests, only the released bits are used. These are pulled directly from the maven.repository.redhat.com, not from an internal source, to be as close to real customer use-case as possible. This means that new versions of Camel Quarkus can only be tested once the release is done and the artifacts are in the maven repository.

OpenShift requirements

There are no specific requirement on OpenShift installation, but pull secret for dockerhub is highly recommended to avoid pull rate limiting.

Test machine requirements

All the required libraries are installed as part of the scenario:

• Git
• Java 17 OpenJDK
• Maven
• Docker/Podman/Buildah

Processes

Contacting PIT team

• Email: pit-qe@redhat.com
• Jira: INTEROP project
• Slack: #forum-qe-layered-product

New version of Camel Quarkus

After a new version of Quarkus is released, you should do the following:

• Update this section of Dockerfile (description is in the comments):

ENV CAMEL_QUARKUS_BRANCH=2.13.x # branch in of the repo which is used to run tests
ENV QUARKUS_VERSION=2.13.8.SP1-redhat-00003 # version of Quarkus BOM associated with the release
ENV QUARKUS_PLATFORM_GROUP_ID=com.redhat.quarkus.platform # group of Quarkus BOM. Unlikely to change
ENV QUARKUS_PLATFORM_ARTIFACT_ID=quarkus-bom # name of Quarkus BOM. Unlikely to change
ENV CAMEL_QUARKUS_VERSION=2.13.8.SP1-redhat-00003 # version of Camel Quarkus BOM associated with the release
ENV CAMEL_QUARKUS_PLATFORM_GROUP_ID=com.redhat.quarkus.platform # group of Camel Quarkus BOM. Unlikely to change
ENV CAMEL_QUARKUS_PLATFORM_ARTIFACT_ID=quarkus-camel-bom # name of Camel Quarkus BOM. Unlikely to change

• Update to preferred JDK in Dockerfile (base image FROM)
• Update ocp client in Dockerfile version to latest released OCP (find at https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/)
• Try to run it locally (see How to run locally#Without OCP deployment) and verify it passes
• Create image and push it to quay.io
• Ensure the image works in OpenShift CI
• After all your checks pass and the image is pushed, create a MR with the updated Dockerfile in this repository.

NOTE: For now, Interop team supports only one job for every product, so we only test the latest CEQ release. They plan to keep N-1 (eg. 4.14.x) and N-2 (eg. 4.13.x) OpenShift certifications, but it should be INTEROP team responsibility.

Adding new tests

Currently, the list of modules to be run is defined in the run script in PROJECTS variable. If you want additional tests to be run, update the list of modules (don't forget to add dependent modules if applicable). You should only add stable tests without random failures to the list! Once done follow the same steps mentioned above (starting from Try to run it locally).

Create an image

NOTE: Do not forget to change the oc_login.sh file back to its original state before you build and push the image to the registry.

NOTE: If you hit issue while building the image BuildKit is enabled but the buildx component is missing or broken. OR the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled, you have to install docker-buildx. For macOS and Colima, you can do it with abiosoft/colima#273 (comment).

Create image

DOCKER_BUILDKIT=1 docker build -t quay.io/rh_integration/camel-quarkus-qe-test-container:latest .

Push image

NOTE: You have to be first logged in to quay.io. You can find under Account settings/User settings/Docker CLI Password/Generate Encrypted password at quay.io.

docker push quay.io/rh_integration/camel-quarkus-qe-test-container:latest

Verify it is present at https://quay.io/repository/rh_integration/camel-quarkus-qe-test-container?tab=tags

This image is later used in the OpenShift CI tests, as defined here.

How to verify image is working at OpenShift CI

You have to firstly decide which strategy we follow. If we are using :latest tag of test images for all tested OpenShift versions or we have multiple different tags. At the time of writing this docs, we are using :latest tag, as we are testing only latest supported GA version of Camel Quarkus product (currently version 3.8).

You can check it in one of .yaml file in https://github.com/openshift/release/tree/master/ci-operator/config/jboss-fuse/camel-quarkus-openshift-interop eg. in https://github.com/openshift/release/blob/master/ci-operator/config/jboss-fuse/camel-quarkus-openshift-interop/jboss-fuse-camel-quarkus-openshift-interop-main__camel-quarkus-ocp4.15-lp-interop.yaml#L5

You should see similar:

base_images:
  camel-quarkus-runner:
    name: camel-quarkus-qe-test-container
    namespace: ci
    tag: latest

Using :latest tag

You can submit a draft PR example to https://github.com/openshift/release that's putting a minor change on the test script, for example echo command, this is only for testing purposes and the actual changes will not be merged. The automation will then recognize the changes and run some checks. Unless you are a trusted user in the openshift-ci group, the PR will automatically get needs-ok-to-test. This needs to be undone, so you need to ask someone from the owners file to comment /ok-to-test in the PR to enable testing.

To trigger the testing workflow, you can post a comment /pj-rehearse <test name> - the bot prints all the tests you can trigger, at the time of writing you can use for example /pj-rehearse periodic-ci-jboss-fuse-camel-quarkus-openshift-interop-main-camel-quarkus-ocp4.15-lp-interop-camel-quarkus-interop-aws.

If you are only testing the changes to the docker image, you can close the PR after the test passes, otherwise after green test results, one of the owners should ack it with /pj-rehearse ack and wait for PR being merged - if it takes too much time, you can approach Slack #forum-qe-layered-product.

Using custom tag

In the past we were testing two versions of GA product. It was 2.13 and 3.2. To differentiate we were using different tags of test image. For example 2.13.x-openjdk11 and 3.2.x-openjdk17. If it would re-happen again, we would have to communicate it with INTEROP team so they create more .yaml files.

In such case the theoretical steps should be:

• Creating PR with new tag added to https://github.com/openshift/release/blob/master/core-services/image-mirroring/supplemental-ci-images/mapping_supplemental_ci_images_ci#L129
• After merged, creating another PR with changes to .yaml files (probably with help of INTEROP team)

How to run locally

Without OCP deployment

NOTE: This is easier approach, but it is not 100% accurate to what happens on OpenShift CI. But usually such verification is enough.

Docker command below will build the image. Then it will run the container, which first logs into your OpenShift, clones testsuite and then run Maven execution of tests. So you will verify that the image works (but you are not running it in OpenShift, so it is not 100% safe way).

• Change oc_login.sh file to login to your existing OCP cluster.
• Go to openshift-ci folder and run:

DOCKER_BUILDKIT=1 docker build -t camel-quarkus-openshift-interop . && docker run camel-quarkus-openshift-interop | tee output.txt

With OCP deployment

• Change oc_login.sh file to login to your existing OCP cluster.
• Create a new public Docker repository (eg quay.io/$USER/test-container)
• Build and save an image for testing (you can use Docker, Podman or Buildah):

podman build --tag=quay.io/$USER/test-container -f openshift-ci/Dockerfile
podman push quay.io/$USER/test-container

• Run the tests:

oc create deployment interop-container --image=quay.io/$USER/test-container

• Clean after yourself:

oc delete deployment interop-container