Add ingress test for opaAuthorizeRequest filter

[wisinghe]

Changes

• Skipper rbac configuration is ordered to be applied before deployment to make sure the service account is created before the deployment.

• Add three new steps in delivery.yaml

1. create-opa-enabled-cluster: creates a new cluster using the same configuration + manifests as the create-cluster step. The only difference in the configuration is skipper_open_policy_agent_enabled flag value. The CLUSTER_ALIAS and LOCAL_ID is adjusted so that it does not conflict with the e2e cluster that is created with create-cluster step.

2. opa-e2e-tests: Run the e2e test for opaAuthorizeRequest filter

3. decommission-opa-enabled-cluster: Decommissions the opa enabled cluster

• Change test/e2e/run_e2e.sh to enable different variation of cluster_creation, e2e testing and cluster_decommissioning.

• opaAuthorizeRequest is initialized and verified in tests

[zalando-robot]

Cannot start a pipeline due to:

Pipeline creation failed with: found undefined alias 'apply_env'
  in "<unicode string>", line 470, column 12:
          env: *apply_env
               ^

Click on pipeline status check Details link below for more information.

[zalando-robot]

Cannot start a pipeline due to:

Pipeline creation failed with: found undefined alias 'apply_env'
  in "<unicode string>", line 509, column 12:
          env: *apply_env
               ^

Click on pipeline status check Details link below for more information.

[zalando-robot]

Cannot start a pipeline due to:

Pipeline creation failed with: found undefined alias 'apply_env'
  in "<unicode string>", line 509, column 12:
          env: *apply_env
               ^

Click on pipeline status check Details link below for more information.

[zalando-robot]

Cannot start a pipeline due to:

Pipeline creation failed with: line 500, col 3: Unknown step id in `depends_on` value: e2e-load-test-result
line 500, col 3: Unknown step id in `depends_on` value: e2e-tests
line 500, col 3: Unknown step id in `depends_on` value: stackset-e2e-tests

Click on pipeline status check Details link below for more information.

[MustafaSaber]

TBH I don't like that we create another cluster in the pipeline; It will add another 30m to each PR, but that's for teapot to decide.

[MustafaSaber]

this PR time

Total steps runtime: 1 h, 34 m, 26 s

other PR times (went through 5 not all all of them)

Total steps runtime: 1 h, 8 m, 47 s

[wisinghe]

Hmm.. I understand that it would be a concern. Will wait for Teapot's feedback for that. However we do not hope to keep these steps forever. It's only until we are confident enough to enable this feature for all the clusters.

Also the overall completion time for a build is not affected as these 3 steps run parallel to other create-cluster, e2e steps.