Test runner: Couldn't initialize clients: unable to find available namespace

[pierDipi]

Describe the bug

Sometimes the integration testing pipeline in https://github.com/knative-sandbox/eventing-kafka-broker/ fails with:

test_runner.go:163: Couldn't initialize clients: unable to find available namespace

Expected behavior

No errors, just create a namespace if there isn't one.

To Reproduce

N/A

Knative release version
devel

Additional context

/area test-and-release

[pierDipi]

Hi @mgencur,
I see this error was introduced in https://github.com/knative/eventing/pull/5337/files,
do you have a suggested fix for this error? Could increasing MaxNamespaceSkip fix this error?

[mgencur]

hi @pierDipi , yes, increasing MaxNamespaceSkip should help. Basically increasing any of these three: https://github.com/knative/eventing/blob/main/test/lib/test_runner.go#L51-L53 . But I'd start with increasing MaxNamespaceSkip to 50.

[pierDipi]

Thanks Martin!

Would you make it configurable with an env var or is it hardcoded by design?

/assign

[mgencur]

These are not parameters that would be changing with each test run or test environment. I believe we can figure out numbers that will be working well for a longer time. No need to introduce new params or env vars, IMO.

[pierDipi]

#5706 didn't work for us.

We're still seen the error above (https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative-sandbox_eventing-kafka-broker/1222/pull-knative-sandbox-eventing-kafka-broker-integration-tests/1435572262957748224)

[mgencur]

@pierDipi can you tell me how you're running the tests? I can see Running go test with args: -tags=e2e -count=1 -race -timeout=30m ./test/e2e_new/... which doesn't seem to exist upstream.

Are you using the REKT framework? If yes, you might need these two?
knative-extensions/reconciler-test#233
knative-extensions/reconciler-test#234

More info would be appreciated.

[pierDipi]

Sure, this happens in knative-sandbox/eventing-kafka-broker for tests that are using the eventing testlib (no rekt).

This is how we run them https://github.com/knative-sandbox/eventing-kafka-broker/blob/f76d520f1759179c756f7a65b07d119444aacd79/test/e2e-tests.sh#L45

Test directory is https://github.com/knative-sandbox/eventing-kafka-broker/tree/main/test/e2e

[pierDipi]

The tests in ./test/e2e_new/ are tests based on REKT which run fine, failed tests are the ones based on eventing testlib which are in test/e2e.

[mgencur]

OK. I'm trying something in knative-extensions/eventing-kafka#859 . Looking at the dump - there are no eventing-e2eX namespaces left after the run. The cleanup seems to work well. So, I believe it's just a matter of running many tests in parallel and the MaxNamespaceSkip number is still too low.

[mgencur]

But, maybe I'm wrong and it will need another fix. Let's see.

[pierDipi]

Right, we run many tests in parallel but the repo where this manifests is https://github.com/knative-sandbox/eventing-kafka-broker (note the trailing -broker, I know it's confusing :))

[mgencur]

Oh. Sorry, Wrong repo. Thanks