Developer productivity: faster test cycle

[aslom]

Problem
When I run tests during development I have to repeat them seeveral times until they succeed (or fail in useful way) because of resource conflicts between multiple test runs, for example test fails because namespace is not cleaned

        channel_header_single_event_helper.go:48: Creating channel
        creation.go:45: Failed to create "InMemoryChannel" "conformance-headers-channel-binary": inmemorychannels.messaging.knative.dev "conformance-headers-channel-binary" is forbidden: unable to create new content in namespace test-must-pass-tracing-headers-in-memory-channel because it is being terminated
        test_runner.go:114: Could not delete the namespace "test-must-pass-tracing-headers-in-memory-channel": Operation cannot be fulfilled on namespaces "test-must-pass-tracing-headers-in-memory-channel": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
FAIL

or previous artifacts like channels were not deleted

        creation.go:45: Failed to create "InMemoryChannel" "conformance-headers-channel-binary": inmemorychannels.messaging.knative.dev "conformance-headers-channel-binary" already exists
FAIL

Persona:
Which persona is this feature for?
Contributors

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

Every e2e test in loca dev mode works or fails quickly and not because of setup/cleanup issues

Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1?

Additional context (optional)
Add any other context about the feature request here.

Every re-run test translates in wasted 30seconds-1minute - tha adds up quickly ...

Example to reproduce in #1813

(base) aslom@m eventing (conformance-test-channel-headers-2) $ go test -v -timeout 30s -tags e2e ./test/conformance -run ^TestMustPassTracingHeaders$ --kubeconfig $KUBECONFIG
2019-09-05T13:25:28.602-0400	info	logging/config.go:43	Successfully created the logger.	{"knative.dev/jsonconfig": "{\n\t  \"level\": \"info\",\n\t  \"encoding\": \"console\",\n\t  \"outputPaths\": [\"stdout\"],\n\t  \"errorOutputPaths\": [\"stderr\"],\n\t  \"encoderConfig\": {\n\t    \"timeKey\": \"ts\",\n\t    \"messageKey\": \"message\",\n\t    \"levelKey\": \"level\",\n\t    \"nameKey\": \"logger\",\n\t    \"callerKey\": \"caller\",\n\t    \"messageKey\": \"msg\",\n\t    \"stacktraceKey\": \"stacktrace\",\n\t    \"lineEnding\": \"\",\n\t    \"levelEncoder\": \"\",\n\t    \"timeEncoder\": \"iso8601\",\n\t    \"durationEncoder\": \"\",\n\t    \"callerEncoder\": \"\"\n\t  }\n\t}"}
2019-09-05T13:25:28.602-0400	info	logging/config.go:43	Logging level set to info
2019-09-05T13:25:28.602-0400	info	logging/config.go:45	Fetch GitHub commit ID from kodata failed: "KO_DATA_PATH" does not exist or is empty
=== RUN   TestMustPassTracingHeaders
=== PAUSE TestMustPassTracingHeaders
=== CONT  TestMustPassTracingHeaders
=== RUN   TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
=== PAUSE TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
=== CONT  TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
--- FAIL: TestMustPassTracingHeaders (1.33s)
    --- FAIL: TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel (1.57s)
        channel_header_single_event_helper.go:43: Running header conformance test with channel "InMemoryChannel"
        test_runner.go:80: namespace is : "test-must-pass-tracing-headers-in-memory-channel"
        channel_header_single_event_helper.go:48: Creating channel
        creation.go:45: Failed to create "InMemoryChannel" "conformance-headers-channel-binary": inmemorychannels.messaging.knative.dev "conformance-headers-channel-binary" already exists
FAIL
FAIL	knative.dev/eventing/test/conformance	1.713s
(base) aslom@m eventing (conformance-test-channel-headers-2) $ k delete inmemorychannel.messaging.knative.dev conformance-headers-channel-binary
Error from server (NotFound): inmemorychannels.messaging.knative.dev "conformance-headers-channel-binary" not found
(base) aslom@m eventing (conformance-test-channel-headers-2) $ go test -v -timeout 30s -tags e2e ./test/conformance -run ^TestMustPassTracingHeaders$ --kubeconfig $KUBECONFIG
2019-09-05T13:25:55.051-0400	info	logging/config.go:43	Successfully created the logger.	{"knative.dev/jsonconfig": "{\n\t  \"level\": \"info\",\n\t  \"encoding\": \"console\",\n\t  \"outputPaths\": [\"stdout\"],\n\t  \"errorOutputPaths\": [\"stderr\"],\n\t  \"encoderConfig\": {\n\t    \"timeKey\": \"ts\",\n\t    \"messageKey\": \"message\",\n\t    \"levelKey\": \"level\",\n\t    \"nameKey\": \"logger\",\n\t    \"callerKey\": \"caller\",\n\t    \"messageKey\": \"msg\",\n\t    \"stacktraceKey\": \"stacktrace\",\n\t    \"lineEnding\": \"\",\n\t    \"levelEncoder\": \"\",\n\t    \"timeEncoder\": \"iso8601\",\n\t    \"durationEncoder\": \"\",\n\t    \"callerEncoder\": \"\"\n\t  }\n\t}"}
2019-09-05T13:25:55.051-0400	info	logging/config.go:43	Logging level set to info
2019-09-05T13:25:55.051-0400	info	logging/config.go:45	Fetch GitHub commit ID from kodata failed: "KO_DATA_PATH" does not exist or is empty
=== RUN   TestMustPassTracingHeaders
=== PAUSE TestMustPassTracingHeaders
=== CONT  TestMustPassTracingHeaders
=== RUN   TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
=== PAUSE TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
=== CONT  TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel
--- FAIL: TestMustPassTracingHeaders (0.25s)
    --- FAIL: TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel (0.41s)
        channel_header_single_event_helper.go:43: Running header conformance test with channel "InMemoryChannel"
        test_runner.go:80: namespace is : "test-must-pass-tracing-headers-in-memory-channel"
        channel_header_single_event_helper.go:48: Creating channel
        creation.go:45: Failed to create "InMemoryChannel" "conformance-headers-channel-binary": inmemorychannels.messaging.knative.dev "conformance-headers-channel-binary" is forbidden: unable to create new content in namespace test-must-pass-tracing-headers-in-memory-channel because it is being terminated
        test_runner.go:114: Could not delete the namespace "test-must-pass-tracing-headers-in-memory-channel": Operation cannot be fulfilled on namespaces "test-must-pass-tracing-headers-in-memory-channel": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
FAIL
FAIL	knative.dev/eventing/test/conformance	0.497s
(base) aslom@m eventing (conformance-test-channel-headers-2) $

[aslom]

/assign @nachocano
/assign @Fredy-Z

[chizhg]

An easy fix could be adding random postfix for the namespace of each test.
WDYT?

[aslom]

Yes - and some easy way to delete all namespace that share the same prefix?

[aslom]

@Fredy-Z on related note: is there some way to run tests without parallel option and/or other option so I can see test logging output if there is timeout? Currently all I get is lot of stack traces but logging is lost :(

Moved to separate issue #1894

[aslom]

@Fredy-Z what do you think about #1824 ?

[aslom]

I was expecting to see log output such as

--- FAIL: TestMustPassTracingHeaders (0.25s)
    --- FAIL: TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel (0.41s)
        channel_header_single_event_helper.go:43: Running header conformance test with channel "InMemoryChannel"
        test_runner.go:80: namespace is : "test-must-pass-tracing-headers-in-memory-channel"
        channel_header_single_event_helper.go:48: Creating channel
...

instead I only getting stack traces because of timeout and no idea how far test got and where it got stack as there is no logging ...

[chizhg]

I was expecting to see log output such as

--- FAIL: TestMustPassTracingHeaders (0.25s)
    --- FAIL: TestMustPassTracingHeaders/TestMustPassTracingHeaders-InMemoryChannel (0.41s)
        channel_header_single_event_helper.go:43: Running header conformance test with channel "InMemoryChannel"
        test_runner.go:80: namespace is : "test-must-pass-tracing-headers-in-memory-channel"
        channel_header_single_event_helper.go:48: Creating channel
...

instead I only getting stack traces because of timeout and no idea how far test got and where it got stack as there is no logging ...

That is an issue of Go, and there is an issue tracking it - golang/go#24929. It has been there for quite a while but still hasn't been fixed..

[aslom]

Moved to #1894

[akashrv]

/milestone v0.10.0

[vaikas]

Looks like there are two issues maybe in this issue IIUC?

1. namespaces
2. logging improvements

Could we split second into its own issue?

[aslom]

@vagababov @Fredy-Z moved logging to new issue: #1894

[Harwayne]

/close

#1896 added the randomness to the namespace names. As far as I can tell, that is all that is being tracked by this issue, as the logging portion was moved to #1894.

[knative-prow-robot]

@Harwayne: Closing this issue.

In response to this:

/close

#1896 added the randomness to the namespace names. As far as I can tell, that is all that is being tracked by this issue, as the logging portion was moved to #1894.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.