Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restructure Conformance Tests #6264

Closed
coryrc opened this issue Dec 19, 2019 · 12 comments
Closed

Restructure Conformance Tests #6264

coryrc opened this issue Dec 19, 2019 · 12 comments
Assignees
@dprotaso
Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/feature Well-understood/specified features, ready for coding. kind/process Changes in how we work lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. triage/accepted Issues which should be fixed (post-triage)

Comments

@coryrc
Copy link
Contributor

coryrc commented Dec 19, 2019

In what area(s)?

/area test-and-release

Describe the feature

Design doc

Summary:
Our API conformance tests were written before we had a pattern to split out tests and before the API specification identified ‘should’ and ‘must’ fields. This resulted in them covering a large swath of functionality in large tests, making it difficult to isolate conformance gaps. They need some help to become useful to people who wish to prove conformance and useful to people to make their implementation conformant.
@coryrc coryrc added the kind/feature Well-understood/specified features, ready for coding. label Dec 19, 2019
@coryrc
Copy link
Contributor Author

coryrc commented Dec 19, 2019

/assign @coryrc

@knative-prow-robot knative-prow-robot added area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/process Changes in how we work labels Dec 19, 2019
@coryrc coryrc mentioned this issue Dec 26, 2019
Closed
@coryrc coryrc mentioned this issue Jan 14, 2020
Closed
@coryrc
Copy link
Contributor Author

coryrc commented Jan 14, 2020

I intend to have something to discuss with serving WG next week.

@coryrc
Copy link
Contributor Author

coryrc commented Jan 22, 2020

Example implementations:
https://github.com/coryrc/serving/blob/restructure-conformance/test/conformance/api/v1/errorcondition_test.go
https://github.com/coryrc/serving/blob/restructure-conformance/test/e2e/errorcondition_test.go
https://github.com/coryrc/serving/blob/restructure-conformance/test/scenarios/container_failures.go
https://github.com/coryrc/serving/blob/restructure-conformance/test/conformance/api/v1/resources_test.go
https://github.com/coryrc/serving/blob/restructure-conformance/test/e2e/resources_test.go

@coryrc coryrc mentioned this issue Jan 30, 2020
Merged
coryrc added a commit to coryrc/serving that referenced this issue Feb 23, 2020
@coryrc
Restructure Conformance API tests to match Spec
0c07394 
Fixes knative#6006 by making TestCustomResourceLimits to be a conformance
test and moving e2e parts out.

For knative#6226 fixes it for TestContainerErrorMsg and
TestContainerExitingMsg, though could be other tests checking errors
incorrectly.

Addresses knative#6264 though continuing work is needed.

Don't mix E2E and conformance:
Strip E2E portions of conformance out into tests in the test/e2e
directory. Commonality goes into test/scenarios or test/v1 depending
on specificity of the common functionality.

Rename to more align with runtime test naming (Must/Should).
@coryrc coryrc mentioned this issue Feb 23, 2020
Closed
coryrc added a commit to coryrc/serving that referenced this issue Feb 28, 2020
@coryrc
Restructure Conformance API tests to match Spec
84acbac 
Fixes knative#6006 by making TestCustomResourceLimits to be a conformance
test and moving e2e parts out.

For knative#6226 fixes it for TestContainerErrorMsg and
TestContainerExitingMsg, though could be other tests checking errors
incorrectly.

Addresses knative#6264 though continuing work is needed.

Don't mix E2E and conformance:
Strip E2E portions of conformance out into tests in the test/e2e
directory. Commonality goes into test/scenarios or test/v1 depending
on specificity of the common functionality.

Rename to more align with runtime test naming (Must/Should).
@knative-housekeeping-robot
Copy link

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

@knative-prow-robot knative-prow-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 21, 2020
@knative-housekeeping-robot
Copy link

Stale issues rot after 30 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.
Rotten issues close after an additional 30 days of inactivity.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle rotten

@knative-prow-robot knative-prow-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 21, 2020
@coryrc
Copy link
Contributor Author

coryrc commented May 28, 2020

#6978 solves this
Anyone interested in doing this would do well to study the work that went into that PR.
/unassign
/remove-lifecycle rotten

@knative-prow-robot knative-prow-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 28, 2020
@mattmoor
Copy link
Member

This may also be relevant to the work @dprotaso has been looking into, but he's out for a few weeks.

@dprotaso
Copy link
Member

I'm back
/assign @dprotaso

@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 17, 2020
@dprotaso
Copy link
Member

dprotaso commented Jan 6, 2021

/remove-lifecycle stale
/lifecycle frozen

Context dump:

Serving's conformance requirements were outlined here: knative/pkg#1563 and a PoC test framework was written knative-sandbox/reconciler-test/pkg/test. Afterwards I was working towards having networking's conformance adopt this framework here: knative/networking#233

At this point the eventing test task force started to come up with their own requirements & frameworks. This started a consolidation around reconciler-test given their needs were a superset of servings. Next steps are to resolve gaps in the consolidate framework.

Those issues are:
knative-extensions/reconciler-test#37
knative-extensions/reconciler-test#64
knative-extensions/reconciler-test#65
knative-extensions/reconciler-test#66

@knative-prow-robot knative-prow-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 6, 2021
@n3wscott n3wscott mentioned this issue Feb 22, 2021
Closed
@evankanderson
Copy link
Member

Should this issue move to https://github.com/knative/specs now that there is a separate repository for conformance-related things (which can be versioned separately from Knative code itself)?

/triage accepted

@knative-prow-robot knative-prow-robot added the triage/accepted Issues which should be fixed (post-triage) label Mar 22, 2021
@coryrc coryrc mentioned this issue Mar 22, 2021
Closed
@coryrc
Copy link
Contributor Author

coryrc commented Mar 22, 2021

Done.

@coryrc coryrc closed this as completed Mar 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dprotaso dprotaso

Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/feature Well-understood/specified features, ready for coding. kind/process Changes in how we work lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. triage/accepted Issues which should be fixed (post-triage)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dprotaso @mattmoor @coryrc @evankanderson @knative-prow-robot @knative-housekeeping-robot