Add opcheck testing for nms #7961

ezyang · 2023-09-13T21:41:04Z

Signed-off-by: Edward Z. Yang ezyang@meta.com

cc @pmeier

Signed-off-by: Edward Z. Yang <ezyang@meta.com>

pytorch-bot · 2023-09-13T21:41:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7961

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 00673be with merge base e3fb8c0 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

Tests / unittests-macos (3.8, macos-m1-12) / macos-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ezyang · 2023-09-13T21:42:04Z

This adds testing for the meta registrations as promised. PyTorch core doesn't have all of the things needed landed yet, so we can't merge it yet pytorch/pytorch#108936

…enerating optest" Richard, I'm curious to see what you think of this. I'm trying to use optest on the torchvision test suite, and after hacking up pytest support in #108929 I noticed that this was 5x'ing the test time... for no good reason. * torchvision nms tests before optests: 60 passed, 4 skipped, 1206 deselected in 11.47s * after optests: 300 passed, 20 skipped, 1206 deselected in 49.85s It's no good reason because torchvision has parametrized the tests to get a spread of various random generation, but for checking schema or fake tensor, we don't actually need to test for different values. This PR hacks up the codegen to replace pytest parametrize markers so that, instead of sampling many values, we sample only one value if you mark it with `opcheck_only_one`. There's a carveout for device parametrization, where we always run all those variants. With this PR: * reduced optests: 88 passed, 4 skipped, 1206 deselected in 13.89s Companion torchvision PR which uses this at pytorch/vision#7961 Signed-off-by: Edward Z. Yang <ezyangmeta.com> [ghstack-poisoned]

Richard, I'm curious to see what you think of this. I'm trying to use optest on the torchvision test suite, and after hacking up pytest support in #108929 I noticed that this was 5x'ing the test time... for no good reason. * torchvision nms tests before optests: 60 passed, 4 skipped, 1206 deselected in 11.47s * after optests: 300 passed, 20 skipped, 1206 deselected in 49.85s It's no good reason because torchvision has parametrized the tests to get a spread of various random generation, but for checking schema or fake tensor, we don't actually need to test for different values. This PR hacks up the codegen to replace pytest parametrize markers so that, instead of sampling many values, we sample only one value if you mark it with `opcheck_only_one`. There's a carveout for device parametrization, where we always run all those variants. With this PR: * reduced optests: 88 passed, 4 skipped, 1206 deselected in 13.89s Companion torchvision PR which uses this at pytorch/vision#7961 Signed-off-by: Edward Z. Yang <ezyangmeta.com> [ghstack-poisoned]

Richard, I'm curious to see what you think of this. I'm trying to use optest on the torchvision test suite, and after hacking up pytest support in #108929 I noticed that this was 5x'ing the test time... for no good reason. * torchvision nms tests before optests: 60 passed, 4 skipped, 1206 deselected in 11.47s * after optests: 300 passed, 20 skipped, 1206 deselected in 49.85s It's no good reason because torchvision has parametrized the tests to get a spread of various random generation, but for checking schema or fake tensor, we don't actually need to test for different values. This PR hacks up the codegen to replace pytest parametrize markers so that, instead of sampling many values, we sample only one value if you mark it with `opcheck_only_one`. There's a carveout for device parametrization, where we always run all those variants. With this PR: * reduced optests: 88 passed, 4 skipped, 1206 deselected in 13.89s Companion torchvision PR which uses this at pytorch/vision#7961 Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #108936 Approved by: https://github.com/zou3519

zou3519 · 2023-09-14T16:18:34Z

test/test_ops.py

+optests.generate_opcheck_tests(
+    TestNMS,
+    ["torchvision"],
+    {},
+    "test/test_ops.py",
+    [],
+    data_dependent_torchvision_test_checks,
+)


Hmm, after pytorch/pytorch#109110 the failures dict is no longer a python dict, generate_opcheck_tests now assumes the existence of a json file.

The NMS tests work on everything, so there are no expected failures. It seems unfortunate that we would need to have a .json file in the repo to use generate_opcheck_tests. But I don't really have a better idea right now, the reason why we require a json file is so that we can automatically update it by writing to it; a string is a bit more difficult because we haven't hooked into the expecttest mechanism.

I don't mind having to have an empty JSON, but I am going to have to do this for each test class (of which there are a lot)

I have a similar problem with fbgemm, where there are a lot of test classes. Maybe everything should roll into the same json file

NicolasHug

Thanks for the PR @ezyang

Our CI is missing the expecttest dep, I pushed a commit for that. Let's see.

pytest.ini

test/test_ops.py

ezyang · 2023-09-25T14:03:19Z

Can't land this yet, because @zou3519 is going to make some more changes to the opcheck API that I want to land before I do this, but the review is appreciated.

NicolasHug · 2023-10-19T19:35:34Z

test/test_ops.py

@@ -462,9 +471,10 @@ def test_boxes_shape(self):

    @pytest.mark.parametrize("aligned", (True, False))
    @pytest.mark.parametrize("device", cpu_and_cuda_and_mps())
-    @pytest.mark.parametrize("x_dtype", (torch.float16, torch.float32, torch.float64), ids=str)
+    @pytest.mark.parametrize("x_dtype", (torch.float16, torch.float32, torch.float64))  # , ids=str)


Had to comment-out the ids part because we're hitting this assert 🤔 https://github.com/pytorch/pytorch/blob/2a40b7efcb273d1689b55763fade49263adcc788/torch/testing/_internal/optests/generate_tests.py#L225

NicolasHug

@ezyang I updated the PR according to the new opcheck, LGTM. LMK if there's anything more you wanted to add on your side, otherwise I'll merge.

Just FYI, we're hitting this "NYI" failure #7961 (comment)

ezyang · 2023-10-20T19:36:31Z

No, thank you so much for finishing it up! Please merge whenever you're ready.

github-actions · 2023-10-20T20:24:52Z

Hey @NicolasHug!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

Summary: Signed-off-by: Edward Z. Yang <ezyang@meta.com> Reviewed By: vmoens Differential Revision: D50789092 fbshipit-source-id: 614de3d6949a84ca576b9e7344de2e8e18152bf3 Co-authored-by: Nicolas Hug <nh.nicolas.hug@gmail.com> Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

Add opcheck testing for nms

7197128

Signed-off-by: Edward Z. Yang <ezyang@meta.com>

facebook-github-bot added the cla signed label Sep 13, 2023

ezyang requested review from NicolasHug and zou3519 September 13, 2023 21:41

ezyang mentioned this pull request Sep 14, 2023

Run only one pytest parametrization when generating optest pytorch/pytorch#108936

Closed

zou3519 reviewed Sep 14, 2023

View reviewed changes

NicolasHug and others added 2 commits September 25, 2023 11:33

Merge branch 'main' into opcheck-nms

03eb099

Install expecttest in CI

b8c5f52

NicolasHug reviewed Sep 25, 2023

View reviewed changes

pytest.ini Outdated Show resolved Hide resolved

test/test_ops.py Outdated Show resolved Hide resolved

NicolasHug and others added 2 commits October 19, 2023 18:19

Merge branch 'main' into opcheck-nms

0eb13d2

Cleanup, fix call, add TestRoiAlign, add json file

71c397e

NicolasHug reviewed Oct 19, 2023

View reviewed changes

NicolasHug added 4 commits October 19, 2023 21:53

Fix path

c44ad4e

Forgot to add file of courseeeeeeee

6a9463c

Use flag() CM instead of custom one

2a7b84e

Remove test for roialign as it's failing, will address in other PR

00673be

NicolasHug approved these changes Oct 20, 2023

View reviewed changes

NicolasHug merged commit 68161e9 into pytorch:main Oct 20, 2023
63 of 64 checks passed

NicolasHug added enhancement module: ops module: tests labels Oct 20, 2023

NicolasHug mentioned this pull request Oct 20, 2023

Add torch.compile support for all ops #8055

Closed

6 tasks

NicolasHug added the pt2 label Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add opcheck testing for nms #7961

Add opcheck testing for nms #7961

ezyang commented Sep 13, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 13, 2023 •

edited

Loading

ezyang commented Sep 13, 2023

zou3519 Sep 14, 2023

ezyang Sep 15, 2023

zou3519 Sep 15, 2023

NicolasHug left a comment

ezyang commented Sep 25, 2023

NicolasHug Oct 19, 2023

NicolasHug left a comment

ezyang commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

Add opcheck testing for nms #7961

Add opcheck testing for nms #7961

Conversation

ezyang commented Sep 13, 2023 • edited by pytorch-bot bot Loading

pytorch-bot bot commented Sep 13, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7961

✅ You can merge normally! (1 Unrelated Failure)

ezyang commented Sep 13, 2023

zou3519 Sep 14, 2023

Choose a reason for hiding this comment

ezyang Sep 15, 2023

Choose a reason for hiding this comment

zou3519 Sep 15, 2023

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

ezyang commented Sep 25, 2023

NicolasHug Oct 19, 2023

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

ezyang commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

ezyang commented Sep 13, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 13, 2023 •

edited

Loading