Port ONNX op test suite. #8
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a port of the generated ONNX operator ("node") tests from https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests.
Major changes / simplifications:
* Focused the README on the ONNX operator tests, avoiding generalizing to other frameworks
* Switched to loading upstream tests from sitepackages instead of a git submodule
* Added `requirements-dev.txt` and `requirements-iree.txt` files so running the tests doesn't require ONNX
* Revised function and variable names and added pytype annotations in the import script
* Added TODOs for error handling and logging
* Stopped generating `.npy` files, favoring `.bin` files
* Limited to just `run_module_io_flags.txt`, no `test_cases.json`
* Limited to only local files, no remote or Git LFS files
* Inlined/simplified Exception subclasses and other helper functions
ScottTodd
commented
Aug 13, 2024
Comment on lines
+121
to
+127
| test_input = test_inputs[i] | ||
| type_proto = model.graph.input[i].type | ||
| converted_data = convert_onnx_proto_to_numpy_array(test_input, type_proto) | ||
| converted_type = convert_onnx_type_proto_to_iree_type_string(type_proto) | ||
| # TODO(scotttodd): raise exception instead of None as flow control? | ||
| if converted_data is None or converted_type is None: | ||
| return False |
There was a problem hiding this comment.
There is a bug fix included here: checking if the converted type is None. Switching using Exceptions instead of a sentinel value would be better.
t = convert_io_proto(test_input, model.graph.input[i].type)
ty = get_io_proto_type(model.graph.input[i].type)
if t is None:
return False
Comment on lines
+112
to
+119
| bytearr = b"" | ||
| if dtype in numpy_to_iree_dtype_map: | ||
| iree_dtype = numpy_to_iree_dtype_map[dtype][1] | ||
| bytearr = struct.pack(f"{len(mylist)}{iree_dtype}", *mylist) | ||
| else: | ||
| print(f"WARNING: unsupported data type in pack_ndarray_to_binary() : '{dtype}'") | ||
| # TODO(scotttodd): raise exception / log to file | ||
| return bytearr |
There was a problem hiding this comment.
This is likely generating buggy test case imports.
ScottTodd
commented
Aug 13, 2024
There was a problem hiding this comment.
Diff with IREE HEAD:
--- a/build_tools/pkgci/external_test_suite/onnx_cpu_llvm_sync.json
+++ b/build_tools/pkgci/external_test_suite/onnx_cpu_llvm_sync.json
@@ -117,8 +117,6 @@
"onnx/node/generated/test_constant_pad_negative_axes",
"onnx/node/generated/test_conv_with_autopad_same",
"onnx/node/generated/test_convtranspose_autopad_same",
- "onnx/node/generated/test_convtranspose_group_2",
- "onnx/node/generated/test_convtranspose_group_2_image_3",
"onnx/node/generated/test_convtranspose_kernel_shape",
"onnx/node/generated/test_convtranspose_output_shape",
"onnx/node/generated/test_cumsum_1d",
@@ -336,7 +334,6 @@
"onnx/node/generated/test_reduce_sum_keepdims_example",
"onnx/node/generated/test_reduce_sum_keepdims_random",
"onnx/node/generated/test_reduce_sum_negative_axes_keepdims_example",
- "onnx/node/generated/test_reduce_sum_negative_axes_keepdims_random",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_example",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_example_expanded",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_random",
@@ -369,7 +366,6 @@
"onnx/node/generated/test_resize_tf_crop_and_resize",
"onnx/node/generated/test_resize_tf_crop_and_resize_axes_2_3",
"onnx/node/generated/test_resize_tf_crop_and_resize_axes_3_2",
- "onnx/node/generated/test_resize_tf_crop_and_resize_extrapolation_value",
"onnx/node/generated/test_resize_upsample_scales_cubic",
"onnx/node/generated/test_resize_upsample_scales_cubic_A_n0p5_exclude_outside",
"onnx/node/generated/test_resize_upsample_scales_cubic_align_corners",
@@ -381,7 +377,6 @@
"onnx/node/generated/test_resize_upsample_sizes_nearest_axes_3_2",
"onnx/node/generated/test_resize_upsample_sizes_nearest_floor_align_corners",
"onnx/node/generated/test_resize_upsample_sizes_nearest_not_larger",
- "onnx/node/generated/test_resize_upsample_sizes_nearest_not_smaller",
"onnx/node/generated/test_rnn_seq_length",
"onnx/node/generated/test_roialign_aligned_false",
"onnx/node/generated/test_roialign_aligned_true",
@@ -452,6 +447,8 @@
"onnx/node/generated/test_slice_neg_steps",
"onnx/node/generated/test_slice_negative_axes",
"onnx/node/generated/test_slice_start_out_of_bounds",
+ "onnx/node/generated/test_softsign",
+ "onnx/node/generated/test_softsign_example",
"onnx/node/generated/test_stft",
"onnx/node/generated/test_stft_with_window",
"onnx/node/generated/test_string_concat",
@@ -527,9 +524,7 @@
"onnx/node/generated/test_qlinearmatmul_3D_int8_float32",
"onnx/node/generated/test_qlinearmatmul_3D_uint8_float16",
"onnx/node/generated/test_qlinearmatmul_3D_uint8_float32",
- "onnx/node/generated/test_reduce_max_empty_set",
"onnx/node/generated/test_reduce_min_empty_set",
- "onnx/node/generated/test_reduce_sum_empty_axes_input_noop",
"onnx/node/generated/test_reduce_sum_empty_set_non_reduced_axis_zero",
"onnx/node/generated/test_resize_downsample_scales_linear_align_corners",
ScottTodd
commented
Aug 13, 2024
Comment on lines
+18
to
+19
| ONNX_PACKAGE_DIR = Path(onnx.__file__).parent | ||
| ONNX_NODE_TESTS_ROOT = ONNX_PACKAGE_DIR / "backend/test/data/node" |
There was a problem hiding this comment.
Thanks for noticing that a git submodule isn't needed here, @zjgarvey !
There was a problem hiding this comment.
BTW, this __file__ solution is a bit cleaner and more robust than https://github.com/nod-ai/SHARK-TestSuite/blob/3b22038d687a13e3c46a8575d3e465133dfcbb7e/alt_e2eshark/onnx_tests/operators/generate_node.py#L29-L32
from site import getsitepackages
base_dir = getsitepackages()[0]
# you can also get the node tests from the git submodule instead by uncommenting:
# base_dir = str(Path(__file__).parents[3]) + "/third_party/onnx"
onnx_node_tests_dir = base_dir + "/onnx/backend/test/data/node/"In particular, there may be multiple site packages directories, and that assumed that onnx was in the first.
There was a problem hiding this comment.
Yeah, that is much cleaner. I'll make that change over in shark test suite, too.
rsuderman
approved these changes
Aug 13, 2024
rsuderman
approved these changes
Aug 13, 2024
ScottTodd
commented
Aug 13, 2024
| ease of use in downstream projects and by developers who prefer to work directly | ||
| with `.mlir` files and native (C/C++) tools. | ||
|
|
||
| ## Quickstart |
There was a problem hiding this comment.
@renxida I took some ideas from nod-ai/SHARK-TestSuite#306 here when updating this README
|
Will also have to pull in ireers (commontools from SHARK-TestSuite) for unified testing/logs between models/onnx, but can do that in future when everything related to iree testing is here |
Possibly, but I'm leaning against that at the moment. I want to use this as a chance to learn from what we've built so far and start over in some ways.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1.
This is a port of the generated ONNX operator ("node") tests from https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests. Major changes / simplifications:
requirements-dev.txtandrequirements-iree.txtfiles so running the tests doesn't require ONNX.npyfiles, favoring.binfilesrun_module_io_flags.txt, notest_cases.json