-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Port ONNX op test suite. #8
Conversation
This is a port of the generated ONNX operator ("node") tests from https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests. Major changes / simplifications: * Focused the README on the ONNX operator tests, avoiding generalizing to other frameworks * Switched to loading upstream tests from sitepackages instead of a git submodule * Added `requirements-dev.txt` and `requirements-iree.txt` files so running the tests doesn't require ONNX * Revised function and variable names and added pytype annotations in the import script * Added TODOs for error handling and logging * Stopped generating `.npy` files, favoring `.bin` files * Limited to just `run_module_io_flags.txt`, no `test_cases.json` * Limited to only local files, no remote or Git LFS files * Inlined/simplified Exception subclasses and other helper functions
test_input = test_inputs[i] | ||
type_proto = model.graph.input[i].type | ||
converted_data = convert_onnx_proto_to_numpy_array(test_input, type_proto) | ||
converted_type = convert_onnx_type_proto_to_iree_type_string(type_proto) | ||
# TODO(scotttodd): raise exception instead of None as flow control? | ||
if converted_data is None or converted_type is None: | ||
return False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a bug fix included here: checking if the converted type is None
. Switching using Exceptions instead of a sentinel value would be better.
t = convert_io_proto(test_input, model.graph.input[i].type)
ty = get_io_proto_type(model.graph.input[i].type)
if t is None:
return False
bytearr = b"" | ||
if dtype in numpy_to_iree_dtype_map: | ||
iree_dtype = numpy_to_iree_dtype_map[dtype][1] | ||
bytearr = struct.pack(f"{len(mylist)}{iree_dtype}", *mylist) | ||
else: | ||
print(f"WARNING: unsupported data type in pack_ndarray_to_binary() : '{dtype}'") | ||
# TODO(scotttodd): raise exception / log to file | ||
return bytearr |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is likely generating buggy test case imports.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Diff with IREE HEAD:
--- a/build_tools/pkgci/external_test_suite/onnx_cpu_llvm_sync.json
+++ b/build_tools/pkgci/external_test_suite/onnx_cpu_llvm_sync.json
@@ -117,8 +117,6 @@
"onnx/node/generated/test_constant_pad_negative_axes",
"onnx/node/generated/test_conv_with_autopad_same",
"onnx/node/generated/test_convtranspose_autopad_same",
- "onnx/node/generated/test_convtranspose_group_2",
- "onnx/node/generated/test_convtranspose_group_2_image_3",
"onnx/node/generated/test_convtranspose_kernel_shape",
"onnx/node/generated/test_convtranspose_output_shape",
"onnx/node/generated/test_cumsum_1d",
@@ -336,7 +334,6 @@
"onnx/node/generated/test_reduce_sum_keepdims_example",
"onnx/node/generated/test_reduce_sum_keepdims_random",
"onnx/node/generated/test_reduce_sum_negative_axes_keepdims_example",
- "onnx/node/generated/test_reduce_sum_negative_axes_keepdims_random",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_example",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_example_expanded",
"onnx/node/generated/test_reduce_sum_square_do_not_keepdims_random",
@@ -369,7 +366,6 @@
"onnx/node/generated/test_resize_tf_crop_and_resize",
"onnx/node/generated/test_resize_tf_crop_and_resize_axes_2_3",
"onnx/node/generated/test_resize_tf_crop_and_resize_axes_3_2",
- "onnx/node/generated/test_resize_tf_crop_and_resize_extrapolation_value",
"onnx/node/generated/test_resize_upsample_scales_cubic",
"onnx/node/generated/test_resize_upsample_scales_cubic_A_n0p5_exclude_outside",
"onnx/node/generated/test_resize_upsample_scales_cubic_align_corners",
@@ -381,7 +377,6 @@
"onnx/node/generated/test_resize_upsample_sizes_nearest_axes_3_2",
"onnx/node/generated/test_resize_upsample_sizes_nearest_floor_align_corners",
"onnx/node/generated/test_resize_upsample_sizes_nearest_not_larger",
- "onnx/node/generated/test_resize_upsample_sizes_nearest_not_smaller",
"onnx/node/generated/test_rnn_seq_length",
"onnx/node/generated/test_roialign_aligned_false",
"onnx/node/generated/test_roialign_aligned_true",
@@ -452,6 +447,8 @@
"onnx/node/generated/test_slice_neg_steps",
"onnx/node/generated/test_slice_negative_axes",
"onnx/node/generated/test_slice_start_out_of_bounds",
+ "onnx/node/generated/test_softsign",
+ "onnx/node/generated/test_softsign_example",
"onnx/node/generated/test_stft",
"onnx/node/generated/test_stft_with_window",
"onnx/node/generated/test_string_concat",
@@ -527,9 +524,7 @@
"onnx/node/generated/test_qlinearmatmul_3D_int8_float32",
"onnx/node/generated/test_qlinearmatmul_3D_uint8_float16",
"onnx/node/generated/test_qlinearmatmul_3D_uint8_float32",
- "onnx/node/generated/test_reduce_max_empty_set",
"onnx/node/generated/test_reduce_min_empty_set",
- "onnx/node/generated/test_reduce_sum_empty_axes_input_noop",
"onnx/node/generated/test_reduce_sum_empty_set_non_reduced_axis_zero",
"onnx/node/generated/test_resize_downsample_scales_linear_align_corners",
ONNX_PACKAGE_DIR = Path(onnx.__file__).parent | ||
ONNX_NODE_TESTS_ROOT = ONNX_PACKAGE_DIR / "backend/test/data/node" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for noticing that a git submodule isn't needed here, @zjgarvey !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, this __file__
solution is a bit cleaner and more robust than https://github.com/nod-ai/SHARK-TestSuite/blob/3b22038d687a13e3c46a8575d3e465133dfcbb7e/alt_e2eshark/onnx_tests/operators/generate_node.py#L29-L32
from site import getsitepackages
base_dir = getsitepackages()[0]
# you can also get the node tests from the git submodule instead by uncommenting:
# base_dir = str(Path(__file__).parents[3]) + "/third_party/onnx"
onnx_node_tests_dir = base_dir + "/onnx/backend/test/data/node/"
In particular, there may be multiple site packages directories, and that assumed that onnx was in the first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that is much cleaner. I'll make that change over in shark test suite, too.
ease of use in downstream projects and by developers who prefer to work directly | ||
with `.mlir` files and native (C/C++) tools. | ||
|
||
## Quickstart |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@renxida I took some ideas from nod-ai/SHARK-TestSuite#306 here when updating this README
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
Will also have to pull in ireers (commontools from SHARK-TestSuite) for unified testing/logs between models/onnx, but can do that in future when everything related to iree testing is here |
Possibly, but I'm leaning against that at the moment. I want to use this as a chance to learn from what we've built so far and start over in some ways.
|
Fixes #1.
This is a port of the generated ONNX operator ("node") tests from https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests. Major changes / simplifications:
requirements-dev.txt
andrequirements-iree.txt
files so running the tests doesn't require ONNX.npy
files, favoring.bin
filesrun_module_io_flags.txt
, notest_cases.json