Enable Onnx Model Tests #159
Conversation
| if os.path.exists(dest_file): | ||
| # model already in cache dir, skip download. | ||
| # TODO: skip model downloading based on some comparison / update flag | ||
| continue |
There was a problem hiding this comment.
I'm setting up code for checking hashes before skipping or redownloading and overwriting in iree_tests, FYI. Could copy or share that: https://github.com/ScottTodd/SHARK-TestSuite/blob/bedc744d6d617aec3ed9d8ff0c6a2cf862f9f11f/iree_tests/download_remote_files.py#L18-L77 (aiming to get that up into a PR and merged in a day or two, need to clean up that commit a bit)
| E2ESHARK_CHECK["postprocess"] = [ | ||
| (torch.nn.functional.softmax, [0], False, 0), | ||
| (torch.topk, [2], True, 1), | ||
| (torch.sort, [], True, 0), | ||
| ] | ||
| # E2ESHARK_CHECK["postprocess"] = [ | ||
| # (torch.nn.functional.softmax, [0], False, 0), | ||
| # (torch.topk, [2], True, 1), | ||
| # (torch.sort, [], True, 0), | ||
| # ] |
There was a problem hiding this comment.
What's the rationale for commenting out these postprocess steps? Would it be better to just remove the postprocessing code from these files? (as a IREE developer working nearly exclusively with native C/C++ tools, this logic in Python is totally foreign to me, so I'd prefer to leave it out if it isn't strictly needed for testing/development)
There was a problem hiding this comment.
You can match the raw value. But for int8 quantized model matching raw floating point output values does not work due to quantization introduced errors. So, typically, doing post processing such as classifying and then comparing is needed to compare with run on CPU
There was a problem hiding this comment.
Until e2eshark has CI (#104), can you share some output from running these new tests? I'd like to see logs (ideally with timestamps)
I'm specifically wondering about what the e2e test flow is for each of these programs, so they can be trusted by downstream users and included in IREE CI. What I'd be looking to see for IREE [testing] is of the form:
iree-import-onnx model.onnx -o model.mlir
iree-compile --iree-hal-target-backends=llvm-cpu model.mlir -o model.vmfb
iree-run-module --module=model.vmfb --device=local-task --input=input0.npy --expected_output=output.npy
When I tried some ONNX models from https://github.com/onnx/models yesterday, I hit issues on the iree-compile step. I'd like to know if this e2eshark flow is following that user flow or if it is doing something different as part of importing/compiling.
There was a problem hiding this comment.
I'll get the CI running with this updated PR today. Should have a report on overall status of everything tomorrow.
There was a problem hiding this comment.
iree-compile seems to be using passes which are different from the torch to linalg pipeline in torch MLIR
There was a problem hiding this comment.
FWIW, I just tried resnet50_vaiq_int8 from here and it imported and compiled successfully. Nice!
I'm wondering what's different about the source .onnx files here and in the onnx/models repo. Where were the Azure .onnx files sourced from? Did you have to edit them in any way?
There was a problem hiding this comment.
Are f32 and int32 models also working? I only see int8 here.
There was a problem hiding this comment.
These are Vitis AI int8 quantized models created off line. There would be _awq_int8 variant added later. For AIE, there will be no FP32 as AIE devices have only bf16 and int support. But having a FP32 model and using bf16 cast would be fine. That way with same model we can test IREE CPU FP32, bf16 and IREE AMD AIE (bf16)
|
Actually one thing, can you add a skip-run file for the models that are not ready yet? Will help for CI :) |
- adds support for downloading and running the e2e tests for onnx models
- currently supported models:
- onnx/models/DarkNet53_vaiq_int8
- onnx/models/DenseNet201_vaiq_int8
- onnx/models/EfficientNet_v2_s_vaiq_int8
- onnx/models/GoogLeNet_vaiq_int8
- onnx/models/Inception_v4_vaiq_int8
- onnx/models/LRASPP_vaiq_int8
- onnx/models/MNASNet_1_3_vaiq_int8
- onnx/models/MobileNetV3_small_vaiq_int8
- onnx/models/RDN_pytorch_vaiq_int8
- onnx/models/RegNet_y_8gf_vaiq_int8
- onnx/models/ResNet152_vaiq_int8
- onnx/models/retinanet_resnet50_fpn_vaiq_int8
- onnx/models/RRDB_ESRGAN_vaiq_int8
- onnx/models/ShuffleNet_v2_x2_0_vaiq_int8
- onnx/models/SqueezeNet_1_1_vaiq_int8
- onnx/models/VGG11_bn_vaiq_int8
- onnx/models/VGG19_vaiq_int8
- onnx/models/VideoResNet_vaiq_int8
- onnx/models/WideResNet_50_2_vaiq_int8
- onnx/models/YoloNetV3_vaiq_int8
- onnx/models/yolov8n_vaiq_int8
- onnx/models/u-net_brain_mri_vaiq_int8
More on the way :)
adds support for downloading and running the e2e tests for onnx models
currently supported models:
More on the way :)