OpenVINO vs Torch Inference

[alexriedel1]

Hello,
considering using a GPU for inferencing, will the inference using a openvino IR outperform the torch cuda inferencing in terms of speed?

Thanks, Alex

[samet-akcay]

Hi @alexriedel1, depends on the model. The use of GPU is advantageous in many cases.
OpenVINO accelerates the CPU inference, though.

Here is an example that I ran some time ago using NVIDIA 3090 GPU and i9-10980XE CPU. Please note that these stats are not official and experimental only.

[ke-dev]

Hi @alexriedel1, depends on the model. The use of GPU is advantageous in many cases. OpenVINO accelerates the CPU inference, though.

Here is an example that I ran some time ago using NVIDIA 3090 GPU and i9-10980XE CPU. Please note that these stats are not official and experimental only.

Hi, @samet-akcay how to get How to get the FP32 and INT8 models? I want to do more similar tests, can you provide some methods?

best regards

[alexriedel1]

Thanks @samet-akcay !
Though these stats are only experimental, I'd be highly interested to see more of these 😄

[samet-akcay]

We plan to add some jupyter notebooks to show this sort of experimentation

[samet-akcay]

Meanwhile, you could use the following benchmarking script to reproduce the throughput results for gpu and cpu
https://github.com/openvinotoolkit/anomalib/blob/development/tools/benchmarking/benchmark.py

[watertianyi]

@samet-akcay
inferencer = OpenVINOInferencer(path=openvino_model_path, metadata=openvino_metadata_path, device="GPU")
super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:104:
Check '!m_device_map.empty()' failed at src/plugins/intel_gpu/src/plugin/plugin.cpp:383:
[GPU] Can't get PERFORMANCE_HINT property as no supported devices found or an error happened during devices query.
[GPU] Please check OpenVINO documentation for GPU drivers setup guide.

[watertianyi]

inferencer = OpenVINOInferencer(path=openvino_model_path, metadata=openvino_metadata_path, device="CPU")

image = cv2.resize(image, tuple(list(self.input_blob.shape)[2:][::-1]))
RuntimeError: Exception from src/core/src/partial_shape.cpp:266:
to_shape was called on a dynamic shape.