YOLOV8 - Quantization Error: Trying to create tensor with negative dimension -37: [0, -37]

[khanonenet]

Hi, I am trying to perform Quantization of YOLOV8 with the notebook provided by openvino. And I am getting this error. Can anyone help me out with it? I am attaching images of the code snippets. Also one thing to mention here that I have trained a Yolov8-seg on a custom dataset.

Notebook being followed for Quantization: https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/230-yolov8-optimization/230-yolov8-optimization.ipynb

[glenn-jocher]

@khanonenet hello! It seems you're encountering an issue with quantization dimensions when using OpenVINO with YOLOv8. This error typically indicates that there's a mismatch between the expected input size of the model and the size being provided during the quantization process.

Here are a few steps you might consider to troubleshoot this issue:

1. Check Input Dimensions: Ensure that the input dimensions specified during the quantization process match those expected by your YOLOv8-seg model. The model expects a certain input size, and providing a different size could lead to negative dimension errors.

2. Model Configuration: Verify that the model configuration file used for training is consistent with the one used during quantization. Any discrepancies could cause issues.

3. Pre-processing: Confirm that the pre-processing steps (like resizing and normalization) are correctly applied to the input data before it's fed into the model for quantization.

4. Model Export: Make sure that the model was correctly exported for inference. If the model is not properly prepared, it might not work as expected with OpenVINO's optimization tools.

5. OpenVINO Compatibility: Check if there are any known compatibility issues with YOLOv8-seg and the version of OpenVINO you're using. It might be helpful to consult the OpenVINO documentation or community for any updates or patches.

6. Custom Layers: If your YOLOv8-seg model uses custom layers, ensure that these are supported by OpenVINO or that you have provided the necessary extensions.

If you continue to face issues, I recommend reaching out to the OpenVINO community or checking their documentation for more specific guidance on quantization with custom models. You can also refer to our documentation for more information on model export and compatibility considerations. Good luck! 🤞