Guidance for performance optimisation on Edge devices

[acode-x]

Hi,

I'm currently running inference in a arm64v8/debian:12-slim Docker container on an edge device, with 2 CPU cores and 1GB of RAM. Initially, I tested ONNX Runtime, but the inference time for a single frame was ~2 seconds.
After switching to tflite-runtime and utilizing the recent fix from examples/YOLOv8-TFLite-Python/main.py, I managed to achieve 1 FPS with the yolo11n_full_integer_quant.tflite model.

I haven't trained any models yet, as yolo11n is working well for my needs.
However, I am looking for further optimization techniques to reach ~10 FPS while detecting only 4 specific classes.

Appreciate any guidance on same.
Thanks!

[UltralyticsAssistant]

👋 Hello @acode-x, thank you for reaching out and being a part of the Ultralytics community 🚀!

It sounds like you're doing some exciting work with your edge device setup. For optimizing inference performance, especially on limited hardware like your edge device, here are a few general tips that might help:

• Model Optimization: You've already switched to tflite-runtime, which is great for edge deployment. You might also want to explore using integer-only quantization or experimenting with smaller model variants to further reduce the load.

• Class Reduction: If you're only interested in detecting 4 specific classes, make sure your model is trained specifically for those classes. This can sometimes offer speed advantages.

• Platform-Specific Optimizations: Check if there are platform-specific libraries or acceleration modes that can be used with your hardware for even faster inference speeds. For ARM processors, look into hardware accelerators or GPU support if available.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

For more detailed guidance or troubleshooting, our Ultralytics engineers will be able to assist you soon.

In the meantime, don't forget to explore our comprehensive Docs for more tips and guidance. You can also join our Discord for real-time support and discussions 🎧, or check out our Discourse and Subreddit for community insights and shared experiences.

Upgrade

Ensure you are using the latest package versions to avoid any known issues. Upgrade to the latest ultralytics package with:

pip install -U ultralytics

Environments

For easy access to testing environments, you might consider doing so in any of the verified environments we support, like Google Colab or Docker.

Status

This badge reflects our current CI status, confirming that all Ultralytics CI tests are passing and YOLO models are functioning correctly across different OS platforms.

Hope these insights are helpful, and we're excited to see how your project progresses! 🚀

[glenn-jocher]

@acode-x to improve performance, consider reducing the input image size, using a smaller model variant, or applying model pruning and quantization techniques. Additionally, ensure that your TFLite model is optimized for your specific hardware, and explore using a Coral Edge TPU for further acceleration.

[acode-x]

Thanks @glenn-jocher !

Currently I have this:

model = YOLO("yolo11n.pt")
model.export(format="tflite", imgsz=640, int8=True)

It is better to prune yolo11n, retrain, export to tflite (or) export yolo11n to tfile, prune, retrain?

[glenn-jocher]

@acode-x it's generally more effective to prune, retrain, and then export to TFLite for optimal performance.