teNNo

The Size Expansion algorithm enables obstacle detection by tracking size changes of objects between frames. This repository contains the code used to perform the experiments performed in the Bachelor's thesis "YOLO-based Obstacle Avoidance for Drones" at Leiden University. It contains a controller for the Ryze Tello drone that was used, enabling the streaming of video and drone control using the keyboard.

Size Expansion uses object detection to find object in images from the drone's camera to avoid them. Multiple detection methods are included. These can be found in the detectors directory

• SIFT: Uses SIFT feature matching to identify objects and calculate sizes. Described in the paper by Al-Kaff et al.
• Darknet: Runs the YOLOv4 model used in the thesis.
• TensorFlow: Can run models found in the Tensorflow object detection model zoo.
• EfficientDet: Test implementation of EfficientDet. Currently only detects objects

How to install

This program is written in Python3. Make sure Python3 and pip3 are installed correctly.

Base + SIFT Detector

Packages needed for the main program:

pip install -r requirements.txt

Install the latest version of TelloPy from source: More information here. Note: the version on PyPi is out of data and does not work!

git clone https://github.com/hanyazou/TelloPy
cd TelloPy
sudo apt-get install python3-setuptools
pip install wheel
python setup.py bdist_wheel
pip install dist/tellopy-*.dev*.whl --upgrade

OpenCV libraries (v4.3.0 used) can also be installed from source here, together with the extra modules (opencv-contrib) necessary for SIFT here. For install directions look here. For installing into a Conda environment see here. Installing via PyPi might also be possible if the wheel is updated to at least v4.3.0. (Note: the most recent version of OpenCV-Python is now uploaded to PyPi)

Installing these packages will allow you run the main controller functionality and the SIFT obstacle detector.

Other detector modules

Other detector modules can be found in the detectors directory. Each contains a separate README with install instructions

Using the gpu version of the CNN object detectors is highly recommended. For this you need to install CUDA.

How to use

Simply run the following command to start the program:

python tenno.py

The following flags are supported:

Option	Action	Choices
-d, --detector	Chooses object detector	{sift, yolodark, tf, eff}
-z, --zfile	Adds a zone weight file for YOLO detector	Path to file
-t, --test	Enables experiment mode	{ratio_size, ratio_kp, avoid}
-o, --output	Output file of experiment results	Path to file
-r, --record	Enables recording of every flight in test modes

To get full functionality using the object detector of choice call:

python tenno.py -d sift
python tenno.py -d yolodark

Keyboard controls

Key	Action
w, a, s, d	Move drone horizontally
Space, Shift	Move drone vertically
q, e	Turn in place
o	Toggle detector on/off
l	Toggle logging on/off
p, 1	Resets current position to (0,0,0)
h	Sets waypoint to (0,0,0)
r	Toggle video recording on/off
2	Toggle waypoint flying on/off
3	Reset position and start flying to waypoint with obstacle detection
-	Toggle HUD on/off
Esc	Land drone and quit program

Test modes

The ratio_size and ratio_kp test modes enable the adjustments of hull size ratio and keypoint size ratio on the fly. Experiments are started using the '3' key. The drone flies forward, flying back when an obstacle is detected, outputting the distance to the obstacle.

Key	Action
m	Increase test ratio by 0.025
n	Decrease test ratio by 0.025

The avoid test enables the default threshold values. The test flies the drone forward, avoiding any detected obstacles.

Key	Action
Home	Confirm successful avoidance
End	Confirm failed avoidance