Hello folks, i had promised to post this a while back, but i got a bit swamped with work; hopefully this will give someone at least a few basic ideas as to what they can do to show off the use of AI Vision with degirum a bit more.
Firstly lets start with the setup, i am using a raspberry pi 5, with the 8gb and a Hailo8L configuration, make sure to install all the degirum and hailo bits necessary for proper function, picamera2 should be installed by default in your raspberry pi, you can check with a quick:
rpicam-hello
You will also need to have pyqt5 installed, which you can do with sudo by:
sudo apt-get install python3-pyqt5
Lastly we need opencv for two lines of code so:
sudo apt-get install opencv-python
Onto the gui
Our imports will look something like
from PyQt5 import QtCore
from PyQt5.QtCore import Qt, QTimer, pyqtSignal, QDir
from PyQt5.QtGui import QImage, QPixmap
from PyQt5.QtWidgets import QApplication, QLabel, QFileSystemModel, QMainWindow, QVBoxLayout, QWidget, QFileDialog, QInputDialog, QHBoxLayout, QPushButton, QMessageBox
from picamera2 import Picamera2
import numpy as np
import degirum as dg, degirum_tools
import cv2
import libcamera
That certainly seems like a lot of thing we wont be using for this tutorial, but if you want to take things further than the scope of this you will certainly need it, so maybe keep it around till then.
The body of the GUI will look something like this:
class AI_Vision(QWidget): #You can swap this to QMainWindow
closed = pyqtSignal()
def __init__(self):
super().__init__()
inference_host_address = '@local' # set to @local if you want to run inference on your local machine, or change if necessary
zoo_url = '/path/to/your/AI/model.json'
model_name = 'model' #match to your models actual name
self.model = dg.load_model(
model_name=model_name,
inference_host_address=inference_host_address,
zoo_url=zoo_url,
output_confidence_threshold = 0.3, #Or whatever you choose as min threshold
overlay_font_scale = 2.5, #I add this because the default font is a bit small
overlay_show_probabilities = True #Also a preference since i enjoy seeing how confident the model is
)# set model name, inference host address, zoo url, token, and image source
self.picam2 = Picamera2()
self.start_camera() #Here we will set the basic configuration for our camera
self.load_gui() #Here we will add all of our gui elements
I am using classes here as i am more used to it, but you are welcome to just make a single file without any classes if you prefer that, its also worht mentioning that this body is made as to have special focus on those of us running local inference on the raspberry pi 5 with its AI hat.
Now lets talk about our gui and camera setup, as i believe here is where we could all take a very different approach and still have good results.
Firstly our camera:
def start_camera(self):
try:
self.picam2.stop() #in case anything else was running
#Here, i add the resolution and mode in a specific manner to avoid having the camera guess and use whatever it prefers
picam2_pConfig = self.picam2.create_preview_configuration(main={"format": 'RGB888',"size": (1920, 1080)}, raw=self.picam2.sensor_modes[1]) #You can verify your camera modes using rpicam-hello --list-cameras
self.picam2.set_controls({"AeFlickerMode":libcamera.controls.AeFlickerModeEnum.Manual}) #Preference, makes it so that camera flicker is null
self.picam2.configure(picam2_pConfig)
self.picam2.start()
self.timer = QTimer(self)
self.timer.timeout.connect(self.update_camera_view) #The main function that will be writing every frame in our case
self.timer.start(1) # Update the view in whatever time frame you choose
except Exception as e:
print(e)
pass
Now our gui:
def load_gui(self):
layout_h = QHBoxLayout()
layout_v = QVBoxLayout()
self.label = QLabel(self) #You need a label to add the image in pyqt5, you can use other methods, but this is my preferred
self.label.setGeometry(0,0,1920,1080) #Make sure to match this resolution to your camera
layout_h.addWidget(self.label)
self.setWindowFlags(Qt.FramelessWindowHint) #You can remove both the flag and state if you prefer a regular floating window
self.setWindowState(QtCore.Qt.WindowFullScreen)
self.setLayout(layout_h)
self.setStyleSheet("background-color: black;") #Block this out if you arent using fullscreen either
self.setContentsMargins(0,0,0,0) #Modify this only if you want the borders around your image to look wider
The gui and its elements can be as diverse as customizable as you want them, you can add a QSlider to control something like brightness with picam2 and its integrated functions picam2.set_controls({"Brightness": self.QSlider.value()}), you can add a QPushButton and connect it to a function that captures pictures or takes a short video.
Onto the next part, which is our update view function, but first a quick explanation as to what you will see next, with the normal vision of the camera you could have a single function for all of it, but given the fact that degirum docs recommend a frame generator function i created a split funcion:
def frame_gen(self):
self.frame = self.picam2.capture_array() #Normally i would have this single line of code in the main camera view function as i mentioned
yield self.frame
def update_camera_view(self):
for result in self.model.predict_batch(self.frame_gen()): #Remember to make sure you are matching your models name
self.frame_drawn = cv2.resize(result.image_overlay, (1920, 1080), interpolation = cv2.INTER_AREA) #We make sure that the image is using the same resolution we are handling in the previous instances
frame_rgb = cv2.cvtColor(self.frame_drawn, cv2.COLOR_BGR2RGB) #Opencv handles the image, which means that the channels are swapped, so we make them standard
height, width, channel = frame_rgb.shape
bytes_per_line = 3 * width
q_image = QImage(frame_rgb.data, self.label.width(), self.label.height(), bytes_per_line, QImage.Format_RGB888) #This is how pyqt5 handles images
pixmap = QPixmap.fromImage(q_image)
self.label.setPixmap(pixmap) #We set the image in our label now
self.label.setContentsMargins(0,0,0,0) #Another line of preference in this code, you can cut it without affecting anything
Now you can finally have an image in a gui showing off your model and raspberry pi capacity, but you can still add a bit more lines of code for proper handling or execution like a close event:
def closeEvent(self, event):
# Clean up
try:
self.timer.stop()
self.closed.emit()
self.picam2.stop()
self.picam2.close()
event.accept()
except Exception as e:
self.picam2.stop()
self.picam2.close()
print(e)
and a if __name__ == "__main__": function to ensure proper running.
If you would like to use this in a kiosk or autostart manner i recommend launching the application or gui with a .sh file including the following line:
sudo /path/to/your/venv/python /path/to/your/app.py
This helps to avoid any Missing Degirum or other similar errors
This is my first time posting a tutorial or how to guide as long as this, so it may be a bit rambly, or i may not have explained things in depth enough, but if you have any questions i can try to help in the comments.