Live detection optimization

General
, ,
1

How Can I Make My Pi Camera Object Detection Code More Responsive?

Hi everyone,

I’m working on a real-time object detection project using the DeGirum model with a Raspberry Pi camera and I’m looking to make the code more responsive particularly in terms of processing speed and display fluidity. Here’s my current code:

import degirum as dg
import degirum_tools
from picamera2 import Picamera2
import cv2

# Initialize Picamera2
camera = Picamera2()
camera.configure(camera.create_video_configuration(main={"format": "RGB888"}))
camera.start()

model = dg.load_model(
    model_name="yolo11n_silu_coco--640x640_quant_hailort_hailo8_1",
    inference_host_address="@local",
    zoo_url="/home/rohan/degirum_project/models"
)

with degirum_tools.Display("Object Detection Camera") as output_display:
    while True:
        frame = camera.capture_array("main")
        result = model.predict(frame)
        result._show_probabilities = True
        output_display.show(result)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

cv2.destroyAllWindows()

Any advice or suggestions would be greatly appreciated!

Thanks in advance.

2 Likes
2

Hi @rohanpatrick568 ,

To greatly speedup the inference you need to use streaming prediction API: model.predict_batch, because single-frame synchronous prediction API model.predict() always waits for result, so no pipelining happens, which reduces FPS.

This streaming API accepts the frame iterator as argument, so we need to wrap the camera frame capturing code into a generator function source(), see code below.

Few more suggestions:

  1. I put overlay_show_probabilities = True into load_model call to enable showing probabilities for all results.
  2. I removed cv2.destroyAllWindows() call because with degirum_tools.Display will close its window.
  3. I removed if cv2.waitKey(1) & 0xFF == ord('q') call because output_display.show does exactly that and it throws KeyboardInterrupt exception on pressing q or x. This exception is then caught in degirum_tools.Display.__exit__ so it will not be visible to you.
def source():
    camera = Picamera2()
    camera.configure(camera.create_video_configuration(main={"format": "RGB888"}))
    camera.start()
    while True:
        frame = camera.capture_array("main")
        yield frame

model = dg.load_model(
    model_name="yolo11n_silu_coco--640x640_quant_hailort_hailo8_1",
    inference_host_address="@local",
    zoo_url="/home/rohan/degirum_project/models",
    overlay_show_probabilities = True
)

with degirum_tools.Display("Object Detection Camera") as output_display:
    for result in model.predict_batch(source()):
        output_display.show(result)
4 Likes
3

Hi,

just put my two cents.

I also used degirum_tools.predict_stream that implement model.predict_batch but found out the best mix for real time scenario implementing Gizmos and playing with allow_drop parameter.

import degirum as dg, degirum_tools
import degirum_tools.streams as dgstreams

video_source="rtsp://10.1.0.42:554/chID=2&streamType=sub"


# Loading a model
model = dg.load_model(
    model_name = "yolo11s_coco--640x640_quant_hailort_hailo8_1",
    inference_host_address = "@local",
    zoo_url = "/home/antonio/zoo",
    overlay_show_probabilities = True,
    overlay_line_width = 2
)

# Gizmo rulez
source = dgstreams.VideoSourceGizmo(video_source)
resize = dgstreams.ResizingGizmo(640, 640)
detection = dgstreams.AiSimpleGizmo(model, allow_drop=True)
display = dgstreams.VideoDisplayGizmo("Detection", allow_drop=True, show_ai_overlay=True, show_fps=True)

dgstreams.Composition(source >> resize >> detection >> display).start(detect_bottlenecks=True)

I

2 Likes
4

@anto.ferro , please be advised that detect_bottlenecks=True is only for debugging. It basically sets allow_drop=True for every queue in the composition. Generally, this is not what you want - a lot of frames may be dropped here and there.

1 Like
5

@vladk , I just removed all the allow_drop=True as you suggested.

I added them to mitigate the delay introduced by preprocess of a 720p frame probably.

After I added the frame resize in the pipeline the allow_drop is not necessary.

Thanks!

6

Hi @anto.ferro

Happy to see that you were able to use gizmos. I just wanted to point out that when resize is outside the AiSimpleGizmo, all overlays are with respect to the resized image (which will be letterboxed). If you need overlay on original image, some extra logic needs to be added.

1 Like
7

Hi @shashi

In my case, the use case doesn’t need to have the overlay on original image because I’m working in parallel to NVR.

I got your point and at the beginning I was strugling working on original image due to some big latency issue.

I will use the overlays on resized image in parallel to the original one without annotation, it is a good compromise for me.

Thanks

1 Like
8

I have to correct my self about the display performance I mentioned before.

If you use remote Xserver in Windows (as I do) exporting DISPLAY from Linux, the Xserver will work synchronously with Gizmo Display and you will get low FPS on display maybe because you moved the window.

I hope it can be useful to evaluate performance in future.