Using the Picamera2 library with TensorFlow Lite

Last week we announced a preview release of the new Picamera2 library, built on top of the open source libcamera framework, which replaced the Picamera library deprecated during the release of Bullseye back in November.

TensorFlow Lite performing real-time object detection using the Raspberry Pi Camera and Picamera2

In the past I’ve spent a lot of time working with TensorFlow and TensorFlow Lite on Raspberry Pi and other platforms and, as a result, I spent a lot of time working with the old Picamera library. So for me, it was time to figure out how to get Picamera2 and TensorFlow talking.

The aim is to put together something that’ll use the Picamera2 library and its QtGL preview window, and overlay real-time object detection on the stream. So, let’s get started!

Installing the Picamera2 library

Right now Picamera2 is in preview release, which means installing it is significantly more complicated than it will eventually be, because you first need to build and install a fork of the libcamera library along with some DRM/KMS bindings directly from GitHub:

$ sudo apt update
$ sudo apt install -y libboost-dev
$ sudo apt install -y libgnutls28-dev openssl libtiff5-dev
$ sudo apt install -y qtbase5-dev libqt5core5a libqt5gui5 libqt5widgets5
$ sudo apt install -y meson
$ sudo pip3 install pyyaml ply
$ sudo pip3 install --upgrade meson
$ sudo apt install -y libglib2.0-dev libgstreamer-plugins-base1.0-dev
$ git clone --branch picamera2 https://github.com/raspberrypi/libcamera.git
$ cd libcamera
$ meson build --buildtype=release -Dpipelines=raspberrypi -Dipas=raspberrypi -Dv4l2=true -Dgstreamer=enabled -Dtest=false -Dlc-compliance=disabled -Dcam=disabled -Dqcam=enabled -Ddocumentation=disabled -Dpycamera=enabled
$ ninja -C build 
$ sudo ninja -C build install
$ cd
$ git clone https://github.com/tomba/kmsxx.git
$ cd kmsxx
$ git submodule update --init
$ sudo apt install -y libfmt-dev libdrm-dev
$ meson build
$ ninja -C build

before then going on to install the Picamera2 library itself:

$ cd
$ sudo pip3 install pyopengl
$ sudo apt install python3-pyqt5
$ git clone https://git@github.com:raspberrypi/picamera2.git
$ sudo pip3 install opencv-python==4.4.0.46
$ sudo apt install -y libatlas-base-dev
$ sudo pip3 install numpy --upgrade
$ cd
$ git clone https://github.com/RaspberryPiFoundation/python-v4l2.git

To make everything run, you will also have to set your PYTHONPATH environment variable. For example, you could put the following in your .bashrc file:

export PYTHONPATH=/home/pi/picamera2:/home/pi/libcamera/build/src/py:/home/pi/kmsxx/build/py:/home/pi/python-v4l2

Installing TensorFlow Lite

Since we’re going to be inferencing, rather than training, from our Python code we can get away with installing the lightweight TensorFlow Lite runtime library along with some other things we’ll need:

$ sudo apt install build-essentials
$ sudo apt install git
$ sudo apt install libatlas-base-dev
$ sudo apt install python3-pip
$ pip3 install tflite-runtime
$ pip3 install opencv-python==4.4.0.46
$ pip3 install pillow
$ pip3 install numpy

This is significantly easier than installing the full TensorFlow package.

Using TensorFlow Lite

Once everything is installed we can go ahead and build our demo, and as everyone knows the obvious objects to look for are apples🍎 and of course bananas🍌. Because the importance of bananas to machine learning researchers can not be overstated.

The code to do this is shown below. Here we start the camera with a preview window, and then repeatedly pass the image buffer to TensorFlow, which will run our object detection model on the image. If we detect any objects we’ll then draw a rectangle around them, and if we passed our code a label file, we’ll label our detected objects.

import tflite_runtime.interpreter as tflite

import sys
import os
import argparse

import cv2
import numpy as np
from PIL import Image
from PIL import ImageFont, ImageDraw

from qt_gl_preview import *
from picamera2 import *

normalSize = (640, 480)
lowresSize = (320, 240)

rectangles = []

def ReadLabelFile(file_path):
  with open(file_path, 'r') as f:
    lines = f.readlines()
  ret = {}
  for line in lines:
    pair = line.strip().split(maxsplit=1)
    ret[int(pair[0])] = pair[1].strip()
  return ret

def DrawRectangles(request):
   stream = request.picam2.stream_map["main"]
   fb = request.request.buffers[stream]
   with fb.mmap(0) as b:
       im = np.array(b, copy=False, dtype=np.uint8).reshape((normalSize[1],normalSize[0], 4))

       for rect in rectangles:
          print(rect)
          rect_start = (int(rect[0]*2) - 5, int(rect[1]*2) - 5)
          rect_end = (int(rect[2]*2) + 5, int(rect[3]*2) + 5)
          cv2.rectangle(im, rect_start, rect_end, (0,255,0,0))
          if len(rect) == 5:
            text = rect[4]
            font = cv2.FONT_HERSHEY_SIMPLEX
            cv2.putText(im, text, (int(rect[0]*2) + 10, int(rect[1]*2) + 10), font, 1, (255,255,255),2,cv2.LINE_AA)
       del im

def InferenceTensorFlow( image, model, output, label=None):
   global rectangles

   if label:
       labels = ReadLabelFile(label)
   else:
       labels = None

   interpreter = tflite.Interpreter(model_path=model, num_threads=4)
   interpreter.allocate_tensors()

   input_details = interpreter.get_input_details()
   output_details = interpreter.get_output_details()
   height = input_details[0]['shape'][1]
   width = input_details[0]['shape'][2]
   floating_model = False
   if input_details[0]['dtype'] == np.float32:
       floating_model = True

   rgb = cv2.cvtColor(image,cv2.COLOR_GRAY2RGB)
   initial_h, initial_w, channels = rgb.shape

   picture = cv2.resize(rgb, (width, height))

   input_data = np.expand_dims(picture, axis=0)
   if floating_model:
      input_data = (np.float32(input_data) - 127.5) / 127.5

   interpreter.set_tensor(input_details[0]['index'], input_data)

   interpreter.invoke()

   detected_boxes = interpreter.get_tensor(output_details[0]['index'])
   detected_classes = interpreter.get_tensor(output_details[1]['index'])
   detected_scores = interpreter.get_tensor(output_details[2]['index'])
   num_boxes = interpreter.get_tensor(output_details[3]['index'])

   rectangles = []
   for i in range(int(num_boxes)):
      top, left, bottom, right = detected_boxes[0][i]
      classId = int(detected_classes[0][i])
      score = detected_scores[0][i]
      if score > 0.5:
          xmin = left * initial_w
          ymin = bottom * initial_h
          xmax = right * initial_w
          ymax = top * initial_h
          box = [xmin, ymin, xmax, ymax]
          rectangles.append(box)
          if labels:
              print(labels[classId], 'score = ', score)
              rectangles[-1].append(labels[classId])
          else:
              print ('score = ', score)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model', help='Path of the detection model.', required=True)
    parser.add_argument('--label', help='Path of the labels file.')
    parser.add_argument('--output', help='File path of the output image.')
    args = parser.parse_args()

    if ( args.output):
      output_file = args.output
    else:
      output_file = 'out.jpg'

    if ( args.label ):
      label_file = args.label
    else:
      label_file = None

    picam2 = Picamera2()
    preview = QtGlPreview(picam2)
    config = picam2.preview_configuration(main={"size": normalSize},
                                          lores={"size": lowresSize, "format": "YUV420"})
    picam2.configure(config)

    stride = picam2.stream_configuration("lores")["stride"]
    picam2.request_callback = DrawRectangles

    picam2.start()

    while True:
        buffer = picam2.capture_buffer("lores")
        grey = buffer[:stride*lowresSize[1]].reshape((lowresSize[1], stride))
        result = InferenceTensorFlow( grey, args.model, output_file, label_file )

if __name__ == '__main__':
  main()

If you want to experiment with TensorFlow Lite, Picamera2 and this code, I’ve pushed it to the Picamera2 GitHub along with the MobileNet V2 model and COCO dataset label file I used to detect apples🍎 and bananas🍌.

15 comments
Jump to the comment form

Avatar

Hi, thank you for this, an up to date PiCamera2 example is exactly what I need.
Did you use the 64 bit or 32 bit Bullseye for this demo?

Reply to Anders

David Plowman

I believe this is using a 32-bit image just because OpenCV installs instantly, whereas on a 64-bit image it seems to involve a multi-hour recompile. But aside from that, this example should, so far as I know, work just the same.

Reply to David Plowman

Avatar

I am interested about the inference FPS for this concrete example.
Further it would be interesting to see whether the inference FPS could be further boosted using TFlite XNNPACK.

Reply to petro

Avatar

2 corrections to “Installing the Picamera2 library”:
– cd kmxss -> cd kmsxx
– cloning picamera2: git clone https://github.com/raspberrypi/picamera2.git

Reply to Ton van Overbeek

David Plowman
Avatar

Cool, but can it tell a bunny from a cat ?

Reply to Ben

Ashley Whittaker

Alasdair we need a bunny/cat test plz. Alex has a bunny. You can find a cat. Wait, your dog is the size of a cat. Perfect.

Reply to Ashley Whittaker

Avatar

What about telling a Pi4B 2g from a Pi 4B 4g? Because that’s something I still haven’t learnt how to do properly!

Reply to Carlos Luna

Avatar

:-)
Some things are important.

——-
Just a thought, thinking about it, it would make a great ‘advanced’ RPFoundation Learn course for an advanced Wildlife/Nature Cam as a next step from the standard Birdbox/Wildlife cam project you have.)

No doubt getting everything converted to just the new modules/software is going to take time, once it is out of testing).

Thanks to all plugging away at this, and the testers.

Reply to Ben

Liz Upton

Eben and I have cats, but they’re black cats and therefore broadly speaking unphotographable. Let’s use Alasdair’s dog.

Reply to Liz Upton

Ashley Whittaker

If Alasdair is stuck on a boat for his holiday… then his cat-sized dog must be at home and [briefly] kidnappable…

Reply to Ashley Whittaker

David Plowman

Just to let everyone know that, as Picamera2 is still a very fast moving target, you also now need to install the python-v4l2 module. The official version is unmaintained and horribly out-of-date, so please use
cd
git clone https://github.com/RaspberryPiFoundation/python-v4l2.git
Also, it will need adding to your PYTHONPATH, so you will have:
export PYTHONPATH=/home/pi/picamera2:/home/pi/libcamera/build/src/py:/home/pi/kmsxx/build/py:/home/pi/python-v4l2
I’ll see if I can get this added to the main text above, but in the meantime hopefully this will be helpful. Thanks!

Reply to David Plowman

Avatar

Please don’t use dollars in the bash examples, so it’s easier to copy and paste.

Reply to Secret-chest

Avatar

Please can this work with the raspberry 3?

Reply to Nathan

Alasdair Allan

Should work out of the box on the Raspberry Pi 3. It’ll run slower, but it’ll work.

Reply to Alasdair Allan

Leave a Comment