Using the Picamera2 library with TensorFlow Lite

Last week we announced a preview release of the new Picamera2 library, built on top of the open source libcamera framework, which replaced the Picamera library deprecated during the release of Bullseye back in November.

TensorFlow Lite performing real-time object detection using the Raspberry Pi Camera and Picamera2

In the past I’ve spent a lot of time working with TensorFlow and TensorFlow Lite on Raspberry Pi and other platforms and, as a result, I spent a lot of time working with the old Picamera library. So for me, it was time to figure out how to get Picamera2 and TensorFlow talking.

The aim is to put together something that’ll use the Picamera2 library and its QtGL preview window, and overlay real-time object detection on the stream. So, let’s get started!

Installing the Picamera2 library

Right now Picamera2 is in preview release, which means installing it is significantly more complicated than it will eventually be, because you first need to build and install a fork of the libcamera library along with some DRM/KMS bindings directly from GitHub:

$ sudo apt update
$ sudo apt install -y libboost-dev
$ sudo apt install -y libgnutls28-dev openssl libtiff5-dev
$ sudo apt install -y qtbase5-dev libqt5core5a libqt5gui5 libqt5widgets5
$ sudo apt install -y meson
$ sudo pip3 install pyyaml ply
$ sudo pip3 install --upgrade meson
$ sudo apt install -y libglib2.0-dev libgstreamer-plugins-base1.0-dev
$ git clone --branch picamera2
$ cd libcamera
$ meson build --buildtype=release -Dpipelines=raspberrypi -Dipas=raspberrypi -Dv4l2=true -Dgstreamer=enabled -Dtest=false -Dlc-compliance=disabled -Dcam=disabled -Dqcam=enabled -Ddocumentation=disabled -Dpycamera=enabled
$ ninja -C build 
$ sudo ninja -C build install
$ cd
$ git clone
$ cd kmsxx
$ git submodule update --init
$ sudo apt install -y libfmt-dev libdrm-dev
$ meson build
$ ninja -C build

before then going on to install the Picamera2 library itself:

$ cd
$ sudo pip3 install pyopengl
$ sudo apt install python3-pyqt5
$ git clone https://[email protected]:raspberrypi/picamera2.git
$ sudo pip3 install opencv-python==
$ sudo apt install -y libatlas-base-dev
$ sudo pip3 install numpy --upgrade
$ cd
$ git clone

To make everything run, you will also have to set your PYTHONPATH environment variable. For example, you could put the following in your .bashrc file:

export PYTHONPATH=/home/pi/picamera2:/home/pi/libcamera/build/src/py:/home/pi/kmsxx/build/py:/home/pi/python-v4l2

Installing TensorFlow Lite

Since we’re going to be inferencing, rather than training, from our Python code we can get away with installing the lightweight TensorFlow Lite runtime library along with some other things we’ll need:

$ sudo apt install build-essentials
$ sudo apt install git
$ sudo apt install libatlas-base-dev
$ sudo apt install python3-pip
$ pip3 install tflite-runtime
$ pip3 install opencv-python==
$ pip3 install pillow
$ pip3 install numpy

This is significantly easier than installing the full TensorFlow package.

Using TensorFlow Lite

Once everything is installed we can go ahead and build our demo, and as everyone knows the obvious objects to look for are apples🍎 and of course bananas🍌. Because the importance of bananas to machine learning researchers can not be overstated.

The code to do this is shown below. Here we start the camera with a preview window, and then repeatedly pass the image buffer to TensorFlow, which will run our object detection model on the image. If we detect any objects we’ll then draw a rectangle around them, and if we passed our code a label file, we’ll label our detected objects.

import tflite_runtime.interpreter as tflite

import sys
import os
import argparse

import cv2
import numpy as np
from PIL import Image
from PIL import ImageFont, ImageDraw

from qt_gl_preview import *
from picamera2 import *

normalSize = (640, 480)
lowresSize = (320, 240)

rectangles = []

def ReadLabelFile(file_path):
  with open(file_path, 'r') as f:
    lines = f.readlines()
  ret = {}
  for line in lines:
    pair = line.strip().split(maxsplit=1)
    ret[int(pair[0])] = pair[1].strip()
  return ret

def DrawRectangles(request):
   stream = request.picam2.stream_map["main"]
   fb = request.request.buffers[stream]
   with fb.mmap(0) as b:
       im = np.array(b, copy=False, dtype=np.uint8).reshape((normalSize[1],normalSize[0], 4))

       for rect in rectangles:
          rect_start = (int(rect[0]*2) - 5, int(rect[1]*2) - 5)
          rect_end = (int(rect[2]*2) + 5, int(rect[3]*2) + 5)
          cv2.rectangle(im, rect_start, rect_end, (0,255,0,0))
          if len(rect) == 5:
            text = rect[4]
            font = cv2.FONT_HERSHEY_SIMPLEX
            cv2.putText(im, text, (int(rect[0]*2) + 10, int(rect[1]*2) + 10), font, 1, (255,255,255),2,cv2.LINE_AA)
       del im

def InferenceTensorFlow( image, model, output, label=None):
   global rectangles

   if label:
       labels = ReadLabelFile(label)
       labels = None

   interpreter = tflite.Interpreter(model_path=model, num_threads=4)

   input_details = interpreter.get_input_details()
   output_details = interpreter.get_output_details()
   height = input_details[0]['shape'][1]
   width = input_details[0]['shape'][2]
   floating_model = False
   if input_details[0]['dtype'] == np.float32:
       floating_model = True

   rgb = cv2.cvtColor(image,cv2.COLOR_GRAY2RGB)
   initial_h, initial_w, channels = rgb.shape

   picture = cv2.resize(rgb, (width, height))

   input_data = np.expand_dims(picture, axis=0)
   if floating_model:
      input_data = (np.float32(input_data) - 127.5) / 127.5

   interpreter.set_tensor(input_details[0]['index'], input_data)


   detected_boxes = interpreter.get_tensor(output_details[0]['index'])
   detected_classes = interpreter.get_tensor(output_details[1]['index'])
   detected_scores = interpreter.get_tensor(output_details[2]['index'])
   num_boxes = interpreter.get_tensor(output_details[3]['index'])

   rectangles = []
   for i in range(int(num_boxes)):
      top, left, bottom, right = detected_boxes[0][i]
      classId = int(detected_classes[0][i])
      score = detected_scores[0][i]
      if score > 0.5:
          xmin = left * initial_w
          ymin = bottom * initial_h
          xmax = right * initial_w
          ymax = top * initial_h
          box = [xmin, ymin, xmax, ymax]
          if labels:
              print(labels[classId], 'score = ', score)
              print ('score = ', score)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model', help='Path of the detection model.', required=True)
    parser.add_argument('--label', help='Path of the labels file.')
    parser.add_argument('--output', help='File path of the output image.')
    args = parser.parse_args()

    if ( args.output):
      output_file = args.output
      output_file = 'out.jpg'

    if ( args.label ):
      label_file = args.label
      label_file = None

    picam2 = Picamera2()
    preview = QtGlPreview(picam2)
    config = picam2.preview_configuration(main={"size": normalSize},
                                          lores={"size": lowresSize, "format": "YUV420"})

    stride = picam2.stream_configuration("lores")["stride"]
    picam2.request_callback = DrawRectangles


    while True:
        buffer = picam2.capture_buffer("lores")
        grey = buffer[:stride*lowresSize[1]].reshape((lowresSize[1], stride))
        result = InferenceTensorFlow( grey, args.model, output_file, label_file )

if __name__ == '__main__':

If you want to experiment with TensorFlow Lite, Picamera2 and this code, I’ve pushed it to the Picamera2 GitHub along with the MobileNet V2 model and COCO dataset label file I used to detect apples🍎 and bananas🍌.


Anders avatar

Hi, thank you for this, an up to date PiCamera2 example is exactly what I need.
Did you use the 64 bit or 32 bit Bullseye for this demo?

David Plowman avatar

I believe this is using a 32-bit image just because OpenCV installs instantly, whereas on a 64-bit image it seems to involve a multi-hour recompile. But aside from that, this example should, so far as I know, work just the same.

petro avatar

I am interested about the inference FPS for this concrete example.
Further it would be interesting to see whether the inference FPS could be further boosted using TFlite XNNPACK.

Ton van Overbeek avatar

2 corrections to “Installing the Picamera2 library”:
– cd kmxss -> cd kmsxx
– cloning picamera2: git clone

David Plowman avatar


Ben avatar

Cool, but can it tell a bunny from a cat ?

Ashley Whittaker avatar

Alasdair we need a bunny/cat test plz. Alex has a bunny. You can find a cat. Wait, your dog is the size of a cat. Perfect.

Carlos Luna avatar

What about telling a Pi4B 2g from a Pi 4B 4g? Because that’s something I still haven’t learnt how to do properly!

Ben avatar

Some things are important.

Just a thought, thinking about it, it would make a great ‘advanced’ RPFoundation Learn course for an advanced Wildlife/Nature Cam as a next step from the standard Birdbox/Wildlife cam project you have.)

No doubt getting everything converted to just the new modules/software is going to take time, once it is out of testing).

Thanks to all plugging away at this, and the testers.

Liz Upton avatar

Eben and I have cats, but they’re black cats and therefore broadly speaking unphotographable. Let’s use Alasdair’s dog.

Ashley Whittaker avatar

If Alasdair is stuck on a boat for his holiday… then his cat-sized dog must be at home and [briefly] kidnappable…

David Plowman avatar

Just to let everyone know that, as Picamera2 is still a very fast moving target, you also now need to install the python-v4l2 module. The official version is unmaintained and horribly out-of-date, so please use
git clone
Also, it will need adding to your PYTHONPATH, so you will have:
export PYTHONPATH=/home/pi/picamera2:/home/pi/libcamera/build/src/py:/home/pi/kmsxx/build/py:/home/pi/python-v4l2
I’ll see if I can get this added to the main text above, but in the meantime hopefully this will be helpful. Thanks!

Secret-chest avatar

Please don’t use dollars in the bash examples, so it’s easier to copy and paste.

Nathan avatar

Please can this work with the raspberry 3?

Alasdair Allan avatar

Should work out of the box on the Raspberry Pi 3. It’ll run slower, but it’ll work.

Comments are closed