Our friends at Google have been busy working on improving LiteRT performance on the Raspberry Pi platform. We are pleased to share this work with you now, providing a robust framework for your general-purpose AI applications.

With special thanks to Google software engineers Lu Wang and Terry Heo, and Igalia graphics software engineer José María Casanova.

Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time, completely offline on a single compact device like a Raspberry Pi. Edge AI unlocks this exact autonomy. It enables developers to build highly secure, self-contained systems — like intelligent robots and local AI agents — with zero cloud dependencies, ultra-low latency, and total data privacy.

We’ve made running edge AI on Raspberry Pi a breeze with LiteRT, Google’s high-performance, production-proven on-device inference runtime. LiteRT allows you to seamlessly deploy everything from classical ML models to state-of-the-art LLMs right out of the box. By delivering optimized execution and hyper-efficient memory usage on both the CPU and the GPU, LiteRT maximizes your Raspberry Pi’s full computing potential.

This hardware–software synergy shines brightest when paired with Gemma, Google’s family of lightweight open models. To show you what’s possible, we’ll demonstrate how running Gemma and LiteRT on a Raspberry Pi 5 can power the Reachy Mini robot to perceive and react to its environment entirely locally in real time. Read on to get started with your own deployment.

Discover the agentic capability of Gemma

Gemma models are well suited for building autonomous agents, smart cameras, and social robots that can reason and execute complex, multi-step workflows directly on your Raspberry Pi. To accommodate different hardware constraints, the Gemma family of models provides several highly efficient options:

• Gemma 3 270M: A hyper-efficient and compact base model designed for task-specific post fine-tuning, enabling high-speed, low-latency features like sentiment analysis or entity extraction in resource-constrained environments.
• EmbeddingGemma 300M: A state-of-the-art text embedding model that produces high-quality embeddings on-device, great for Retrieval Augmented Generation (RAG), semantic search, and classification.
• Gemma 3 1B: A lightweight and multilingual text-only model that balances compact size with strong generative capabilities, making it ideal for a wide range of on-device reasoning, summarization, and content creation tasks.
• Gemma 4 E2B: Tailored specifically for mobile and tight edge environments, it features memory-mapped per-layer embeddings, and is ideal for continuous monitoring, fast text/image/audio inference, and edge-based speech processing where saving RAM is absolutely critical.
• Gemma 4 E4B: The sweet spot for performance and size. This model delivers noticeably stronger reasoning capabilities and frontier-level edge performance while remaining compact. It is the perfect choice for complex multi-step planning without overwhelming the Raspberry Pi’s resources.

Gemma performance on Raspberry Pi CPU

Through LiteRT-LM, a specialized orchestration layer on top of LiteRT, developers can seamlessly deploy Gemma right out of the box. Under the hood, sophisticated CPU acceleration via LiteRT and XNNPACK ensures the Gemma family of models is highly optimized for resource efficiency and low-latency execution directly on the Raspberry Pi.

Model	Framework (CPU)	Prefill (tokens/sec)	Decode (tokens/sec)	Peak memory (MB)
Gemma 4 E2B	LiteRT-LM (QAT)	99	9	1432
	llama.cpp (Q4_0)	24	4	4406

• Hardware: Raspberry Pi 5 (8 GB RAM variant)
• Benchmark setup: 1024 prefill tokens and 256 decode tokens; the CPU runs with four threads
• llama.cpp is benchmarked using llama-bench with gemma-4-E2B-it-Q4_0.gguf
• LiteRT-LM is benchmarked using litert lm benchmark with gemma-4-E2B-it.litertlm

On a Raspberry Pi 5, LiteRT-LM delivers robust performance for Gemma 4 E2B, achieving 99 tokens/sec for prefill and 9 tokens/sec for decode, all while maintaining a remarkably low peak memory footprint of just 1432 MB. This brings Gemma’s highly responsive, general-purpose intelligence to Raspberry Pi. 

Thanks to Gemma 4 E2B’s highly efficient tokenizer, which packs more text into fewer tokens (averaging ~4.2 characters per token), LiteRT-LM achieves an impressive end-to-end generation speed of ~27.3 characters per sec, roughly 300 words per minute (wpm), in the Reachy Mini voice demo. This throughput makes Gemma 4 E2B excellent for real-time speech and translation tasks, delivering text at twice the speed of normal human speech (~150 wpm)

Explore more ready-to-use open models to run on Raspberry Pi from the LiteRT Hugging Face Community.

Execute on Raspberry Pi GPU with LiteRT

On Raspberry Pi 5, the quad-core Arm Cortex-A76 CPU is a raw computing powerhouse, delivering ~153.6 GFLOPS (FP32) and up to ~2.0 TOPS (INT8). In comparison, the integrated Broadcom VideoCore VII GPU is clocked at 800 MHz and offers a peak of ~76.8 GFLOPS (FP32) and ~0.24 TOPS (INT8).

While the CPU possesses a massive capacity advantage, the GPU introduces heterogeneous parallel execution, a paradigm critical for real-time edge applications. Rather than saturating the CPU, developers can delegate tasks across both processors to optimize overall system and thermal efficiency. For example, by offloading continuous vision or audio models to the VideoCore VII GPU, it preserves high-priority CPU cycles for overall system monitoring, pipeline orchestration, or computationally demanding LLM inference.

As such, we have enabled GPU inference on Raspberry Pi 5 with LiteRT’s WebGPU (Vulkan) backend via ML Drift. This integration allows you to run a wide range of computer vision, audio, and embedding models directly from the LiteRT Hugging Face Community with exceptional, low-latency performance. This includes seamless support for popular MediaPipe models, Ultralytics YOLO models, Moonshine, and much more.

The table below demonstrates the CPU and GPU latency of running classic computer vision and segmentation models via LiteRT:

Model	Task	Input size	CPU latency (ms)	GPU latency (ms)
MediaPipe Selfie Segmenter	Segmentation	256 × 256	7.88	26.36
YOLO26n	Object detection	640 × 640	101.26	375.73
EfficientNet-Lite0	Image classification	224 × 224	29.15	87.85
Moonshine-tiny	Speech recognition	5 sec	148.67	409.03

• Hardware: Raspberry Pi 5 (8 GB RAM variant)
• LiteRT CPU runs with four threads
• LiteRT GPU running with WebGPU (Vulkan) on updated Mesa V3DV Vulkan drivers

Deep dive: Reachy Mini pipeline powered by LiteRT

The Reachy Mini pipeline is a powerful showcase of low-latency, real-time edge AI inference running entirely on Raspberry Pi 5. By leveraging LiteRT, the system splits intensive vision and language workloads into a concurrent, dual-processing architecture across the CPU and GPU.

Here is how the parallel architecture works under the hood to ensure seamless interactions:

• Object detection (YOLO on GPU): Camera frames are streamed to the Raspberry Pi, where a YOLO detector (yolox-tiny) runs continuously on the GPU, avoiding resource contention and freeing up the CPU.
• Speech recognition (Moonshine on CPU): When the user speaks, the ASR component transcribes the audio into text directly on the CPU.
• Reasoning and action (Gemma 4 E2B on CPU): The Gemma 4 E2B model processes the resulting transcript alongside the latest visual metadata to generate low-latency streaming responses, such as speech replies and physical robotic gestures.
• Text-to-speech (TTS on CPU): The TTS component synthesizes the generated text into audio in streaming. The system streams the synthesized voice back to the Reachy Mini robot.

See the full source code from the Reachy demo in the LiteRT Samples Github repo.

Agentic coding with LiteRT on Raspberry Pi

LiteRT provides a comprehensive suite of tools that covers the entire development cycle: conversion, quantization, benchmark, and inference. For a fast, frictionless setup, the most straightforward approach is to use the LiteRT CLI tool. Rather than requiring developers or coding agents to manually manage multiple independent libraries, the LiteRT CLI aggregates core edge workflows into a single, unified command set.

You can now supercharge your development cycle by adding the LiteRT CLI skill and other advanced LiteRT skills into your AI coding agent, such as Google Antigravity. This empowers agents to autonomously orchestrate and execute complex, multi-stage machine learning workflows on your behalf. For example, you can easily build your own voice translator completely offline on a Raspberry Pi, like the Gemma Translator shown below. 

An ultra-lean binary footprint for IoT devices

For resource-constrained IoT devices, minimizing storage and memory overhead is critical. Without special optimization, generic AI runtimes often bundle heavy desktop or server dependencies. In contrast, LiteRT is engineered specifically for on-device deployment, maintaining an exceptionally lean and modular distribution.

The table below compares the download footprint required to run LLM inference on a Raspberry Pi (Arm64 Linux).

Tool	Download size	Architecture and dependency
LiteRT CLI (with LLM Inference)	~25 MB	Modular and on-demand: A lightweight pure-Python CLI frontend (~100 KB) that dynamically installs only the specific runtime modules (e.g. inference or conversion) optimized for the target hardware.
Ollama	~1.44 GB	Monolithic: A single, pre-compiled binary that statically bundles all execution runtimes and heavy server-class acceleration drivers, regardless of the actual capabilities of the target host.

Running your first model

You can install the LiteRT CLI and run your first model on a Raspberry Pi 5 with just a few simple commands.

1. Install the LiteRT CLI

To get started, install the LiteRT CLI via pip (ideally within a virtual environment):

pip install litert-cli

2. Run the model 

Download and run any compatible model directly from the LiteRT Hugging Face Community. The code snippet below demonstrates how to execute Gemma 4 E2B (e.g. gemma-4-E2B-it-litert-lm) on Raspberry Pi 5.

Run the model by providing your Hugging Face authentication token:

export HUGGING_FACE_HUB_TOKEN=<your_hugging_face_token_here>
litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --attachment=image.jpg \
  --prompt="You are Reachy Mini. Identify the main object in front of you, state its location (Left/Right/Center), and suggest one arm action in 10 words or less."

What’s next?

We are excited to share that LiteRT integration and Gemma models are coming soon to Hailo AI accelerators! This update will allow you to seamlessly offload model inference to the Raspberry Pi AI HAT+ and AI HAT+ 2, delivering massive hardware acceleration benefits through the exact same LiteRT workflows you use today.

Explore our resources and start your journey with LiteRT:

• Official documentation: Access installation guides, API references, and quick-start tutorials on the LiteRT Developer Site.
• GitHub repos: Find the latest source code, implementation details, and updates on the LiteRT and LiteRT-LM GitHub repos.
• Samples and templates: Check out the LiteRT-Samples GitHub repo for reference code. Kick-start your own application using the Google AI Edge Gallery app.
• Ready-to-use models: Download optimized, open-weight models like the lightweight and powerful Gemma 4 E2B directly from the LiteRT Hugging Face Community.

We value your input. Please share your thoughts, feedback, or feature requests by opening an issue on our GitHub Issue Tracker. We can’t wait to see what you build!

Acknowledgements

Google: Changming Sun, Chintan Parikh, Cormac Brick, Daisuke Majima, Dillon Sharlet, Erin Walsh, Frank Barchard, Glenn Cameron, Ian Ballantyne, Jingjiang Li, Jun Jiang, Kimish Patel, Lu Wang, Matthias Grundmann, Rodney Witcher, Sachin Kotwani, Sasha Denisov, Scott Loftin, Shuangfeng Li, Somdatta Banerjee, Terry (Woncheol) Heo, Volodymyr Kysenko, Weiyi Wang, Yi-Chun Kuo, Yu-hui Chen, and the gTech team.

Raspberry Pi, Hailo, and Igalia: Ashley Whittaker, Eldad Rubinstein, José María Casanova, Naushir Patuck, and Sarah Cunningham.

Ultralytics: Francesco Mattioli, Lakshantha Dissanayake, and Onuralp Sezer.

Moonshine AI: Pete Warden

Appendix

Model	Info density (chars/tok)	Framework (CPU)	Model size (MB)	Prefill (tokens/sec)	Decode (tokens/sec)	Generation speed (chars/sec)	Peak memory (MB)
Gemma 4 E2B	~4.2	LiteRT-LM	2468	99	9	~27.3	1432 MB
		llama.cpp	2709	58	7	~12.8	4406 MB
Gemma 3 270M	~2.8	LiteRT-LM	278	433.17	22.58	~34.4	680 MB
		llama.cpp	278	462	39	~49.2	685 MB

• gemma-3-270m-it-Q8_0.gguf
• gemma-4-E2B-it-Q4_0.gguf

When running Gemma 3 270M, LiteRT-LM delivers impressively fast performance of 433 prefill tokens/sec and 23 decode tokens/sec. This speed makes it exceptionally well-suited for low-latency, real-time applications, such as the fluid interactive behaviors required by the Reachy Mini robot.