# LiteRT-LM **Repository Path**: simonxing/LiteRT-LM ## Basic Information - **Project Name**: LiteRT-LM - **Description**:  LiteRT-LM  是用于在边缘平台上高效运行语言模型的 C++ 库 - **Primary Language**: C/C++ - **License**: Apache-2.0 - **Default Branch**: main - **Homepage**: https://www.oschina.net/p/litert-lm - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 1 - **Created**: 2026-11-06 - **Last Updated**: 2026-11-06 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # LiteRT-LM LiteRT-LM is Google's **production-ready** orchestration layer to run LLMs with LiteRT, engineered for **high-performance**, **cross-platform** execution. 🔗 [Product Website](https://ai.google.dev/edge/litert-lm) | 💬✨ [Chat Demo](https://google-ai-edge.github.io/LiteRT-LM/web_demos/chat/index.html) | 🔎✨ [Embedding Demo](https://google-ai-edge.github.io/LiteRT-LM/web_demos/embedding_search/index.html) ## 🔥 What's New: v0.19.0 - * **🎯 Accelerated Logit Masking**: Enables faster constrained decoding on GPUs, with full support for MTP-enabled decoding. - * **🌐 Web SDK**: @litert-lm/core now stream-loads all models, defaults to the GPU backend, and supports image and audio inputs. - * **🎙️ Speech Models**: Added support for multiple ASR and TTS models, including Qwen3-ASR, Kokoro, and Qwen3-TTS. - * **🧱 C API 1.0.0 (Breaking)**: Functions that can fail now return LiteRtLmStatusCode with results in out_* parameters, and only litert_lm_* symbols are exported. - * **⚙️ Performance & Platform**: GPU sampling now matches CPU for the same seed, ModelInfo is faster and uses less memory. 👉 Try Gemma4-E4B with MTP on Linux, macOS, Windows or Raspberry Pi with the [LiteRT-LM CLI](https://ai.google.dev/edge/litert-lm/cli): ```bash litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E4B-it-litert-lm \ gemma-4-E4B-it.litertlm \ --backend=gpu \ --speculative-decoding=on \ --prompt="What is the capital of France?" ``` ## 🌟 Key Features - 📱 **Cross-Platform Support**: Android, iOS, Web, Desktop, and IoT (e.g. Raspberry Pi). - 🚀 **Hardware Acceleration**: Peak performance via GPU and NPU accelerators. - 👁️ **Multi-Modality**: Support for vision and audio inputs. - 🔧 **Tool Use**: Function calling support for agentic workflows. - 📚 **Broad Model Support**: Gemma, Llama, Phi-4, Qwen, and more. ![](./docs/api/kotlin/demo.gif) -------------------------------------------------------------------------------- ## 🚀 Production-Ready for Google's Products LiteRT-LM powers on-device GenAI experiences in **Chrome**, **Chromebook Plus**, **Pixel Watch**, and more. You can also try the [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) app to run models immediately on your device. **Install the app today from Google Play** | **Install the app today from App Store** :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------: Get it on Google Play | Download on the App Store ### 📰 Blogs & Announcements Link | Description :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------- [Bringing Gemma 4 12B to your Laptop: Unlocking Local, Agentic Workflows with Google AI Edge](https://developers.googleblog.com/bringing-gemma-4-12b-to-your-laptop-unlocking-local-agentic-workflows-with-google-ai-edge/) | Bring agentic, multimodal AI capabilities to everyday laptops, enabling local data processing and visual insight generation. [Blazing-fast on-device GenAI with LiteRT-LM](https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/) | Unlock Gemma 4's full potential with blazing speed and incredible efficiency using newly added Swift, JavaScript, and Flutter APIs. [Accelerating Gemma 4: faster inference with multi-token prediction drafters](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/) | An overview of how Multi-Token Prediction (MTP) drafters are making Gemma 4 models up to 3x faster at inference. [Bring state-of-the-art agentic skills to the edge with Gemma 4](https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/) | Deploy Gemma 4 in-app and across a broader range of devices with stellar performance and broad reach using LiteRT-LM. [On-device GenAI in Chrome, Chromebook Plus and Pixel Watch](https://developers.googleblog.com/on-device-genai-in-chrome-chromebook-plus-and-pixel-watch-with-litert-lm/) | Deploy language models on wearables and browser-based platforms using LiteRT-LM at scale. [On-device Function Calling in Google AI Edge Gallery](https://developers.googleblog.com/on-device-function-calling-in-google-ai-edge-gallery/) | Explore how to fine-tune FunctionGemma and enable function calling capabilities powered by LiteRT-LM Tool Use APIs. [Google AI Edge small language models, multimodality, and function calling](https://developers.googleblog.com/google-ai-edge-small-language-models-multimodality-rag-function-calling/) | Latest insights on RAG, multimodality, and function calling for edge language models. -------------------------------------------------------------------------------- ## 🏃 Quick Start ### 🔗 Key Links - 👉 [Technical Overview](https://ai.google.dev/edge/litert-lm/overview) including performance benchmarks, model support, and more. - 👉 [LiteRT-LM CLI Guide](https://ai.google.dev/edge/litert-lm/cli) including installation, getting started, and advanced usage. ### ⚡ Quick Try (No Code) Try LiteRT-LM immediately from your terminal without writing a single line of code using [`uv`](https://docs.astral.sh/uv/getting-started/installation/): ```bash uv tool install litert-lm litert-lm run \ --from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \ gemma-3n-E2B-it-int4 \ --prompt="What is the capital of France?" ``` -------------------------------------------------------------------------------- ### 📚 Supported Language APIs Ready to get started? Explore our language-specific guides and setup instructions. Language | Status | Best For... | Documentation :------------------- | :-------------- | :---------------------- | :------------ **Python** | ✅ Stable | Prototyping & Scripting | [Python Guide](https://ai.google.dev/edge/litert-lm/python) **Kotlin** | ✅ Stable | Android apps & JVM | [Kotlin Guide](https://ai.google.dev/edge/litert-lm/android) **Swift** | 🚀 Early Preview | Native iOS & macOS | [Swift Guide](https://ai.google.dev/edge/litert-lm/swift) **JavaScript (web)** | 🚀 Early Preview | Browser environments | [JavaScript Guide](https://ai.google.dev/edge/litert-lm/js) **Flutter** | 🚀 Community | Cross-platform mobile | [Flutter Guide](https://ai.google.dev/edge/litert-lm/flutter) **C++** | ✅ Stable | High-performance native | [C++ Guide](https://ai.google.dev/edge/litert-lm/cpp) #### 🏗️ Build From Source This [guide](./docs/getting-started/build-and-run.md) shows how you can compile LiteRT-LM from source. If you want to build the program from source, you should checkout the stable [![Latest Release](https://img.shields.io/github/v/release/google-ai-edge/LiteRT-LM)](https://github.com/google-ai-edge/LiteRT-LM/releases/latest) tag. --------------------------------------------------------------------------------