Reading
OlliteRT is an innovative open-source Android app that allows users to turn their phones into OpenAI-compatible local large language model (LLM) inference servers. Built on Google's LiteRT runtime, it supports multimodal inference, tool calling, and streaming responses, enabling the operation of models like Gemma and Qwen on the device without cloud connectivity.
Section 01
OlliteRT is an innovative open-source Android app built on Google's LiteRT runtime, which can turn an Android phone into an OpenAI-compatible local LLM inference server. It supports multimodal inference, tool calling, and streaming responses, allowing models like Gemma and Qwen to run without cloud connectivity—protecting user privacy and lowering the hardware barrier for AI applications.
Section 02
OlliteRT was created by developer NightMean with the design philosophy of being the "Android version of Ollama". By downloading a model and launching the app, users can make their phones provide OpenAI-compatible HTTP API services via the LiteRT runtime. Its core advantage is full localization: no cloud dependency, no API key required, no subscription fees, and data always stays on the device—meeting privacy needs.
Section 03
Built on Google's LiteRT runtime (formerly TensorFlow Lite) and the NanoHTTPD lightweight server, it provides OpenAI-compatible interfaces. It supports downloading models from HuggingFace or importing local .litertlm models; recommended models include the Gemma 4 series (multimodal), Gemma3 1B (text-only for low-end devices), etc. It features multimodal processing (text/visual/audio), experimental tool calling, and streaming responses. It also has a built-in performance testing tool, a real-time monitoring dashboard, and supports Prometheus metric export.
Section 04
Compared to traditional GPU servers that consume over 300 watts, running on a phone only uses 5-10 watts, making it suitable for long-term use of old phones. It supports auto-start on boot, enabling "set once and run long-term". The developer reminds users to avoid running it in enclosed environments (like under a blanket) during high loads to prevent device overheating.
Section 05
Using an OpenAI-compatible API format, it can work with mainstream clients like OpenWebUI, OpenClaw, Home Assistant, Python SDK, and curl. Simply configure the server address (e.g., http://[phone IP]:8000/v1) to use local models.
Section 06
Current limitations: Only one model can be loaded at a time; tool calling is done via prompt injection; token counting is based on character estimation. Future plans include supporting on-demand model loading, allowing dynamic model switching via API requests without manual operation.
Section 07
It is open-sourced under the Apache 2.0 license with transparent code. It offers three build versions: stable, beta, and development. Documentation is comprehensive (model guides, client tutorials, API docs, etc.). It supports developer contributions (build instructions, HuggingFace OAuth integration).
Section 08
OlliteRT represents a new paradigm for edge AI, bringing LLM capabilities to mobile devices while protecting privacy and lowering the barrier to use. It is suitable for privacy-sensitive users, those who want to utilize idle devices, and edge AI developers. As edge technology advances, such tools will become more powerful and user-friendly, and OlliteRT has already taken a solid step forward.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.