platform_external_llamacpp: An On-Device LLM Inference Solution Built for AOSP

platform_external_llamacpp is an on-device LLM inference solution built for the Android Open Source Project (AOSP). By adapting llama.cpp to the AOSP build system, it fills the standardization gap for on-device LLM inference in the Android ecosystem. The project provides Soong build rules, JNI bridge layer, and automated model download scripts, supporting Qwen2.5 series models from 0.5B to 7B parameters, and offers native LLM capabilities for AOSP and AAOSP (Automotive Android).

With the development of LLM technology, on-device inference has become a mobile AI trend. iOS provides mature support via Core ML and Neural Engine, but the AOSP ecosystem has long lacked a standardized, easily integrable local LLM inference solution. platform_external_llamacpp aims to fill this gap by adapting llama.cpp to the AOSP build system, providing native LLM inference capabilities for Android devices, and serving as a key infrastructure for on-device AI implementation in the Android ecosystem.

The project's core contribution lies in its complete AOSP integration solution:

1. Android.bp Build Rules: Adapted to the Soong system, defining build configurations for the static library libllama and JNI shared library libllm_jni, correctly passing header file paths and compiler flags.
2. JNI Bridge Layer: jni/llm_jni.cpp connects Java/Kotlin and C++ code, enabling Android framework layer system services to call llama.cpp inference capabilities via JNI. The layered architecture is clear: App → Binder IPC → System Service → JNI → llama.cpp, with low coupling.

The project supports Qwen2.5 series models by default, tiered based on device memory:

Device Memory	Recommended Model	Model Size	Context Length
12GB+	Qwen2.5 7B Q4_K_M	~4.4GB	8192 tokens
8GB+	Qwen2.5 3B Q4_K_M	~2.0GB	4096 tokens
4-8GB	Qwen2.5 1.5B Q4_K_M	~1.1GB	2048 tokens
<4GB	Qwen2.5 0.5B Q8_0	~0.5GB	1024 tokens
An download_model.sh script is provided, which uses --tier to specify the model level, and downloads to the $ANDROID_PRODUCT_OUT/data/local/llm/ directory by default.

Automated Upstream Sync: The sync_upstream.sh script updates the llama.cpp version while preserving local adaptation layers (Android.bp, JNI, etc.). System Property Configuration: Customize model path, context window, number of GPU acceleration layers, and inference thread count via adb shell setprop. AAOSP Integration: As the underlying dependency for AAOSP LLM system services, it meets automotive scenario requirements: privacy protection (data not sent to the cloud), low latency, offline availability, and cost control.

The project follows AOSP conventions for usage steps:

1. Source Code Sync: cd external/llama.cpp && ./scripts/sync_upstream.sh
2. Model Download: ./scripts/download_model.sh --tier high (choose based on device memory)
3. Build Compilation: m libllama libllm_jni
4. Device Installation: adb push $ANDROID_PRODUCT_OUT/data/local/llm/*.gguf /data/local/llm/

The project clearly defines licenses for each component:

• This packaging project: Apache 2.0 (consistent with AOSP)
• llama.cpp upstream: MIT License
• Qwen2.5 model: Apache 2.0 All are business-friendly licenses, reducing compliance risks for enterprises.

Current Limitations:

• Only Qwen2.5 series is supported by default (llama.cpp itself supports more models)
• GPU acceleration requires manual adjustment of the persist.llm.gpu_layers property
• Model download depends on external networks (Hugging Face) Potential Extensions:
• Support more model architectures (Llama, Gemma, Mistral, etc.)
• Integrate Android Neural Networks API (NNAPI) or Qualcomm QNN for hardware acceleration
• Provide Kotlin/Java high-level wrapper libraries to simplify app integration
• Support model hot update and A/B testing