LLM Inference Explorer: Real-time Observation of the Full Lifecycle of Large Model Inference via Streamlit

This article introduces LLM Inference Explorer, a lightweight Streamlit application that connects to a local Ollama instance to real-time display the complete inference process of large language models. It visualizes pre-filling, decoding loops, token streaming, and performance metrics, helping developers and researchers intuitively understand the internal mechanisms of LLM inference and solve the "black box" dilemma of the inference process.

The inference process of large language models (LLMs) is a mysterious black box for many developers. After entering a prompt and waiting for results, what happens in between? Why are response speeds different? How are tokens generated? This opacity affects the understanding of model behavior and hinders inference optimization and performance tuning. To master LLM technology, one needs to "see" the internal mechanisms of the inference process.

The core features of LLM Inference Explorer include:

1. Local model connection and management: automatically connects to Ollama instances and supports model switching;
2. Real-time token streaming display: receives tokens via SSE and displays them one by one, observing generation speed and rhythm;
3. Performance metric monitoring: shows tokens per second, first token time, total time consumed, etc.;
4. Inference process explanation: the sidebar explains stages such as pre-filling, decoding loop, and SSE transmission.

The project uses a layered architecture:

• UI layer: Streamlit for quickly building interactive interfaces;
• HTTP client: httpx for handling asynchronous requests and streaming responses;
• Inference runtime: Ollama (based on llama.cpp under the hood);
• Dependency management: uv tool.

Usage steps:

1. Environment preparation: Python 3.12+, uv, local Ollama instance;
2. Download model: ollama pull llama3.2;
3. Start the application: make dev (visit http://localhost:8501). Note on containerized deployment: Containerized Ollama on macOS disables Metal GPU acceleration; Apple Silicon users are advised to run ollama serve directly.

The tool is suitable for:

1. Educational learning: helping beginners understand abstract concepts like pre-filling and decoding;
2. Model evaluation: comparing the inference speed and quality of different models;
3. Performance debugging: locating bottlenecks in pre-filling or decoding stages;
4. Prompt engineering: observing the impact of prompts on the inference process.

Current limitations: only supports Ollama backend, basic performance metrics, benchmark module under development, no batch inference support. Future expansions: multi-backend integration (vLLM, TGI), advanced analysis (token latency, memory monitoring), comparison mode, history records, custom metrics.

LLM Inference Explorer achieves educational value with a minimalist design, making the abstract inference process visible and measurable. It lowers the threshold for observing LLM inference, helps developers deeply understand model behavior, and builds more reliable and efficient applications. In the rapid development of AI, maintaining insight into underlying mechanisms is crucial, and this tool is a practical choice for cultivating such insight.