Reading
Pravāha is an LLM inference engine built from scratch using pure Python. It not only implements vLLM-level continuous batching and paged attention mechanisms but also innovatively integrates an intelligent cluster of 51 autonomous agents, supporting ReAct reasoning loops, self-repair auditing, and persistent memory.
Section 01
Pravāha is an LLM inference engine built from scratch using pure Python. It not only implements vLLM-level continuous batching and paged attention mechanisms but also innovatively integrates an intelligent cluster of 51 autonomous agents, supporting ReAct reasoning loops, self-repair auditing, and persistent memory.
Section 02
Pravāha (Sanskrit for "flow") is a high-performance large language model inference engine built from scratch using pure Python. Unlike existing tools such as vLLM, Ollama, and llama.cpp, Pravāha not only provides production-grade inference performance but also innovatively integrates an intelligent cluster of 51 autonomous agents, elevating the inference engine to an entirely new level of intelligence.
The core design philosophy of the project is "no black boxes"—all components remain fully transparent and customizable. From the custom Naive KV-Cache implementation to deterministic memory control, developers can precisely understand and regulate every behavior of the system. The project aims to provide full visibility into the inference process while maintaining a streaming latency of <10 milliseconds.
Section 03
Pravāha adopts a clear layered architecture, extending from the user interface to the underlying Rust performance core:
Layer 1: Interaction Interface Provides CLI (based on Typer), FastAPI services, WebSocket real-time communication, and a Textual-based terminal dashboard (TUI), even including pixel-style avatar animations to make the command-line experience more engaging.
Layer 2: Engine Core AsyncPravahaEngine is the core of asynchronous inference, working with the EventBus event bus and RequestQueue request queue to achieve efficient task scheduling.
Layer 3: Inference Pipeline Starting from the Tokenizer, it goes through the Scheduler, Decoder, and finally reaches the Sampler, forming a complete inference processing chain.
Layer 4: Memory Plane This is one of Pravāha's technical highlights. PagedKVCache implements paged KV cache management, BlockManager handles memory block allocation, PrefixTrie (implemented in Rust) supports prefix sharing, LRU Swapping enables intelligent page swapping, and the Preemption mechanism handles priority preemption. This design achieves vLLM-level memory usage efficiency.
Layer 5: Intelligent Cluster (51 Agents) This is the core feature that distinguishes Pravāha from other inference engines. The 51 agents are divided into four categories: 20 Execution Agents, 12 Audit Agents, 10 Security Agents, and 9 Design Agents. All of them work based on the ReAct (Reasoning + Action) loop, with tool usage capabilities and persistent memory.
Layer 6: Extended Features Built-in RAG (Retrieval-Augmented Generation) pipeline, visual routing, conversation branching, plugin system, and safety guardrails.
Layer 7: Observability Integrates Prometheus metrics, Tracer tracking, CostEstimator for cost estimation, and SelfBenchmark self-test tools.
Layer 8: Rust Performance Core Key components such as BlockAllocator, PrefixTrie, and AllocatorStats are implemented in Rust, achieving near-native performance while maintaining the convenience of Python development.
Section 04
Pravāha's agent system is its most innovative feature. Each agent follows the ReAct loop: THINK → ACT → OBSERVE → THINK again... until an answer is reached. This is not a simple prompt wrapper but a true autonomous decision-making system.
Section 05
PlannerAgent Responsible for task decomposition, breaking down complex requests into executable sub-steps.
CoderAgent Performs code generation and validation, and can call Python executors, file readers, and web search tools.
DebuggerAgent Conducts root cause analysis and automatic repair, locating issues by executing code and reading files.
ResearcherAgent Performs web research and cross-validation, collecting information using web_search and fetch_url tools.
ReasoningAgent Handles chain-of-thought and mathematical validation, verifying logical correctness via Python executors.
Other Execution Agents include: CriticAgent (quality criticism), ValidatorAgent (output validation), SummarizerAgent (text summarization), ExpanderAgent (content expansion), TranslatorAgent (language translation), MergerAgent (output merging), RouterAgent (task routing), MemoryAgent (memory management), ToolAgent (tool orchestration), JudgeAgent (quality evaluation), RefinerAgent (output refinement), ClassifierAgent (task classification), ExtractorAgent (data extraction), NarratorAgent (narrative writing), EnsembleAgent (multi-model integration).
Section 06
Audit Agents adopt a static regex-first analysis strategy to detect code issues with zero LLM cost:
SyntaxAuditAgent Detects 7 syntax risks: eval/exec, bare except, star imports, mutable default parameters, global keyword abuse, assert statements.
TypeSafetyAgent Focuses on 3 type safety issues: isinstance chains, bare type() calls, overuse of Any type.
LogicFlawAgent Identifies 4 logical flaws: == None comparisons, while True infinite loops, unreachable code, empty catch blocks.
PerformanceProfilerAgent Analyzes 3 types of performance issues: nested loops, string concatenation, repeated calculations.
Other Audit Agents include: ConsistencyGuardAgent (output consistency check), HallucinationHunterAgent (fact verification), EdgeCaseHunterAgent (edge condition detection), OutputVerifierAgent (final quality gating), PatchApplierAgent (automatic repair), SelfReflectionAgent (metacognitive review), TestGeneratorAgent (test generation), RegressionGuardAgent (regression detection).
Section 07
Security Agents provide enterprise-level code security auditing, with partial support for CVSS scoring:
SecurityAuditAgent Detects 12 high-risk patterns, including eval/exec/pickle, and maps to CWE standards.
InjectionScannerAgent Scans 10 types of injection attacks: SQL injection, XSS, XXE, command injection, template injection.
AuthAuditAgent Checks 5 authentication issues: JWT, session fixation, hard-coded credentials.
CryptoAuditAgent Identifies 8 encryption weaknesses: MD5/SHA1/DES/RC4/ECB/weak keys.
DependencyAuditAgent Monitors 6 dangerous dependencies: pickle/marshal/ctypes/telnet.
SecretsScannerAgent Uses entropy analysis to detect over 8 types of secret leaks: AWS/GitHub/OpenAI/Slack keys.
Other Security Agents include: NetworkSecurityAgent (network security), PrivilegeAuditAgent (privilege audit), APISecurityAgent (API security), ComplianceAgent (compliance check).
Section 08
Design Agents focus on UI/UX design automation:
UIDesignerAgent Responsible for layout, visual, and interaction specification design.
ComponentBuilderAgent Generates React/HTML/CSS component code.
LayoutAgent Handles CSS Grid/Flexbox layouts.
StyleAgent Manages the design token system.
AccessibilityAgent Ensures WCAG 2.1 AA-level accessibility compliance.
UXReviewerAgent Conducts reviews based on Nielsen's 10 heuristic principles.
DesignCriticAgent Scores designs from five dimensions.
PrototypeAgent Builds single-file HTML prototypes.
DesignSystemAgent Maintains tokens and pattern libraries.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.