Reading
A local LLM inference toolkit for AMD APU + NVIDIA discrete GPU hybrid environments, enabling flexible dual-GPU scheduling via the Vulkan backend and resolving ROCm compatibility issues on older architectures.
Section 01
A local LLM inference toolkit for AMD APU + NVIDIA discrete GPU hybrid environments, enabling flexible dual-GPU scheduling via the Vulkan backend and resolving ROCm compatibility issues on older architectures.
Section 02
For users who want to run large language models locally, hardware configuration often involves compromises. Many people have desktops or laptops equipped with AMD APUs and NVIDIA discrete GPUs. This heterogeneous environment faces numerous challenges when running LLMs on Linux: ROCm has limited support for older AMD GPUs, Vulkan is universal but complex to configure, and there is little documentation on how to make the two work together.
The LLM-Toolkit project was created to address this specific scenario. It is not a general LLM deployment solution, but a deeply optimized toolkit for the specific hardware combination of AMD Ryzen APU + NVIDIA discrete GPU.
Section 03
The project author's actual hardware environment is quite representative:
The challenge with this configuration is: Vega 8 belongs to the GCN architecture, while ROCm officially only supports RDNA2 (gfx1030+) and newer architectures. This means ROCm/HIP backend cannot be used to accelerate inference on the APU.
Section 04
After in-depth research and testing, the project author finally chose Vulkan as the unified backend. This decision was based on the following key findings:
Section 05
In the Linux kernel 6.17 environment, ROCm/HIP has serious driver-level issues on Vega 8. Both the ROCm 5.7 included with Ubuntu and ROCm 6.4.4 tested via Docker crash at the amdgpu driver level (MODE2 GPU reset caused by no-retry page fault). This is a kernel driver bug that cannot be fixed from user space.
Section 06
As a cross-platform graphics API, Vulkan has broader hardware support. The project's test data shows performance differences across different backends (using the Llama 2 7B Chat Q4_K_S model):
| Backend | Device | Prompt Processing Speed | Generation Speed |
|---|---|---|---|
| Vulkan | RTX 5090 | 2,117 tokens/s | 273 tokens/s |
| Vulkan | Vega 8 iGPU | 49 tokens/s | 14 tokens/s |
| CPU-only | Ryzen 5700G | 55 tokens/s | 12 tokens/s |
The data reveals several key insights:
Section 07
LLM-Toolkit provides a series of carefully designed startup scripts covering different usage scenarios:
Section 08
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
A self-hosted MCP server that provides cross-session persistent coordination, task management, event bus, and observability for AI agents. It supports 35 tools and can be integrated into clients like Claude Code and Cursor with zero code changes.