Reading
A plugin designed for Hermes Agent that provides token-by-token cost accounting for local large language model inference, considering hardware depreciation, power consumption, and performance benchmarks to help users make informed decisions between local deployment and cloud APIs.
Section 01
Hermes Local Rig Accounting is a plugin designed for Hermes Agent, aiming to solve the problem of opaque cost for local LLM inference. It helps users make data-driven decisions between local deployment and cloud APIs by accounting for hardware depreciation, power consumption, performance benchmarks, and other dimensions on a token-by-token basis.
Section 02
Many developers only focus on one-time hardware investment when considering local deployment, ignoring ongoing operational costs. The real cost includes hidden costs such as hardware depreciation, power consumption, maintenance costs, opportunity costs, and performance differences. The core idea of this tool is to make these hidden costs explicit, helping users make informed decisions.
Section 03
The plugin provides core features such as token-by-token cost accounting, hardware cost modeling (total configuration cost, service life, power consumption, etc.), power cost calculation (supports automatic/manual electricity prices), performance benchmarking (measuring TPS), and community leaderboard (submitting comparison results).
Section 04
The cost model includes: Depreciation cost (GPU dedicated cost/(number of service life years × 8766 hours) × inference hours), energy cost ((average power consumption in watts /1000) × electricity price × operating hours), cost per million tokens (total hourly cost/(TPS ×3600) ×1e6). Example: GPU cost $1500, 3-year service life, 450W power consumption, $0.12 per kWh, TPS 50, resulting in a cost of $0.62 per million tokens, which can be compared with cloud APIs.
Section 05
Installation methods: Via the hermes plugins install command or manual repository cloning. Configuration: Modify config.yaml to set hardware cost, service life, power consumption, and electricity price (supports automatic query). Command-line tools include /rig-benchmark (measure TPS), /rig-summary (device overview), /rig-cost (cumulative cost), /rig-rates (check electricity price), /rig-submit (submit to leaderboard), supporting multi-device configuration and LLM tool integration.
Section 06
Privacy protection: Local computing, no telemetry, transparent open-source formulas. Application scenarios: Cost quantification for individual developers, finding cost balance points for small teams, enterprise IT budget planning, and comparing performance optimization opportunities for researchers.
Section 07
Decision-making needs to consider cost comparison, performance requirements, data privacy, flexibility, and maintenance burden. This tool helps users make scientific decisions by making hidden costs explicit, which is an important progress in AI infrastructure management and worth trying for users deploying LLMs locally.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.