TokenJuice: A Terminal Output Compression Tool to Slim Down AI Programming Agents

TokenJuice is an open-source tool designed to save context window space for AI programming agents (such as Claude Code and Codex) by intelligently compressing terminal command outputs. It uses a non-intrusive design that does not alter the command execution process, compresses redundant outputs via a rule engine while retaining key information, and supports native integration with mainstream frameworks along with a safety valve mechanism to ensure access to original outputs.

When AI programming agents execute commands, terminal outputs often contain redundant information like duplicate lines and formatting noise, occupying valuable context window space and increasing token consumption and inference costs. TokenJuice emerged to address this; its core philosophy is "your LLM needs a diet", reducing the number of tokens sent back to the AI via an intelligent compression layer.

TokenJuice adheres to the principle of "never altering original command execution". Its workflow consists of three stages: 1. Original Execution: The command is passed to the shell unchanged; 2. Intelligent Compression: The rule engine identifies and removes redundant content while retaining key information; 3. Compressed Result Return: The result is sent back to the agent via existing hooks, making the process transparent to the agent with no need for adaptation.

TokenJuice's compression is based on a hierarchical rule engine, with priorities from lowest to highest: built-in rules, user-level rules, and project-level rules. Each rule defines matching conditions and strategies in JSON. Compared to LLM summarization solutions, it offers advantages like strong determinism, low latency, and no additional costs, and developers can customize rules.

TokenJuice supports native integration with Claude Code and Codex CLI; installation can be completed with a single command (e.g., tokenjuice install claude-code), which intelligently preserves existing user settings. It provides diagnostic tools (tokenjuice doctor hooks) and verification tools (tokenjuice verify) to ensure proper integration.

TokenJuice provides multiple layers of safety valves: 1. The --raw/--full flags skip compression; 2. The --store flag saves the original output locally, which can be viewed via tokenjuice cat; 3. Machine callers can set "raw":true to globally disable compression, avoiding information bottlenecks.

Compression effects vary by command type: build tool outputs have a compression rate of 60%-80%, while test framework output compression rates depend on the ratio of passed/failed cases. It is suitable for scenarios with frequent terminal interactions like daily development and CI/CD auxiliary reviews, with no negative impact on concise commands.

TokenJuice fills a gap in the AI programming agent toolchain, providing a practical solution through its rule engine, non-intrusive integration, and safety valves. The project is in an active development phase with ongoing expansion of the built-in rule set, and it will play a more important role as AI agents become more widespread in the future.