Back to Basics in Agent Search: Is grep Sufficient?

This study explores the choice of retrieval strategies in agent search scenarios through systematic empirical analysis. Key findings include: 1) Simple grep text matching often outperforms vector retrieval in accuracy; 2) The choice of tool invocation methods and agent frameworks has a significant impact on final performance. This research challenges the industry assumption that vector retrieval is inherently superior, providing new practical insights for agent search system design.

Recent advances in large language model agents have enabled complex agent workflows—models can autonomously retrieve information, invoke tools, and reason over large-scale corpora to complete tasks entrusted by users. Retrieval-Augmented Generation (RAG) is increasingly adopted in agent search systems, yet existing literature lacks systematic comparisons of the interaction between retrieval strategy choices, agent architectures, and tool invocation paradigms. Important practical dimensions, such as how tool outputs are presented to the model and how performance changes when searches have to deal with more irrelevant contextual text, remain under-explored in agent loops.

This study designed two complementary experiments to answer the core questions. Experiment 1 compared grep and vector retrieval on 116 question samples from LongMemEval, using the custom agent framework Chronos and multiple provider-native CLI frameworks (Claude Code, Codex, and Gemini CLI), testing two presentation methods: inline tool results and file-based tool results. Experiment 2 compared the performance of pure grep and pure vector retrieval under conditions where additional irrelevant conversation history was gradually mixed in, simulating real-world scenarios where queries are surrounded by distracting information.

The experimental results are surprising: in comparisons between Chronos and various provider CLI frameworks, grep usually yields higher accuracy than vector retrieval. This finding challenges the common industry assumption that vector embedding retrieval is inherently superior to traditional keyword matching in semantic understanding tasks. The study shows that in specific agent search scenarios, precise text matching may be more reliable than approximate semantic retrieval, especially when precise location of specific information is needed.

However, the study also reveals a deeper truth: even when the underlying conversation data is identical, the overall score still strongly depends on the agent framework used and the tool invocation style. This means that the choice of retrieval strategy is only one of the factors affecting agent search performance; the design of the agent architecture, the presentation method of tool results, and the interaction mode between the model and tools are equally crucial. This finding reminds developers to take a more comprehensive perspective when optimizing RAG systems.

This research has important practical implications for the design of agent search systems. It suggests that before pursuing complex technical solutions, developers should first evaluate whether simple methods (such as grep) can already meet the needs. At the same time, the design of tool invocation interfaces and the choice of agent frameworks should receive the same level of attention as retrieval strategies. For teams with limited resources, this means that effective agent search systems can be built with a simpler technology stack.

The study also points out the limitations of the current experiments, including the limited scope of testing and the characteristics of specific datasets. Future research can explore validating these findings on larger-scale and more diverse tasks, and deeply analyze the complementarity between grep and vector retrieval for different types of queries. In addition, developing adaptive systems that can dynamically select retrieval strategies based on query characteristics is also a direction worth exploring.