Case Study of RAG-Based Agentic System Engineering Verification Workflow

This article introduces a case study of an agentic verification workflow for the system engineering domain. The core is to use RAG technology to implement intelligent review of architecture documents, evidence retrieval, and credibility assessment, providing reusable architectural patterns for automated verification of complex system engineering. This case comes from the GitHub project (camirian/agentic-systems-verifier-case-study), which is publicly available in the form of documents and architectural patterns, and does not contain production source code or sensitive materials. Its core objectives are to assist system engineers in tasks such as parsing architectural inputs, retrieving requirement contexts, evaluating the credibility of conclusions, and exposing traceability gaps, supporting manual review rather than replacing it.

In the modern system engineering domain, complex cyber-physical systems generate a large number of artifacts such as requirement documents and architecture models scattered across multiple tools/platforms. Traditional manual review faces three major challenges: slow speed, high error rate, and difficulty in scaling. Especially in safety-critical fields such as aerospace, automotive, and medical devices, strict traceability requirements (design decisions need to be supported by evidence, requirements need to correspond to implementations) make manual review a bottleneck.

This case demonstrates an agentic verification workflow that combines system engineering, model-based architecture review, and RAG-enhanced assessment. This project is a pattern-first public case and does not contain production source code, datasets, or sensitive materials. Its core objectives include: parsing structured architectural inputs, retrieving relevant requirement contexts, evaluating the credibility of answers relative to source materials, exposing traceability gaps, and supporting manual review (not replacing it).

The project adopts a modular layered architecture with clear responsibilities for each layer:

1. Ingestion Layer: Receives source documents and structured model artifacts, converts them into a unified internal representation;
2. Parsing Layer: Performs model-oriented text parsing and requirement record extraction, establishing semantic representations;
3. Retrieval Layer: Core application of RAG technology, context-aware evidence selection;
4. Verification Layer: LLM-as-reviewer performs credibility, precision, and recall scoring;
5. Review UI Layer: Provides an interactive interface to check conclusions and evidence chains, supporting manual confirmation.

The key technologies of the project include:

• RAG-enhanced evidence retrieval: Combines vector embedding and re-ranking models to accurately locate evidence that supports/refutes architectural decisions;
• Credibility assessment mechanism: Faithfulness-style checks to evaluate whether LLM conclusions are faithful to source materials and avoid hallucinations;
• Traceability guarantee: Each verification conclusion is accompanied by a complete evidence chain (source document, paragraph, retrieval score, etc.) to ensure transparency and auditability.

Applicable Scenarios: Compliance review (meeting industry standards/regulatory requirements), change impact analysis (assessing the scope of design change impacts), requirement completeness check (identifying ambiguity/conflicts/omissions), architecture consistency verification (alignment between high-level architecture and detailed design). Public Deliverables: Case study document (docs/CASE_STUDY.md), release boundary description (docs/RELEASE_BOUNDARY.md), verification plan (VERIFICATION_PLAN.md), demo video link. Excluded content: Production source code, proprietary PDFs, cloud deployment configurations and credentials.

The core insight provided by this case: Agents should enhance rather than replace manual review. Providing traceable evidence through RAG, enabling flexible deployment via modular architecture, and maintaining human-machine collaboration with a transparent interface—these principles have universal significance for building trustworthy AI-assisted system engineering tools. For teams exploring AI applications, this project demonstrates a pragmatic path: starting with documents and patterns, gradually building verification capabilities, and maintaining human supervision to achieve scalable review.