From Real Data to Production-Grade RAG: A Generative AI Engineer's Practical Portfolio

This article introduces the open-source portfolio sierra-genai-engineering, which includes 9 projects and over 10,000 real records. All data comes from real-time APIs (not synthetic) and covers scenarios like RAG knowledge bases, document classification, and clinical trial analysis. This portfolio addresses the problem of relying on toy datasets in the current NLP field and provides a production-grade NLP pipeline reference for LLM technology implementation.

Most current NLP projects rely on Kaggle CSVs or synthetic datasets, which struggle to reflect real-world complexity (data noise, API rate limits, data drift, etc.). The core philosophy of this portfolio is the 'real data pipeline from scratch': each project starts with real-time API calls, covering authoritative data sources like arXiv, PubMed, and ClinicalTrials.gov, forcing developers to handle real data engineering challenges (XML/JSON parsing errors, fault-tolerant retries, data version management, etc.).

The portfolio integrates multi-source data:

• arXiv API: 2646 ML/AI/NLP research abstracts
• PubMed: 500 biomedical literature entries
• ClinicalTrials.gov: 500 clinical trial records
• Congress.gov: 496 bills from the 118th U.S. Congress
• U.S. Community Survey: 3222 county-level demographic data points
• Oyez API: 59 U.S. Supreme Court case metadata entries
• BLS: 72 months of employment time series

The multi-source design simulates real scenarios of enterprise-level knowledge bases (technical documents, medical literature, legal provisions, etc.) rather than simple data accumulation.

Technical Architecture: Three-stage retrieval

1. Vector retrieval: all-MiniLM-L6-v2 encoded into 384-dimensional vectors, FAISS IVF-Flat index (100 centroids), single retrieval latency of 1.37ms
2. Cross-encoder reordering: ms-marco-MiniLM-L-6-v2 re-ranks the top 100 results, improving precision by 40%
3. Generation enhancement: inject retrieval results into LLM context to answer questions based on real literature

Performance Metrics: End-to-end latency of 60-80ms (embedding + FAISS + reordering), full response takes approximately 180ms; t-SNE visualization shows clear category clustering in the embedding space, capturing the boundaries of research fields.

Processes 991 documents from arXiv/PubMed/Wikipedia, classified into 6 categories. Uses TF-IDF + Random Forest architecture, with advantages: fast speed, strong interpretability, low cost.

The project emphasizes a 'when to upgrade' decision framework: only consider transformer models when category boundaries are blurred, deep semantic understanding is needed, or training data is extremely scarce. This kind of engineering judgment is more important than pursuing technical novelty.

Vertical Domain Projects:

• Clinical trial analysis: tracks trial trends from 500 records
• Congressional bill analysis: policy research on 496 bills
• Supreme Court voting analysis: research on judge voting patterns in 59 cases
• MLOps model registry: integrates 3222 county-level data, including version control and A/B testing frameworks

Engineering Practices: All projects include Jupyter Notebooks (EDA), Streamlit dashboards (demonstration), and complete dependency configurations; data acquisition scripts are automated to ensure reproducibility; it declares 'zero synthetic data' with no manually constructed samples.

Insights: Generative AI engineers need to master embedding model principles, vector index optimization, precision/recall trade-offs, multi-source data integration—not just API calls. Enterprise teams can learn from: two-stage retrieval, multi-source fusion, confidence calibration, latency optimization.

Conclusion: The hype around generative AI will eventually fade, but solid engineering capabilities will never go out of style. The value of this portfolio lies in showing the path to implementing LLMs into reliable, maintainable, and scalable production systems, serving as a reference roadmap for engineer growth.