Enterprise AI Platform Lab: A Complete Practice from Bare Metal to Production-Grade LLM Inference Stack

Hello everyone! Today I'm sharing an enterprise AI platform lab project—a complete practice from bare metal to production-grade LLM inference stack. This project is based on a 3-node Proxmox virtualization cluster and uses Terraform, Ansible, and ArgoCD to build a complete LLM inference infrastructure including Vault key management, Traefik ingress, monitoring system, and AI cost attribution system. It is not only learning material but also a production-ready deployment template that covers best practices for modern AI infrastructure.

Background and Infrastructure Foundation

The project chooses Proxmox VE as the virtualization layer due to its high availability (workload migration when a single node fails), resource pooling (unified management of CPU/memory/storage), and flexible scalability. On top of this, the k3s lightweight Kubernetes distribution is deployed, with advantages including low resource consumption (only 512MB of memory per node), built-in core components (Flannel, CoreDNS, etc.), simplified installation (single binary file), and production readiness (CNCF certified). For automated deployment: Terraform is responsible for infrastructure as code (defining virtual machines, networks, storage), and Ansible handles configuration management (installing k3s and its dependencies).

Detailed Explanation of Core Components

1. ArgoCD: Core of GitOps workflow, storing application configurations in Git as the single source of truth, continuously monitoring and syncing cluster state, supporting model deployment version control, automatic sync, multi-environment promotion, and quick rollback.
2. Vault: Centralized secret management, providing dynamic secret generation, automatic rotation, fine-grained access control, and audit logs. Integration with Kubernetes is achieved via the Kubernetes Auth Method for Pod authentication, with the External Secrets Operator syncing secrets.
3. Traefik: Ingress controller supporting automatic service discovery, dynamic configuration, middleware (authentication/rate limiting, etc.), and Let's Encrypt integration, used for routing inference services, API version management, and WebSocket support.
4. cert-manager: Cooperates with Traefik to automatically apply/renew Let's Encrypt certificates and store them as Kubernetes Secrets.
5. Prometheus+Grafana: Monitoring stack that collects time-series data (including GPU utilization, inference latency/throughput), visualizes via Grafana, and sets up alerts.

AI Cost Attribution System (Project Highlight)

In enterprises, AI resource costs need to be allocated by team/project/user. This system implements:

• Attaching metadata (team, project, user) to inference requests;
• Recording processing time and resource consumption;
• Aggregating cost data by dimension and generating reports/budget alerts; Tech stack: OpenTelemetry distributed tracing, correlating tracing data with resource metrics, and displaying cost dashboards via Grafana.

Deployment Process (IaC Principles)

1. Infrastructure preparation: Configure Proxmox cluster → Terraform define VM specs → Create VM → Ansible configure OS;
2. Kubernetes deployment: Install k3s server on the first node → Other nodes join as agents → Configure kubectl → Verify cluster;
3. Core service deployment: Install ArgoCD → Initialize Vault → Deploy Traefik+cert-manager → Prometheus+Grafana monitoring stack;
4. LLM inference stack: Deploy model services (vLLM/TGI) → Configure routing rules → Set up auto-scaling → Cost/performance monitoring.

Production-Ready Features

• High availability: k3s server HA (embedded etcd), Traefik multiple replicas, Vault Raft mode, monitoring component redundancy;
• Security: TLS encryption for component communication, Vault managing sensitive credentials, RBAC access control, network policies limiting Pod communication;
• Observability: Log collection (e.g., Loki), distributed tracing, metric monitoring and alerts, cost attribution reports;
• Maintainability: GitOps configuration management, declarative infrastructure, automatic certificate management, documented operation and maintenance processes.

Learning Value and Application Scenarios

Learning Objectives: Understand the enterprise AI platform tech stack, GitOps and IaC, Kubernetes AI workload management; Practical Applications: Reference architecture for internal enterprise AI platforms, quick start for AI project infrastructure, technical selection evaluation; Expansion Directions: Multi-cluster federation, MLOps pipeline integration (Kubeflow/MLflow), complex cost allocation models, model version management and A/B testing.

Summary

This project demonstrates the complete construction process from bare metal to production-grade LLM inference services, covering key links such as virtualization, container orchestration, GitOps, secret management, ingress control, monitoring, and cost management. For enterprise AI infrastructure deployment teams, it provides valuable practical experience and technical selection references. Using modern DevOps tools to achieve version control, automated deployment, and repeatable construction of AI infrastructure will become a key support for enterprise digital transformation.