Alchemyst Cloud Cartographer: A Distributed LLM Inference Deployment Solution on GCP

This article introduces the open-source project Alchemyst Cloud Cartographer, a production-grade distributed LLM inference deployment solution based on GCP. The project ensures security through public/private subnet isolation, implements Infrastructure as Code (IaC) using Terraform, uses the iii framework for distributed communication, supports Gemma 3 270M model inference, provides a complete operation and maintenance testing and expansion path, and serves as a secure, scalable, and maintainable reference for enterprises and developers to deploy LLMs in the cloud.

With the rapid development of open-source LLMs, enterprises face three core challenges when deploying LLMs: security (requiring protections like network isolation and access control), scalability (handling traffic fluctuations with reasonable costs), and maintainability (needing IaC, automated testing, and monitoring). This project is a complete reference implementation designed to address these issues.

The project adopts a layered public-private subnet architecture:

• The public subnet (10.10.1.0/24) hosts the gateway VM (with a public IP), runs the iii framework engine and caller process, exposes HTTP APIs externally, and incoming traffic is protected by Cloud Armor WAF.
• The private subnet (10.10.2.0/24) hosts the inference VM (without a public IP), runs the Gemma 3 270M model inference process, and outgoing traffic goes through Cloud NAT.
• Communication between subnets uses internal VPC WebSocket, with strict firewall access restrictions. The iii framework is chosen as a lightweight RPC communication tool, requiring no complex orchestration, running as a systemd service, and supporting OpenAI-compatible response formats. Security measures include Cloud Armor, VPC firewall, IAP SSH access, Shielded VM, etc.

The project implements IaC based on Terraform with a modular design (network, iam, compute, observability modules) for reusable code. It integrates a CI/CD pipeline that automatically performs Terraform format checks, configuration validation, static analysis (tflint), and security scanning (tfsec, checkov) to ensure safe and standardized changes.

The project provides a multi-dimensional test suite:

• Smoke test: End-to-end API test to verify normal link operation;
• Isolation test: Confirm that the inference VM cannot be directly accessed from the internet;
• Chaos test: Kill the inference process to verify systemd automatic recovery;
• Load test: Use k6 to evaluate high-concurrency performance. Through the observability module, Cloud Monitoring dashboards and alerts are configured to monitor key metrics such as API latency, VM resource usage, and iii health status.

The monthly cost of the project is approximately $153 (gateway-vm: $13, inference-vm: $98, Cloud NAT: $3, Cloud Router: $36, etc.), and the GCP free trial credit can cover about 60 days. The expansion path has four stages: vLLM optimization → TensorRT-LLM compilation → Triton Inference Server → NVIDIA Dynamo distributed inference, to gradually improve performance and throughput.

This architecture is suitable for scenarios such as enterprise internal AI services, model evaluation platforms, edge AI gateways, and development/test environments. The project is not only a technical implementation but also a collection of best practices for production-grade LLM deployment, providing a validation starting point for teams building their own LLM inference capabilities and helping to transform model capabilities into business value.