Reading
A complete Terraform infrastructure project demonstrating how to deploy vLLM inference services on Amazon EKS to achieve production-grade streaming LLM inference capabilities.
Section 01
A complete Terraform infrastructure project demonstrating how to deploy vLLM inference services on Amazon EKS to achieve production-grade streaming LLM inference capabilities.
Section 02
With the popularization of Large Language Models (LLMs) in enterprise applications, efficiently deploying inference services in cloud-native environments has become a key challenge. The open-source vllm-on-eks project by Nicolas-Richard provides a complete solution, demonstrating how to deploy vLLM on Amazon Elastic Kubernetes Service (EKS) to achieve production-grade streaming LLM inference capabilities.
As a supporting code repository, this project complements the blog post Streaming LLM inference on EKS and offers readers a complete practical path from infrastructure setup to application deployment.
Section 03
vLLM is an open-source LLM inference and serving engine developed by the research team at the University of California, Berkeley. Its core innovations include:
Section 04
Amazon EKS is a managed Kubernetes service provided by AWS, offering enterprise-grade container orchestration with:
Section 05
The project uses a clear layered architecture, dividing the infrastructure into two Terraform subprojects:
This layer is responsible for building and configuring the EKS cluster itself, including:
This layer deploys specific application components on top of the EKS cluster:
Section 06
The project adopts an intelligent image building and pushing strategy:
terraform_data resourceThis design ensures consistency between image versions and code versions while avoiding unnecessary repeated builds.
Section 07
The project encapsulates daily operations through the Makefile in the root directory, providing a concise command-line interface:
Section 08
| Command | Description |
|---|---|
make deploy |
Full deployment process: Bootstrap ECR repository, then perform complete platform-apps application deployment |
make ecr-bootstrap |
Only create ECR repository; required before first deployment |
make terraform-apply |
Execute Terraform apply in infra/platform-apps |
make destroy |
Destroy platform-apps resources (retains EKS cluster) |
make gateway-url |
Output the public NLB URL of the gateway |
make gateway-token |
Output the access token |
make gateway-info |
Output both URL and token |
make gateway-test |
Streaming chat completion test; output raw SSE blocks (for debugging) |
make gateway-chat |
Streaming chat completion; output only assistant text to stdout |
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.