CUDA Inference Engine: High-Performance GPT-2 Inference Engine Implementation Using C/C++

This project implements a GPT-2 inference engine using pure C/C++ and CUDA, inspired by Andrej Karpathy's llm.c. It addresses LLM inference performance bottlenecks in production by leveraging low-level optimizations, providing both educational value (clear Transformer implementation without framework abstraction) and practical benefits (high throughput/low latency, minimal dependencies for edge deployment).

Performance Challenges: Python-based inference faces bottlenecks from GIL, dynamic typing, and hidden optimizations, while modern GPUs require fine-grained memory and kernel management.

Inspiration: Derived from llm.c (C-based GPT-2 implementation for education), this project extends it with CUDA to unlock GPU parallelism. Key values: educational (direct view of model computation), optimization potential (control over memory layout/kernel scheduling), deployment flexibility (standalone binaries for resource-limited environments).

Components: llmc directory (core C/C++ logic), dev tools, training/test scripts, CMake config.

Model Loading: Converts PyTorch/TensorFlow weights to contiguous memory; uses pinned memory for GPU transfer, supports weight sharding for large models.

Transformer Forward Pass: Implements layer norm (manual C), self-attention (CUDA parallelization with shared memory, online Softmax), feed-forward (custom CUDA kernels for GELU/linear), residual connections.

Tokenizer: C-based BPE with Unicode support.

Memory Management: Pre-allocated pools, activation reuse, weight sharing, optional INT8 quantization.

Kernel Fusion: Merges operations (linear+GELU+layer norm) into single kernels to reduce memory access and kernel launch overhead.

Shared Memory: Uses shared memory to cache Q/K blocks in attention, reducing global memory latency.

FP16 Support: Leverages Tensor Cores for FP16 (mixed precision: FP16 for weights/activations, FP32 for sensitive ops like Softmax).

Batch Optimization: NHWC layout for batch data to improve GPU cache hit rate.

Expected Performance: Lower first-token latency (20-40% vs PyTorch), higher throughput (50%+ in batch), small binary size (tens of MB).

Application Scenarios: Embedded AI, real-time interactive systems, batch text generation, education/research.

Limitations: Only supports GPT-2; no auto-diff (limited training); less flexible than Python for frequent model updates.

• llm.c: This project adds CUDA support (vs llm.c's CPU focus); both educational but this targets production GPU performance.
• vLLM: vLLM has PagedAttention and broader model support but depends on Python; this is lighter for minimal deployment.
• TensorRT-LLM: NVIDIA's closed-source high-performance solution; this is open-source and modifiable.

Insights: Performance requires hardware understanding; dedicated implementations can outperform flexible frameworks; education and utility can coexist.

Future: Support larger models (Llama, Mistral), advanced quantization (GPTQ/AWQ), multi-GPU distributed inference.

Conclusion: This project demonstrates high-performance LLM inference without heavy frameworks, offering a valuable reference for developers/researchers needing low-level control or edge deployment.