DeepStack: The 'Design Navigator' for 3D Stacked AI Chips, 100,000x Acceleration in Finding Optimal Solutions

DeepStack is a framework for distributed 3D stacked AI accelerators, addressing the memory wall problem and solving the exponential complexity of design space exploration (DSE). Key benefits: 100,000x faster DSE than detailed simulators, 9.5x throughput improvement over baseline designs, and ability to find optimal configurations in a 250-trillion design point space.

AI models face the "memory wall"—growing model size (from billions to trillions of parameters) outpaces memory bandwidth. 3D stacking (vertical compute/memory integration) solves this with higher bandwidth and lower latency, but distributed 3D inference introduces complex tradeoffs (hardware: DRAM layers, connections; system: model splitting, parallel strategies). The design space is up to 1e14+ points, making brute force impossible.

DeepStack balances accuracy and speed (ms-level per design point, 2-12% error vs simulators). Key components:

• Hardware modeling: Transaction-aware bandwidth, bank activation constraints, buffer limits, thermal-power modeling.
• System modeling: Supports data/model/pipeline/tensor/hybrid parallelism and scheduling.
• Innovations: Dual-stage network abstraction (speed + critical path accuracy), tile-level compute-communication overlap.

• Accuracy: Consistent with real 3D chips; 2.12% error vs NS-3 network simulator; 12.18% error vs vLLM on 8xB200 GPUs.
• Speed: 100,000x faster than state-of-the-art detailed simulators.
• DSE Outcomes: 9.5x throughput improvement over baseline. Key findings: batch size drives architecture choices; parallel strategy must align with hardware; optimal 3D layer count exists; interconnect topology is critical.

• Chip Architects: Pre-tapeout evaluation of 3D stack configurations (e.g., DRAM layer count impact) in seconds.
• System Engineers: Optimize deployment (parallel strategy, batch size) for specific models/workloads.
• Researchers: Fast validation of new AI architectures/parallel strategies without hardware prototypes.

• Open Source: Plan to release framework, pre-trained models, DSE tools, and benchmarks.
• Future Work: Support HBM/CXL/in-memory computing; extend to distributed training; auto-optimization via ML; multi-objective optimization (performance + cost + power).