ArchitectureAdvanced
GPU Cold Starts & Serverless AI Inference: vLLM, TensorRT-LLM, and Triton
Direct Answer & Overview
An in-depth analysis of GPU cold-start mechanics during serverless model scaling, comparing modern inference serving runtimes (vLLM PagedAttention, TensorRT-LLM, and Triton Inference Server) and container optimization techniques.
1.Anatomy of an LLM Cold Start: Model Weights to VRAM
A 70B parameter model in FP16 format occupies ~140 GB of disk space. When a serverless container spins up from zero, it must download weights over the network (10-30s), load them into system RAM, and transfer them across PCIe/NVLink into GPU HBM3 memory (5-15s). Total cold start can easily exceed 45 seconds without optimized warm pools.
2.Inference Engines: vLLM vs. TensorRT-LLM vs. TGI
• vLLM: Pioneered PagedAttention, treating KV cache like virtual memory pages to eliminate 96% of memory fragmentation. Highest developer ergonomics and continuous batching.
• TensorRT-LLM: NVIDIA's compiled C++ engine; achieves maximum FLOP efficiency and kernel fusion on H100/H200 hardware.
• HuggingFace TGI: Battle-tested production server with built-in token streaming, FlashAttention-2, and watermarking.
3.Techniques for Sub-Second Cold Starts
High-performance infrastructure uses local NVMe caches, pre-baked Docker images, Safetensors memory-mapped files (`mmap`), and keep-alive warm instance pools to achieve instantaneous inference readiness.
Deploying High-Throughput Model with vLLM and PagedAttentionbash
# Launch vLLM server with Tensor Parallelism across 2 GPUs
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.95 \
--max-model-len 8192 \
--port 8000Frequently Asked Questions
What is PagedAttention in vLLM?
PagedAttention partitions the transformer KV cache into non-contiguous memory blocks, mirroring operating system virtual memory to virtually eliminate memory waste.
Why are Safetensors faster to load than PyTorch checkpoints?
Safetensors avoids Python pickle deserialization and allows zero-copy direct memory mapping (mmap) straight from disk to RAM.
Does API100 suffer from cold starts?
No, API100 maintains continuously warm enterprise clusters, guaranteeing sub-second response times without serverless spin-up delays.
A100
API100 Engineering Team
Infrastructure & Latency Research