GPU Cold Starts & Serverless AI Inference: vLLM, TensorRT-LLM, and Triton

Overview #

An in-depth analysis of GPU cold-start mechanics during serverless model scaling, comparing modern inference serving runtimes (vLLM PagedAttention, TensorRT-LLM, and Triton Inference Server) and container optimization techniques.

Anatomy of an LLM Cold Start: Model Weights to VRAM #

A 70B parameter model in FP16 format occupies ~140 GB of disk space. When a serverless container spins up from zero, it must download weights over the network (10-30s), load them into system RAM, and transfer them across PCIe/NVLink into GPU HBM3 memory (5-15s). Total cold start can easily exceed 45 seconds without optimized warm pools.

Inference Engines: vLLM vs. TensorRT-LLM vs. TGI #

• vLLM: Pioneered PagedAttention, treating KV cache like virtual memory pages to eliminate 96% of memory fragmentation. Highest developer ergonomics and continuous batching.
• TensorRT-LLM: NVIDIA's compiled C++ engine; achieves maximum FLOP efficiency and kernel fusion on H100/H200 hardware.
• HuggingFace TGI: Battle-tested production server with built-in token streaming, FlashAttention-2, and watermarking.

Techniques for Sub-Second Cold Starts #

High-performance infrastructure uses local NVMe caches, pre-baked Docker images, Safetensors memory-mapped files (mmap), and keep-alive warm instance pools to achieve instantaneous inference readiness.

Code Example: Deploying High-Throughput Model with vLLM and PagedAttention #

# Launch vLLM server with Tensor Parallelism across 2 GPUs
python3 -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 8192 \
    --port 8000

Frequently Asked Questions #

Q: What is PagedAttention in vLLM?

PagedAttention partitions the transformer KV cache into non-contiguous memory blocks, mirroring operating system virtual memory to virtually eliminate memory waste.

Q: Why are Safetensors faster to load than PyTorch checkpoints?

Safetensors avoids Python pickle deserialization and allows zero-copy direct memory mapping (mmap) straight from disk to RAM.

Q: Does API100 suffer from cold starts?

No, API100 maintains continuously warm enterprise clusters, guaranteeing sub-second response times without serverless spin-up delays.