Tensor Parallelism vs Pipeline Parallelism: How 70B and 405B Models Run Across GPUs

Overview #

A technical breakdown of distributed GPU orchestration, comparing Tensor Parallelism (TP, intra-node GPU matrix splitting) and Pipeline Parallelism (PP, inter-node layer partitioning) to serve massive 70B, 405B, and 671B foundation models at scale.

Tensor Parallelism (TP): Splitting Matrix Multiplications #

Tensor Parallelism (Megatron-LM style) partitions individual weight matrices across multiple GPUs within a single physical server connected via high-bandwidth NVLink (900 GB/s on H100). Column-parallel and row-parallel linear layers split the computation, requiring an All-Reduce communication collective after each layer. TP minimizes latency and is typically scaled to 2, 4, or 8 GPUs per node.

Pipeline Parallelism (PP): Partitioning Across Server Nodes #

When a model exceeds the memory capacity of a single 8-GPU node (such as Llama 3.1 405B in FP16 requiring 800+ GB of VRAM), Pipeline Parallelism divides the model's sequential transformer layers across multiple physical nodes. Node 1 processes layers 1-32, Node 2 processes layers 33-64, etc. Micro-batching algorithms (like 1F1B) keep GPU utilization high while minimizing pipeline bubble bubbles.

Context Parallelism (CP) for Million-Token Sequences #

For extreme context windows (128k to 1M+ tokens), the attention matrix itself exceeds single GPU memory. Context Parallelism partitions the sequence length dimension across GPUs using Ring-Attention, enabling processing of massive documents without OOM errors.

Code Example: Configuring Tensor Parallelism in PyTorch Distributed #

import os
import torch
import torch.distributed as dist

def init_distributed_tensor_parallel():
    # Initialize process group across 8 GPUs on an HGX H100 node
    dist.init_process_group(backend="nccl")
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    world_size = dist.get_world_size()
    print(f"GPU Node Initialized: Rank {local_rank} of {world_size} using NCCL NVLink")

# Invoked across torchrun worker processes

Frequently Asked Questions #

Q: Why can't Tensor Parallelism run across Ethernet or standard network cables?

TP requires thousands of All-Reduce collective operations per second. Running TP over slow networking creates massive communication bottlenecks; it requires ultra-high bandwidth NVLink (900 GB/s) or InfiniBand.

Q: What is an HGX server?

An NVIDIA HGX server is an 8-GPU integrated motherboard (e.g. 8x H100 or H200 80GB) connected by an NVLink switch mesh, providing 640GB of unified GPU memory.

Q: How does API100 scale distributed inference?

API100 clusters utilize optimized Tensor Parallelism with continuous batching across high-density GPU nodes to deliver sub-50ms token generation.