ArchitectureAdvanced

Tensor Parallelism vs Pipeline Parallelism: How 70B and 405B Models Run Across GPUs

Direct Answer & Overview

A technical breakdown of distributed GPU orchestration, comparing Tensor Parallelism (TP, intra-node GPU matrix splitting) and Pipeline Parallelism (PP, inter-node layer partitioning) to serve massive 70B, 405B, and 671B foundation models at scale.

1.Tensor Parallelism (TP): Splitting Matrix Multiplications

Tensor Parallelism (Megatron-LM style) partitions individual weight matrices across multiple GPUs within a single physical server connected via high-bandwidth NVLink (900 GB/s on H100). Column-parallel and row-parallel linear layers split the computation, requiring an All-Reduce communication collective after each layer. TP minimizes latency and is typically scaled to 2, 4, or 8 GPUs per node.

2.Pipeline Parallelism (PP): Partitioning Across Server Nodes

When a model exceeds the memory capacity of a single 8-GPU node (such as Llama 3.1 405B in FP16 requiring 800+ GB of VRAM), Pipeline Parallelism divides the model's sequential transformer layers across multiple physical nodes. Node 1 processes layers 1-32, Node 2 processes layers 33-64, etc. Micro-batching algorithms (like 1F1B) keep GPU utilization high while minimizing pipeline bubble bubbles.

3.Context Parallelism (CP) for Million-Token Sequences

For extreme context windows (128k to 1M+ tokens), the attention matrix itself exceeds single GPU memory. Context Parallelism partitions the sequence length dimension across GPUs using Ring-Attention, enabling processing of massive documents without OOM errors.

Configuring Tensor Parallelism in PyTorch Distributedpython

import os
import torch
import torch.distributed as dist

def init_distributed_tensor_parallel():
    # Initialize process group across 8 GPUs on an HGX H100 node
    dist.init_process_group(backend="nccl")
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    world_size = dist.get_world_size()
    print(f"GPU Node Initialized: Rank {local_rank} of {world_size} using NCCL NVLink")

# Invoked across torchrun worker processes

Frequently Asked Questions

Why can't Tensor Parallelism run across Ethernet or standard network cables?

TP requires thousands of All-Reduce collective operations per second. Running TP over slow networking creates massive communication bottlenecks; it requires ultra-high bandwidth NVLink (900 GB/s) or InfiniBand.

What is an HGX server?

An NVIDIA HGX server is an 8-GPU integrated motherboard (e.g. 8x H100 or H200 80GB) connected by an NVLink switch mesh, providing 640GB of unified GPU memory.

How does API100 scale distributed inference?

API100 clusters utilize optimized Tensor Parallelism with continuous batching across high-density GPU nodes to deliver sub-50ms token generation.

A100

API100 Engineering Team

Infrastructure & Latency Research

Author Profile →

Previous Guide

Speculative Decoding Explained: How Draft Models Accelerate LLM Inference 3x

Next Guide

Batch API vs Real-Time Inference: Getting 50% Discounts on Non-Urgent AI Tasks