2025-05-04 · Updated 2026-07-15

Scaling Large Language Models: Multi-GPU and Multi-Node Strategies That Hold Up in Practice

Large-model workloads move beyond one GPU for different reasons. A training job may run out of memory on optimizer state, another on long-sequence activations, and a model that fits may still miss its throughput target. Each problem calls for a different partition and communication pattern.

This is a practical walk-through of the main parallelism strategies and the constraints behind them, informed by Hugging Face’s Ultra-Scale Playbook. The goal is to show what each split buys, what it communicates, and when combinations become necessary.

TL;DR. Replicated data parallelism buys training throughput when one replica fits. Fully sharded data parallelism partitions model state but adds parameter all-gathers and gradient reduce-scatters. Tensor, pipeline, context, and expert parallelism split layer math, depth, sequence, and MoE experts. Combine them only after identifying the binding memory or communication constraint.

This guide assumes you are comfortable with backpropagation, Transformer layers, and a standard PyTorch training loop.

Start with two memory budgets

Training and inference do not have the same footprint.

training peak ≈ parameters
              + gradients
              + optimizer state
              + saved activations
              + temporary buffers
              + communication buffers
              + allocator headroom

inference peak ≈ resident weights
               + KV cache
               + runtime workspace
               + communication buffers
               + allocator headroom

A 70-billion-parameter model has a 140 GB decimal lower bound for BF16 weights alone. That number says little about training, where gradients, optimizer state, master weights, and activations may dominate. It also does not size serving, where cache policy, sequence length, batch concurrency, and quantization matter.

Profile the exact architecture, precision, sequence length, micro-batch, optimizer, checkpointing policy, and runtime. Record peak allocated and reserved memory, tokens per second, time in kernels, and time exposed in collectives.

Distributed scaling starts from a measured bottleneck

Every parallel dimension makes a trade

The useful question is not “Which technique is best?” It is “Which tensor dimension is split, which state is replicated, and which collective enters the critical path?”

Strategy	Splits	Primary relief	Communication introduced
Replicated data parallelism	batch	training throughput	gradient all-reduce
Fully sharded data parallelism	parameters, gradients, optimizer state across a DP group	model-state memory	parameter all-gather, gradient reduce-scatter
Tensor parallelism	matrix or attention dimensions inside layers	layer weights and activations	collectives within transformer blocks
Pipeline parallelism	layer groups	model depth and per-stage state	point-to-point activations plus scheduling bubbles
Context parallelism	sequence dimension	long-sequence activation memory	key/value or attention exchange across the sequence group
Expert parallelism	MoE experts and routed tokens	expert capacity per rank	token dispatch and combine, commonly all-to-all

Memory relief is not a fixed multiplier. It depends on sharding degree, what remains replicated, transient unsharded state, activation policy, padding, imbalance, and buffers.

Replicated data parallelism: throughput without capacity

Replicated data parallel training

Distributed data parallelism keeps a full training replica on every rank. Each rank handles a different micro-batch, and gradients are synchronized before the optimizer step.

Use it when the full training state fits with safe headroom and the global batch can grow or gradient accumulation can adjust. Its main advantages are simple semantics and mature overlap between backward computation and bucketed gradient reduction.

Adding ranks can hurt if the local batch becomes too small, the network cannot hide the all-reduce, input delivery stalls, or the desired optimization batch does not scale.

Fully sharded data parallelism: state memory for collectives

Fully sharded data parallel execution

Fully sharded data parallelism stores parameter, gradient, and optimizer shards across a group. A layer’s parameters are all-gathered for computation and can be resharded afterward; gradients are reduce-scattered back to owners.

Current PyTorch documentation distinguishes FSDP2’s fully_shard API from the older FullyShardedDataParallel wrapper. FSDP2 groups communication by the modules to which fully_shard is applied and recommends bottom-up application so layer groups can overlap communication and computation.

from torch.distributed.fsdp import fully_shard


# Apply bottom-up: each block becomes a communication group.
for block in model.transformer.blocks:
    fully_shard(block)

# Shard remaining root parameters such as embeddings and output projection.
fully_shard(model)

# Construct the optimizer after parameters have become sharded DTensors.
optimizer = AdamW(model.parameters(), lr=learning_rate)

This is a structural sketch, not a complete launcher. Device meshes, mixed precision, checkpointing, initialization, optimizer state, and distributed checkpoints must match the training stack.

Sharding is attractive when model state is the binding constraint and layer compute can hide enough collective traffic. It may be a poor trade for small models, slow links, tiny layers, or layouts whose shard group crosses the wrong topology boundary.

Tensor parallelism: partition layer math

Tensor parallel layer partition

Tensor parallelism partitions linear algebra inside a layer—for example, column-parallel and row-parallel projections. Partial results require collectives within transformer blocks, so latency and bandwidth matter repeatedly throughout the forward and backward passes.

Use it when a layer or its activations do not fit, or when the matmuls are large enough for partitioned kernels to outperform one rank. Map the tensor-parallel group to the fastest communication domain available, then measure. High degree can shrink each local matrix until kernel efficiency falls while collective overhead grows.

Sequence parallelism is often paired with tensor parallelism to avoid replicating some activation work; it is distinct from context parallelism over the model’s full input sequence.

Pipeline parallelism: partition depth and schedule time

Pipeline parallel stages and micro-batches

Pipeline parallelism places different layer groups on different stages and sends activations between them. Micro-batches keep stages working concurrently.

It relieves per-stage model state and can reduce the volume of communication crossing a slower boundary compared with per-layer tensor collectives. Its costs are bubbles, activation transfers, stage imbalance, more complex scheduling, and harder recovery and checkpointing.

For a simple balanced GPipe-style schedule with p stages and m micro-batches, the idealized forward bubble fraction is approximately:

(p - 1) / (m + p - 1)

Real schedules may use one-forward/one-backward, interleaving, or zero-bubble variants, and unequal layer cost can dominate the formula. Choose stage boundaries from measured time and memory, not equal layer counts.

Context parallelism: partition long-sequence activations

Context parallel attention exchange

Context parallelism distributes the sequence dimension. Each rank owns a sequence shard, while attention exchanges the information needed to preserve full-context semantics. Implementations may use point-to-point rings, all-gather, all-to-all, or hierarchical combinations.

It reduces activation memory for long-context training but replicates weights across the context group and introduces attention communication. Benefit depends on attention type, causal masking, sequence length, recomputation, and how context groups compose with tensor and data parallel groups.

Do not select it from a universal 8K, 32K, or 100K threshold. Profile activation memory and attention communication for the actual architecture.

Expert parallelism: only for an MoE architecture

Expert-parallel token routing

Expert parallelism distributes experts in mixture-of-experts layers. The router sends token representations to selected experts and combines their results. Only selected experts compute for each token, but total expert weights still require storage and serving placement.

This is not an optimization switch for a dense model. It is part of an MoE architecture and brings load balance, capacity, token all-to-all, dropped or padded tokens, auxiliary losses, and failure skew. Track tokens per expert, routing entropy, capacity overflow, communication time, and quality by route.

Compose a layout from the topology

Large training systems combine dimensions. The total world size commonly follows a product such as:

world size = DP × TP × PP × CP

Expert parallelism may share or fold dimensions differently, so verify the framework’s supported mesh rather than multiplying blindly.

Build the layout in this order:

Draw communication domains: GPU-to-GPU links, switches, NUMA boundaries, node fabric, oversubscription, and storage path.
Place frequent latency-sensitive collectives, commonly TP, in the fastest suitable domain.
Choose FSDP or replicated DP groups from remaining capacity and bandwidth.
Add PP when depth placement or cross-domain traffic benefits, balancing measured stage time and memory.
Add CP only for the sequence constraint, and EP only for the model’s expert topology.
Confirm divisibility of heads, hidden dimensions, layers, experts, batch, and sequence for the candidate mesh.
Benchmark several valid meshes; topology-aware heuristics choose candidates, not winners.

Two clusters with the same GPU count can prefer different layouts because link bandwidth, switch hierarchy, CPU attachment, and network contention differ.

Training and serving need separate decisions

Inference usually does not carry gradients or optimizer state, so FSDP-style training layouts do not transfer automatically.

For serving, ask:

Does one replica fit weights, KV cache, workspace, and target concurrency?
Is throughput better served by more independent replicas or by sharding one replica?
Does TP reduce per-rank weight and cache pressure enough to repay per-layer communication?
Is PP supported efficiently for the model and request scheduler?
How do prefill and decode stress compute, memory bandwidth, and interconnect differently?
What happens to tail latency when requests have different prompt and output lengths?

Benchmark the complete server with the scheduler, quantization, context distribution, batching policy, and traffic shape. Training tokens per second cannot predict serving time to first token or inter-token latency.

Measure a scaling layout honestly

For each candidate, record:

model, code, runtime, kernels, and topology identity
global and local batch, sequence distribution, and token count
peak memory by category where available
useful tokens per second and model FLOP utilization when calculated consistently
exposed time in all-reduce, all-gather, reduce-scatter, all-to-all, and point-to-point operations
input stalls, checkpoint time, restart behavior, and straggler distribution
training loss or serving-output parity against the baseline

Compare weak and strong scaling deliberately. Weak scaling increases total work with rank count; strong scaling holds total work constant. A percentage labeled “scaling efficiency” is meaningless without that denominator and baseline.

Conclusion

Parallelism is a mapping from a measured bottleneck to a tensor dimension and a communication pattern. Replication, sharding, layer partitioning, staging, sequence partitioning, and expert routing each relieve a different constraint and create a different failure mode.

Inventory the workload, draw the topology, generate valid meshes, and profile them. The winning layout is the one that fits with headroom and minimizes exposed communication for the job you actually run.