AllReduce Explained: Algorithms, Performance, and Distributed Training

AllReduce is a collective communication operation that combines values from multiple processes and returns the final result to every participant. In distributed machine learning, it is commonly used to combine gradients so that every GPU applies the same model update.

The basic idea is simple. Each worker contributes data, the system reduces those values using an operation such as sum, average, minimum, or maximum, and every worker receives the result. The difficult part is making that process fast. As training expands from one GPU to several GPUs, and eventually across multiple servers, communication becomes harder to hide. The way AllReduce is implemented can determine whether adding more hardware improves training speed or simply gives GPUs more time to wait.

Understanding AllReduce helps explain how distributed training stays synchronized, why Ring AllReduce is so common, and why communication eventually becomes one of the biggest limits on training performance.

What Is AllReduce?

AllReduce is a collective operation, which means every process in a communication group participates in it. Each process begins with its own tensor or value. Those inputs are combined using a reduction operation, and the completed result is returned to every process.

The operation can be written as:

In this formula, (P) represents the number of participating processes. The symbol (\oplus) represents the selected reduction operation, such as addition, multiplication, minimum, or maximum.

Unlike a standard reduce operation, AllReduce does not send the result to only one process. Every participant receives the same completed output. This makes it useful whenever multiple workers need to agree on a shared result.

A Simple AllReduce Example

Imagine four GPUs each holding a single value.

If the selected reduction operation is sum, the result is:

After AllReduce finishes, every GPU holds the value 20.

The same process can be applied to large tensors containing millions or billions of values. Instead of adding one number from each GPU, the system combines matching elements across entire tensors.

How AllReduce Is Used in Distributed Training

AllReduce is most closely associated with synchronous data-parallel training. In data parallelism, each GPU holds a complete copy of the model but processes a different portion of the training data. This allows several batches to be processed at the same time.

A typical training step works like this:

1. Every GPU starts with the same model parameters.
2. Each GPU receives a different local batch.
3. Each GPU performs a forward pass.
4. Each GPU calculates gradients during backpropagation.
5. AllReduce combines the gradients from every GPU.
6. Each GPU performs the same optimizer update.

Without gradient synchronization, each model replica would update itself using only its local batch. The replicas would quickly begin moving in different directions. AllReduce keeps them aligned by making sure every worker uses the same combined gradients. Depending on the framework and configuration, the gradients may be summed and then divided by the number of workers, or they may be averaged as part of the synchronization process. Either way, every worker ends the operation with matching gradient values.

Why AllReduce Matters for Scaling

Adding GPUs increases the amount of computation a system can perform in parallel. Communication does not improve automatically just because more compute is available. At smaller scales, the forward and backward passes usually account for most of the training step. Gradient synchronization may take only a small amount of time. As more GPUs are added, each worker processes a smaller share of the total work. The local computation finishes faster, but the workers still need to exchange and combine gradients before the training step can continue.

Eventually, communication begins taking up a larger percentage of the step.

This does not mean the gradient tensor becomes larger every time another GPU is added. The model still produces roughly the same amount of gradient data per worker. The challenge comes from coordinating more participants, completing more communication stages, dealing with network contention, and waiting for slower workers.

At that point, the workload may stop being limited by computation and start being limited by communication.

Common AllReduce Algorithms

AllReduce describes the result of the operation, not the exact method used to produce it. Communication libraries can choose from several algorithms depending on the message size, number of participants, available interconnects, and physical network topology.

Reduce Followed by Broadcast

One of the simplest approaches is to perform a reduce operation first and then broadcast the result. During the reduce phase, data from every process is sent toward a root process and combined. The root then broadcasts the completed result back to everyone else.

This method is easy to understand, but it can place too much pressure on the root. That process may need to receive, reduce, and redistribute a large amount of data while the other workers wait. It can work well at small scale, but it becomes less attractive as the number of participants and the size of the messages increase.

Ring AllReduce

Ring AllReduce avoids relying on a single central process. The participating GPUs are arranged in a logical ring. Each GPU communicates directly with two neighbors, sending data in one direction and receiving data from the other.

The gradient tensor is divided into chunks. Those chunks move around the ring in two main phases:

1. Reduce-scatter
2. All-gather

During reduce-scatter, chunks circulate through the ring and are partially reduced at each step. By the end of this phase, every GPU owns one fully reduced chunk of the original tensor. During all-gather, those completed chunks circulate again. Once the second phase ends, every GPU has reconstructed the full reduced tensor.

Because communication is spread across all participants, there is no single root process that must handle everything.

Tree AllReduce

Tree-based algorithms organize workers into a communication tree. During the reduction phase, values move from lower levels of the tree toward the root. During the distribution phase, the result travels back down to the other workers.

The main advantage is latency. A tree can complete communication in a logarithmic number of stages rather than passing every chunk through every participant. This often makes tree-based AllReduce useful for smaller messages or very large process groups, where the number of communication steps matters more than maximizing raw bandwidth.

Recursive Doubling

Recursive doubling pairs processes together across a series of communication rounds. During each round, a process exchanges data with a partner and combines the received values with its current result. The distance between partners increases after each step until every process has incorporated data from the entire group. This approach can perform well for smaller messages because it requires relatively few communication stages. It is commonly discussed in MPI systems and high-performance computing environments.

Hierarchical AllReduce

Real GPU clusters often contain more than one type of connection. GPUs within the same server may communicate through NVLink, PCIe, or another high-speed local interconnect. Communication between servers may travel through InfiniBand or Ethernet.

Hierarchical AllReduce treats these communication levels differently. The system might first reduce gradients among GPUs inside each server. It then exchanges results between servers before distributing the completed output back to the local GPUs.

This avoids sending every individual GPU message directly across the slower inter-node network. It also allows the communication algorithm to take advantage of the actual hardware layout.

Why Ring AllReduce Is So Common

Ring AllReduce is popular because it uses bandwidth efficiently. Every GPU continuously sends and receives chunks instead of waiting for one central worker to collect everything. When the tensors are large, this steady communication pattern can keep high-speed interconnects busy and make good use of the available bandwidth.

This makes Ring AllReduce a strong choice for large gradient tensors, particularly when GPUs are connected through fast and relatively balanced links. It is also predictable. Each participant sends and receives roughly the same amount of data, so the communication workload is spread evenly across the group.

For these reasons, libraries such as NCCL often consider ring-based approaches when handling large messages.

Ring AllReduce Communication Cost

Suppose there are (P) participating workers and each worker holds a tensor containing (N) values.

Ring AllReduce requires two phases:

• Reduce-scatter takes (P – 1) steps.
• All-gather takes another (P – 1) steps.

Each worker sends approximately:

values during the complete operation.

As the number of workers increases, this amount approaches (2N). This means the communication volume per worker does not grow without limit simply because more workers join the group. The larger concern is the number of communication stages. Ring AllReduce requires a linear number of steps relative to the number of participants.

This leads to its main tradeoff:

Ring AllReduce is highly efficient with bandwidth, but it is not always the best algorithm for latency.

Limitations of Ring AllReduce

Ring AllReduce can perform extremely well, but it is not ideal in every situation. Each chunk must pass through several neighboring workers. As the number of participants grows, the operation requires more communication stages.

This becomes more noticeable when messages are small. For small tensors, the startup cost of each communication step may matter more than the amount of data being transferred. Ring performance can also suffer when the hardware topology is uneven. A ring that crosses a slow network link may be limited by that link, even if the rest of the connections are much faster.

Other common problems include:

• Network congestion
• Slow inter-node connections
• Poor GPU placement
• Uneven workload completion
• One delayed worker holding up the entire group
• Communication paths that do not match the physical topology

At larger scales, tree, hierarchical, or hybrid algorithms may provide better performance.

AllReduce and Stragglers

AllReduce is synchronous. Every participant must take part in the operation. If one worker reaches the collective late, the others cannot complete it without that worker. This makes distributed training sensitive to stragglers. A worker may fall behind because of uneven data loading, slower hardware, thermal throttling, network problems, or differences in the amount of computation performed during the step.

The collective may appear slow in a profiler even when the communication library is functioning correctly. In reality, several workers may simply be waiting for the final rank to arrive. This is why AllReduce timing should not always be treated as pure network time. It can also include synchronization delay caused by differences elsewhere in the training pipeline.

PyTorch AllReduce

PyTorch provides direct access to AllReduce through torch.distributed.all_reduce.

A basic example looks like this:

import torch
import torch.distributed as dist
tensor = torch.tensor([1.0], device="cuda")
dist.all_reduce(
    tensor,
    op=dist.ReduceOp.SUM
)
print(tensor)

Each process begins with its own local tensor. After all_reduce() completes, the tensor is updated in place with the sum of the values from every process in the group.

The function can also use other reduction operations, including minimum, maximum, and product.

Important arguments include:

• tensor, which contains the input and receives the output
• op, which selects the reduction operation
• group, which selects the participating process group
• async_op, which allows asynchronous execution

When async_op=True is used, PyTorch returns a work handle rather than waiting for the operation to finish immediately. The program can perform other work before calling wait() on that handle.

PyTorch DDP and Gradient Synchronization

Most PyTorch users do not manually call all_reduce() for every model gradient. DistributedDataParallel, commonly called DDP, manages gradient synchronization automatically. During backpropagation, it organizes gradients into buckets and launches communication as those buckets become ready. This allows some AllReduce operations to overlap with the remaining backward computation.

For example, gradients from later layers may become available first. DDP can begin reducing those gradients while earlier layers are still completing their backward calculations.

This overlap matters because communication that runs at the same time as computation may be partially or completely hidden. Only the portion that remains unfinished after backward computation ends becomes exposed on the critical path before the optimizer step. Bucket size can affect this balance. Larger buckets may use bandwidth efficiently but start later. Smaller buckets may begin communicating sooner but create more operations and additional overhead.

NCCL and AllReduce

On NVIDIA GPUs, PyTorch often relies on the NVIDIA Collective Communications Library, or NCCL, to perform AllReduce.

NCCL handles much of the low-level work, including:

• Dividing tensors into chunks
• Selecting communication paths
• Mapping operations onto GPU interconnects
• Coordinating transfers between ranks
• Choosing between available algorithms and protocols

This means torch.distributed.all_reduce() does not have one fixed performance profile. The result depends on the number of GPUs, tensor size, process placement, network hardware, NCCL configuration, and the topology connecting the devices. The same PyTorch code can perform very differently on two clusters because the underlying communication environments are different.

MPI_Allreduce

AllReduce existed in high-performance computing long before modern deep learning frameworks.

MPI, the Message Passing Interface, provides the standard MPI_Allreduce collective for distributed scientific and numerical workloads.

A typical call looks like this:

MPI_Allreduce(
    sendbuf,
    recvbuf,
    count,
    datatype,
    op,
    comm
);

The function combines values from every process in the communicator and returns the result to every participant.

MPI also provides related forms:

• MPI_Allreduce performs a blocking collective.
• MPI_Iallreduce starts a nonblocking operation.
• MPI_Allreduce_init creates a persistent collective request.

The basic semantics are the same as AllReduce in machine learning frameworks. Every process contributes data, the values are reduced, and every process receives the result.

Deep learning libraries build on many of the same distributed communication ideas that have been used in HPC systems for decades.

AllReduce Compared With Other Collectives

AllReduce is only one of several collective communication operations used in distributed systems.

The easiest way to remember the difference is to ask two questions:

1. Are values being combined?
2. Who receives the final result?

AllReduce combines values and sends the completed result to everyone.

All-gather does not combine values. It collects and concatenates them.

All-to-all redistributes different pieces of data between workers.

Reduce-scatter combines the data but leaves each worker with only part of the reduced result.

AllReduce Versus All-Gather

AllReduce and all-gather are sometimes confused because both leave every worker with information collected from the full process group. The difference between the two is reduction.

AllReduce combines matching values. If four workers contribute matching tensor elements, those elements may be summed or averaged into one result.

All-gather preserves the original values. It collects each worker’s input and joins the inputs together.

All-gather is commonly used when model or activation shards must be reconstructed. AllReduce is more common when workers need to agree on shared gradient values.

AllReduce Versus Reduce-Scatter

Reduce-scatter performs the reduction but does not return the entire result to every worker.

Instead, the reduced output is divided into shards. Each worker receives one shard.

This is important in sharded training systems, where no individual GPU needs to store the complete optimizer state, gradient tensor, or model parameter set.

Ring AllReduce can be understood as a reduce-scatter followed by an all-gather. The first phase creates the reduced shards, and the second phase distributes those shards so every worker receives the complete result.

AllReduce Versus All-to-All

All-to-all does not reduce values. It changes which worker owns each piece of data. Every participant divides its local input into several pieces and sends a different piece to every other participant.

This operation is common in mixture-of-experts models, distributed embedding systems, and workloads where tokens or features must be routed to different devices.

A useful mental model is:

• AllReduce creates agreement.
• All-gather reconstructs data.
• All-to-all redistributes ownership.

What Determines AllReduce Performance?

AllReduce performance is shaped by several connected factors.

Tensor Size

Small tensors are often limited by latency. The time required to launch and coordinate the operation may be larger than the time needed to transfer the data.

Large tensors are more likely to be limited by bandwidth. In that case, algorithms that keep network links busy may perform better.

Number of Workers

More workers increase the amount of coordination required. Some algorithms also need more communication stages as the process group grows. The effect depends heavily on the selected algorithm. Ring and tree approaches scale differently.

Interconnect Speed

NVLink, PCIe, InfiniBand, and Ethernet provide very different levels of bandwidth and latency. Communication within one server is often much faster than communication between servers. This is why a training job may scale well from two to eight GPUs inside one machine but slow down sharply when it expands to another node.

Hardware Topology

The physical arrangement of GPUs and network adapters matters. Two GPUs may appear to be part of the same system while communicating through a slower PCIe path. Another pair may have a direct high-speed connection. Topology-aware communication libraries attempt to choose paths that match the hardware, but placement and configuration problems can still limit performance.

Algorithm Selection

Ring, tree, recursive doubling, and hierarchical algorithms make different tradeoffs. The best choice may change when the tensor size, world size, or network topology changes. This is one reason performance can shift suddenly even when the model itself stays the same.

Worker Balance

AllReduce cannot finish until every worker participates. Uneven batch processing, data loading delays, or hardware differences can make the collective appear slower because faster ranks spend time waiting for slower ones.

Ways to Reduce AllReduce Overhead

AllReduce is difficult to eliminate from synchronous data-parallel training, but its exposed cost can often be reduced.

Overlap Communication With Computation

Frameworks can start reducing gradient buckets before the backward pass has completely finished. The more communication that runs alongside useful computation, the less of it appears in the final step time.

Tune Gradient Bucket Sizes

Larger buckets may provide better bandwidth efficiency, while smaller buckets may start earlier and allow more overlap. The best setting depends on the model, network, and order in which gradients become ready.

Use Gradient Accumulation

Gradient accumulation processes several local microbatches before synchronizing. This reduces how often AllReduce is called, although each synchronization still transfers the full gradient data. It can improve communication efficiency, but it also changes the effective batch size and may require learning-rate adjustments.

Use Mixed Precision

Using lower-precision gradient formats can reduce the number of bytes transferred. However, communication savings depend on how the framework stores, reduces, and updates gradients. Some systems still maintain higher-precision copies for numerical stability.

Use Gradient Compression

Compression methods reduce communication by quantizing, sparsifying, or approximating gradients. These techniques can lower network traffic, but they may introduce additional compute overhead or affect convergence.

Use Hierarchical Collectives

Hierarchical strategies reduce traffic across slower inter-node links by combining values locally before communicating between servers. This can provide a major improvement when local GPU connections are much faster than the cluster network.

Use Sharded Training

Some distributed strategies replace full gradient AllReduce with combinations of reduce-scatter and all-gather. This can lower memory usage and change the way communication is distributed across the training step.

Improve the Network

Sometimes the most direct solution is better technical infrastructure.

Faster network adapters, improved GPU placement, direct GPU communication, and properly configured InfiniBand can reduce synchronization time without changing the model.

How to Diagnose Slow AllReduce Performance

When AllReduce dominates a training profile, the collective itself is not always the only problem. Start by separating communication issues from workload imbalance.

Compare single-node and multi-node performance. If scaling drops sharply only after crossing server boundaries, the inter-node network is a likely bottleneck. You should also check whether every worker reaches the collective at roughly the same time. If one rank arrives much later, investigate data loading, model branches, hardware differences, and uneven batch processing.

Useful troubleshooting steps include:

• Measure collective bandwidth with tools such as nccl-tests.
• Compare intra-node and inter-node performance.
• Confirm NCCL is using the intended network interface.
• Check whether InfiniBand is active or communication has fallen back to sockets.
• Inspect the GPU and PCIe topology.
• Look for ranks that finish backward computation later than others.
• Measure how much communication overlaps with backpropagation.
• Test different DDP bucket sizes.
• Check for network contention from other workloads.
• Confirm all ranks call collectives in the same order.

Collective operations must be called consistently. If one process calls a different collective, uses a different tensor shape, or reaches operations in a different order, the program may hang or fail.

Can Distributed Training Avoid AllReduce?

Some systems reduce their dependence on global AllReduce, but they still need a way to coordinate workers. Gradient accumulation lowers the frequency of synchronization. Sharded strategies divide communication across reduce-scatter and all-gather operations. Decentralized training may replace global agreement with local communication between neighboring workers.

Gossip-based methods allow workers to exchange information gradually rather than synchronizing the full group at every step. These approaches can reduce the cost of strict global coordination, but they introduce other tradeoffs. Model replicas may temporarily disagree, convergence behavior may change, or additional training steps may be needed.

When a system claims to avoid AllReduce, it usually replaces global synchronization with another communication pattern rather than removing coordination completely.

Why AllReduce Appears So Often in Training Profiles

AllReduce is closely connected to the critical path of synchronous training.

The optimizer cannot safely update model parameters until the required gradients have been synchronized. Although frameworks can overlap some communication with backpropagation, any remaining work must finish before the next step can proceed. That makes exposed AllReduce time easy to see in performance traces.

A large AllReduce bar does not automatically mean the collective library is poorly optimized. It may indicate that the communication cannot be fully hidden, the workers are arriving at different times, or the cluster network cannot keep up with the available GPU compute.

In many large training systems, AllReduce is not an accidental overhead. It is the cost of keeping distributed model replicas in agreement.

Key Takeaways

AllReduce combines values from every participating process and returns the completed result to all of them. In synchronous data-parallel training, it is commonly used to keep gradients and model updates consistent across GPUs.

Ring AllReduce is widely used because it distributes communication evenly and uses bandwidth efficiently. Its main weakness is latency, since the number of communication stages grows with the number of participants.

Tree and recursive-doubling algorithms may perform better for smaller messages or larger process groups. Hierarchical approaches are useful when communication within a server is much faster than communication between servers.

PyTorch DDP uses gradient buckets and overlaps communication with backpropagation when possible. NCCL handles much of the lower-level algorithm selection and topology mapping on NVIDIA GPUs. MPI provides similar AllReduce operations for scientific and high-performance computing workloads.

When distributed training stops scaling, AllReduce is one of the first areas worth examining. The problem may come from network bandwidth, communication latency, poor topology, uneven workers, or communication that cannot be hidden behind useful computation.

The goal is not always to remove AllReduce. In most synchronous training systems, that is unrealistic. The goal is to understand where the communication time comes from and reduce the portion that prevents the GPUs from doing useful work.