The Price of Locality:
Why Forward-Forward Underperforms Backpropagation?

Geoffrey Hinton introduced the Forward-Forward Algorithm in 2022: it trains each layer through positive and negative forward passes with local objectives (original paper).

BP versus FFA

Backpropagation and Forward-Forward use the same forward computation but organize learning signals in fundamentally different ways. BP waits for the task loss, then transports one coordinated signal through the full chain rule. FFA attaches an independent contrastive objective to every layer and updates each block without gradients crossing layer boundaries.

One cat image, two FFA passes

This supervised CIFAR-10 example uses two forward passes: cat photo + “cat” label is positive, and the same cat photo + “truck” label is negative. Here “negative” means an incorrect image–label pair, not a photograph of a truck. The photos and three-layer flow are illustrative, not actual CIFAR-10 samples or measured scores.

Positive pass · correct label

+ CAT label

Same cat image + encoded “cat” label overlay → forward through the layers

L1: goodness ↑L2: goodness ↑L3: goodness ↑

This pass contributes a loss that pushes each layer's goodness above its threshold.

Negative pass · wrong label

+ TRUCK label

Same cat image + encoded “truck” label overlay → forward through the layers

Class cue only. A truck photo with the true label “truck” would be positive in its own example.

L1: goodness ↓L2: goodness ↓L3: goodness ↓

This pass contributes a loss that pushes each layer's goodness below its threshold.

Photos: cat by Halved sandwich and truck by Mattes (public domain).

FFA underperforms BP

The starting point of this work is empirical: FFA remains substantially less accurate than BP even when both are trained on the same fixed backbone and dataset. In the strictly local endpoint of our CNN12/CIFAR-10 intervention, the gap is 32.65 percentage points.

We answer this question by identifying two distinct but coupled failure modes: concurrent optimization on shifting representations and geometric representation collapse. The later experiments test how the gap changes with locality, model scale, and update geometry.

Two coupled failure modes

1. Learning from a moving representation

Each FFA block receives its inputs from the blocks before it. Because all blocks train at the same time, those inputs change whenever an earlier block updates. A downstream block can reduce its local loss for the current representation, only to receive a different representation at the next step. Thus, every block may make local progress without the network following one coordinated descent direction.

The multi-layer theorem captures this interference through a linearly decaying optimization term plus a coupling residual.

2. Geometric representation collapse

Representations become alike

One intuition is that each forward block can slightly reduce differences between samples' feature directions. Repeating such blocks can accumulate these small reductions, making initially distinct representations increasingly alike. Consequently, the local goodness gradients become increasingly redundant, and the network loses its ability to distinguish between classes.

Imagine comparing every pair of training images after each layer. Let $h_i^{(\ell)}$ be the representation (a.k.a. activation or feature) of sample $i$ after layer $\ell$, and let $\widetilde h_i^{(\ell)}=\operatorname{LN}(h_i^{(\ell)})$ be its normalized version. The representation kernel $\Sigma^{(\ell)}$ is the resulting $N\times N$ similarity table: entry $(i,j)$ compares images $i$ and $j$ at depth $\ell$. Using the representation matrix $H^{(\ell)}$ defined above (with the iteration index suppressed), Then $$[\Sigma^{(\ell)}]_{ij} =\frac{\langle\widetilde h_i^{(\ell)},\widetilde h_j^{(\ell)}\rangle}{d}, \qquad \Sigma^{(\ell)}\in\mathbb R^{N\times N}.$$ Intuitively, $\Sigma^{(\ell)}$ can evaluate how similar the network's representations are for different samples. In the extreme case, if all normalized representations point in the same direction, $\Sigma^{(\ell)}$ becomes the all-ones matrix $J_N$. The paper shows that this stronger contraction can happens when the specified kernel-product conditions hold. At that time, it becomes hard to distinguish between samples.

Why BP can preserve distinct learning signals

Imagine that cat, dog, and truck images have almost the same features at one layer. The network gives them similar predictions, although their correct labels call for different corrections.

FFA: Each layer forms a new signal from its local goodness objective. If those activations align, their corrections also point along nearly the same line; positive and negative examples may reverse the sign, but add no new direction. This is the analyzed collapsed-gradient case, not a property of every local goodness.

BP: The final loss uses each image's label. With three classes, the resulting output errors can span more than one direction even when the predictions are nearly identical. Backward Jacobians carry these errors to earlier layers; if they preserve those directions, BP can send varied corrections despite similar current features. A Jacobian can also suppress directions, so this is a possibility rather than a guarantee.

$\Sigma$ measures similarity among the features. To see whether the corrections have also become redundant, we use their second-moment matrix $\Gamma$: effective rank near one means that most correction energy lies along one line. Effective learning capacity (ELC) sums the effective rank above this one-direction baseline across layers. It measures the diversity of available learning signals, not a guarantee of task accuracy. In this paper, we show that ELC of BP can increase with depth, while ELC of FFA can remain near one, which also explains the accuracy gap.

Potential repairs: richer local targets and flatter updates

Give each block a supervised readout

One route gives every block more label information: attach a trainable readout to its output and train that block with cross-entropy against the true label or next token. This is Local Cross-Entropy (LCE). Like FFA, LCE stops gradients at block boundaries, but each block now receives a task-label signal through its own readout rather than a positive/negative goodness signal. In the language-model scale sweep, each block has its own vocabulary readout; the displayed parameter counts cover the shared Transformer backbone and exclude these extra heads.

Spread each weight update across more directions

Another route changes the update geometry directly. For an update $\Delta W=U\operatorname{diag}(s_i)V^\top$, interpolate its squared singular values toward their mean: $\lambda_i(p)^2=(1-p)s_i^2+p\overline{s^2}$. This keeps the update's Frobenius norm fixed while flattening its spectrum as $p$ increases. The mechanism test below applies this rule to convolutional updates, then measures whether the error-signal spectrum $\Gamma$ and task accuracy improve.

Explore the measured evidence

The controls below expose the measured sweeps behind the headline results. In the network diagrams, orange dots mark where each block receives local supervision, while the blue dot marks the task loss at the final layer. Orange segments show separate local gradients; the blue reference shows the end-to-end BP pathway.

Vision: controlled locality intervention

The task is ten-class CIFAR-10 image classification with a fixed 12-layer convolutional network. Grouped FFA trains on true-label positive and wrong-label negative overlays; the slider changes how many contiguous layers share each local goodness objective. Architecture and training protocol stay fixed, so this sweep isolates the block partition.

Language modeling: a scale sweep

The task is next-token prediction on an OpenWebText subset with 50 million training tokens and 0.5 million validation tokens. Decoder-only Transformers range from 2-layer Tiny to 12-layer XLarge, use GPT-2 tokenization and 256-token contexts, and are compared by validation perplexity under scale-dependent token budgets. The slider selects a model size; it does not change how a fixed model is partitioned.

Mechanism test: flattening the update spectrum

On CNN3/CIFAR-10, we apply the spectral interpolation described above to each convolutional update. The backbone, data, each method's objective and optimizer, and the training budget stay fixed across $p$.

What should better local learning preserve?

The diagnosis suggests that improving the scalar goodness alone is not enough. A scalable local method must optimize locally without losing the distinctions needed by the global task.

• Stabilize the target seen by each block. Limit upstream drift or explicitly track it during concurrent updates.
• Allow selective cross-block communication. Even a limited global correction can coordinate otherwise independent objectives.
• Preserve diverse learning directions. Monitor representation and error-signal effective rank, not only local loss.
• Evaluate the actual task. Local convergence is an optimization diagnostic, not a substitute for downstream accuracy or perplexity.

BibTeX

@inproceedings{wu2026price,
  title     = {The Price of Locality: Why Forward-Forward Underperforms Backpropagation?},
  author    = {Wu, Zhaoxian and Liu, Haichuan and Chen, Tianyi},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2609.33240}
}