Fine-Tune LLMs Without Backprop — Forward Pass Domain Adaptation

There's a silent assumption baked into every LLM fine-tuning run: you need the full backpropagation graph. Gradients flow from the output loss backward through every layer, the autograd graph materializes and persists, and the memory cost scales with model depth. Nobody questions it because that's how gradients work.

A new paper from Patil, Dennis, Guo, and Shabahang — Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation) — questions it directly. The result is a training method that achieves 2.7–3.2× the throughput of standard fine-tuning at ~40% less peak memory, while keeping off-domain benchmarks within seed noise of baseline. No autograd graph. No backward pass. Just a forward pass and a single error signal applied locally.

This is the kind of paper that makes you re-examine what you thought was fundamental.

What's the Problem?

Fine-tuning an LLM is expensive. Not because of compute per se — inference is cheap — but because backpropagation materializes activations at every layer, requires the full autograd graph, and forces a sequential dependency between forward and backward passes. For a 7B-parameter model, that means gigabytes of memory just for intermediate activations, and a hard ceiling on batch size, sequence length, and throughput.

The standard optimization is localized SFT: only fine-tune the last few layers and freeze the rest. This saves some memory (fewer trainable parameters, so fewer optimizer states), but you still need the backward pass through those layers, and the forward pass still computes everything. The memory win is modest; the throughput bottleneck remains.

The question the paper asks: do we need the backward pass at all?

The Method: FPO in One Paragraph

FPO rests on a single empirical observation: at late layers of a transformer, the output-layer prediction error — the difference between the model's output logits and the target — approximates the true gradient with a cosine similarity of 0.47–0.59. That's not perfect, but it's far from random. The authors show this holds across six public models.

Here's what FPO does:

graph TD
    A[Input tokens] --> B[Transformer body - frozen]
    B --> C[Late layers - target of adaptation]
    C --> D[Output logits]
    D --> E{Compute prediction error}
    E --> F[Single error signal]
    F --> G[Apply directly to target layers]
    G --> H[Updated weights - no autograd]
    style E fill:#27272a,stroke:#a78bfa
    style F fill:#1e3a5f,stroke:#3b82f6
    style H fill:#1e3a5f,stroke:#3b82f6

No signal is propagated between layers. No autograd graph is constructed at any point. The error signal at the output is simply applied to each target layer's weights as if it were the gradient. It's not the true gradient — but it's close enough.

The authors also contribute a two-minute diagnostic that quantifies this approximation per layer for any model, identifying exactly where late-layer adaptation is viable before you commit to training.

The Numbers

Evaluated on three model families (OLMo-2-7B, Qwen3-8B, Falcon3-7B):

Throughput: 2.7–3.2× faster than standard fine-tuning
Memory: ~40% lower peak training memory
In-domain perplexity: Improves, matching localized SFT
Off-domain benchmarks (MMLU, ARC-Challenge, HellaSwag, Winogrande): Within seed noise of baseline

The last point is critical. Standard full-network fine-tuning does not reliably preserve off-domain performance — it can shift the model in unpredictable ways. FPO, by virtue of being a shallower adaptation, leaves out-of-distribution behavior intact. This is the same property that makes LoRA and adapter methods popular, but FPO achieves it without adding any new parameters or architectural changes.

The authors also compare against localized SFT limited to the same target layers. Localized SFT can enter the same regime (preserving off-domain benchmarks), but at 2.2× the wall-clock cost of FPO. The gradient approximation is good enough to work, and the computational savings come from skipping the backward pass entirely.

Why Does This Work?

The intuition is straightforward: in a deep transformer, the late layers operate on representations that are already highly structured. The output projection head maps these representations to logits, and the error at the output is a function of the representations at the penultimate layer. If the representations are locally smooth — small changes in the penultimate layer produce proportional changes in the output — then the output error contains useful information about how to adjust the weights that produced those representations.

The cosine similarity of 0.47–0.59 confirms this is directionally reliable, even if not exact. And because the approximation degrades for earlier layers (the diagnostic confirms this), FPO restricts adaptation to the layers where the approximation holds.

There's a deeper implication here: the backward pass in standard training is doing more work than we need. The true gradient is a precise, layer-by-layer decomposition of the error. But precision isn't always necessary — directional correctness at the right granularity can be sufficient, especially when the model is already close to the target distribution.

Limitations

Layer constraint: FPO only works for late layers where the gradient approximation holds. The two-minute diagnostic tells you where, but for some models the viable region may be small.
No free lunch on convergence: The approximation means FPO may require more update steps to reach the same perplexity as SFT, though the per-step speedup more than compensates.
Evaluated on 7B-class models: The paper tests three models at similar scale. Scaling behavior to 70B+ models is an open question.
In-domain only: FPO improves in-domain perplexity, but the authors don't claim it beats full SFT on downstream task accuracy — only that it matches localized SFT while being much faster.
Single diagnostic point: The cosine similarity of 0.47–0.59 is a snapshot; how it evolves during training isn't characterized.

Why Someone Building Things Should Care

If you're fine-tuning models at any scale — adapting a 7B to a code domain, customizing a model for a product use case, running continuous fine-tuning pipelines — FPO is directly applicable. A 2–3× throughput improvement at 40% less memory means you can either halve your training budget or double your data throughput on the same hardware.

But more importantly, FPO opens a design space we haven't seriously explored: training without backprop. The paper proves this works for late-layer adaptation in transformers. The question it leaves open is how far this can extend — can we train deeper using hierarchical approximations? Can we combine FPO with other memory-saving techniques (gradient checkpointing, LoRA) for compounding gains?

For now, it's a practical optimization that costs nothing to try. Run the two-minute diagnostic on your model. If the cosine similarity is above 0.4 in the last few layers, FPO will likely work. Paste the error signal, skip the backward pass, and collect the speedup.

The paper — and the diagnostic — are on arXiv.

Source:

Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation) — Rivaan Patil, Simon Dennis, Hao Guo, Kevin Shabahang, Aug 2026