What If Your LLM's Parallel Reasoning Threads Could Actually Talk to Each Other?

A new attention mechanism lets concurrent reasoning paths share insights mid-generation — and the model learns to do it without any human-written training data.

~6 min read · Rutgers University / Amazon AGI Labs · April 21, 2026 · Architecture

TL;DR LACE (Lattice Attention for Cross-thread Exploration) retrofits transformers with a lightweight side path that lets parallel reasoning threads communicate at the token level during inference. Unlike standard Best-of-N sampling — where threads independently fail in the same ways — LACE threads can flag dead ends, share breakthroughs, and elect the best solution on the fly. On challenging math benchmarks, this coordination beats independent parallel sampling by over 7 points, using less than 1% extra parameters.

Here's the dirty secret of test-time compute scaling: when you sample four reasoning paths from the same model on the same problem, they often fail in exactly the same way. They're not four independent attempts — they're four correlated guesses. The diversity you're paying for doesn't exist.

LACE, from researchers at Rutgers and Amazon AGI Labs, attacks this problem at the architectural level rather than the prompting level. The core observation is disarmingly simple: standard transformer attention is one-dimensional. It flows across time — token to token within a single thread — but never across threads. LACE adds a second dimension.

💡 The Core Idea Standard LLMs reason like K workers solving a puzzle in separate, soundproofed rooms. LACE tears down the walls. By extending causal attention into a 2D "lattice" — time on one axis, thread identity on the other — each reasoning path can observe the progress of its peers and adjust course accordingly.

How Lattice Attention Actually Works

The architecture is deliberately non-disruptive. Rather than replacing the standard causal attention mechanism, LACE grafts a lightweight side path onto it.

flowchart LR
    A[Input Tokens] --> B[Standard Causal Attention\nper thread]
    B --> C[SDPA Output\nprojects to low-dim]
    C --> D[3D RoPE\nencode token + thread position]
    D --> E[Cross-Thread Attention\nacross all N threads]
    E --> F[Gated Fusion\nlearned gate σ]
    B --> F
    F --> G[Output\nper thread]

The process: each thread's standard attention output (already rich with within-thread context) gets projected to a lower-dimensional space and fed into a cross-thread attention module. A learned gating mechanism then decides how much cross-thread information to blend back into each thread's representation. At inference time, this means each token generation step can incorporate signals from all other parallel threads.

Three design choices keep the overhead minimal. First, working on attention outputs rather than raw embeddings inherits the effective context length of standard attention for free. Second, lattice layers are inserted only in the middle-to-late layers of the base model (inspired by ControlNet), where cross-thread communication matters most for complex reasoning. Third, the gate allows the model to dial down cross-thread influence entirely when thread-independent processing is more appropriate. Total additional parameter count: under 1% of the original model.

The Data Problem — And a Clever Fix

A framework for cross-thread reasoning is useless without training data that exhibits cross-thread reasoning. That data doesn't exist naturally.

The authors' solution is a synthetic pipeline built on three properties. First, diversity: for each training problem, an iterative generation procedure explicitly bans previously seen solution approaches via a "Solutions Cache" in the prompt, forcing the model to explore genuinely different strategies. Second, cross-thread synergy: threads are formatted to include summaries and self-evaluation tags ([[best]], [[success]], [[fail]]), so the training signal explicitly rewards the model for reading its peers' conclusions and adjusting its own. Third, appropriate difficulty: only problems where the base model can produce both correct and incorrect solutions are included — problems that are too easy breed sameness; problems that are too hard yield no positive signal.

The resulting data format looks like:

Thread 1: [reasoning] <summary>...</summary> [[best]]
Thread 2: [reasoning] <summary>...</summary> [[success]]
Thread 3: [reasoning] <summary>...</summary> [[fail]]

An LLM-as-judge assigns these labels based on cross-thread comparison, creating supervision that is by definition cross-thread aware.

Training: Three Stages, One Novel RL Reward

After continuous pre-training to initialize the lattice layers, the model undergoes SFT with a "random thread shuffling" trick: reasoning steps from different threads are randomly permuted during training, forcing the model to rely on cross-thread cues rather than intra-thread order.

The most interesting piece is Lattice GRPO, their multi-thread extension of the GRPO reinforcement learning algorithm. Two reward signals are combined:

$$r^{(g)} = R_{\text{acc}}(y^{(g)}) + \lambda_{\text{div}} \cdot R_{\text{div}}(y^{(g)})$$

In plain English: each group of parallel completions receives a combined reward — one term for whether the model correctly identifies its best reasoning path, and one term for how different the threads' reasoning paths are from each other (measured by embedding cosine distance). A single shared advantage signal is then broadcast to all threads in a group, reinforcing collaborative behavior rather than individual thread performance.

The diversity reward acts as a training-time temperature: it prevents entropy collapse (mode collapse during RL), which the baseline model suffers after roughly 150 training steps. LACE with the diversity reward shows a rebound in entropy and a steady rise in thread diversity instead.

Results

The gain is largest for the 4B model on AIME 2024 — a full 10 percentage points over voting-based selection from independent threads. Crucially, LACE's learned self-selection (the model picking its own best thread via [[best]] tags) also substantially beats majority voting as a post-hoc selection strategy, confirming that real cross-thread communication — not just ensemble diversity — is driving the gains.

What Emerges Without Being Trained For

Perhaps the most striking finding is behavioral. When visualizing gate scores (how much cross-thread information each token draws on), a clear pattern emerges: cross-thread attention peaks at the onset of exploration steps and at self-assessment tags. The model has learned, without explicit supervision, to look around at peers precisely when it needs to: when starting a new reasoning branch and when reaching a verdict.

Even more striking is an emergent early-stopping behavior. In a worked example with four threads on an AIME problem, once Thread 1 arrives at the correct answer efficiently, the other threads detect this and simply annotate themselves as "follows a similar approach to the reference" and tag themselves [[success]] without redundantly completing their own derivations. Collaborative intelligence — learned from synthetic data generated entirely by a language model.

What It Costs

The latency hit (roughly 30–40% slower step time at 4 threads) is real and memory-bandwidth bound rather than compute bound. This is the practical cost of synchronizing information across threads during generation. Whether that tradeoff is worth it depends entirely on the task: for hard mathematical reasoning, the answer appears to be yes.

⚠️ Watch out for

• Latency overhead is ~30–40% at 4 threads and grows to ~55–96% at 128 threads — this is not free parallelism.
• Memory overhead for the 4B model is ~10–15%, which may matter for tight deployments.
• All experiments use Qwen3 (1.7B and 4B). The 8B results in the appendix are described as "preliminary," and behavior on non-math tasks remains limited to one agent benchmark.
• The synthetic training pipeline requires a powerful frontier model (Qwen3-235B-A22B is used here) to generate diverse, high-quality multi-thread data — not a trivial bootstrapping cost.

Why It Matters

The standard scaling playbook for inference-time compute — sample more, vote harder — has a ceiling. You can't escape correlated failures by adding more correlated samples. LACE is an early, concrete demonstration that cross-thread communication during generation can break that correlation in a principled way, without requiring external orchestration or a separate verifier model.

The approach is also modular: it's a post-training intervention, not a from-scratch architecture. That makes it potentially applicable to any existing transformer base model. And because the self-selection behavior is baked into generation rather than bolted on afterward, it adds no extra decoding steps — LACE at 4 threads delivers results competitive with judge-based selection methods while being notably faster end-to-end.

The paper frames this as turning "redundant samples into interactive exploration." That framing is right, and it points toward a broader principle: the value of parallel computation in reasoning may depend not just on how many paths you explore, but on whether those paths can inform each other.

Source: LACE: Lattice Attention for Cross-thread Exploration Authors: Yang Li, Zirui Zhang, Yang Liu, Chengzhi Mao Published: 2026-04-21 PDF: https://arxiv.org/pdf/2604.15529