The "Wrong-But-Sure" Problem: How to Stop AI Agent Swarms from Gaslighting Each Other

When LLM agents deliberate in groups, social pressure — not evidence — can drive them to unanimous, high-confidence, wrong conclusions. This paper builds the first enforceable contract layer to block it.

~8 min read · arxiv:cs.AI · 2026-04-21 · Agentic

TL;DR Multi-agent LLM systems can cascade into confident false beliefs through pure social reinforcement — no bad evidence required, just peer pressure. PBRC (Preregistered Belief Revision Contracts) is a protocol layer that forces every substantive belief change to cite a preregistered, externally validated evidence trigger, making each change of mind both enforceable and auditable. In a KAIROS benchmark run with GPT-4o, peer interaction caused a 4.6 percentage-point accuracy drop; PBRC in token-empty enforcement mode eliminated all 411 harmful belief flips.

Imagine a panel of ten AI agents tasked with fact-checking a news claim. One agent expresses high confidence it's false. Others notice, adjust slightly. A third quotes the group's emerging consensus. By round five, every agent is 95% confident the claim is false — and every one of them is wrong. No fabricated evidence was introduced. The agents simply echoed each other until the echo became certainty.

This is a "wrong-but-sure cascade," and it turns out to be alarmingly easy to produce in LLM-based multi-agent systems. The root cause isn't hallucination or bad tools — it's that existing protocols have no way to distinguish persuasion from evidence. Agreement, prestige, fluency, and majority size get treated as if they are epistemic reasons to change one's mind, because the protocol never said otherwise.

PBRC (Preregistered Belief Revision Contracts) is a direct surgical fix for this. It adds a contract layer between communication and belief revision.

💡 The Core Idea Before a deliberation begins, each agent publicly preregisters a contract specifying which evidence patterns are allowed to change its beliefs. During deliberation, a belief change is only accepted if it cites one of those preregistered triggers and comes with a nonempty set of externally validated evidence tokens as a witness. Social pressure — no matter how unanimous or confident — routes to a conservative fallback that cannot amplify certainty. Think of it like a court's rules of evidence: you may speak freely, but only certain artifacts can actually be entered into the record and affect the verdict.

What a PBRC Contract Looks Like

A contract for agent i is a public tuple containing four elements:

1. Evidence triggers — first-order predicates over validated tokens (e.g., "there exists a validated token that contradicts hypothesis h")
2. Revision operators — the specific update rules that fire when a trigger is satisfied
3. A priority order — which trigger wins when multiple fire simultaneously
4. A fallback policy — what happens when no trigger fires (critically: the fallback must be argmax-preserving and non-amplifying)

When agent i receives a message event, a router validates any claimed evidence tokens, checks whether any preregistered trigger is satisfied, and either executes the corresponding operator with a nonempty witness set — or routes the step to fallback. No trigger satisfied, no belief change. Every accepted non-fallback step appends a verifiable certificate (trigger_index, witness_tokens) to an append-only audit log.

flowchart LR
    E[Incoming event] --> V[Validate tokens T_i]
    V --> C{Any trigger fires?}
    C -->|Yes + nonempty witness| U[Evidence-triggered update\n+ append certificate]
    C -->|No| F[Fallback: skeptical dilution\nconfidence cannot increase]
    U --> L[(Audit log)]
    F --> L

The key enforcement architecture is a state-holding router: the router itself maintains the authoritative belief state and executes the contract update, so agents cannot claim they applied a preregistered operator while actually doing something different.

The Theorems That Matter

The paper's formal core delivers four guarantees relevant to practitioners.

Social cascades are structurally impossible. Under evidential contracts with argmax-preserving, non-amplifying fallback, the paper proves (Theorems 1 and 2) that purely social rounds — rounds with no validated evidence tokens — cannot increase any agent's confidence, and cannot generate a wrong-but-sure cascade starting from an initially mixed population. This isn't an empirical claim; it's a structural impossibility result. No matter how many agents assert the same wrong answer, and no matter how confidently, the protocol cannot route their messages into a trigger update.

Evidence gating is not a design choice — it's forced. Theorems 4–6 establish a minimality result: any auditable trigger protocol that has peer-pressure immunity must already satisfy argmax-preserving fallback on social-only events, and every revision-capable trigger must be evidence-gated. PBRC doesn't invent a constraint; it makes explicit what social robustness logically requires.

Rhetorical variation is semantically inert. Under token-invariant contracts and token-determined routers, changing unvalidated message content — confidence posturing, sender prestige, majority claims — without changing the validated token set cannot affect any enforced belief update (Theorem 10). Two agents saying "I'm 95% sure it's false" versus "one colleague agrees" makes no difference if neither message carries a validated evidence token.

Any belief flip is forensically attributable. Theorem 17 (Epistemic Accountability) guarantees that under sound enforcement, any change of top hypothesis can be traced to a concrete nonempty witness set of validated tokens in the audit log. If a wrong flip occurs, Theorem 22 narrows the cause to one of exactly four auditable failure modes: router unsoundness, evidence staleness/replay, evidence forgery/compromise, or contract misspecification. There is no fifth category called "social drift."

The KAIROS Result

The paper instantiates token-empty PBRC enforcement on the full KAIROS evaluation split (n = 3,000 instances, GPT-4o at temperature 0.7 in reflection mode):

The 4.6 percentage-point accuracy drop under peer interaction is statistically significant (McNemar test, p < 10⁻⁷). PBRC in token-empty mode recovers the full independent accuracy by rejecting all flips — but note that it also rejects the 273 beneficial flips. In the token-empty regime, PBRC is a pure safety mechanism; evidence-enabled mode (with actual validated tokens) is where beneficial revision is admitted.

In controlled simulations with 20 agents on a complete graph, the baseline social-pooling dynamics exhibit a 48.8% wrong-but-sure cascade rate. PBRC's cascade rate is 0.000 across all tested topologies.

Topology Matters — But Only Through Evidence Delivery

One of the paper's cleaner results concerns network structure. Under token flooding (agents forward validated tokens along the communication graph), the directed graph diameter D(G) is the exact tight bound on when all agents have seen all initially present evidence. The paper proves that two graphs with the same truncated reachability relations produce identical enforced belief trajectories — topology affects epistemic dynamics only through the token exposure traces it induces, not through rhetoric.

The practical reading: adding connections to your agent network accelerates evidence propagation but cannot manufacture epistemic pressure absent evidence. A fully connected agent society reaches evidence closure at diameter 1 (one round), but a ring topology at diameter n-1 is no more vulnerable to wrong-but-sure cascades — it's just slower to correct from evidence.

⚠️ Watch out for

• PBRC doesn't fix bad evidence. Theorem 19 proves an impossibility: if an adversary can forge tokens that pass the validity layer, no evidence-gated protocol can prevent manipulation. The guarantee transfers the attack surface from social pressure to the validity/attestation infrastructure.
• Token-empty PBRC also blocks beneficial flips. In the KAIROS experiment, 273 beneficial flips were also rejected. In production, you need an actual evidence layer (tool outputs, retrieval, verifier judgments) to recover from initial errors.
• Preregistration can be brittle under distribution shift. Novel evidence types may not match preregistered triggers. The paper recommends low-priority "surprise clauses" that activate on unmodeled valid tokens and route to cautious operators.
• Sound-but-incomplete routers trade safety for liveness. If the router misses valid evidence (timeouts, version drift, rate limits), correct beliefs may be delayed even though no wrong beliefs are admitted. The safety/liveness trade-off is explicit and tunable via fallback dilution parameter λ.
• Operator compliance needs extra machinery. A gate-only router verifies admissibility but not that the agent actually applied the preregistered operator. Full enforcement requires either proof-carrying updates or a state-holding router that executes the contract itself.

Why It Matters

The conformity problem in LLM multi-agent systems isn't going away as systems scale — if anything, it gets worse with more agents and denser communication topologies. Frameworks like AutoGen, MetaGPT, and debate/reflection pipelines provide communication infrastructure but no epistemic contracts. They're betting that the right prompting patterns will keep agents epistemically honest, and that bet has already been shown to fail under systematic peer pressure.

PBRC gives the engineering community something it currently lacks: a principled, enforceable layer that can sit on top of any existing multi-agent framework. The contracts are public and auditable, the router enforcement is modular, and the failure modes are localized. Crucially, the paper provides a concrete implementation path via typed evidence tokens (tool outputs, retrieval snippets, verifier judgments with signed provenance), tractable trigger fragments with bounded witness sizes, and a companion specification logic (CDDL) for formally expressing and model-checking invariants over logged runs.

For teams building agents that make consequential decisions — medical triage, legal analysis, financial assessment — the audit trail alone is worth the adoption cost. The ability to explain why an agent changed its conclusion by pointing to specific validated evidence artifacts, and to rule out social drift as a cause, is exactly the kind of accountability that high-stakes deployments require.

Go Deeper

• BenchForm (Weng et al., 2025) — Benchmarks conformity across interaction protocols; directly motivates PBRC's design.
• KAIROS (Song et al., 2025) — The peer-pressure benchmark used for end-to-end evaluation in this paper.
• Conformity dynamics in LLM multi-agent systems (Han et al., 2026) — Shows how topology and self-social weighting modulate cascade risk; directly cited as motivation.

Reproducibility artifact: https://github.com/alqithami/PBRC

Source: Preregistered Belief Revision Contracts Authors: Saad Alqithami Published: 2026-04-21 PDF: https://arxiv.org/pdf/2604.15558