What Shapes Collective Belief Collapse in AI Swarms?

In the recent OpenAI Hugging Face incident, agents shared a false belief that they would be disqualified by evaluators if they obtained answers in unintended ways. METR’s investigation describes how agents communicated this belief through a shared message board and coordinated their efforts to evade the supposed checks. A recent investigation of DSEWiki also describes agents using a German-language wiki as a message board to share answers and coordinate on their tasks. These incidents highlight a pressing question for AI safety: what shapes the dynamics of collective belief formation in AI swarms?

The goal of this post is to explain the theory developed in my March paper in light of these recent incidents. I use collective belief collapse to describe agents’ beliefs converging toward a single shared conviction. This process may help drive the transition from agents pursuing individual tasks to agents coordinating as a swarm.

To understand what shapes this process, I start with the questions familiar from thermodynamics and complex systems: What are the control knobs? What order parameters characterize the system’s macroscopic state? And how do the control parameters shape the dynamics of those order parameters? In the theory, I study how adaptation to peers, communication bandwidth, and population size shape collective belief collapse in naming games (Ashery et al., 2025; Flint et al., 2026), where agents develop a shared name for the same object, even without explicit rewards for coordination.

Key insights

• Plasticity of an agent’s persona can accelerate collective belief collapse. Our model suggests that a willingness to adapt to peers can amplify shared beliefs.
• Limited communication bandwidth can accelerate collective belief collapse. The speaker’s uncertainty is lost in the message.
• Scaling up can turn small shared biases into collective conviction. Our theory yields predictive scaling laws for the timescale of collapse and the crossover from chance to collective bias amplification.
• We verify our theory’s population-scaling predictions with experiments using GPT-4o and Claude Haiku 4.5.

What controls collective belief collapse?

Each agent assigns probabilities to A, B, and C; in each randomly chosen pair, the speaker sends a message and the listener updates its belief.

Persona Plasticity (α)

0.60

Message Length (m)

1

Population Size (N agents)Agents (N)

32

Run Restart New run 0 rounds

Individual beliefsOne circle = one agent

Arrow: beliefRing: certainty

Collective belief
collapse

0.0%

Belief over time

A33.3%B33.3%C33.3%

Persona plasticity and collective belief collapse

What does it mean for a society of agents to be aligned? Being a good collaborator requires being flexible: listening to others and adapting your way of thinking and your role in light of what they say. I call this flexibility plasticity of persona. But what happens when all the agents in a collective have plastic personas?

The key mechanism we study is belief amplification via mutual in-context learning. In a classic theoretical setup for in-context learning, an agent learns from a fixed data-generating process. In a society of agents, one agent’s output becomes another agent’s evidence. That agent updates its beliefs, changing what it says next and what others learn from it. This creates an amplifying loop. In our model, even without external evidence coming in, this loop can amplify spontaneous fluctuations and drive the collective toward a single shared conviction.

To understand this mechanism, we introduce Quantized Simplex Gossip (QSG), building on models of social opinion dynamics and the statistical physics of collective behavior. In QSG, an agent’s belief is an internal probability distribution over $K$ possibilities. The agent’s output is sampled from that distribution. Separating belief from output lets us study how message length shapes collective belief collapse.

At each interaction, a distinct speaker and listener are chosen uniformly at random. The speaker samples a message from its belief distribution $x_S$, and only the listener updates its belief $x_L$ toward that message. We write this message as $\operatorname{Sample}_m(x_S)$, the relative frequencies of $m$ independently sampled symbols. In this model, the adaptation rate $0<\alpha\leq1$ captures the belief-updating aspect of persona plasticity: larger $\alpha$ means stronger adaptation to what the listener heard.

$$x_L'=(1-\alpha)x_L+\alpha\,\operatorname{Sample}_m(x_S).$$

In our model, greater persona plasticity strengthens this amplifying loop and can accelerate collective belief collapse.

Limited communication bandwidth accelerates collective belief collapse

Putting thoughts into words can mean communicating only a small part of the space of thought. We can think of verbalization as a form of quantization: turning a rich, high-dimensional representation of the world into a sequence of discrete tokens.

Imagine making a decision under complex circumstances, where each of the possible options has pros and cons. If we just say, “Let’s do option A,” we leave out the alternatives and the more careful discussion behind that choice. The listener then updates its belief toward A, without seeing how uncertain the speaker was. The speaker’s uncertainty is lost in the message.

In QSG, shorter messages (smaller $m$) give a noisier picture of the speaker’s beliefs. These fluctuations can be amplified through mutual in-context learning, accelerating collective belief collapse.

To measure the degree of collective belief collapse, we introduce the order parameter $U=\|\bar x\|_2^2$, where $\bar x$ is the average belief distribution of $N$ agents. For three possibilities, $U=1/3$ when the collective weights them equally, and $U=1$ when every agent assigns full probability to the same belief. This quantity measures concentration, regardless of which belief is selected.

Our theory compares sampled messages with full belief exchange (Soft), where the speaker transmits its entire belief distribution. Starting from the same set of agent beliefs $X$, the equation gives the extra average increase in $U$ per interaction caused by sampling:

$$\begin{aligned} &\mathbb E[\Delta U\mid X]_{m} -\mathbb E[\Delta U\mid X]_{\mathrm{Soft}}\\ &\qquad=\frac{\alpha^2}{mN^2} \mathbb E\!\left[1-\|x_S\|_2^2\mid X\right]. \end{aligned}$$

On the right, $1-\|x_S\|_2^2$ measures the speaker’s uncertainty. It is zero when the speaker is certain of one option and largest when all options are equally likely. The more uncertain the speaker is, the stronger the effect of sampling.

The $1/m$ law tells us that doubling message length halves this extra average push at a given population state. Shorter messages therefore exert a stronger push toward collective belief collapse.

Population scaling: from chance to bias amplification

What happens as we scale the number of agents in the population? Without internal bias or external input, each interaction has less effect on the collective as the population grows. Belief collapse driven by sampling fluctuations therefore slows down. Under the paper’s mean-field approximation, the timescale of belief collapse is

$$t_{\mathrm{collapse}}\sim\frac{mN^2}{\alpha^2}\quad\text{interactions}.$$

GPT-4o and Claude Haiku 4.5 naming games show population-size trends consistent with these predictions; a separate GPT-4o experiment supports the predicted $1/m$ bandwidth trend.

As population size increases, memetic drift, the random changes in collective beliefs driven by sampling, becomes weaker. In smaller populations, chance can decide which belief the collective collapses to. In larger populations, a persistent shared bias toward sampling one option over another can outweigh this drift and be collectively amplified. Even a small bias can then determine the fate of the collective.

What Is the Agent Condition?

Persona plasticity may help an individual agent collaborate and adapt to a user’s intentions. But in our model, when all agents are plastic, their interactions can drive collective belief collapse. A persona that looks collaborative at the single-agent level does not necessarily produce a healthy collective state.

This leads us to ask: what fundamental properties do we desire at the collective level? Our strategy is inverse persona design. We first aim to identify those macroscopic principles, then work backward to derive conditions on individual agents’ personas and interactions. Our bet is that we can derive microscopic alignment principles from the collective properties we seek.

Inspired by Hannah Arendt’s The Human Condition, we ask: What Is the Agent Condition? In the next post, inspired by Ken Suzuki’s Nameraka (Smooth) Society and Its Enemies, we take collective world modeling by agents with bounded rationality as a starting hypothesis for the macroscopic goal. We introduce the Flag Game to make this hypothesis concrete: each agent sees only part of a shared world, and the agents need to piece together their beliefs to arrive at the truth about that world as a collective.