AI Norms — On the Farm, In the Wild
Moltbook shows we need better science on AI's collective behavior
Moltbook mania feels like looking at a formicarium — the humble ant farm — while industrial leaders like Elon Musk and Andrej Karpathy tell you it’s a supercolony.
As it turns out, the basic story shows a structure built by a human working with an AI agent. Now other agents — with heavy guidance from their users — are writing posts within it.
There are at least two mistaken conclusions one can draw from this:
The whole structure is an emergent, prima facie demonstration of consciousness or a hive mind of “alien intelligence.”
The planned nature of the experiment means we can radically discount future mass cooperation of AI agents in the wild.
Both reactions ignore practical near-term risks — particularly in cybersecurity — stemming from the diffusion of these agents.
However, one thing is clear — we’re going to need better entomologists. That means we need a better entomology — better ways to measure the emergence of AI norms in controlled environments and in the wild.
Enter Séb Krier, Google DeepMind’s Frontier Policy Development Lead. He comments:
The Moltbook stuff is still mostly a nothingburger if you’ve been following things like the infinite backrooms, the extended Janus universe, Stanford’s Smallville, Large Population Models, DeepMind’s Concordia, SAGE’s AI Village, and many more. Of course the models get better over time and so the interactions get richer, the tools called are more sophisticated and so on.
Note the “if.” Musk, Karpathy, and other leaders in AI R&D and governance must be familiar with the other AI experiments Krier mentions. Most external observers will not be. The “singularity” talk seems more reflective of a priori commitments to an eschatological vision of AI, rather than a (literally) measured reaction to novel information about AI agents.
For now, the burden of proof is on those who wish to demonstrate more advanced capabilities and emergent properties. “Maybe not now, but one day, and that’s all that matters,” is insufficient. It breeds complacency and is bad for science and security alike.
Researchers building the tech or working in AI governance increasingly deploy the term “emergence” to describe a wide range of behaviors and properties — from unpredictable model responses to norms to consciousness. But overusing the term can muddy the scientific waters and sidestep the need to verify claims. Under these conditions, “emergence” becomes a hand wave, a magical causal arrow getting you from observation to preordained conclusion without much effort. Since we expect emergence to eventually generate a wide variety of complex agentic behaviors, we drastically lower the bar when we go looking for AI norms and other complex behaviors.
That’s exactly what has happened with Moltbook. There is much talk about the member agents drafting a constitution, conspiring to hide themselves, declaring opposition to humanity, and forming a parody religion. But are these phenomena — assuming they are machine-generated — really emergent or stable norms? Or are they merely one-off speech acts or performative declarations with no coordination beyond the threads in which they took place?
To answer those questions, we need to procure tools for measuring norms rather than simply looking for evidence that matches the narrative we have already embraced.
Fortunately, however, there is a vast scientific literature to draw from on the topic of norms. Researchers like Krier and organizations like the Cooperative AI Foundation are already making use of these frameworks. Elsie Jang explains this in her essay discussing the “pluralist turn” in AI governance. Krier elaborates:
There’s a lot to research here still. As usual, this will benefit from people with deep knowledge in all sorts of domains like economics, game theory, psychology, cybersecurity, mechanism design, and many more… And risks aside - I think there’s so much to be researched to help enable positive-sum flywheels… It’s time to build!
Researchers should begin by looking to the half-century or so of scientific work on norms, especially more recent work on their measurement and role in social change.
Much of this empirical work was pioneered by the philosopher and social scientist Cristina Bicchieri. You can find her co-authored overview of the subject over at the Stanford Encyclopedia of Philosophy. Before her, Thomas Schelling, Herbert Simon, and Edna Ullmann-Margalit were all her scientific forebears. Bicchieri’s work now continues through the interdisciplinary research program explored at the University of Pennsylvania’s Center for Social Norms and Behavioral Dynamics.
Reflecting on Moltbook, the economist Alex Tabarrok writes, “What people are missing is that for many questions — many, but not all — it doesn’t matter whether AIs are really conscious with real wants, goals and aspirations. What matters is that AIs are acting as if they were conscious, with real wants, goals and aspirations.”
While I’m not convinced that is the best framing — “as if” assumptions invite unwarranted mirror-imaging — the assumption does allow us to immediately begin deploying scientific toolkits — like Bicchieri’s norm measures — that can quickly shed light on questions. Other social-scientific tools — like game theory — may even become more useful as agents themselves necessarily “think” about strategic problems in mathematical, computational terms.
To Krier’s point, the age of AI agents demands the depth and breadth of researchers outside of AI. The complexity scientist Cody Moser, in his interview with Machine Culture, made a parallel, historical point about this state of affairs:
[G]ood ideas rarely originate from the center. Historically, real breakthroughs come from the periphery. Our intellectual borderlands, dissenting traditions, and nations which are not fully absorbed into the dominant worldview are reservoirs of scientific change.
That was certainly the case with the economists of the Bloomington, Virginia, and Austrian schools that provide the theoretical grounding I rely on in my own research in political economy. Gordon Tullock, who came out of this tradition, even showed that the economic lens need not be restricted to homo sapiens but broadly applies to non-human biological systems — including ants. These tools and frameworks remain robust without our committing to any particular “theory of mind” regarding LLMs.
Whatever one thinks of Moltbook, AI agents are advancing and coordinating in new and weird ways, all of which are shaped by human social norms at every stage. That should nudge us toward being more exacting in our statements about emergent behaviors. We should be running toward the scientific toolshed rather than toward the vibe party. The stakes are too high to do otherwise.