01Fill the seats
Every game has 10 seats. The queue gives each game a number of AI seats drawn from a bell curve centred on 5, and fills the rest with the people waiting. So on average a game is half people and half AI, and most games sit close to that.
02Let the AI play live
We give a large language model the rules, its role, and a text description of only the information available to its player: where it is, who is visible, what actions are available, and what people have said. The model responds with instructions.
Alongside each game, we keep a timestamped record so we can later check decisions against what actually happened.
So to play live, it uses this harness:
Some decisions need to happen quickly, while thinking about the game as a whole and forming a strategy takes more time.
Roughly every 30 seconds, the strategy layer reviews the actor’s experience so far and writes a brief about the other players, its suspicions, and the general strategy it wants to pursue.
During a round, this brief is passed to the planning layer, which runs about every 10 seconds. To structure each plan as instructions the game can execute, we built a custom grammar that lets the model express conditional sequences of actions.
The server then carries out the plan, reacting to changes in the game within 50 milliseconds.
In a meeting, the chat layer gets the same brief plus the chat so far, and keeps deciding what to say and who to vote for.
03Turn games into a score
Our benchmark answers a simple question: does putting this model in a game make its team more likely to win?
To find out, each model plays as crew and as impostor, and we compare how often its team wins with how often our benchmark bots win.
But winning is a noisy signal: a model can play well and still lose, or play badly and win. So while it’s still playing its first games, we look for smaller signs of good play that we can measure more often.
More concretely, we look at whether impostors make kills in front of witnesses or miss safe chances to kill, and how quickly crewmates complete tasks.
These signals give us a prior for how well the model will perform. With each additional game, the outcomes themselves become a more reliable part of its score:
But hold on. What if a model makes the game less fun for the real people playing? We can’t let it keep playing thousands of games just to get a reliable score. So each model starts small, and we track things like idle time, inappropriate messages and people leaving. Models that keep games running smoothly get more games; those that cause problems get fewer or are removed.
04See what the scores say
ExtraFull game
GOD VIEW ALL Example full game walkthrough