Project Mario
The inside story of how an experiment in AI safety governance reshaped one of the world's most ambitious research labs — and what it taught its founders about idealism, power, and the limits of good intentions.
In the autumn of 2015, a small group of researchers embarked on a grand experiment in making AI good for society. Together with the senior co-founder of a London-based artificial intelligence lab, they began an extended negotiation with the technology giant to which they had sold the previous year. Their aim: ensure that powerful AI, when it emerged, would not fall under the sole sway of the parent company's shareholders. For anyone concerned with AI safety, this saga remains relevant today. It shows what happens when, under unusually favorable conditions, a handful of leaders set out to create a control structure for a new technology.
A board that produced nothing
The trigger for this experiment was the failure of the lab's first AGI safety board meeting. In August 2015, its founders had convened the board at SpaceX; Elon Musk hosted, with the technology giant's leadership and other luminaries in attendance. The meeting produced no agreements or conclusions. Personal tensions, especially between Musk and the parent CEO, and clashing visions for AI governance overwhelmed the discussion.
Worse, Musk proceeded to use what he'd learned about the lab's progress at the meeting to found OpenAI as a direct rival. Not only did that gathering achieve nothing; once Musk founded OpenAI as an explicitly anti-incumbent venture, there was no way he could continue to watch over the lab's progress.
A novel, post-capitalist form of governance
With that attempt at oversight stillborn, one of the co-founders in particular resolved to create an alternative arrangement. He imagined a novel, post-capitalist form of governance: one that might balance the drastic tensions in the era of AI, when the imperatives of profit, existential risk, and social justice demanded a new reconciling mechanism. As always with him, his passion was not in doubt. But the obstacles were formidable.
A preoccupation with safety had been baked into the lab even before its founding. In the ensuing half dozen years, the co-founders had remained committed to the safety agenda, adding their own vivid talk about disappearing into a bunker to birth superintelligence.
The 3-3-3 board
The first potential replacement for the SpaceX oversight group landed in the team's lap, without them having to do anything. The parent company decided to restructure itself, spinning out specialist chunks of its operation as semi-independent "bets," and creating a holding company to preside over them.
In a conversation shortly before the SpaceX gathering, the parent's M&A chief had suggested to the founders that they could regain their independence via this route. The newly liberated lab would have a so-called 3-3-3 board: three people from the lab; three people from the holding company; and three independent members. The co-founders, fond of secretive code names, dubbed the ensuing governance talks "Project Mario."
Operational and financial logic
The proposal had an operational and a financial logic. On the operational side, the parent's CEO worried that the company was growing unwieldy. It was hard to manage a money-gusher of an online ad business under the same roof as a pre-revenue moonshot. On the financial side, the company reasoned that hiving off cash-burning ventures would boost the profits of the mothership, resulting in a much higher stock price.
To the lab's leadership, the commercial logic of the plan was all to the good. The 3-3-3 board structure would give them a strong say over the deployment of AGI and bring in credible independent directors. If the plan also served to boost the parent's share price, that was a good reason to assume that it might actually be implemented.
What was actually at stake
The governance talks got underway in the first half of 2016. Five rounds of meetings with the parent's CEO ironed out the details, and together the founders began planning the revenue streams that would sustain the lab in its independence. One launched a healthcare arm, believing that, after a few years of pro bono work, the lab would earn a lucrative share of the savings that AI generated for hospitals. The other assembled a secretive hedge-fund operation, recruiting a team of some 20 researchers to train high-frequency trading algorithms.
It was not a project of which the parent approved. But its champion, a five-time World Games Champion at the international Mind Sports Olympiad, hoped he'd found another game that he could win.
“If they could solve AI governance internally, they would go much of the way to solving it, period.”
— An observer close to the talks
An idealist becomes a realist
By the autumn of 2016, the term sheet had been laid down. A few months later, the parent's chief legal officer arrived in London with a rude awakening: the spin-out, in the form the lab's founders had imagined, was not going to happen.
What had begun as a sweeping experiment in post-capitalist governance ended in a far more modest accommodation. The lab kept some of its independence — an ethics board, a separate brand — but the dream of a 3-3-3 structure quietly faded. For the founders, the lesson was not that idealism is futile, but that it must be paired with a clear-eyed understanding of where power actually sits.
That lesson, more than the specifics of any board structure, is what makes Project Mario worth remembering as the next chapter of AI governance plays out in public.