Who Actually Governs AI: Part 5 - Dissent

In May 2024, Daniel Kokotajlo sat down to sign OpenAI's standard exit paperwork and refused. The document asked him to promise never to criticize the company, for the rest of his life, or forfeit roughly $2 million in equity he had already earned over four years there.

He wasn't leaving over money. He was leaving because he no longer trusted OpenAI to handle increasingly powerful AI responsibly, and signing away his voice on the way out would have erased that judgment from the record along with his severance.

What happened next, the apology, the partial reversal, and the much longer list of resignations, Senate testimony, and public warnings that followed at OpenAI, Google DeepMind, and Anthropic, traces one question through the entire AI industry: inside the companies building the most powerful technology in a generation, who actually has the power to say no, and what does it cost them to try.

The Contract Nobody Was Supposed to Read

Kokotajlo worked in OpenAI's governance division from 2022 to 2024. When he decided to leave, he found himself facing a document that was less a standard NDA than a lifetime gag order with a price tag attached: sign away your right to ever criticize the company, or lose the vested equity you already own.

He refused. He went public instead, and journalist Kelsey Piper at Vox obtained internal documents showing OpenAI had used this exit structure widely, not as an unusual edge case.

Think of it less like a severance agreement and more like a non-compete that follows you for life, except the thing you're barred from doing isn't working for a rival. It's talking.

Did You Know?

Much of OpenAI's staff equity wasn't ordinary stock. It came as Profit Participation Units, a structure that ties an employee's payout to company profits rather than straightforward ownership, with gains historically capped at a multiple of the original grant. In late 2025, as OpenAI restructured into a public benefit corporation, it began converting those units into ordinary shares, removing the cap entirely.

An Apology That Came With an Asterisk

Sam Altman said he was "genuinely embarrassed" once the clawback provision became public, and claimed he hadn't known about it. Leaked emails obtained by Vox later showed senior OpenAI leadership, including Altman, had in fact signed off on the underlying documents.

OpenAI reversed course days later: it released departed employees from their existing non-disparagement obligations and removed the clawback language from future paperwork, stating it would never revoke vested equity again.

The reversal fixed the paperwork going forward. It did nothing to undo the years the threat had already been hanging over anyone who left quietly rather than risk it, which is most people, since most departing employees never find out whether a threat like that would have held up in court.

When the Whole Team Walks

Two weeks after Kokotajlo's story broke, OpenAI dissolved Superalignment, the team it had built a year earlier with a mandate to solve how humans keep control over AI systems smarter than their own creators. Its co-lead, Jan Leike, resigned first and wrote that "safety culture and processes have taken a backseat to shiny products," adding that his team had spent months "sailing against the wind" for basic computing access. He later joined Anthropic to keep doing the same work.

Five months after that, Miles Brundage, OpenAI's senior advisor on AGI readiness, resigned too, and the small team he ran was folded into other groups. He wrote that neither OpenAI nor any other frontier lab, nor the world, was actually ready for what it was building.

Both departures happened while every major lab, OpenAI included, kept racing to ship the AI agents that can operate a computer the way a person does, the exact kind of capability jump Superalignment existed to get ahead of. The team built to slow that race down didn't outlast a single product cycle.

That same June, thirteen current and former employees of OpenAI, Google DeepMind, and Anthropic, six of them anonymous, signed an open letter titled "A Right to Warn About Advanced Artificial Intelligence." It asked companies to stop enforcing non-disparagement clauses and to create a protected channel for raising safety concerns. AI researchers Yoshua Bengio, Geoffrey Hinton, and Stuart Russell endorsed it publicly.

Testifying Costs More Than Quitting

Quitting is one way to dissent. Testifying under oath is another, and it asks for more.

In September 2024, William Saunders, a former OpenAI technical staff member, told the US Senate Judiciary Committee he had resigned because he'd "lost faith that by themselves they will make responsible decisions about AGI." He described periods when hundreds of employees could have accessed OpenAI's most advanced systems without proper safeguards in place, and called for legal whistleblower protections built specifically for AI workers, since existing law generally only protects reports of illegal conduct, not speculative future harm.

The costliest case belongs to someone who never got to testify at all. Suchir Balaji, a researcher who spent four years at OpenAI, left the company in August 2024 over concerns that training ChatGPT on copyrighted material violated the law, concerns he later laid out publicly to the New York Times. He was found dead in his San Francisco apartment that November, at 26. The city's medical examiner ruled it a suicide with no evidence of foul play; his parents have publicly disputed that finding and say he had told them he intended to testify in ongoing copyright litigation against his former employer. Whatever the full explanation, his death remains the starkest illustration in this story of the gap between speaking up and living with the consequences of it.

The Law Trying to Catch Up

Government eventually noticed that a personal decision to speak up was, for years, the only mechanism holding this in check. California's SB 53, signed into law in September 2025 and effective this year, now requires large frontier AI developers, those with over 500 million dollars in annual revenue, to set up an internal system for anonymous safety reports and to protect employees who escalate concerns about catastrophic risk to regulators. Violations can cost a company up to a million dollars each.

It's a narrow law. It protects future disclosures about specific categories of catastrophic harm, not general criticism, and it doesn't touch the mountain of past exit agreements already signed under the old rules. But it exists at all because the industry racing to spend hundreds of billions of dollars on the compute buildout behind these systems had, until very recently, no legal obligation to make room for an employee who thought it was moving too fast.

Dissent Migrates, It Doesn't Disappear

Here's the part that should complicate any tidy villain, either company narrative: Anthropic itself exists because a group of OpenAI's own safety-minded staff, including siblings Dario and Daniela Amodei, left in 2021 over disagreements about how fast and how carefully OpenAI should move. They started a company explicitly built around taking safety more seriously.

Five years later, the same pattern is showing up inside the company they built. In September 2026, Jacob Coxon, a 27-year-old researcher who had worked at both OpenAI and Anthropic, resigned and posted on Slack that unchecked competition between labs was pushing toward self-improving AI systems that could, in his words, "kill us all by the end of the decade." His departure post drew tens of millions of views within a day, helped along by a same-day interview with the Wall Street Journal. Anthropic's own alignment lead, Evan Hubinger, publicly backed the substance of the warning and put his personal estimate of AI-driven human extinction risk above 10 percent within the next decade.

Months earlier, Anthropic's head of safeguards research, Mrinank Sharma, had resigned too, writing in a public letter that "the world is in peril," and that he had repeatedly watched how hard it is "to truly let our values govern our actions" inside the organization itself. He left the field entirely to study poetry.

The company founded by people who once walked out of OpenAI over exactly this argument now has its own safety staff walking out the same door.

Knowlegic Perspective

None of the people in this story set out to be martyrs. Kokotajlo wanted his last paycheck without a lifetime gag order attached. Leike and Brundage wanted their teams properly resourced. Saunders wanted a legal channel to raise a concern without losing his career over it. Nobody here asked for the outcome they got.

That's what makes the pattern worth taking seriously. Sincerity was never the missing ingredient. What these researchers consistently lacked was structural power: the ability to slow a product roadmap, redirect a compute budget, or hold a company to a promise it made about itself, without first giving up their paycheck, their equity, their next job in the field, or, in Balaji's case, according to his own family's account, far more than that.

Anthropic being founded by people who lived through exactly this fight at OpenAI, only to watch a version of it recur inside their own company five years later, is the clearest evidence that this isn't really a story about any one company's culture. It's a story about an industry where the people with the clearest view of the risk keep discovering they have the least power to stop it, whichever building they happen to be standing in.

The ones who try anyway pay for it, in equity, in relationships, in some cases in far more than that, and the industry keeps moving at very close to the same pace regardless of who just walked out the door.

Read All Articles of this Series:

Who Actually Governs AI: Part 1 - Copyright

Who Actually Governs AI: Part 2 - Labor

Who Actually Governs AI: Part 3 - Defense

Who Actually Governs AI: Part 4 - Surveillance

Sources & References

• OpenAI sends internal memo releasing former employees from controversial non-disparagement agreements, CNBC (2024)
• OpenAI disbands another safety team, as head advisor for 'AGI Readiness' resigns, CNBC (2024)
• OpenAI former safety leader Jan Leike joins rival AI startup Anthropic, CNBC (2024)
• Written Testimony of William Saunders, Former Member of Technical Staff, OpenAI, US Senate Committee on the Judiciary (2024)
• OpenAI whistleblower Suchir Balaji dead at age 26; family seeks answers as death ruled suicide, CBS News (2024)
• Anthropic Safety Leader Resigns, Warns 'the World Is in Peril', eWeek (2026)