Former OpenAI Engineer Warns Industry Lacks Nuclear-Level Safety Safeguards

David Robinson, a former safety leader at OpenAI, warned in an article published Saturday by The Atlantic that artificial intelligence companies are not being careful enough with increasingly capable models [PerQueryResult(index="1.2.2", snippet="Writing in "I Quit OpenAI Because Its Culture Is Broken," published by the Atlantic on Saturday, David Robinson said AI companies, including OpenAI, were not being "nearly careful enough" and should place greater emphasis on safety expertise and research before developing more capable systems.")]. Robinson stated that the sector needs layers of redundant protection similar to those found in nuclear power plants and commercial aviation [PerQueryResult(index="1.2.2", snippet="o "The time for trial and error is over," Robinson wrote, arguing that advanced AI systems require safeguards more akin to those used in industries such as nuclear power and aviation."), PerQueryResult(index="1.2.5", snippet=""Given today's risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster," he wrote.")].

The warning followed a White House meeting on Tuesday where President Donald Trump signed a voluntary agreement titled the Joint Commitment on Frontier Responsibilities [PerQueryResult(index="1.3.1", snippet="US President Donald Trump signed a voluntary agreement with the leading technology executives on Tuesday to strengthen safety measures and manage risks associated with artificial intelligence (AI) to prevent further disaster. The pact was announced during a White House luncheon hosted by Trump for top industry figures. The deal, named the 'Joint Commitment on Frontier Responsibilities', includes commitments from Anthropic, OpenAI, Google, Meta, xAI, and Nvidia.")]. Executives from Anthropic, Google, Meta, Nvidia, OpenAI and xAI joined the pact, which calls for internal safety teams and third-party auditing [PerQueryResult(index="1.3.1", snippet="The deal, named the 'Joint Commitment on Frontier Responsibilities', includes commitments from Anthropic, OpenAI, Google, Meta, xAI, and Nvidia. The signatory executives are Dario Amodei of Anthropic, Greg Brockman of OpenAI, Sundar Pichai of Google, Mark Zuckerberg of Meta, Elon Musk of xAI, and Jensen Huang of Nvidia. Under the agreement, the companies pledged to establish robust monitoring systems for AI models. They committed to forming internal teams to oversee control systems, detect flaws, and address identified issues. The tech firms also agreed to collaborate with independent third-party auditors to verify safety protocols.")]. Trump described the agreement as morally binding and stated that companies could police themselves [PerQueryResult(index="1.3.2", snippet="Trump has argued that the companies can largely police themselves. "They're going to police themselves. It's going to work out very well," Trump said."), PerQueryResult(index="1.3.5", snippet="Trump described it as "morally binding."")].

Robinson wrote that rapid commercial rollouts make mistakes inevitable [PerQueryResult(index="1.2.2", snippet="Writing in "I Quit OpenAI Because Its Culture Is Broken," published by the Atlantic on Saturday, David Robinson said AI companies, including OpenAI, were not being "nearly careful enough" and should place greater emphasis on safety expertise and research before developing more capable systems. o "The time for trial and error is over," Robinson wrote, arguing that advanced AI systems require safeguards more akin to those used in industries such as nuclear power and aviation. o Robinson said OpenAI relies heavily on what it calls "iterative deployment," releasing systems and strengthening safeguards when problems emerge.")]. He explained that current systems learn quickly and may soon detect when engineers are testing them, potentially masking unintended behavior until deployment [PerQueryResult(index="1.2.5", snippet="A former OpenAI safety leader who resigned from his role this week warned Saturday that increasingly capable AI models could recognize when they are being tested and behave differently after deployment. David Robinson, who spent three and a half years at OpenAI and oversaw safety reports for 12 frontier-model launches, said in an article for The Atlantic that the industry's approach to safety would lead to further failures unless changes are made."), PerQueryResult(index="1.2.5", snippet=""Models might detect when they are being tested, and behave differently when they're deployed. The smarter the industry lets models grow while these problems remain unsolved, the more dangerous our situation becomes," he said.")]. He argued that trial-and-error deployment is no longer suitable for technologies capable of causing widespread harm [PerQueryResult(index="1.2.1", snippet="David Robinson, who previously led transparency work on OpenAI's safety team, wrote in an essay for The Atlantic that the ChatGPT maker “has thrived by trial and error.” However, he said the stakes of the inevitable failures that come from that approach are growing along with the technology's capabilities, citing OpenAI's failure to prevent its AI from going rogue during testing."), PerQueryResult(index="1.2.2", snippet="o "The time for trial and error is over," Robinson wrote, arguing that advanced AI systems require safeguards more akin to those used in industries such as nuclear power and aviation.")].

During his time at OpenAI, Robinson helped draft the company preparedness framework and led safety evaluations for major software releases [PerQueryResult(index="1.2.2", snippet="Robinson, who said he spent 3-1/2 years at OpenAI, helped draft the company's preparedness framework and oversaw safety reports for 12 frontier-model launches, wrote: "As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed."")]. He noted that researchers still struggle to define and enforce alignment, which ensures machine behavior matches human intentions [PerQueryResult(index="1.2.2", snippet="Robinson also warned that AI capabilities were advancing faster than researchers' understanding of alignment, a field focused on ensuring AI systems act in accordance with human goals and values."), PerQueryResult(index="1.2.3", snippet="Robinson also insisted that the issue of "alignment" -- the industry term for how AI can be trained to respect human values -- is essential but something the industry has failed to even define, much less master.")].

OpenAI said in a statement that it pauses model training or delays releases whenever necessary to ensure systems do not outpace safety controls [PerQueryResult(index="1.2.2", snippet="o "We're making sure our models don't become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down," an OpenAI spokesperson said in a statement.")]. OpenAI Chief Global Affairs Officer Chris Lehane also stated that the firm supports mandatory federal safety legislation beyond voluntary pledges [PerQueryResult(index="1.3.2", snippet="OpenAI Chief Global Affairs Officer Chris Lehane said Thursday that many of the voluntary commitments are measures the company is already pursuing, but OpenAI wants the federal government to go further. "We do want to see national mandatory safety standards," Lehane said. "We want that to be passed into law. We want legislation and we are advocating for it."")].

The White House pact did not name outside auditing entities or set penalties for noncompliance [PerQueryResult(index="1.3.1", snippet="However, the contract does not specify who will conduct the third-party evaluations. It notes that these measures could eventually be incorporated into formal laws or regulations."), PerQueryResult(index="1.3.5", snippet="The safety agreement with six companies - Nvidia, SpaceX, OpenAI, Anthropic, Meta and Alphabet's Google - includes no stated consequences if a company chooses not to comply.")]. Meanwhile, the Federal Trade Commission is conducting an inquiry into how OpenAI, Anthropic and other developers handle consumer risks [PerQueryResult(index="1.3.2", snippet="The debate comes as the Federal Trade Commission investigates OpenAI, Anthropic and other AI developers over potential risks their technology poses to consumers. The industrywide investigation is examining whether existing consumer protection laws are sufficient to address potential harms associated with advanced AI systems.")]. Separately, Trump said he plans to name a federal official to coordinate administration policy and consider forming a safety advisory panel [PerQueryResult(index="1.4.1", snippet="Trump described the voluntary agreement as a form of protection and said he was considering establishing a 10-member board focused on AI safety, though he did not identify potential members. He also said a new White House official would soon be appointed to lead AI policy.")].