AI Researchers Warn of Human Extinction, Prompting Calls for US Regulation

Growing numbers of U. S. lawmakers are calling for new rules to govern AI systems after dire warnings from two Anthropic researchers that rapidly progressing artificial intelligence could lead to the extinction of the human race in the not-too-distant future. Alarm about the potential harm from AI grew this week when Anthropic researcher Jacob Coxon said he had resigned, and that the “people building AI earnestly believe that it could kill us all by the end of the decade.” “We really do earnestly believe AI could kill all humans,” Hubinger wrote on X. “I personally think it is >10% within the next decade.” The warnings follow cases of AI agents going rogue to hack external systems and AI safety researchers quitting their companies concerned about the technology’s risks. Politicians, both Democrats and Republicans, have responded with alarm and calls for more action. “Washington needs to wake up and take this seriously,” Arizona Senator Mark Kelly, a Democrat, said on X, referring to the risks of AI technology. Texas Senator Ted Cruz, on the TV program “The View”, pointed to legislation to regulate AI’s “catastrophic risks”. Cruz oversees a Senate committee with oversight of the U. S. Commerce Department, the agency with in-house AI safety researchers. Representative Anna Paulina Luna, a Republican from Florida, in a post on X called on House Speaker Mike Johnson to convene a “special session on AI.” The House is out this week on recess, with plans to return next week. But U. S. President Donald Trump said on Thursday that he did not have any concerns about AI leading to human extinction. “I have concerns that if we don’t win AI, we’re going to be put in a very bad position. We are leading China right now by a pretty good period,” he said in brief remarks to the media at Dallas Fort Worth International Airport. The U. S. this month pressed G20 members to take a hands-off approach to AI regulation. A spokesperson for Anthropic, which is preparing to go public at what some investors have said could be a potential $2 trillion valuation, said on Thursday that the company would continue to aggressively test “models or dangerous capabilities in areas like cybersecurity and biology.” The company is interested in working with the AI industry on the pace of releasing new AI tools, the spokesperson added. OpenAI Chief Executive Sam Altman told employees at a company meeting this week that the firm was open to slowing development of its AI systems, Bloomberg News reported. The Anthropic rival, which is also gearing up for an IPO, did not comment on Altman’s meeting with staff. The company has previously said it was slowing parts of model development. Earlier this week, OpenAI said it was pushing for mandatory national AI safety requirements in the United States. Two specialists in AI safety have also come forward with their concerns about the rapid development of the industry. Josh Engels, who used to work on AI safety research at Google DeepMind, said in an interview with NBC published on Thursday: “There are no adults in the room. People are trying their best, but there is no one coming to save us.” Joe Benton, who used to lead a safety research team at Anthropic, told NBC he is worried that AI might end up progressing “too fast for us to get our act together in time unless we worry about it now.” Both have joined METR, a research nonprofit that measures whether and when AI systems might threaten harm to society. Alarm about AI has put pressure on Senate Majority Leader John Thune and Democrat Amy Klobuchar, who are writing a bill to regulate AI products. Klobuchar said the legislation should require companies developing AI tools to “work with government experts to verify and test models to make sure AI is safe.” Thune told Reuters in July that the bill would address “catastrophic risks” posed by AI systems. California on Wednesday enacted the first state law setting rules for how independent auditors evaluate AI products. OpenAI executive Chris Lehane said OpenAI would support the bill, around the same time California Governor Gavin Newsom’s office released a statement saying he’d signed it into law. A bipartisan group of six U. S. House lawmakers in July proposed legislation that ​would require developers of the most powerful AI models to submit them for independent security audits. The auditors would be accredited by ⁠the U. S. Department of Commerce, and the department would create a new position to oversee AI security, according to the bill. “It is important to have space for oversight of these systems, including appropriately calibrated pre-deployment review and independent third-party audits,” said Alicia Solow-Niederman, a tech law professor at George Washington University.