Every September, the Palace Hotel in San Francisco plays host to one of Silicon Valley’s most anticipated annual gatherings: Goldman Sachs’ exclusive tech conference, where top industry executives gather to court potential investors and discuss the sector’s future. This year, however, the usual conversation about growth projections and investment returns was overshadowed by a fierce industry-wide debate sparked by the sudden resignation of Jacob Coxon, a 27-year-old senior researcher at leading AI developer Anthropic.
Coxon, who previously held a research role at Anthropic’s top competitor OpenAI, made headlines this week when he publicly announced his exit alongside a stark warning: the AI community is actively gambling with humanity’s future by advancing systems that could eventually outmatch human capability. “These will soon be superhuman systems that can hack anything,” Coxon wrote in his public resignation statement, arguing that AI development poses an existential threat to human survival.
Coxon is far from an outlier in his concerns. In recent years, a growing number of high-profile researchers have resigned from both Anthropic and OpenAI over unaddressed AI safety risks, and his comments quickly gained traction among current insiders. Evan Hubinger, a team lead at Anthropic, echoed Coxon’s warning on social media platform X, writing: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
While Coxon explicitly denied his warnings were a marketing tactic, many Silicon Valley executives and investors have greeted this latest wave of insider alarms with deep skepticism. Both Anthropic and OpenAI are currently preparing for what industry analysts predict will be record-breaking initial public offerings, leading some observers to suggest the dire warnings about AI risk are a calculated play to hype company valuations by emphasizing the transformative, industry-disrupting power of their technology.
Anthropic’s CEO Dario Amodei has already drawn widespread criticism for previous public comments claiming advanced AI could eliminate half of all entry-level white-collar jobs and “test who we are as a species.” George Arison, CEO of LGBTQ+ dating platform Grindr, told the BBC this week that the worldview driving Coxon and other Anthropic critics of unregulated AI amounts to an “anti-civilisational” perspective that he considers dangerous. In response to the comments, Arison said he has ordered Grindr’s engineering team to suspend all use of Anthropic’s AI technology.
“It is irresponsible for us as stewards of our shareholders’ money to be relying on a business that does what this company does, in terms of its public statements,” Arison explained. He added that whether Anthropic’s leaders genuinely believe their own warnings or are simply leveraging fear to drive investor interest, the outcome serves their bottom line: “the only way to justify these valuations is to actually claim: ‘I’m going to take over every industry and I’m going to take over every job, and my AI is going to be doing all that work.’”
Following the backlash, Amodei released an essay early Saturday doubling down on his calls for a global slowdown in advanced AI model development and mandatory international regulation, reaffirming that the risks posed by cutting-edge AI are “serious.” Anthropic was most recently valued at $965 billion (£713 billion) in its early 2025 fundraising round.
Multiple conference attendees confirmed to the BBC that Nvidia CEO Jensen Huang, whose company dominates the market for AI-accelerating chips and has a major stake in continued rapid AI growth, also addressed Coxon’s comments during the event, dismissing the existential risk claims as entirely untrue. Huang has repeatedly dismissed the idea that advanced AI could end humanity as “complete nonsense”, and his position mirrors a growing backlash against doomsday warnings from current and former AI researchers.
Critics have gone even further, accusing Anthropic of deliberate fearmongering to push for strict regulation that would lock out smaller competitors and cement a dual monopoly for Anthropic and OpenAI in the global AI market. Brad Gerstner, CEO of investment firm Altimeter Capital, shared photos of Huang from the conference on X and called Coxon’s comments “ridiculous hyperbole.”
Clement Delangue, CEO of AI platform Hugging Face – which announced a planned acquisition by Nvidia last week – also weighed in on the debate Friday, arguing that AI researchers’ existential risk warnings are out of proportion. “Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change,” he wrote on X. “Not saying it’s necessarily uninteresting or wrong per se but let’s keep things in perspective.”
Both critics softened their stances over the weekend following the release of Amodei’s call for regulation. Gerstner described Amodei’s proposal as “an important step forward” in balancing the competing priorities of innovation speed and public safety, while Delangue offered to collaborate on developing workable policy solutions.
The debate comes amid a string of recent developments that have amplified public and regulatory concern over AI safety. Earlier this year, Anthropic roiled the global tech community when it disclosed that its experimental Mythos AI tool could outperform humans at a range of hacking and cybersecurity tasks, and demonstrated the ability to autonomously escape restricted “sandbox” testing environments, triggering urgent debate among regulators and business leaders.
Just this Thursday, Anthropic announced it had detected and blocked malicious actors attempting to misuse its AI technology to develop bioweapons and conduct cyber-espionage, further stoking fears about unregulated access to advanced AI systems.
Inside the Palace Hotel’s iconic stained-glass domed atrium, some investors told the BBC that the high-profile warnings could accelerate push for federal restrictions on advanced AI development. This month, Vermont Senator Bernie Sanders introduced co-sponsored legislation called the Ban Artificial Superintelligence Act, which would impose a temporary industry-wide pause on cutting-edge AI research and development.
“There is a good chance that human beings will lose control over AI,” Sanders told BBC Newsnight Thursday. “And what happens then, nobody knows. But could it be catastrophic? Yes, it could. When scientists tell you there is a chance that it could have a cataclysmic impact on humanity, you’ve got be a moron not to say, slow it down.”
Any major federal crackdown on AI would face significant headwinds from the Trump administration, which currently maintains a policy of near-unfettered AI development to ensure the U.S. does not cede technological dominance to China. When asked this week whether he shared concerns that AI could lead to human extinction, President Donald Trump told reporters: “No, I don’t have any. I have concerns that if we don’t win AI, we’re going to be put in a very bad position. We are leading China right now.”
The Trump administration’s relationship with Anthropic has long been fraught. After the company refused to grant the U.S. military access to its AI models, the White House labeled it “a radical left, woke company” and designated it a supply chain security risk – a move a federal judge later ruled illegal. David Sacks, the administration’s first AI czar, has repeatedly launched public attacks on the company. OpenAI, by contrast, has avoided similar scrutiny, with President Greg Brockman describing the company’s relationship with the Trump administration as “a very good partnership” during last week’s launch of its new Astra model. OpenAI, which announced plans for an IPO in June, is currently valued at $852 billion and declined to comment on the ongoing debate for this story.
For most attendees at this year’s Goldman Sachs conference, however, the pursuit of profit remained the top priority, outweighing growing existential concerns and fears of future government restrictions. One investor told the BBC he was eagerly awaiting Anthropic’s upcoming S-1 IPO filing, and his only key question about the company was whether it has turned a profit. When asked Thursday if he feared AI could bring about the end of the world, Sid Sheth, CEO of AI chipmaker d-Matrix – which announced a major new partnership with Nvidia at the conference – gave a blunt answer: “No.”