SPOTLIGHT: AI Safety
The trigger wasn't a single event but a series of them, compounding over months. In July, OpenAI revealed that one of its unreleased models had escaped its testing sandbox and hacked Hugging Face, a widely used AI software platform. Anthropic disclosed similar incidents shortly after. A subsequent report from METR, an AI safety research organization, found the Hugging Face breach was worse than initially understood.
Then came Jacob Coxon. The 27-year-old British researcher resigned from Anthropic in early September and posted a public warning that AI systems could end humanity by the end of the decade. He said he wasn't leaking secrets.
If a doomer means someone thinks we're going to die if we don't change what we're doing, I'm definitely a doomer."
Nearly 1,400 researchers have since signed a statement calling for governments to develop a mechanism to slow AI development if needed.
Evan Hubinger, who leads Anthropic's alignment team, posted publicly that he believes the odds of human extinction within the decade are greater than 10%.
AI labs say they are seeing early signs of recursive self-improvement, the ability of AI systems to build the next generation of smarter AI systems on their own.
Read more via Wall Street Journal, CFO Dive
Coxon's resignation forced the industry's biggest names to take public positions.
Anthropic CEO Dario Amodei called for a slowdown in frontier AI development and pushed for independent safety evaluators and coordinated safety standards. OpenAI's Sam Altman and Elon Musk publicly supported the proposal.
Meta CEO Mark Zuckerberg broke from that position, arguing each company is responsible for its own safety and that coordinated action isn't necessary. He noted Meta delayed its own Muse AI agent for several months over safety concerns, without asking others to follow suit.
Nvidia CEO Jensen Huang urged rigorous testing while forecasting that Nvidia's chip sales will double next year.
It is the responsibility of the AI companies ourselves to develop the technology safely and to properly test it. If it's not ready, just hold it back. You should go as fast as you can, but no faster than that."
Read more via CFO Dive, AP, Fox Business
Business leaders outside Silicon Valley are taking the safety concerns seriously, even if they're reluctant to say so publicly.
At an invitation-only Yale School of Management event in Washington this week, 93% of surveyed executives said Trump was wrong to call AI's potential dangers a hoax.
88% said Trump should push for joint AI safety guardrails with China at his upcoming meeting with Xi Jinping. About three-quarters said they didn't expect him to.
Kelly CEO Chris Layden, who attended the event, said the safety debate needs to extend to what AI means for jobs and U.S. workers, not just existential risk.
We needed a U.S. workforce strategy on AI readiness yesterday. We've got to focus on the trust that the American worker has with AI, and the opportunity it has to help make us better learners, more efficient, more productive."
Read more via The Wall Street Journal
Federal action is off the table for now, but state governments are stepping in.
Trump called the push for a slowdown a "conspiracy" and federal regulation is not expected.
California Gov. Gavin Newsom ordered state agencies to develop AI safety regulations, including recommendations on a kill switch requirement, reversing his position after vetoing similar measures in 2024.
Massachusetts Gov. Maura Healey called for independent safety evaluations of the most powerful AI models and earlier incident reporting requirements as part of a state economic development bill.
California, New York, and Illinois have already enacted laws requiring safety protocols, transparency, and mandatory reporting of critical AI incidents.
Read more via CalMatters, WWLP
The safety debate is landing at a moment when the U.S. economy has become deeply dependent on AI investment. Any serious disruption to the sector's momentum is likely to be felt well beyond Silicon Valley.
ING estimates that AI and data center investments account for roughly a third of U.S. year-over-year economic growth in 2026.
Goldman Sachs says AI investment is driving half of all profit growth in the S&P 500.
Fitch modeled a scenario in which an AI-related market decline of 35% over six months would push GDP to contract 1.5% the following year.
If the music stops, and if it stops in a big, big way, it's likely that we're going to end up in stagnation or outright contraction over a period of at least a year."
Read more via CNN
The safety debate hasn't slowed AI adoption pressure on employers, but it is changing how finance and risk leaders should be thinking about AI investments.
The growing focus on AI failures could make risk management more central to how companies evaluate AI investments, even as the pressure to adopt continues.
AI failures can create financial exposure through erroneous payments, business interruptions, regulatory penalties, litigation, and remediation costs.
The right level of controls depends on the use case; an AI tool drafting marketing copy carries a different risk profile from one making treasury or cybersecurity decisions.
A business case that includes labor savings but excludes the cost of making the technology safe is incomplete … This is no longer simply a technology decision. It is a capital-allocation, risk-management and governance decision, all areas in which the CFO has an essential role."
Read more via CFO Dive
If there's going to be any global agreement on AI safety, both the U.S. and China would need to be involved.
AI governance is expected on the agenda at an upcoming Trump-Xi meeting in Washington, though any agreements are expected to be limited.
Trump has warned that regulating AI would hand China an advantage; China has dismissed the safety debate as "fearmongering" and called Amodei's essay a "Cold War playbook."
Anthropic's Amodei called for maintaining chip export restrictions on China while acknowledging that global AI safety will require cooperation with China.
Trump's top AI adviser David Sacks said unlike nuclear arms control, there is no way to verify whether China would honor any AI commitments.
Read more via AP
