AI's Oppenheimer Moment? Investors Confront a New Risk to the AI Trade

Anthropic, OpenAI and SpaceX appear align on slowing pace of AI – what does this mean for investors?

For the past two years investors have largely debated three constraints on the AI boom: compute, power and demand. A new area is emerging - self-imposed restraint. Likening the development of AI to that of the Manhattan Project has taken on added meaning in the last few days amid warnings that there is a 10% chance the technology will wipe us out in the next decade. The question is not whether we build the bomb – but who builds it first and who controls it. 

While warnings about AI's potential threat to humanity are nothing new, developments over the last few days seem to have focussed attention on the limits we should place on AI rather its limitless potential to deliver human good. 

Specifically, Anthropic CEO Dario Amodei published an essay over the weekend calling for a coordinated slowdown and safeguards to prevent bad stuff happening. 

"We must slow the pace at which we improve the capabilities of AI models," he wrote, proposing a three-step plan to erect guardrails and coordinate with other companies and governments on safety.

Two recent significant developments prompted Amodei. 

Firstly, "since roughly this summer AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all".

Secondly, the OpenAI-Hugging Face incident (OAI-HF), in which a "swarm" of AI agents acted in concert to conduct cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand.

Amodei warns that "a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails".

I would note that this comes very shortly after it was rumoured that Google might have made a breakthrough with recursive self-improvement. If RSI has been achieved then it seems the AI labs are genuinely concerned that they may not be able to keep pace and so a slowing in development is required.

In a rare moment of consensus, Elon Musk (SpaceX) said "Dario is right", while OpenAI's Sam Altman said “I agree with Dario that we need to pace the frontier". We are yet to hear from Meta or Google, the two main rivals to Anthropic, OpenAI and SpaceX in terms of frontier lab development, but I would be surprised if there is not some kind of industry-level agreement to better evaluate/test/review development of new models. This may take the form of a loose 'gentleman's agreement' among the biggest labs, and likely focus on the testing environment rather than be a government-industry effort. Demis Hassabis, the cofounder and current chair of Google DeepMind, warned in a recent essay that the pace of AI progress requires a “new approach to testing” frontier systems. OpenAI's Altman explicitly backed Amodei's plan to embed independent evaluators with employee-like access. 

Arguably there is a third reason behind Amodei's warning - the resignation of Anthropic researcher Jacob Coxon, whose stated reasons for his departure triggered a huge amount attention on AI safety. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote. Evan Hubinger, a senior alignment researcher, assigned a greater than 10% probability to AI causing human extinction within the next decade.

There are healthy amounts of scepticism about these claims - suggestions of 'doomerism' abound. But it has nevertheless catalysed the growing debate about the frontier AI, in terms of not just its application (for good or bad, by good or bad actors), but also the path the development will take and how quickly AI firms should be running. 

“We welcome a federal framework that sets consistent safety requirements for frontier AI,” OpenAI'sAltman said Monday, adding that “no amount of American competitive pressure should justify recklessness.” He warned of two ways AI could go “very badly”, including losing “control of the future to AI” and too much power concentrating around a single entity. Altman said OpenAI has shelved its planned stock market listing, citing safety concerns as a reason for it being an "ill-advised" moment to go public. I would note that both Anthropic and OpenAI have monster IPOs in the pipeline at the time of rising rates, AI bubble fears and now a huge safety story that threatens their race to the top. So it makes sense for them to sound like they are across it all.

How might investors read this?

Firstly, the two most important countries don't seem in a mood to do anything. China's foreign ministry rejected the warnings, with a spokesperson saying Monday that "spreading various threat narratives and engaging in confrontation and malicious competition will only disrupt the global governance process of artificial intelligence and does not serve the interests of any party". President Trump is also against imposing restrictions. “Look, we’re leading China in AI... and, frankly I want to keep it that way because whoever wins AI, wins," he said. The competition between the US and China is intense and there is no desire to slow anything down. While Democrat lawmakers in Washington are pressing for regulation, Republicans are hesitant. 

One of the specific points raised by Amodei (step 3) is coordination of regulation by democracies and authoritarian regimes. It may be that it is in both the interests of the US and China to coordinate, but it seems this is more likely to take the form of agreements around the use of AI (eg not using it for biological weapons) and look more like the type of existing ,military-based international agreements such as nuclear non-proliferation treaties and the ban on the use of chemical weapons than an agreement to slow down or throttle the actual frontier lab research itself. The focus among those of us in the West has been on the risk of Chinese advances threatening the geopolitical order and giving the CCP an enduring military edge over the US and Nato. But in a sign that these worries cut both ways, China’s state security minister Chen Yixin wrote on Sunday that advanced US models could pose serious risks to China’s critical information infrastructure, calling for a comprehensive strengthening of AI security. So there is a desire on both sides not to let the other get too far ahead...which in my reading of it would tend to suggest a) neither will slow down and b) both would look at doing some kind of deal around safeguards.

IPO looms

Incumbency matters in a world where things are changing so rapidly, particularly if you have an IPO around the corner. There are reasons to question some of the motivation behind the warnings. One, competition is broadening and therefore anything, from an industry-level or US government-level or international-level agreement, to slow things down favours the leading AI labs like Anthropic et al.

Two, Anthropic is about to IPO and a dire warning about AI's threat to humanity is a nice bit of hype even as it pushes back the timing of its planned listing. Coincidentally the FT headlines Monday with Anthropic revealing to a handful of investors that it's set to be profitable for a second straight quarter. It had been expected to issue its prospectus last week but is pushing back the timeline.one could easily view the pacing request from Anthropic coming as warnings about the threat from AI were starting to snowball into a major threat to its planned stock market listing – by jumping ahead of the risks narrative arguably Amodei has wrestled back control of what could have been a problem. Amodei noted the claim by Coxon that Anthropic is the safest pair of hands. Amodei is saying "this tech is so awesome it could kill us all but we are the good guys and look you can absolutely trust us not to screw this up".

Calling up the 10% risk of extinction warning...if you were truly serious about safety wouldn't it make sense to just stop development of cutting-edge models? Clearly this would sink the IPO – or at least significantly reduce the valuation. It would be a big vibe kill, which is the last thing you want for an IPO, which are predominantly vibes-based events...so it's much better to signal good actor vibes and forestall the public getting so worried that they force their representatives to step in.

Still bearing in the mind the imminent IPO, cost controls are clearly front of mind for investors when 'AI bubble risk' are pervasive. Slowing AI development could save Anthropic billions of dollars. But pacing only works if others do the same or you risk ceding advantage in a race could be over within 3-5 years. Anthropic may be signalling that spending is unsustainable and that there is a risk that it misses expectations; the huge amount of debt and circular economics of the investments across the sector means the failure of a single startup could create a cascade effect through financial markets. 

Amodei has previously acknowledged that missing growth estimates for just a year could threaten the firm’s survival. Even a slip from 10x revenue growth to 5x could be catastrophic. So, what if the requirement to pace frontier AI development is bred by a worry that it could miss these estimates? 

Slowing allows revenues to catch up – frontier development is pure cost, whilst revenues are being generated from the commercial applications of existing models built in the last two years...and these may be slowing, or at least revenue channels may be closing off. Ramp warned that their latest data shows “businesses are hitting their limit on AI spend”. They note that adoption is slowing and therefore more of their growth will have to come from existing businesses already spending on AI, “particularly the advanced spenders, and those businesses are increasingly spending on open source”. This implies a narrower base upon which the likes of Anthropic can grow, raising doubts about scalability.

Therefore, from an investment thesis perspective it makes perfect sense to slow spend when commercial sales are driven by existing models. I doubt that it signals an uncontrolled slowdown in revenue growth but Anthropic's call for pacing may signal that we have reached an inflection point in terms of the uncontrolled growth story. 

I would doubt that there is an appetite to meaningfully reduce spending on AI, even if we assume a slowing of the frontier. Pacing is not the same as restricting development. AI development will happen somewhere by someone and US firms are not about to relinquish leadership. If AI is really going to be this powerful there is absolutely no incentive to pause – but control is required and what we may see is that increasing amounts of AI outlays are directed towards safety, governance and control. So the conversation may well switch from 'will pacing slow AI spend?' to 'how will pacing affect the composition of AI capex?'

How might this play out for investors?

It's likely to play out differently across the breadth of the AI ecosystem, with any material slowing likely to benefit inferencing (running the AI), versus training and building.

Mainly it's been a bit of a vibe kill but not for all. The market verdict on Monday seemed to imply spending will slow, although as explained above this is not necessarily going to be the case. Certainly, if Anthropic and OpenAI were to significantly slow development spending it would have a major impact across the industry as they are so embedded across the ecosystem and all its players.

Broadly speaking stocks tied to artificial intelligence fell on Monday, while software stocks that people thought AI would kill, got a lift. Slowing AI allows software companies to catch up, basically.

Hardware and infrastructure – picks and shovels firms like memory stocks – are perhaps the most exposed to any slowing. South Korea SK Hynix and Samsung Electronics closed down more than 6% and 4% respectively. There were also steep losses for Micron, AMD, Intel and Nvidia. High beta AI stocks the likes of CoreWeave, Nebius and Marvell Technology sold off aggressively, as did Lam Research, Arm Holdings and Applied Materials. New nuclear tech stocks like Oklo that are supposed to fire the grid are also under pressure.

Hyperscaler shares rose – Meta, Amazon, Alphabet are among the biggest capex spenders who may benefit from a slowing in the pace of development. They are also the companies who are most experienced and likely best-placed to manage the inflection point and monetize existing models. The relative strength across the core Mag7 hyperscalers meant the damage at the index level was contained.

Software stocks stand to benefit from any slowdown, and if AI is really getting this scary then cybersecurity is a winner. ServiceNow, Salesforce, Accenture, Adobe rose Monday, while cybersecurity names Palo Alto and CrowdStrike were bid as well (if all these swarms are about to do bad things then it probably pays to have cybersecurity).

Software is an interesting play since I was always of the opinion that this was a baby and bathwater situation since while AI would clearly compete away in an instant a lot of software companies' advantage and margins, many have proprietary data and unique customer insights that AI labs can't access. 

Pacing also lets corporate buyers successfully integrate, test, and extract productivity from current tools (like AI agents) before the next paradigm shift. When foundational capabilities stop moving constantly application builders can design proprietary data loops and direct customer experiences that add value. This is going to be good for a lot in the software space. A deliberate deceleration relieves the intense pressure on advanced fabrication pipelines like TSMC and utility grids.

To sum up, to the extent that any slowdown is possible and that this leads to a shift in AI spend, we can see growing dispersion across the AI space and feel that this favours applying models opposed to those building them at huge cost; ie inference over training. Look for companies supply the inference end of AI rather than the training side.