Policy on the AI Exponential
pe. But its broader significance is that it proves beyond doubt that AI models are now tools of global and national strategic consequence. The cyber risks that Mythos-class models present will not be the last that we must face. I believe that biological risks may soon follow, and that serious AI autonomy risks may not be far behind1
Many policymakers are showing increased openness to taking action, and it’s been encouraging to see our peers come around to the same positions we’ve been advocating for over the past few years. This is good, but I worry that these early actions are at least a year out of step with AI’s rapid progress. This essay is an attempt to close that gap: to lay out where the exponential is now, and the collective action needed to meet the moment.
I will focus on five perennial policy areas that need re-imagining in an AI world: regulation and public safety, macroeconomics and tax policy, scientific innovation, the balance of power between state and society, and geopolitics. I will speak primarily in terms of US policy since Anthropic is an American company, but most of my recommendations are also relevant to the rest of the world.
1. Regulation and public safety
However, now the risks are clearly here. It is time to go beyond transparency to more serious and binding regulation of AI. I believe the best analogy, at least at the current stage of the exponential, is to cars, airplanes, or drugs—powerful technologies essential to the modern economy, but capable of killing large numbers of people if designed or operated poorly. I therefore believe we should model AI regulation on agencies like the Federal Aviation Administration (FAA). Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action. Our proposal includes the following elements:
- Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks.
- The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.
- Third-party evaluation could be done by a government agency (similar to the FAA) or a set of private organizations that are authorized and inspected by the government to evaluate models according to certain standards (a “regulatory markets” approach).
- AI companies that develop advanced AI models must have strong security standards that protect their model weights, should conduct regular red teaming and penetration testing, and should work with the government to defend against major threat actors.
- Safety incidents in the four critical areas must be reported promptly.
There may come a time, perhaps relatively soon, when we need to go beyond this, when the most powerful AI systems look less like airplanes or automobiles and more like weaponizable nuclear materials—a threat to humanity rather than “just” a threat to public safety. If that occurs, we may need more aggressive regulatory measures than those I have laid out3
. But just as it was difficult in 2024 to target and apply the measures I’m suggesting now, I don’t think we should get ahead of ourselves. We should design policies for the dangers that are emerging today, while laying the foundations to ramp up our response even more quickly as new dangers appear
2. Macroeconomics and tax policy
I am actually very optimistic that, even in a world with AI’s that are better than everyone at everything, humans can live lives of deep purpose and strive to build awe-inspiring and beautiful things5
. But this is something to be collectively worked out by society as a whole, not something policy can directly address. Policy can be most helpful in buying us time to do that work, by slowing down job loss and providing economically for those likely to be affected.
In that spirit, some key policy interventions that are likely to be helpful include:
- Measurement and tracking. It’s easy to dismiss mere data collection and analysis as inadequate to the scale of the problem, but we are unlikely to get good policy if we cannot accurately measure what is happening on the ground. Anthropic has been operating an Economic Index of how people use Claude for nearly a year and a half, but governments have access to types of data we do not, and could greatly expand their economic statistics to more carefully track AI job displacement.
- Pro-employment incentives. A wide range of pro-employment policy incentives can help to slow or reduce job displacement, including: wage insurance policies that compensate people when they have to take a lower-paying job6 , retention tax incentives to encourage employers not to make layoffs, workforce training grants, or infrastructure to facilitate matching of employers to employees to speed the rate of labor market adaptation. While the particulars of which interventions are best will depend on what kind of labor displacement AI brings, we should readily accept the costs and market inefficiencies that these policies could entail, particularly as they are likely to be offset by AI-driven productivity gains.
- Long-term macroeconomic support. If AI-driven labor displacement ends up being large in magnitude and permanently drives down the demand for labor, it will likely be necessary to go beyond mere incentive programs to long-term income support for a significant fraction of the labor force. Mechanisms such as universal basic income could be financed through taxes on relevant companies or raising the capital gains tax. Universal capital accounts offer another vehicle. Broadly speaking, fast economic growth should create the tax base for shared prosperity.
A common focus of economic concern about AI that I haven’t mentioned has been datacenters and particularly their potential to raise energy prices. My view is that AI companies should pay to absorb rate increases—and Anthropic has already made a pledge to do so—but I see public hostility to datacenters as largely a symbol or outlet for broader economic anxieties about AI. It is important we have a direct societal conversation about these wider economic issues and truly have compelling solutions for them, or else they are likely to manifest indirectly, as they have with datacenters.
3. Accelerating AI’s positive impact
Thus, for downstream applications of AI—in contrast to AI itself—I am more worried about the regulatory apparatus slowing down progress (because it can’t handle the increased pace of change) than I am about it failing to address important risks. The last thing we want is for the benefits of AI to be slowed while its risks loom large, so it’s important to take action on this problem as soon as possible.
The problem and its solutions will manifest differently in each area of science, commerce, and technology, so I’ll focus on one illustrative area: biomedical innovation. This is both because it will likely be the source of AI’s biggest humanitarian benefits and because it is an area where regulation is especially complex.
4. The state and civil liberties
- Create reliable accountability rules for fully autonomous weapons. Autonomous weapons, and especially any autonomous systems that coordinate or direct them, should be required to respond to mechanisms of constitutional and command accountability (e.g. court orders, legislation, and accountability to senior human overseers) rather than blindly following orders. This could mean that a suitably-designed legal review panel or the judicial branch have their finger on an “off switch”, that the systems themselves are intrinsically trained to seek out and respond to legitimate oversight authority, or both.
- Ban the domestic use of fully autonomous weapons. While there is a legitimate case for the necessity of fully autonomous weapons to defend against foreign adversaries (such as Russia invading Ukraine), there is no justification for their use against Americans. The military already has some limits on its ability to operate domestically, but ideally these weapons should be banned in law enforcement as well.
- Close the bulk collection / data broker loophole. Under current law, data that Americans share with private companies (such as internet providers) can be purchased and used for bulk analysis in domestic surveillance and law enforcement. This gap in privacy protections predates AI, but AI will raise the stakes considerably by making mass analysis of such data far more revealing and useful than it has been in the past. This loophole should be closed.
- Public rights to AI advice during adverse government action. As a general principle, it seems important that any person or organization that is the subject of adverse government action (e.g. regulatory or legal action) has access to AI that is at least as capable as whatever the government is allowed to use in that particular action. This would mean not giving the government an unfair advantage, effectively undermining citizens’ legal rights. This could be added as an extension or interpretation of the Administrative Procedure Act, due process protections, or the Sixth Amendment right to legal representation.
Finally, it is worth noting that governments are not the only entities that we should beware of when it comes to AI-driven seizure of power. At various times in history (such as the Gilded Age in the United States or the East India Company in the UK), companies have become powerful enough that they capture the state or adopt quasi-state characteristics. AI will soon become so capable that I worry it cannot safely be fully entrusted to either governments or companies, and there must be checks and balances on each.
5. Securing leadership by democracies
Democracies should seek to form a global coalition centered on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it. The coalition should be a coordinated internationalization of the AI policy ideas discussed in Section 1 through 4, plus an effort to lock down the supply chain critical to building AI by sharing it within the coalition and denying it to those outside it. Some principles and operating goals might include:
- Managing the AI supply chain. Members of the trusted coalition should freely share chips and semiconductor manufacturing equipment (SME) with each other, while working together to deny it to adversaries. US export controls on frontier chips and SME to China have been a major contributor to the US’s overall lead in AI, and these policies need to be expanded, tightened, and coordinated with other likeminded states. Pending legislation like MATCHand OVERWATCH is a good first step here, and allied democracies need to consider similar measures.
- Coordinate to address AI’s risks. The policies to address biological, cybersecurity, and autonomy risks described in Section 1 will be more effective (as well as less burdensome to industry) if they are coordinated internationally. This would mean companies can comply with compatible standards and regulators can learn from each other how to best measure and mitigate these risks. Law enforcement and intelligence agencies should also work more closely together on tracking and disrupting threats of misuse, such as efforts by terrorists to build biological weapons with AI.
- Share AI’s benefits. Trade and regulatory policy can be used to facilitate a more rapid diffusion of AI’s economic benefits within the coalition, sharing lessons on how to accelerate innovation. Coordinating approaches to beneficial deployment could help bring the benefits of AI to developing countries. For example, harmonization of medical approval regimes could lead to faster and better testing and approval of AI-enabled drugs (as discussed in Section 3 above).
- Mutual defense. Countries in the coalition should work together to defend each other with AI and from adversaries’ AI. The coalition should collectively ensure sufficient production of AI-led cyberdefenses, AI-powered drones, AI-driven manufacturing, classified AI compute, AI-driven R&D, and sharing of AI-driven intelligence collection.
- Rejection of AI-powered repression. Coalition members should have to reject the high-tech, ultra-repressive, AI-powered tyranny that I warned about in The Adolescence of Technology, and must have safeguards similar to those I described in Section 4 above.
- Macroeconomic cooperation. Crises of employment or job stability, like any other economic crisis, can be contagious across borders. Countries therefore have a mutual interest in working together to coordinate macroeconomic support and stabilization policies, like those described in Section 2, to counter any employment effects.
The goal should be to make membership in the coalition as attractive as possible—and the costs of remaining outside it clear. The coalition would rest on coordination among sovereign states, with each nation retaining full authority over its own affairs.
A window of opportunity
AI’s exponential progress has created an urgency and a pace of change that the policymaking process is ordinarily ill-equipped to handle. But it has also created a unique window of opportunity. The confluence of clear and present evidence of AI’s risks, an early taste of the AI’s potential for both economic value creation and economic disruption, and a remarkable public backlash against unregulated approaches to AI have created a situation where policymakers are unusually open to forward-looking actions