Artificial intelligence safety regulation is a delicate balancing act. A new study published in the Proceedings of the National Academy of Sciences suggests that weak or poorly targeted rules may be worse than having no rules at all. The research, led by Benjamin Laufer and a team from Cornell and Carnegie Mellon University, uses game theory to show that when governments focus only on downstream AI applications, they unintentionally encourage general-purpose AI developers to skimp on safety.
Key Facts From the Study
- The study uses theoretical economics and game theory to model AI safety regulation.
- Weak regulation that targets only downstream companies can lead to a 'free-riding' effect, where AI model developers offload safety responsibilities.
- Strict regulation covering the entire AI supply chain can mutually benefit all players, improving both safety and financial returns.
- The situation is described as a classic prisoner's dilemma, where cooperation yields the best outcome for all parties.
The researchers created a theoretical model to test how different regulatory approaches impact the safety of AI products. They divided the AI ecosystem into two main groups: general-purpose AI model developers such as OpenAI, Google, and Anthropic, and downstream domain specialists who apply these models in fields like AI medical diagnostics, e-commerce customer service chatbots, autonomous vehicles, and hiring tools. The model assumes that each group makes strategic decisions about how much to invest in safety measures like third-party audits, red-teaming, and robustness testing.
The Problem With Regulating Only End-Users
At first glance, regulating specific AI use cases might seem logical. Governments can address known risks in high-stakes areas like healthcare, finance, or criminal justice without slowing down fundamental research. However, the study's authors argue that this approach can backfire. When regulators focus exclusively on downstream companies, general-purpose AI developers lower their own safety investment. They expect the downstream specialists to carry the burden of making the final product safe. This free-riding behavior reduces the overall safety level of the AI system, even if the downstream company adds its own safeguards.
Laufer explained in a statement that there is a 'free-riding behavior' that occurs, adding that 'the regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.' The downstream companies, in turn, may not have full visibility into the model's training data, design choices, or known vulnerabilities, making their guardrails incomplete. The result is a final product that appears regulated but is actually less safe than an unregulated one, because no one takes full responsibility for the core model's risks.
The Game Theory of AI Safety
To understand why weak regulation fails, the researchers turned to game theory, specifically the prisoner's dilemma. In this classic model, two rational decision-makers can either cooperate or betray each other. If both cooperate, they achieve the best collective outcome. If both betray, they get a mediocre outcome. But if one cooperates while the other betrays, the cooperator gets the worst possible result. Faced with uncertainty, rational players often choose to betray, leading to an outcome that is worse for everyone than if they had cooperated.
In the context of AI regulation, the general-purpose developer and the downstream specialist face a similar choice. They can each invest heavily in safety, or they can cut corners and rely on the other party to fill the gaps. Without strict regulation that holds both accountable, each party is tempted to freeride. The general-purpose developer assumes the downstream specialist will catch any issues, while the downstream specialist may assume the model is already safe because it comes from a reputable lab. This mutual distrust leads to underinvestment in safety on both sides, creating a product that is more dangerous than one produced under a clear, strict regulatory framework.
However, the study also reveals a more hopeful scenario. When regulation is strong and applies to every actor in the supply chain, it aligns incentives. Both parties are required to meet meaningful safety standards, and they can trust that their counterpart is doing the same. This trust enables cooperation, which improves the safety of the end product and also increases the utility that both parties derive from their investments. Utility in the model is defined as revenue share minus investment cost. So stricter, well-placed regulation does not have to sacrifice profit for safety; it can actually enhance the economic value of AI systems.
The Broader Regulatory Debate
The study arrives at a time when the United States government and Silicon Valley are deeply divided over how to regulate artificial intelligence. Two main camps have emerged. On one side are anti-regulation technologists who favor lighter federal guardrails, largely aligning with the Trump administration's approach to AI governance. This group argues that the AI industry should be free from unnecessary restrictions to innovate as quickly as possible, because that is the only way the United States can win the global AI race against China. They often dismiss stricter safety advocates as 'doomers' or accuse them of engaging in regulatory capture to protect incumbent firms.
On the other side are proponents of stricter federal AI regulation. They claim that the pursuit of wider profit margins leads the AI industry to underestimate or undersell the risks of under-regulated development. These risks include AI psychosis, data center health impacts on local communities, algorithmic bias, and a potential unemployment crisis as AI adoption spreads. The debate has become polarized, but the researchers behind the new study suggest that safety versus revenue is a false dichotomy. Their model indicates that well-designed regulation can benefit all stakeholders, creating an environment where innovation continues without compromising public safety.
Why the Supply Chain Matters
One of the report's central messages is that AI cannot be treated as a single object. As Laufer noted, 'People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology. To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.'
This supply chain includes not only model developers and end-users but also data providers, cloud infrastructure operators, fine-tuning services, and even open-source distributors. Each layer introduces its own risks and opportunities for safety investment. For example, a hospital deploying an AI diagnostic tool relies on the model developer for the core algorithm, the cloud provider for secure hosting, and its own IT team for integration and monitoring. If regulation only addresses the hospital, the other players may have no incentive to improve their safety practices.
Historical and Global Context
The debate over AI regulation is not new. Since the early days of artificial intelligence, researchers have worried about unintended consequences. In the 2010s, the rise of deep learning led to calls for ethical guidelines and impact assessments. The European Union has taken a more proactive approach with the EU AI Act, which categorizes AI applications based on risk and imposes obligations on providers and deployers. The act attempts to cover the full supply chain, requiring foundation model developers to conduct safety evaluations and document their systems. The new study supports this kind of comprehensive approach, suggesting that piecemeal regulations will likely fail.
In the United States, federal action has been limited. Several executive orders and agency guidance documents have encouraged voluntary commitments from leading AI companies, but these lack enforcement power. Some states have introduced their own laws, creating a patchwork of rules that may confuse companies and create loopholes. The study's authors argue that only federal regulation that sets clear, mandatory standards for all AI supply chain participants can resolve the prisoner's dilemma and prevent free-riding.
Potential for Mutual Benefit
The researchers emphasize that strong regulation is not merely a constraint; it can also be a catalyst for innovation. When companies know that their competitors are subject to the same safety rules, they can invest in differentiating themselves through superior safety, rather than cutting corners to reduce costs. This could lead to new markets for safety auditing, interpretability tools, and robust testing services. It also levels the playing field between companies that prioritize safety and those that use lax practices to gain a temporary advantage.
The model shows that a regulatory sweet spot exists when regulators expect both general-purpose AI producers and downstream companies to make sufficient investments to meet meaningful safety standards. At that point, the benefits of cooperation outweigh the temptation to defect. The authors suggest that policymakers should design regulations that are not just strict but also transparent and predictable, so that all actors can plan their investments accordingly.
As the global AI race intensifies, the stakes could not be higher. The United States and other nations must decide whether to pursue a path of minimal oversight or to embrace robust, supply-chain-wide regulation. The new study offers a data-driven argument that weak regulation is the worst of both worlds: it creates an illusion of safety while allowing dangerous gaps to remain. The authors hope that their findings will encourage a more nuanced conversation, one that moves beyond the false choice between innovation and caution and focuses on how to design regulations that genuinely protect the public without stifling progress.
Source: Gizmodo News