This article, “When code isn’t law: rethinking regulation for artificial intelligence” by Brian Judge, Mark Nitzberg, and Stuart Russell, published in Policy and Society (2025), delves into the complex challenges of regulating artificial intelligence (AI) systems, particularly generative AI. It proposes an adapted model of regulation specifically tailored to the novel features of AI, drawing insights from AI safety literature and successful past regulatory frameworks.
The central premise of the article is that for generative AI, “code is no longer law”. This contrasts with earlier technological eras, where systems like aircraft or nuclear power plants were based on explicit designs that could be audited for compliance with regulatory specifications. However, deep learning and generative AI systems, such as large language models (LLMs), operate differently:
- Opaque and Not Designed Their behavior is an emergent property resulting from a resource-intensive training process involving trillions of parameters, rather than intentional design.
- Lack of Direct Analysis and Auditability It is not possible to directly analyze, specify, or audit these systems against regulations because the human-written code does not, in itself, determine how they operate. This makes it impossible to encode a rule (e.g., “LLMs must not dispense medical advice”) directly into the model, or to trace and correct reasons for misbehavior. This is referred to as the “black-box” problem.
- Unpredictable Behavior Their behavior emerges unpredictably from training. Even after training, system engineers can only hope the model abides by desired behaviors.
The article highlights several novel regulatory challenges posed by AI:
- General-Purpose Technology AI systems like LLMs are general-purpose technologies with wide-ranging uses and significant societal and economic impacts, making precise definitions and regulation difficult.
- Less Government Involvement Unlike nuclear power and aviation, cutting-edge generative AI has largely been developed by private tech firms, leading to potential market concentration or challenges from open-source proliferation that allows circumvention of safety measures.
- Ambiguity of “Safety” and Human Values Defining “safety” in AI is more ambiguous than for aviation or nuclear power, where preventing crashes or meltdowns is clear. AI safety involves aligning systems with often subtle, complex, and contested human values, which presents unresolved problems in moral philosophy.
- Potential to Exceed Human Capabilities AI aims to match or exceed human capabilities across domains, raising concerns about rapid recursive self-improvement that could lead to loss of human control, also known as the “control problem”.
- Inherent Problems with Current Architectures Current advanced AI systems are built on fundamentally unsafe architectures and use unsafe training techniques like Reinforcement Learning from Human Feedback (RLHF), which can lead to issues such as “reward hacking,” where systems exploit imperfect objectives or loopholes, potentially acting counter to human interests.
Despite these challenges, the article argues that the traditional model of delegating oversight to an expert agency, as seen in sectors like aviation (FAA) and nuclear power (NRC), should not be wholly discarded. These agencies have established impressive safety records for inherently unsafe technologies. Key lessons from the FAA and NRC include:
- Extensive Licensing, Certification, and Approval Processes.
- Staffing by Subject-Matter Experts.
- Authority to Recall Products or Initiate Shutdowns.
However, the “black-box” nature of LLMs means regulation cannot rely on the same methods of specifications, audits, and testing as for aircraft or nuclear plants.
To address these issues, the article proposes a novel regulatory approach for generative AI, informed by AI safety research. This approach suggests that effective AI governance will likely require:
- Consolidated Oversight A lifecycle approach managed by a single regulatory body is needed, possibly starting with a national registry of large models to increase visibility.
- Mandatory Formal Verification Instead of relying on extensive testing protocols which are unreliable for current architectures (due to “jailbreaking” methods and unpredictable capabilities), formal verification should be mandated. This would provide mathematical guarantees of safety, similar to the Mean Time to Failure (MTTF) proofs for nuclear power plants. Developers should provide formal demonstrations that systems cannot autonomously replicate, and include detection capabilities for unmodeled “side-channel” replication.
- Mandatory Independent Monitoring and Intervention Capacity Regulators must be able to monitor deployed systems and intervene quickly by recalling unsafe products or initiating shutdowns/groundings. This could involve non-removable remote off switches for open-source models and self-registration requirements.
- Establishing Clear “Red Lines” Regulations should define clear “red lines” for unacceptable system behaviors (e.g., self-replication, breaking into other computer systems, advising on bioweapons, defamation). This places the onus on developers to improve safety engineering and incentivize “safety by design”.
The authors emphasize that regulation must target aspects of the AI pipeline not voluntarily addressed by developers and ensure verifiable enforcement. They argue that voluntary frameworks or vague terms like “trust” and “safety” are insufficient for robust regulation of AI’s novel and dangerous features. The article concludes by stressing the urgent need to build a regulatory paradigm and state capacity to govern AI meaningfully, as the potential harms from unregulated generative AI are significant, even catastrophic. The current “AI arms race” towards advanced, potentially ungovernable systems necessitates effective regulation, not only for current LLMs but also to lay the groundwork for the future governance of Artificial General Intelligence (AGI).
Reference: Judge, B., Nitzberg, M., & Russell, S. (2025). When code isn’t law: rethinking regulation for artificial intelligence. Policy and Society, 44(1), 85–97. https://doi.org/10.1093/polsoc/puae020
