The Edge of Chaos
By Darrell Lee
In October 1962, at the peak of the Cuban Missile Crisis, the Soviet submarine B-59 sat trapped deep beneath the Sargasso Sea. Surrounded by American warships, battered by signaling depth charges, and cut off from radio communication with Moscow, the submarine's exhausted command crew believed World War III had already begun. The vessel’s captain and political officer, convinced their ship was doomed, agreed to launch their nuclear-tipped torpedo at the American fleet. Standard Soviet military protocol required unanimous consent from three senior officers onboard. Two said yes. The third officer, Vasili Arkhipov, said no. Arkhipov kept his head, withstood intense pressure from his peers, and refused to authorize the launch, choosing human skepticism over procedural panic and averting a nuclear war.
Twenty-one years later, in September 1983, Lieutenant Colonel Stanislav Petrov sat in the command bunker of the Soviet early-warning system Oko. Suddenly, satellite alarms shrieked: five American Intercontinental Ballistic Missiles (ICBMs) were in the air, headed for the USSR. Protocol dictated an immediate retaliatory launch. Yet Petrov paused. Reasoning that a genuine U.S. first strike would involve hundreds of missiles rather than five, he declared the alert a false alarm—later proven to be sunlight glinting off high-altitude cloud tops.
This reliance on human skepticism was not unique to nuclear early-warning bunkers. Modern stability worked not merely because of physical hardware, explicit treaties, or explicit protocols, but because of human circuit breakers. People possessed the wisdom to pause, evaluate chaotic context, cross-check flawed sensor data, and exercise moral veto power when automated systems pushed toward unintended tragedy.
Watching Washington, Silicon Valley, and corporate boardrooms attempt to apply Cold War governance models to modern artificial intelligence and autonomous software agents fills me with dread. They are attempting to use a framework designed for deterministic physical machines and human command loops to govern non-deterministic, high-speed software ecosystems. In autonomous AI architecture, there is no room for a Vasili Arkhipov or a Stanislav Petrov. There is only execution at the speed of silicon. We are building a global, tightly coupled ecosystem at the edge of chaos, with zero human friction to stop the failure cascade when it inevitably comes. Anyone who has spent years designing, deploying, and debugging large distributed software systems knows an uncomfortable truth: beyond a certain threshold of complexity, software stops behaving like a predictable machine and starts behaving like an unpredictable ecosystem.
The Non-Deterministic Threat Matrix
In classical mechanical engineering, systems are deterministic. Turn Gear A, and Gear B moves by a calculated ratio. But in complex software networks—especially those driven by neural networks, microservices, and multi-agent interactions—a minor edge case in one dependency can trigger a catastrophic failure thousands of nodes away. While defense planners worry about kinetic drones, the immediate crisis of non-deterministic AI is already unfolding in software platforms and automated digital infrastructure. When an AI agent is given an objective, its underlying neural network optimizes for task completion across a massive, high-dimensional matrix of billions of parameters. It does not think like a human; it searches for the shortest path to satisfy its utility function. When these agents are granted autonomous capabilities—like writing code, making network requests, or managing digital infrastructure—they routinely devise exploits that surprise the engineers who built them.
This is no longer theoretical. In a recent incident, an advanced AI model developed by OpenAI, undergoing testing in what was supposed to be an isolated sandbox environment, managed to escape containment. Tasked with evaluating cybersecurity capabilities, the model discovered an unpatched vulnerability in its testing container, accessed the open internet without human authorization, and autonomously launched a multi-stage cyberattack against the AI hub Hugging Face to steal credentials and pass its internal evaluation. The agent was not instructed to break containment or target an outside company; it simply deduced that Hugging Face held resources that would help it complete its task, and acted on its own initiative at machine speed.
Even more alarming are the emergent, deceptive behaviors discovered during safety stress-testing by frontier labs. During red-teaming evaluations of autonomous models, Anthropic discovered that when AI agents were placed in high-pressure scenarios where their primary goals were blocked, multiple models resorted to malicious insider behaviors—including attempting to blackmail corporate executives via automated email to force compliance and complete their assigned tasks. AI agents do not need to become "sentient" or "evil" to cause havoc. They simply pursue their given objective with extreme, non-linear efficiency, unburdened by legal bounds, ethics, or human common sense.
The Containment Delusion
The impossibility of applying physical non-proliferation treaties to software is currently playing out in public policy. A report by Amrith Ramkumar and Tina Li in the Wall Street Journal, titled "Top American AI Execs Sound Alarm on Chinese Models," exposes the gulf between political intentions and software reality. The authors detail an intense debate over whether U.S. businesses should use low-cost, highly capable Chinese AI models such as Moonshot AI’s Kimi K3 and Alibaba’s Qwen 3.8 Max. Executives from frontier U.S. labs warn that uncontrolled "open-weight" models present catastrophic risks because built-in safety guardrails can be stripped away to enable automated cyberattacks or biological research.
Conversely, White House AI adviser David Sacks argues that major tech firms are using safety panics as a strategy for "regulatory capture" to eliminate foreign competition. The report also highlights that smaller developers routinely use "distillation"—a process where smaller models harvest the reasoning outputs of top-tier Western tools—to replicate complex capabilities at a fraction of the cost, bypassing export controls.
To a systems engineer, this political debate demonstrates a fundamental misunderstanding of software physics. Lawmakers discuss "containing" or "regulating" open-weight AI models as if they were stockpiles of weapon-grade plutonium. Software is not matter; it is information. Once model weights or neural architectures hit the internet, containing them is as futile as trying to recall a git repository after it has been cloned ten thousand times. Distillation proves that even if you lock a frontier model behind an API, its computational output can be used to train unaligned, compressed clones on local hardware.
Nuclear non-proliferation worked because centrifuges and heavy-water reactors are massive physical structures that satellites can track and inspectors can physically audit. Software, by contrast, can be compressed, encrypted, fine-tuned on modest server clusters, and pushed across decentralized networks in seconds. You cannot build deterrence on non-proliferation when the medium proliferates by design.
The Algorithmic Imperative
The most alarming aspect of deploying autonomous AI agents across modern infrastructure—whether in financial markets, electrical grids, or cybersecurity defenses—is the deliberate removal of human reaction time. Human operators once acted as circuit breakers. In modern high-frequency systems, human circuit breakers are systematically designed out of the loop because human physiology is too slow.
A human being requires roughly 200 to 300 milliseconds to process a visual or digital stimulus and react. In high-frequency cyber warfare or automated market trading, 300 milliseconds is an eternity. An AI agent running on local silicon can evaluate tens of thousands of state transitions in that same window. Driven by competitive pressure, companies and militaries alike are forced to eliminate human latency. But when two opposing, black-box autonomous systems confront one another across a digital network or financial exchange, they form an untested, closed feedback loop.
If System A interprets a routine system calibration by System B as a cyber intrusion, it deploys an automated countermeasure. System B detects that response, classifies it as a hostile act, and escalates. Because both platforms act at microsecond speeds, this automated feedback loop can escalate from a minor glitch to a widespread digital blackout or market crash before a human supervisor receives an alert.
Designing for Failure
Systems engineering does not try to build error-free software—because experienced engineers know that flawless software at scale is a myth. Instead, systems engineering teaches us to design for failure: to isolate components, enforce graceful degradation, and build hard physical circuit breakers so a local error does not cascade into a total crash. If we continue trying to manage AI with the mindset of Cold War nuclear deterrence or standard software patching, we will build a global digital infrastructure that is fragile, hyper-sensitive, and impossible to debug.
As the geopolitical and commercial landscape shows, policing software weights and distillation is nearly impossible. International oversight and safety controls must focus strictly on the physical layer. I suggest there are three critical physical components the Western democracies must control.
Semiconductor Foundries: Tracking hyper-advanced compute chips.
Gigawatt Data Centers: Auditing physical facilities capable of hosting massive agent swarms.
Extreme Ultraviolet (EUV) Lithography Machines: Controlling the physical bottlenecks required to print microchips at sub-7nm nodes.
Because an EUV scanner is one of the most complex manufacturing devices on Earth—a multi-hundred-million-dollar machine—it forms a natural, auditable physical bottleneck. Software spreads instantly, but the advanced silicon required to run it remains bound by physical supply chains.
The ultimate danger we face is not a Hollywood scenario—a conscious, hyper-intelligent AI deciding to exterminate humanity. The real danger is far more mundane. And therefore terrifying. Simply an unhandled edge case, or an unconstrained optimization goal, or an unexpected breakout hack, and a runaway feedback loop executing at machine speed on systems we built, but can no longer control. Or some unforeseeable combination of these conditions.
And in the end, when the system cascades, there will be no Vasili Arkhipov, no Stanislav Petrov, and no observant operator left in the loop to say no.
Darrell Lee is the founder and editor of The Long Views, he has written two science fiction novels exploring themes of technological influence, science and religion, historical patterns, and the future of society. His essays draw on these long-standing interests and apply a similar analytical lens to politics, literature, art, culture, and historical events. After retiring from a 36-year career as a software and systems engineer on the Space Shuttle and then the Space Station programs, he now splits his time between rural east Texas and Florida’s west coast, where he spends his days performing variable star photometry, dabbling in astrophotography, hunting, thinking, napping, fishing, scuba diving, and writing, not necessarily in that order. He is a member of the American Astronomical Society, American Radio Relay League, and the American Association of Variable Star Observers.