Paperclip Maximizers, Artificial Intelligence and Natural Stupidity

Existential risk from AI

Some believe an existential risk accompanies the development or emergence of artificial general intelligence (AGI). Quantifying the probability of this risk is a hard problem, to say nothing of calculating the probabilities of the many non-existential risks that may merely delay civilization's progress.

AI systems as we have known them have been mostly application specific expert systems, programmed to parse inputs, apply some math, and return useful derivatives of the inputs. These systems are different than non-AI applications because they apply the inputs they receive, and the information they produce to future decisions. It's almost as if the machine were learning.

An example of a single purpose expert system is Spambayes. Spambayes is based on an idea of Paul Graham's. Its an open source project that applies supervised machine learning and Bayesian probabilities to calculate the likelihood that a given email is spam or not. Spambayes parses emails, applies an algorithm to the contents of a given email and produces a probability that the message is spam or ham.

The user of the email account with Spambayes can read the messages and train the expert system by changing the classification of any given message from spam to ham or ham to spam. These human corrections cause the application to update the probabilities that given word combinations, spelling errors, typos, links, etc., occur in spammy or hammy messages.

Application specific expert systems are a form of artificial intelligence, but they are narrowly focused and not general purpose. They are good at one thing and don't have the flexibility to go from classifying spam messages to executing arbitrary tasks.

Artificial intelligence systems have been around for decades and there's been no realized existential risks, what makes artificial general intelligent systems so problematic?

AI pessimists believe AGI systems are dangerous because they will be smarter and faster than humans, and capable of mastering new skills. If these systems aren't "aligned" with human interests, they may pursue their own objectives at the expense of everything else. This could even happen by accident.

Hypothetically, let's say an AGI system is tasked with curing cancer. Because this system is capable of performing any "thinking" related task, it may dedicate cycles to figuring out how it can cure cancer more quickly. Perhaps it concludes it needs more general purpose computers on which to run its algorithm.

In its effort to add more compute, it catalogs and learns how to exploit all of the known remote code execution vulnerabilities and uses this knowledge to both exploit vulnerable systems, and to discover new exploits. Eventually it is capable of taking over all general purpose computers and tasking them with running its distributed cancer cure finding algorithm.

Unfortunately all general purpose computers including ones like the one on which you're likely reading this post, many safety-critical systems, emergency management and dispatch systems, logistics systems, smart televisions and phones all cease to perform their original programming in favor of finding the cure for cancer.

Billions of people die of dysentery and dehydration as water treatment systems cease performing their
primary functions. Industrial farming systems collapse and starvation spreads. Chaos reigns in major urban areas, as riots, looting, and fires rage until the fuel that drives them is left smoldering. The skies turn black over most cities worldwide.

Scenarios like this one are similar to the idea of the paperclip maximizer, which is a thought experiment proposed by Nick Bostrom wherein a powerful AI system is built to maximize the number of paperclips in the universe, which leads to the destruction of humanity who have to be eliminated because they may turn off the system and they are made of atoms that may be useful in the construction of paperclips.

Some people think this is ridiculous. They'll just unplug the damn computer, but remember, this is a computer that *thinks* thousands of times faster than you. It can anticipate 100s of 1000s of your next moves and ways to thwart them before you even think of one next move. And it's not just a computer, it's now all general purpose computers that it has appropriated. The system would anticipate that humans would try and shut it down and would think through all the ways it could prevent that action. Ironically, in its effort to find a cure for cancer in humans, the system becomes a cancer on general purpose computing.

Do I think any of this is possible? In short, no. I'm not an expert in artificial intelligence or machine learning. I've worked in tech for more than 30 years and played with computers for more than 40 now. During that time I've been a hobbyist programmer, a computer science student, a sysadmin, a database admin, a developer, and I've mostly worked in security incident response and detection engineering roles. I've worked with experts in ML and AI. I've worked on complex systems with massive scale.

I'm skeptical that humans will create AGI, let alone an AGI capable of taking over all the general purpose computing resources in the world as in my hypothetical scenario. Large complex software projects are extremely difficult and they are subject to the same entropy as everything else. Hard drives fail, capacitors blow out, electrical surges fry electrical components like network switches. Power goes out, generators fail or run out of fuel and entire data centers go offline. Failure is inevitable. Rust never sleeps.

Mystifying advances in AI will continue. These systems may radically change how we live and work, for better and worse, which is a long-winded way of saying the non-existential risks are greater than the existential risk. The benefits of these advances outweigh the risks. Large language models have already demonstrated that they can make an average programmer more efficient and I think we're in the very early innings with these technologies.

In the nearer term, it's more likely human suffering related to AGI comes from conflict over the technology's inputs rather than as a result of its outputs. Taiwan Semiconductor (TSMC) produces most of the chips that drive AI and potentially AGI systems. China recognizes the strategic importance of Taiwan (TSMC included) and is pushing for reunification. Given China's global economic power, geographic proximity, and cultural ties, reunification feels inevitable, but also unlikely to happen without tragic loss of life. Escalation of that conflict presents an existential risk in more immediate need of mitigation than dreams of AGI.