Three: Carl Shulman on the economy and national security after AGI

The human brain runs on just 20 watts — a fraction of a cent worth of electricity per hour. What would happen if AI could do the same?

Plenty of people have toyed with this question. But perhaps nobody has followed through and considered all the implications as much as Carl Shulman, whose behind-the-scenes work has greatly influenced how leaders in artificial general intelligence (AGI) picture the world they’re creating.

Carl simply follows the logic to its natural conclusions, leading to a world where:

• One cent of electricity buys what costs hundreds of dollars today — medical advice, company management, scientific research — triggering a scramble to manufacture chips and apply them to the most lucrative forms of intellectual labour
• The world’s supply of AI researchers explodes from 10,000 to 10 million or more, enormously accelerating further AI progress
• Companies operated entirely by AIs are much faster and cheaper than those that depend on people for decision making, and humans are progressively driven out of business
• The technical challenges of robotics are rapidly overcome — leading to strong, fast, precise, and tireless robot workers capable of any physical work, and a rush to build billions of them 
• Human population levels become irrelevant to economic growth, which now depends on how quickly machines can replicate their components. Given how quickly complex biological systems can reproduce — some in a matter of days — doubling every few months may be a conservative estimate
• No country can afford not to participate in the economic explosion. Delay, and your rivals’ economies grow 10x, 100x, then 1,000x larger than yours, leaving you ultimately disempowered

As the economy grows, each person could afford the equivalent of a team of hundreds of machine ‘people’ to help them with every aspect of their lives.

But with growth rates this high, it doesn’t take long to reach Earth’s physical limits — the toughest to engineer around being the planet’s ability to release waste heat. If this machine economy and its insatiable demand for power generates more heat than the Earth radiates into space, the planet will rapidly heat up and become uninhabitable for biological organisms.

This eventually creates pressure to move economic activity off-planet. There’s little need for computer chips to be on Earth, and solar energy and minerals are more abundant in space. So you could develop populations of billions of digital scientific researchers orbiting in space, sending the results of their work — like drug designs — back to Earth.

These are just some of the wild implications if AGI could merely match what evolution has already managed.

In this interview with host Rob Wiblin, Carl explains the above, and Rob pushes back on whether that’s realistic or just a cool story:

• If this is where we’re heading, how come economic growth remains slow now and isn’t really increasing?
• Why have computers and computer chips had so little effect on economic productivity so far?
• Are self-replicating biological systems a good comparison for self-replicating machine systems?
• Isn’t this just too crazy and weird to be plausible?
• What bottlenecks might we encounter supplying energy and resources to this growing economy?
• Could there be severely declining returns to bigger ‘brains’ and more training?
• Wouldn’t humanity get scared and pull the brakes if such a transformation kicked off?
• If this is right, why don’t economists agree?

In the last section of the episode, Carl addresses the moral status of machine minds themselves. Would they be conscious or otherwise have a claim to moral rights? And how might humans and machines coexist with neither side dominating or exploiting the other?
Learn more and read the full transcript on the 80,000 Hours website.

This episode is the first part of Rob’s marathon interview with Carl Shulman in 2024. The second episode is on government and society after AGI, and you can listen to them in either order.

Chapters:

• Cold open (00:00:00)
• Rob’s intro (00:01:00)
• Transitioning to a world where AI systems do almost all the work (00:05:21)
• Economics after an AI explosion (00:14:25)
• Objection: Shouldn’t we be seeing economic growth rates increasing today? (00:59:12)
• Objection: Speed of doubling time (01:07:33)
• Objection: Declining returns to increases in intelligence? (01:11:59)
• Objection: Physical transformation of the environment (01:17:39)
• Objection: Should we expect an increased demand for safety and security? (01:29:14)
• Objection: “This sounds completely whack” (01:36:10)
• Income and wealth distribution (01:48:02)
• Economists and the intelligence explosion (02:13:31)
• Baumol effect arguments (02:19:12)
• Denying that robots can exist (02:27:18)
• Classic economic growth models (02:36:12)
• Robot nannies (02:48:27)
• Slow integration of decision-making and authority power (02:57:39)
• Economists’ mistaken heuristics (03:01:07)
• Moral status of AIs (03:11:45)
• Rob’s outro (04:11:47)

Producer and editor: Keiran Harris
Audio engineering lead: Ben Cordell
Technical editing: Simon Monsour, Milo McGuire, and Dominic Armstrong
Transcriptions: Katy Moore