The missing half of AI futurism debates

Why we should think a little harder about what it takes to build a Dyson Sphere

This post is part of Epoch AI’s Gradient Updates newsletter, which shares more opinionated or informal takes on big questions in AI progress. These posts solely represent the views of the authors, and do not necessarily reflect the views of Epoch AI as a whole.

---

The Dyson Sphere in the room

If you follow AI discourse, you’ll inevitably run into some version of the following claim: a few years after we automate AI research, the world becomes unrecognizable, with nanotech, Dyson swarms, and near-light-speed spacecraft. For example, Ajeya Cotra argues that:

“[A] vast population of superhuman AI agents will likely invent new technologies radically faster than humans could on our own, including developing even more powerful successors. A year or two into this process, the fast-evolving AI civilization will likely develop truly sci-fi technologies like near-light-speed spacecraft or molecular nanotechnology.”

Where does such a bold claim come from? If you squint, the argument has two parts. First, once we automate all of AI research, AIs will get wildly better very quickly, enough to leave modern-day human experts in the dust. We’ve done (and read) empirical work related to this, and as far as we can tell, this is actually quite plausible — or at least the bottlenecks don’t clearly seem strong enough to prevent this.

The second part of the argument is that with enough of these hyper-proficient AIs, you can build these truly sci-fi technologies in a couple years. Put another way, the primary bottleneck to wild technological progress is just that we don’t have enough really cracked workers. But is that actually true?

We don’t know for sure, but unless AIs can do more or less anything, the timeline to any futuristic technology will depend quite a bit on how hard they are to develop. Regardless of who you are, it’s probably much harder to develop a nuclear weapon than an axe. And yet we rarely see people working out just how hard it is to build Dyson Spheres, nanotech, or super bioweapons. This is the missing half of AI futurism debates: we’ve thought hard about how good the AIs might be, but not about how hard it’ll be to develop specific technologies.

So what should we do? Our answer is that some people should work on a new research direction based on a three-step procedure:

1. Pick a specific technology and define it concretely, like self-replicating interstellar probes: machines that can land in a new star system, build copies of themselves from local materials, and launch those copies onward at a large fraction of light speed.

2. Make some assumptions about AI. For example, we could assume that AIs are capable enough to be “drop-in remote worker replacements”, while needing the runtime compute of an H100 GPU. Or we could specify how many AI workers there are, how fast they can be run, what kinds of physical actuators they have access to, yada yada yada.

3. Estimate how long this AI would take to develop the technology, or what resources it would need.

To be clear, this isn’t the kind of thing that everyone should be thinking about. It’s just that almost nobody’s thinking about it right now, and if the intelligence explosion really might happen (and possibly within a few years), we should probably try a bit harder than that.

Now let’s address the objections…

Objection 1: Haven’t people done this already?

It’s true that people have thought about bottlenecks to the real-world impacts of superintelligence, but they mostly haven’t done so in the way that we’re envisioning.

For example, some people (including us) like to think about AI’s technological impacts in terms of economic metrics. The textbook example is to ask how AGI would impact GDP growth — if it accelerates growth ten-fold, it’d be like compressing a century of technological progress into a decade. The issue is that metrics like GDP might be poor proxies for what we care about,¹ like the timelines to specific technologies that are especially important. For instance, if AGI raised world GDP one hundred-fold, would humans have spread to the stars? Would we be able to upload our brains? Could the AIs develop bioweapons or molecular nanotech to take over the world? These questions matter a ton for human welfare, and yet can be hard to infer (or even largely divorced) from GDP.

Some people have also thought about how fast the physical world could change by looking at growth rates in biological systems, like how fruit fly populations can double in days. This is nice as an existence proof that physical things can double super fast, but who knows if this analogy actually holds for AI? On its own it’s weak evidence and prone to reference class tennis.

The closest literature to what we have in mind is exploratory engineering (a.k.a. scientific roadmapping), where you come up with detailed and plausible engineering pathways to some future technology, like nanotech, neural recording, positional chemistry, or brain emulation. A striking example is what Armstrong and Sandberg did with the replicating probes we mentioned earlier: they took a probe design published in the 1980s,² a procedure for disassembling planets, and worked through the physics to argue that within decades, humanity would be able to launch a project to colonize the entire reachable universe!

The difference between these ideas and our proposal is that we want some people to do exploratory engineering with explicit AI assumptions bolted on. How much sooner could you get nanotechnology with an army of AIs that can match human abilities and work at human speeds? What if they work a hundred times faster?

We can only think of one example of someone doing this publicly, namely Damon Binder’s work on post-AGI energy production. He designs a minimal solar power system that an economy with abundant robot labor could build, and finds it could double on the scale of weeks. In general, we’re envisioning similar analyses which further emphasize assumptions about future AI, and which vary the assumptions to see how the conclusions change.

Objection 2: This won’t tell us anything useful, because the future is too uncertain

Another form of pushback is to say, “exploratory engineering: the trick that never works”. Well we’d disagree, because there are cases where this does work! Here are some examples:

• Rockets: In 1903, Konstantin Tsiolkovsky argued that rockets could achieve the speeds needed for space travel, a whole 41 years before the first rocket went to outer space.³ Essentially, he derived an equation for rocket motion, and calculated that liquid fuels (like liquid Hydrogen and Oxygen) would shoot out of the rocket fast enough to help it escape the Earth’s gravity. In contrast, solid fuels like gunpowder wouldn’t make the cut.

• Satellites: Arthur C. Clarke was about 19 years ahead of the curve on using satellites to reliably communicate information around the world. The idea was to put three satellites in fixed positions in the sky above the equator, which would collectively relay things like radio, TV, and telephone signals with near-global coverage.

Image source: Dreams of Space blog via Arthur C. Clarke’s “The Young Traveler in Space”.

This isn’t to say that the future is certain. Our examples are selected to illustrate our point, and the picture might look far less rosy if we could somehow see all the failed attempts at this kind of futurism. But we shouldn’t be too pessimistic either. If you look at the track record of forecasts from futurists, it really doesn’t look so bad. And in some ways the task we’re talking about is easier — it’s more about showing that there’s a plausible engineering pathway to some futuristic tech given enough resources, rather than saying when exactly it’ll exist.

Objection 3: But it’s really hard to model superintelligence!

A final class of objections goes along the lines of “how can we know the impacts of superintelligence if we don’t know what it’ll look like?” Maybe its capabilities will be spiky in ways that are hard to predict. Or it’ll take galaxy-brained actions that we humans couldn’t possibly wrap our heads around — it’d be like a beginner in chess trying to predict the moves of a grandmaster.

There’s some truth to this, but just because we don’t know exactly what superintelligence would look like doesn’t mean that we can’t get useful insights about it. For example, suppose you had a billion AIs that were each at least as good as top human experts at virtually all cognitive tasks.⁴ If you could show that these AIs could build molecular nanotechnology within a few years, then surely a billion much smarter AIs could too, even if you don’t know how to model them. And if you can show that these AIs probably can’t do this, then at least this helps us identify cruxes, focusing debates on more concrete AI capabilities.

In fact, existing arguments about AI futurism already do this. We don’t know exactly what superintelligence will look like, but we can ask what happens to world GDP if AI can merely match human capabilities. This is how people often argue that advanced AI could lead to over 30% per year growth in world GDP. So these kinds of models and arguments have already proven useful for thinking about explosive economic growth.

What’s more, these influential analyses about explosive growth were done by just a few people. The point of this post is really to say that “more people should do this kind of analysis for specific, high-stakes technologies after AGI”. And if you’re interested in doing this, you should really pounce at this opportunity, because it may be the one case where thinking about constructing Dyson Spheres is actually useful.

---

If you’re interested in doing the sort of work described in this essay, please reach out to js@epoch.ai.

1. Metrics like real GDP also have some pretty bizarre properties — “you could have full automation and really explosive growth in every intuitive sense of the term and yet real GDP growth could go down”.

2. They also looked at other replicators, like those from biology.

3. By “outer space” we mean anything beyond the Kármán line, which is everything with an altitude over 100 km above mean sea level.

4. This is what the AI Futures Project refers to as “Top-human-Expert-Dominating AI (TED-AI)”.