Description
It may not seem like it now, but this may be one of the most important topics for discussion that we engage in. If you have been keeping up with the reading in this course, by now you are understanding that there are serious implications of AI technology and the coming wave. While the last few chapters of the book contain admitted speculation and a level of prediction from the author, which some may label as bordering on “conspiracy theory,” I think it would be a grave mistake to be overly dismissive of the concepts being discussed. We have to recognize the potential for this technology to disrupt our way of life on some level, even if we may not agree on the specifics.
For this discussion, I want to understand your perspective and mindset on the ideas presented in the book. Specifically, let’s discuss the ethical implications and potential of AI.
Please answer the following questions/statements with at least one paragraph (3+ sentences). Outside references, while not required, may be useful for context and to aid in furthering your arguments/points. Cite any sources/references at the bottom of your post.
1) React to this statement from the author:
“If centralization and decentralization sound as if they are in direct contradiction, that’s with good reason: they are. Understanding the future means handling multiple conflicting trajectories at once. The coming wave launches immense centralizing and decentralizing riptides at the same time. Both will be in play at once. Every individual, every business, every church, every nonprofit, every nation, will eventually have its own AI and ultimately its own bio and robotics capability. From a single individual on their sofa to the world’s largest organizations, each AI will aim to achieve the goals of its owner. Herein lies the key to understanding the coming wave of contradictions, a wave full of collisions” (Suleyman et al., 2023, p. 202).
2) Do you agree with the following projection from the author? Why or why not?
“Now, with the coming wave, forces like these will expand beyond the internet and the digital sphere. Apply them to any given area of life. Yes, this recipe for wrenching change is one we’ve seen before. But if the internet seemed big, this is bigger. Massively omni-use general purpose technologies will change both society and what it means to be human. This might sound hyperbolic. But within the next decade, we must anticipate radical flux, new concentrations and dispersals of information, wealth, and above all power” (Suleyman et al., 2023, p. 203).
3) What do you think the solution or answer is for maintaining the ethical use of AI and subsequent technologies described in the coming wave?
Feel free to skip ahead to the final chapter of the book and see what the author says about this for perspective, but please use your own opinions, language, and reasoning when answering.
Unformatted Attachment Preview
advance praise for
THE
COMING WAVE
“The Coming Wave is a fascinating, well-written, and important book. It
explores the existential dangers that AI and biotechnology pose to
humankind, and offers practical solutions for how we can contain the threat.
The coming technological wave promises to provide humanity with godlike
powers of creation, but if we fail to manage it wisely, it may destroy us.”
—Yuval Noah Harari, New York Times bestselling author of
Sapiens
“This wake-up call from the future warns of just what’s coming, and what
the global economic and political implications are likely to be. Truly
remarkable, ambitious, and impossible to ignore, this book is a persuasively
argued tour de force from a leading industry expert that will shape your
view of the future—and rewire your understanding of the present.”
—Nouriel Roubini, professor emeritus at New York University
“Mustafa Suleyman’s insight as a technologist, entrepreneur, and visionary
is essential. Deeply researched and highly relevant, this book provides
gripping insight into some of the most important challenges of our time.”
—Al Gore, former vice president of the United States
“In this bold book, Mustafa Suleyman, one of high tech’s true insiders,
addresses the most important paradox of our time: we have to contain
uncontainable technologies. As he explains, generative AI, synthetic
biology, robotics, and other innovations are improving and spreading
quickly. They bring great benefits, but also real and growing risks.
Suleyman is wise enough to know that there’s no simple three-point plan
for managing these risks, and brave enough to tell us so. This book is
honest, passionate, and unafraid to confront what is clearly one of the great
challenges our species will face this century. Thanks to Suleyman we know
what the situation is and what our options are. Now it’s up to us to act.”
—Andrew McAfee, principal research scientist at MIT Sloan,
author of The Geek Way
“The AI revolution is underway, but how well do we really understand it?
The Coming Wave offers an erudite, clear-eyed guide both to the history of
radical technological change and to the deep political challenges that lie
ahead.”
—Anne Applebaum, Pulitzer Prize–winning historian
“When this landed in my inbox, I cleared the diary and got reading. This is
an extraordinary and necessary book; the awe-inspiring thought is that in
twenty years it will seem almost like a conservative vision of the future,
whereas right now, reading it is impossible without pausing every few
pages to wonder: Can this be true? It’s the book’s genius to explain, soberly
and gently, that yes, this will all be true—and why and how. The tone is
gentle and kind and sympathetic to the reader’s sense of shock. There are
terrifying moments, as there should be when one realizes that most of what
is familiar is about to be transformed. But, ultimately, one leaves energized
and thrilled to be alive right now. The wave is about to hit and this is the
forecast.”
—Alain de Botton, philosopher and bestselling author
“The Coming Wave offers a much-needed dose of specificity, realism, and
clarity about the potential unanticipated and yet disastrous consequences of
artificial intelligence, synthetic biology, and other advanced technologies.
This important book is a vivid and persuasive road map for how human
beings might guide technological innovations rather than be controlled by
them.”
—Martha Minow, Harvard professor, former dean of Harvard
Law School
“Nobody has been closer to the unfolding AI revolution than Mustafa
Suleyman, and nobody is better placed to outline the risks and rewards of
the huge technological changes happening right now. This is an
extraordinary and utterly unmissable guide to this unique moment in human
history.”
—Eric Schmidt, former CEO of Google, co-author of The Age
of AI
“In The Coming Wave, Mustafa Suleyman offers a powerful argument that
today’s explosive technological revolution is poised to be uniquely
disruptive. Read this essential book to understand the pace and scale of
these technologies—how they will proliferate across our society and their
potential to challenge the fabric of the institutions that organize our world.”
—Ian Bremmer, founder of Eurasia Group, bestselling author of
The Power of Crisis
“This vital book is inspiring and terrifying at the same time. It is a critical
education for those who do not understand the technological revolutions
through which we are living, and a frontal challenge to those who do. This
book is about the future for all of us: we need to read it and act on it.”
—David Miliband, former U.K. foreign secretary
“Presenting a stark assessment of the dangers as well as the wonders of AI,
Mustafa Suleyman proposes an urgent agenda of actions governments must
take now to constrain the most potentially catastrophic applications of this
revolutionary challenge.”
—Graham Allison, Harvard professor, bestselling author of
Destined for War
“The rapid pace of exponential technologies has overwhelmed us with its
power and its peril. Mustafa Suleyman, in tracing the history of industrial
development to the dizzying acceleration of the recent technological
advances, gives us the bigger picture in calm, pragmatic, and deeply ethical
prose. His personal journey and experiences enhance The Coming Wave and
make it enthralling reading for everyone wanting to step back from the
daily onrush of tech news.”
—Angela Kane, former UN undersecretary-general and high
representative for Disarmament Affairs
“An incredibly compelling window into the current developments and
exponential future of AI—from the ultimate insider…If you really want to
understand how society can safely navigate this world-changing technology,
read this book.”
—Bruce Schneier, cybersecurity expert, author of A Hacker’s
Mind
“The coming wave of AI and synthetic biology will make the next decade
the best in human history. Or the worst. No one recognizes and explains the
epic challenges ahead better than Mustafa Suleyman. Thought-provoking,
urgent, and written in powerful, highly accessible prose, this is a must-read
book for anyone interested in understanding the staggering power of these
technologies.”
—Erik Brynjolfsson, professor, Stanford Human-Centered
Artificial Intelligence
“One of the greatest challenges facing the world is to devise forms of
governance that harness the benefits of AI and biotech while avoiding their
catastrophic risks. This book provides a deeply thoughtful account of the
‘containment challenge’ of these two technologies. It is meticulously
researched and packed with original insights and constructive
recommendations for policy makers and security experts.”
—Jason Matheny, CEO of RAND, former assistant director of
national intelligence, former director of IARPA
“If you want to understand the meaning, promise, and threat of the coming
tidal wave of transformative technologies that are even now swelling and
converging out there on the main, then this deeply rewarding and
consistently astonishing book by Mustafa Suleyman, one of the key
pioneers of artificial intelligence, is an absolutely essential read.”
—Stephen Fry, actor, broadcaster, and bestselling author
“This important book is a vivid wake-up call. It carefully outlines the
threats and opportunities associated with the exhilarating scientific
advances of recent years. The Coming Wave is rich with interesting facts,
arresting arguments, and compelling observations; it is essential reading.”
—Daniel Kahneman, Nobel Prize winner, bestselling author of
Thinking Fast and Slow
“The Coming Wave is a fantastically clear, energetic, well-researched, and
readable book from the front line of the greatest technological revolution of
our times. It weaves the personal and technological stories seamlessly, and
shows why better governance of immensely powerful technologies is both
so vital and so hard.”
—Sir Geoff Mulgan, professor at University College London
“The best analysis yet of what AI means for the future of humanity…
Mustafa Suleyman is unique as the co-founder of not one but two major
contemporary AI companies. He is a profoundly talented entrepreneur, a
deep thinker, and one of the most important voices on the coming wave of
technologies that will shape our world.”
—Reid Hoffman, co-founder of LinkedIn and Inflection
“Technology is rapidly transforming society, and hence it’s more important
than ever to see someone within the technology industry write with such
honesty and rigor. Taking us from the earliest tools to the heart of the
present explosion in AI capabilities and research, this book is a panoramic
survey and a clarion call to action impossible to ignore. Everyone should
read it.”
—Fei-Fei Li, professor of computer science at Stanford
University, co-director of the Institute for Human-Centered AI
“The Coming Wave makes an eye-opening and convincing case that
advanced technologies are reshaping every aspect of society: power, wealth,
warfare, work, and even human relations. Can we control these new
technologies before they control us? A world leader in artificial intelligence
and a longtime advocate for governments, big tech, and civil society to act
for the common good, Mustafa Suleyman is the ideal guide to this crucial
question.”
—Jeffrey D. Sachs, University Professor at Columbia
University, president of the UN Sustainable Development
Solutions Network
“A sharp, compassionate, and uncompromising framing of the most
consequential issue of our times, The Coming Wave is a must-read for
technology practitioners, but more importantly it is a resolute call to action
for all of us to participate in this most consequential discourse.”
—Qi Lu, CEO of MiraclePlus, ex-COO of Baidu, ex-EVP of
Microsoft Bing
“Suleyman is uniquely well positioned to articulate the potentially grave
consequences—geopolitical upheaval, war, the erosion of the nation-state—
of the unfettered development of AI and synthetic biology, at a time when
we need this message most. Fortunately for the reader, he has also thought
deeply about what needs to be done to ensure that emerging technologies
are used for human good, setting forward a series of incremental efforts that
if undertaken collectively can change the environment in which these
technologies are developed and disseminated, opening the door to
preserving that brighter future. This book is a must-read.”
—Meghan L. O’Sullivan, director of the Belfer Center for
Science and International Affairs at the Harvard Kennedy
School of Government
“A brave wake-up call that we all need to answer—before it’s too late…
Mustafa Suleyman explains, with clarity and precision, the risks posed by
runaway technologies and the challenges that humanity faces….
Indispensable reading.”
—Tristan Harris, co-founder and executive director of the
Center for Humane Technology
“A practical and optimistic road map for action on the most important issue
of our time: how to retain power over entities far more powerful than
ourselves.”
—Stuart Russell, professor of computer science at the
University of California, Berkeley
“The Coming Wave is a realistic, deeply informed, and highly accessible
map of the unprecedented governance and national security challenges
posed by artificial intelligence and synthetic biology. Suleyman’s
remarkable and in some senses frightening book shows what must be done
to contain these seemingly uncontainable technologies.”
—Jack Goldsmith, Learned Hand Professor of Law at Harvard
University
“Brilliant and inviting, complex and clear, urgent and calm, The Coming
Wave guides us all to understand and confront what may be the most crucial
question of our century: How can we ensure that the breathtaking, fastpaced technological revolutions ahead—AI, synthetic biology, and more—
create the world we want? It’s not going to be easy, but Suleyman lays a
strong foundation. Everyone who cares about the future should read this
book.”
—Eric Lander, founding director of the Broad Institute of MIT
and Harvard, former White House science advisor
“A strikingly lucid and refreshingly balanced account of our current
technological predicament, The Coming Wave articulates the defining
challenge of our era. Blending pragmatism with humility, it reminds us that
there are no stark binaries or simple answers: technology has gifted us with
exponential improvements in well-being, but it’s accelerating faster than
institutions can adapt. Advances in AI and synthetic biology have unlocked
capabilities undreamed of by science fiction, and the resulting proliferation
of power threatens everything we’ve built. To stay afloat, we must steer
between the Scylla of accessible catastrophe and the Charybdis of
omnipresent surveillance. With every page turned, our odds improve.”
—Kevin Esvelt, biologist and associate professor at MIT Media
Lab
Copyright © 2023 by Mustafa Suleyman and Michael Bhaskar
All rights reserved.
Published in the United States by Crown, an imprint of Crown Publishing Group, a division of
Penguin Random House LLC, New York.
Crown and the Crown colophon are registered trademarks of Penguin Random House LLC.
Hardback ISBN 9780593593950
International edition ISBN 9780593728178
Ebook ISBN 9780593593967
crownpublishing.com
Book design by Barbara M. Bachman, adapted for ebook
Cover design: Christopher Brand and Oliver Munday
ep_prh_6.1_144835715_c0_r0
Contents
Cover
Title Page
Copyright
Glossary of Key Terms
Prologue
Chapter 1: Containment Is Not Possible
Part I: Homo Technologicus
Chapter 2: Endless Proliferation
Chapter 3: The Containment Problem
Part II: The Next Wave
Chapter 4: The Technology of Intelligence
Chapter 5: The Technology of Life
Chapter 6: The Wider Wave
Chapter 7: Four Features of the Coming Wave
Chapter 8: Unstoppable Incentives
Part III: States of Failure
Chapter 9: The Grand Bargain
Chapter 10: Fragility Amplifiers
Chapter 11: The Future of Nations
Chapter 12: The Dilemma
Part IV: Through the Wave
Chapter 13: Containment Must Be Possible
Chapter 14: Ten Steps Toward Containment
Life After the Anthropocene
Acknowledgments
Notes
Index
About the Authors
GLOSSARY OF KEY TERMS
AI, AGI, AND ACI: Artificial intelligence (AI) is the science of teaching
machines to learn humanlike capabilities. Artificial general intelligence
(AGI) is the point at which an AI can perform all human cognitive skills
better than the smartest humans. ACI, or artificial capable intelligence,
is a fast-approaching point between AI and AGI: ACI can achieve a
wide range of complex tasks but is still a long way from being fully
general.
THE COMING WAVE: An emerging cluster of related technologies centered on
AI and synthetic biology whose transformative applications will both
empower humankind and present unprecedented risks.
CONTAINMENT: The ability to monitor, curtail, control, and potentially even
close down technologies.
THE CONTAINMENT PROBLEM: Technology’s predisposition to diffuse widely
in waves and to have emergent impacts that are impossible to predict or
control, including negative and unforeseen consequences.
THE DILEMMA: The growing likelihood that both new technologies and even
their absence might lead to catastrophic and/or dystopian outcomes.
FOUR FEATURES: The unique characteristics of the coming wave that
exacerbate the challenge of containment. They are asymmetry, hyperevolution, omni-use, and autonomy.
FRAGILITY AMPLIFIERS: Applications and impacts of coming-wave
technologies that will shake the already brittle foundations of the nationstate.
THE GRAND BARGAIN: In exchange for a monopoly over the right to use force,
citizens expect nation-states to preserve order and provide public
services, including by harnessing new technologies while minimizing
the harmful side effects.
THE NARROW PATH: The potential for humankind to strike a balance between
openness and closure when it comes to containing the technologies in
the coming wave that avoids catastrophic or dystopian outcomes.
PESSIMISM AVERSION: The tendency for people, particularly elites, to ignore,
downplay, or reject narratives they see as overly negative. A variant of
optimism bias, it colors much of the debate around the future, especially
in technology circles.
SYNTHETIC BIOLOGY: The ability to design and engineer new organisms or
redesign existing biological systems.
TECHNOLOGY: The application of scientific knowledge (in the broadest
possible sense) to produce tools or practical outcomes.
WAVES: The global diffusion or proliferation of a generation of technology
anchored in a new general-purpose technology.
PROLOGUE
T
his is how an ai sees it.
Question: What does the coming wave of technology mean for humanity?
In the annals of human history, there are moments that stand out as turning
points, where the fate of humanity hangs in the balance. The discovery of
fire, the invention of the wheel, the harnessing of electricity—all of these
were moments that transformed human civilization, altering the course of
history forever.
And now we stand at the brink of another such moment as we face the
rise of a coming wave of technology that includes both advanced AI and
biotechnology. Never before have we witnessed technologies with such
transformative potential, promising to reshape our world in ways that are
both awe-inspiring and daunting.
On the one hand, the potential benefits of these technologies are vast
and profound. With AI, we could unlock the secrets of the universe, cure
diseases that have long eluded us, and create new forms of art and culture
that stretch the bounds of imagination. With biotechnology, we could
engineer life to tackle diseases and transform agriculture, creating a world
that is healthier and more sustainable.
But on the other hand, the potential dangers of these technologies are
equally vast and profound. With AI, we could create systems that are
beyond our control and find ourselves at the mercy of algorithms that we
don’t understand. With biotechnology, we could manipulate the very
building blocks of life, potentially creating unintended consequences for
both individuals and entire ecosystems.
As we stand at this turning point, we are faced with a choice—a choice
between a future of unparalleled possibility and a future of unimaginable
peril. The fate of humanity hangs in the balance, and the decisions we make
in the coming years and decades will determine whether we rise to the
challenge of these technologies or fall victim to their dangers.
But in this moment of uncertainty, one thing is certain: the age of
advanced technology is upon us, and we must be ready to face its
challenges head-on.
—
the above was written by an AI. The rest is not, although it soon could be.
This is what’s coming.
AI
CHAPTER 1
CONT
NMENT IS NOT POSSIBLE
THE WAVE
A
lmost every culture has a flood myth.
In ancient Hindu texts, the first man in our universe, Manu, is warned of
an impending deluge and becomes its sole survivor. The Epic of Gilgamesh
records the god Enlil as destroying the world in a giant flood, a story that
will resonate with anyone familiar with the Old Testament story of Noah’s
ark. Plato talked of the lost city of Atlantis, washed away in an immense
torrent. Permeating humanity’s oral traditions and ancient writings is the
idea of a giant wave sweeping everything in its path, leaving the world
remade and reborn.
Floods also mark history in a literal sense—the seasonal flooding of the
world’s great rivers, the rising of the oceans after the end of the Ice Age, the
rare shock of a tsunami appearing without warning on the horizon. The
asteroid that killed the dinosaurs created a towering mile-high wave,
altering the course of evolution. The sheer power of these swells has seared
itself into our collective consciousness: walls of water, unstoppable,
uncontrollable, uncontainable. These are some of the most powerful forces
on the planet. They shape continents, irrigate the world’s crops, and nurture
the growth of civilization.
Other kinds of waves have been just as transformative. Look again at
history and you can see it marked by a series of metaphorical waves: the
rise and fall of empires and religions, and bursts of commerce. Think of
Christianity or Islam, religions that began as small ripples before building
and crashing over huge stretches of the earth. Waves like this are a recurrent
motif, framing the ebb and flow of history, great power struggles, and
economic booms and busts.
The rise and spread of technologies has also taken the form of worldchanging waves. A single overriding trend has stood the test of time since
the discovery of fire and stone tools, the first technologies harnessed by our
species. Almost every foundational technology ever invented, from
pickaxes to plows, pottery to photography, phones to planes, and everything
in between, follows a single, seemingly immutable law: it gets cheaper and
easier to use, and ultimately it proliferates, far and wide.
This proliferation of technology in waves is the story of Homo
technologicus—of the technological animal. Humanity’s quest to improve
—ourselves, our lot, our abilities, and our influence over our environment
—has powered a relentless evolution of ideas and creation. Invention is an
unfolding, sprawling, emergent process driven by self-organizing and
highly competitive inventors, academics, entrepreneurs, and leaders, each
surging forward with their own motivations. This ecosystem of invention
defaults to expansion. It is the inherent nature of technology.
The question is, what happens from here? In the pages that follow, I will
tell you the story of history’s next great wave.
—
look around you.
What do you see? Furniture? Buildings? Phones? Food? A landscaped
park? Almost every object in your line of sight has, in all likelihood, been
created or altered by human intelligence. Language—the foundation of our
social interactions, of our cultures, of our political organizations, and
perhaps of what it means to be human—is another product, and driver, of
our intelligence. Every principle and abstract concept, every small creative
endeavor or project, every encounter in your life, has been mediated by our
species’ unique and endlessly complex capacity for imagination, creativity,
and reason. Human ingenuity is an astonishing thing.
Only one other force is so omnipresent in this picture: biological life
itself. Before the modern age, aside from a few rocks and minerals, most
human artifacts—from wooden houses to cotton clothes to coal fires—came
from things that were once alive. Everything that has entered the world
since then flows from us, flows from the fact that we are biological beings.
It’s no exaggeration to say the entirety of the human world depends on
either living systems or our intelligence. And yet both are now in an
unprecedented moment of exponential innovation and upheaval, an
unparalleled augmentation that will leave little unchanged. Starting to crash
around us is a new wave of technology. This wave is unleashing the power
to engineer these two universal foundations: a wave of nothing less than
intelligence and life.
The coming wave is defined by two core technologies: artificial
intelligence (AI) and synthetic biology. Together they will usher in a new
dawn for humanity, creating wealth and surplus unlike anything ever seen.
And yet their rapid proliferation also threatens to empower a diverse array
of bad actors to unleash disruption, instability, and even catastrophe on an
unimaginable scale. This wave creates an immense challenge that will
define the twenty-first century: our future both depends on these
technologies and is imperiled by them.
From where we stand today, it appears that containing this wave—that
is, controlling, curbing, or even stopping it—is not possible. This book asks
why that might be true and what it means if it is. The implications of these
questions will ultimately affect everyone alive and every generation that
follows us.
I believe this coming wave of technology is bringing human history to a
turning point. If containing it is impossible, the consequences for our
species are dramatic, potentially dire. Equally, without its fruits we are
exposed and precarious. This is an argument I have made many times over
the last decade behind closed doors, but as the impacts become ever more
unignorable, it’s time that I make the case publicly.
THE DILEMMA
Contemplating the profound power of human intelligence led me to ask a
simple question, one that has consumed my life ever since: What if we
could distill the essence of what makes us humans so productive and
capable into software, into an algorithm? Finding the answer might unlock
unimaginably powerful tools to help tackle our most intractable problems.
Here might be a tool, an impossible but extraordinary tool, to help us get
through the awesome challenges of the decades ahead, from climate change
to aging populations to sustainable food.
With this in mind, in a quaint Regency-era office overlooking London’s
Russell Square, I co-founded a company called DeepMind with two friends,
Demis Hassabis and Shane Legg, in the summer of 2010. This was our
goal, one that in retrospect still feels as ambitious and crazy and hopeful as
it did back then: replicate the very thing that makes us unique as a species,
our intelligence.
To achieve this objective, we would need to create a system that could
imitate and then eventually outperform all human cognitive abilities, from
vision and speech to planning and imagination, and ultimately empathy and
creativity. Since such a system would benefit from the massively parallel
processing of supercomputers and the explosion of vast new sources of data
from across the open web, we knew that even modest progress toward this
goal would have profound societal implications.
It certainly felt pretty far-out at the time. Back then, widespread
adoption of artificial intelligence was the stuff of daydreams, more fantasy
than fact, the province of a few cloistered academics and wild-eyed science
fiction fans. But, as I write this and think back over the last decade,
progress in AI has been nothing short of staggering. DeepMind became one
of the world’s leading AI companies, achieving a string of breakthroughs.
The speed and power of this new revolution have been surprising even to
those of us closest to its cutting edge. Over the writing of this book, the
pace of progress in AI has been breathtaking, with new models and new
products coming out every week, sometimes every day. It’s clear this wave
is accelerating.
Today, AI systems can almost perfectly recognize faces and objects. We
take speech-to-text transcription and instant language translation for
granted. AI can navigate roads and traffic well enough to drive
autonomously in some settings. Based on a few simple prompts, a new
generation of AI models can generate novel images and compose text with
extraordinary levels of detail and coherence. AI systems can produce
synthetic voices with uncanny realism and compose music of stunning
beauty. Even in more challenging domains, ones long thought to be
uniquely suited to human capabilities like long-term planning, imagination,
and simulation of complex ideas, progress leaps forward.
AI has been climbing the ladder of cognitive abilities for decades, and it
now looks set to reach human-level performance across a very wide range
of tasks within the next three years. That is a big claim, but if I’m even
close to right, the implications are truly profound. What had, when we
founded DeepMind, felt quixotic has become not just plausible but
seemingly inevitable.
From the start, it was clear to me that AI would be a powerful tool for
extraordinary good but, like most forms of power, one fraught with
immense dangers and ethical dilemmas, too. I have long worried about not
just the consequences of advancing AI but where the entire technological
ecosystem was heading. Beyond AI, a wider revolution was underway, with
AI feeding a powerful, emerging generation of genetic technologies and
robotics. Further progress in one area accelerates the others in a chaotic and
cross-catalyzing process beyond anyone’s direct control. It was clear that if
we or others were successful in replicating human intelligence, this wasn’t
just profitable business as usual but a seismic shift for humanity,
inaugurating an era when unprecedented opportunities would be matched
by unprecedented risks.
As the technology has progressed over the years, my concerns have
grown. What if the wave is actually a tsunami?
—
in 2010 almost no one was talking seriously about AI. Yet what had once
seemed a niche mission for a small group of researchers and entrepreneurs
has now become a vast global endeavor. AI is everywhere, on the news and
in your smartphone, trading stocks and building websites. Many of the
world’s largest companies and wealthiest nations barrel forward, developing
cutting-edge AI models and genetic engineering techniques, fueled by tens
of billions of dollars in investment.
Once matured, these emerging technologies will spread rapidly,
becoming cheaper, more accessible, and widely diffused throughout society.
They will offer extraordinary new medical advances and clean energy
breakthroughs, creating not just new businesses but new industries and
quality of life improvements in almost every imaginable area.
And yet alongside these benefits, AI, synthetic biology, and other
advanced forms of technology produce tail risks on a deeply concerning
scale. They could present an existential threat to nation-states—risks so
profound they might disrupt or even overturn the current geopolitical order.
They open pathways to immense AI-empowered cyberattacks, automated
wars that could devastate countries, engineered pandemics, and a world
subject to unexplainable and yet seemingly omnipotent forces. The
likelihood of each may be small, but the possible consequences are huge.
Even a slim chance of outcomes like these requires urgent attention.
Some countries will react to the possibility of such catastrophic risks
with a form of technologically charged authoritarianism to slow the spread
of these new powers. This will require huge levels of surveillance along
with massive intrusions into our private lives. Keeping a tight rein on
technology could become part of a drift to everything and everyone being
watched, all the time, in a dystopian global surveillance system justified by
a desire to guard against the most extreme possible outcomes.
Equally plausible is a Luddite reaction. Bans, boycotts, and
moratoriums will ensue. Is it even possible to step away from developing
new technologies and introduce a series of moratoriums? Unlikely. With
their enormous geostrategic and commercial value, it’s difficult to see how
nation-states or corporations will be persuaded to unilaterally give up the
transformative powers unleashed by these breakthroughs. Moreover,
attempting to ban development of new technologies is itself a risk:
technologically stagnant societies are historically unstable and prone to
collapse. Eventually, they lose the capacity to solve problems, to progress.
Both pursuing and not pursuing new technologies is, from here, fraught
with risk. The chances of muddling through a “narrow path” and avoiding
one or the other outcome—techno-authoritarian dystopia on the one hand,
openness-induced catastrophe on the other—grow smaller over time as the
technology becomes cheaper, more powerful, and more pervasive and the
risks accumulate. And yet stepping away is no option either. Even as we
worry about their risks, we need the incredible benefits of the technologies
of the coming wave more than ever before. This is the core dilemma: that,
sooner or later, a powerful generation of technology leads humanity toward
either catastrophic or dystopian outcomes. I believe this is the great metaproblem of the twenty-first century.
This book outlines exactly why this terrible bind is becoming inevitable
and explores how we might confront it. Somehow we need to get the best
out of technology, something essential to facing a daunting set of global
challenges, and also get out of the dilemma. The current discourse around
technology ethics and safety is inadequate. Despite the many books,
debates, blog posts, and tweetstorms about technology, you rarely hear
anything about containing it. I see this as an interlocking set of technical,
social, and legal mechanisms constraining and controlling technology
working at every possible level: a means, in theory, of evading the dilemma.
Yet even technology’s harshest critics tend to dodge this language of hard
containment.
That needs to change; I hope this book shows why, and hints at how.
THE TRAP
