According to scientists, superintelligence may be just “a few thousand days” away.

In 2014, British philosopher Nick Bostrom published a book on the future of artificial intelligence (AI) titled Superintelligence: Paths, Dangers, Strategies. The book greatly influenced the idea that advanced AI systems—”superintelligences” more capable than humans—might someday take over the world and destroy humanity.

A decade later, OpenAI’s CEO, Sam Altman, claims that superintelligence might be just “a few thousand days” away. A year ago, Altman’s OpenAI co-founder, Ilya Sutskever, created a team within the company to focus on “safe superintelligence.” Now, Sutskever and his team have raised $1 billion to start their own company pursuing this goal.

But what are they really talking about? Broadly speaking, superintelligence is anything smarter than humans. However, unraveling what that might mean in practice can be quite complex.

In my opinion, the most useful way to think about the different levels and types of intelligence in AI was developed by American computer scientist Meredith Ringel Morris and her colleagues at Google.

Their framework lists six levels of AI performance: no AI, emergent, competent, expert, virtuoso, and superhuman. It also makes an important distinction between narrow systems, which can carry out a small range of tasks, and more general systems.

A narrow, no-AI system would be something like a calculator: it performs various mathematical tasks based on explicitly programmed rules.
Many low-level narrow AI systems already exist and have been very successful. Morris cites the chess program Deep Blue, which defeated world champion Garry Kasparov in 1997, as an example of a low-level virtuoso AI system.

Some narrow systems even exhibit superhuman capabilities. One example is AlphaFold, which uses machine learning to predict the structure of protein molecules and whose creators won the Nobel Prize in Chemistry this year.

General-purpose systems are software that can tackle a much broader range of tasks, including things like learning new skills.

A general-purpose no-AI system might resemble Amazon’s Mechanical Turk: it can perform a wide variety of tasks but does so by asking real people to complete them.

In general, general AI systems are far less advanced than their narrow counterparts. According to Morris, the cutting-edge language models behind chatbots like ChatGPT are general AI but are currently at the “emergent” level (meaning they are “on par with or slightly better than an untrained human”) and have yet to reach the “competent” level (as good as 50% of trained adults).

So, by this calculation, we are still some distance from general superintelligence.
As Morris points out, determining precisely where a given system stands would require reliable tests or benchmarks.

Depending on our benchmarks, an image generation system like DALL-E might be at the virtuoso level (because it can produce images that 99% of humans could not draw or paint) or might still be emerging (because it makes mistakes no human would, like mutant hands or impossible objects).

There is significant debate even about the capabilities of current systems. A prominent 2023 paper argued that GPT-4 displays “sparks of artificial general intelligence.”

OpenAI claims its latest language model, o1, can “perform complex reasoning” and “rivals expert human performance” on many benchmarks.
However, a recent paper by Apple researchers found that o1 and many other language models struggle with genuine mathematical reasoning problems. Their experiments show that these models’ results resemble sophisticated pattern recognition rather than true advanced reasoning. This suggests that superintelligence may not be as imminent as some have suggested.

Some believe that the rapid pace of AI progress in recent years will continue or even accelerate. Tech companies are investing hundreds of billions of dollars into AI hardware and capabilities, making this possibility not entirely out of reach.
If this happens, we might see a general superintelligence within the “few thousand days” proposed by Sam Altman (that’s a decade or more in less scientific terms). Sutskever and his team mentioned a similar timeframe in their paper on superalignment.

Many recent successes in AI have been achieved through a technique called “deep learning,” which, simply put, finds associative patterns in massive datasets. In fact, this year’s Nobel Prize in Physics was awarded to John Hopfield and AI “godfather” Geoffrey Hinton for their invention of Hopfield networks and the Boltzmann machine, which underpin many powerful deep learning models used today.

General-purpose systems like ChatGPT have relied on human-generated data, much of it in the form of texts extracted from books and websites. Improvements in their capabilities have largely resulted from scaling up systems and the volume of training data.

However, there may not be enough human-generated data to take this process much further (though efforts to use data more efficiently, generate synthetic data, and improve skill transfer across domains could drive improvements). Even if there were enough data, some researchers argue that language models like ChatGPT are fundamentally incapable of achieving what Morris would call general competence.

A recent paper suggests that an essential feature of superintelligence would be its openness, at least from a human perspective. It would need to continuously generate outcomes that a human observer would find novel and could learn from.

Existing foundational models are not trained openly, and existing open systems are quite limited. This paper also emphasizes that novelty or learning ability alone are not enough. A new kind of open foundational model is required to achieve superintelligence.

What does all this imply for AI risks? At least in the short term, we don’t need to worry about a superintelligent AI taking over the world.

But that doesn’t mean AI poses no risks. Once again, Morris and her colleagues have thought about this: as AI systems gain more capabilities, they can also gain greater autonomy. Different levels of capability and autonomy present different risks.

For example, when AI systems have low autonomy and people use them as a kind of consultant (when we ask ChatGPT to summarize documents, for instance, or let YouTube’s algorithm shape our viewing habits), we might face the risk of over-relying on or becoming too dependent on them.

Meanwhile, Morris highlights other risks to watch for as AI systems become more capable, ranging from people forming parasocial relationships with AI systems to mass job displacement and societal boredom.

Suppose we eventually develop autonomous, superintelligent AI agents. Would we risk them concentrating power or acting against human interests?

Not necessarily. Autonomy and control can go hand in hand. A system can be highly automated yet still offer a high degree of human oversight. Like many in the AI research community, I believe safe superintelligence is achievable. However, its creation will require a complex, multidisciplinary effort, with researchers venturing into uncharted territory to accomplish it.