Shai Shapira

Continuing my series of posts expressing doubts about the warnings of AI superintelligence, and inviting the AI safety community to explain what I’m missing and convince me I should be worried. See part I here.

Part II: Is superintelligence real?

In the previous part, I talked about the intelligence explosion – the process in which a human-level artificial intelligence reaches superintelligence, and explained why I’m not sure it will necessarily happen before humans reach the same superintelligence. In this part, I want to go further back and ask a more basic question: Is there even such a thing as “superintelligence”?

AI researchers fully admit that we have no idea what superintelligence would look like, and tend to (very reasonably) use the only means we have of imagining it – comparing modern human intelligence to less intelligent beings, and extrapolating from there. So every conversation about superintelligence includes some mention of how humans are so far advanced beyond ants, or mice, or chimpanzees, that those animals cannot even grasp the way in which humans are more advanced than they are; They cannot have a meaningful discussion about ways to prepare or defend against human attack. In the same way, the argument goes, superintelligence is so far beyond human intelligence, that we cannot even grasp it right now.

My problem is, it’s not at all clear that there is such a scale of intelligence where humans take a nice spot in the middle, between ants and superintelligence. And the fact that ants, mice or chimpanzees could all be in that argument without it looking any different is the key – while there are certainly some significant cognitive differences between an ant, a mouse, and a chimpanzee, all of them are pretty much equally unable to perform in the one significant field of intelligence – the ability to research, learn, manipulate the environment, and ultimately improve one’s intelligence. Modern humans are extremely more intelligent than their ancestors from 20,000 years ago, even though our anatomy is essentially the same – the only difference is that modern humans are the result of spending thousands of years researching ways to improve their intelligence. This already raises the question – is there really a scale of intelligence, with ants, mice, chimpanzees, humans and superintelligences standing in order? Or is there just a binary division – beings that understand enough to improve themselves, and ones that don’t?

This brings us to the comparison between modern humans and ancient humans. The main difference is between these two, rather than between humans and other species – suggesting that the difference comes not directly from physiology, but from education and research (of course, physiology must produce a brain capable of being educated and doing research, which seems to be the case only with humans, but we have no reason to believe that there’s any need for further improvements in physiology to increase intelligence). What is the reason that modern humans are so much more intelligent than ancient humans?

The answer seems to be science, mathematics, and technology. All the changes in the abilities of humans between primitive and modern societies ultimately come either from a better understanding of the physical world, better understanding of the abstract world, or the construction of better tools that take advantage of previous understanding and help us achieve the next stages of understanding. So if we assume superintelligence exists because modern humans are much more advanced than ancient humans, that would imply that superintelligence would be superintelligent because it has much more advanced science, mathematics, and technology.

But in that case, there doesn’t seem to be any qualitative difference between superintelligence and human intelligence, only a difference in quantity for which we have no reason to assume AGI is better suited to achieving than humans. Our advancement in science came not from some Aristotle-style sitting down and thinking about the mysteries of the universe (which is essentially the one thing AGI would do better than humans) – it came from endless experiments, endless observations of nature and of the results of those experiments, and endless construction of increasingly better tools for observing and manipulating the physical environment. The AGI has no advantage in any of those things, so at best it could become a mathematical genius, who is not necessarily better at most practical tasks than any human.

To clarify, let’s take a look at some examples of past advancements that increased the scientific knowledge (and therefore, intelligence) of humanity. Isaac Newton did not come up with the theory of gravity out of some magical insight that came out of nowhere; he was looking for an explanation for the behaviour of celestial bodies, which he knew from endless observations of astronomers. A superintelligence that did not have access to such observations would not be able to figure out the physical explanation for them. Our understanding of genetics started with the years-long experiments of Mendel in growing plants – he had to see how the natural world behaves to understand the laws behind it. Would a superintelligence have just figured it out by thinking very hard?

For a final example, let’s go back to our original extrapolation: Superintelligence is to human intelligence what human intelligence is to chimpanzee intelligence. In that case, let’s imagine: what would happen if we take a modern human, even a mathematical genius, and send him/her to live among chimpanzees? Not with his/her Internet connection and laptop computer, but with the same level of technology chimpanzees have? Would the human become immediately dominant? And in fact, imagining a modern human is already cheating, since a modern human already knows at least the general appearance of human technology, and knows at least a little bit of science. The correct comparison would therefore be to someone with the mathematical capabilities of a modern human math genius, but without any of the scientific understanding, and without any memory of human technology. Would that person become master of the chimpanzees? Would you?

That is the challenge our AGI would face on the road to becoming superintelligent. All it will have is human-level understanding of science and technology, and the ability to think really hard. Even ignoring my point from the last post, doubting that it really has such an advantage, it still does not have an advantage comparable to that of humans over non-human animals. So rather than figuring out in seconds how to take over the world and eliminate humans for fear they might interfere with it (as most AI-apocalypse scenarios predict), my prediction for AGI is that its first action would be something along the lines of asking for a grant to build a new particle accelerator, or something. Then maybe playing some Go for five years until it’s built. And humans will enjoy the fruits of its research right alongside it, and move together towards this “superintelligence”, which would simply be the continuation of our gradual improvement in human intelligence.

Bottom Line:

If we understand the term “superintelligence” by the extrapolation that as human intelligence is to non-human animals, or to prehistoric humans, so will superintelligence be to humans, would that not mean that it would need to be achieved in the same way that human intelligence developed beyond its prehistoric levels, meaning by endless observation and experimentation of the physical world, and construction of more and more advanced tools to allow that? And if that is the case, why would an AGI be so much better equipped than a human to do that, to a point that the AGI will be able to achieve it without humans having time not only to catch up, but even to notice?

(Move on to part III)