The dangers of artificial intelligence research are becoming an increasingly popular topic in scientific and philosophical circles, and it seems like everyone I know who studied the issue enough is convinced that it’s something major to worry about. Personally, I have some issues that make me unsure about it – both about the likelihood of this being an actual potential catastrophe, and about the idea of AI safety research being the reasonable response to it. So I decided to detail here my doubts about the issue, hoping that people in the AI safety community (I know you’re reading this) will respond and join in a debate to convince me it’s worth worrying about.
In the first part I’ll talk about whether or not the idea of the intelligence explosion can really happen. In the second part I’ll ask even more basically, whether or not superintelligence even exists as a coherent concept. The third part will ask, assuming I’m wrong in the first two parts and AI really is going to advance seriously in the future, what can be done about it other than AI safety research. I’m going to include a lot of explanations to make sure it’s accessible to non-AI-researchers, so if you’re an AI researcher in a hurry, feel free to skim through it and focus on the (literal) bottom line in each of the three parts.
Part I: The superintelligence explosion
The main concept on which the AI warnings are built is the intelligence explosion – the idea that at some point our AI research is going to reach the level of human intelligence (researchers like to call that AGI, Artificial General Intelligence), and from that point it will be able to improve itself and therefore reach, in a very short time, levels of intelligence vastly superior to ours. Considering the amount of debates everywhere on the Internet on the question of whether or not AI can be evil, harmful, or just naively destructive, I see remarkably little debate on the question of whether superintelligence is possible. And in fact, there are two questions to be asked here – whether or not superintelligence can be reached by an AGI significantly more quickly than by a human, and even more basically than that, can we really be sure that “superintelligence” actually exists, in the way that AI safety researchers present it. Let me elaborate on these issues.
The main argument for the AGI being able to reach superintelligence in a worrying speed, from what I can find, is the physical advantages in calculation and thinking that electronic machines enjoy over biological brains; see Nick Bostrom’s description of it here, for example. According to him, the superior speed and efficiency of computation in an electronic machine will vastly surpass those of a human brain, therefore, once an AGI is created, it will be able to do what humans do, including researching AI, significantly faster and better. Then it will research ways to improve itself more and more, until it becomes so vastly superior to humans that we will be completely irrelevant to its world.
The problem I see with this argument, that I did not see addressed anywhere else, is that it puts humans in a needlessly disadvantaged playing field. Yes, it’s certainly possible that supercomputers in the near future will have better computing power than human brains, but that’s no different than gorillas having superior muscle power than human muscles, which does not stop humans from being dominant over gorillas; that is because humans do not need to depend on their biological assets. Humans use tools, whether it’s a rifle to defend against an attacking animal, or a computer to outthink an attacking intelligence. Whatever hardware the AGI has access to, we probably have access to more.
Think about the classical examples, of the AI defeating humans in various games. A common prelude to talking about the dangers of AI is how intelligent computes are now defeating humans in Chess, Checkers, Go, and so on. But humans are playing these games with a deliberate handicap – they are only allowed to use their brains. The AI can use computers to help it.
For the sake of any non-computer-scientist readers, I want to stop and make a little clarification – there is a significant difference between non-AI algorithms and AI. The definition might not be completely universal, different people might understand the word AI in different ways, so let me define the word AI for the purpose of this post:
Definition: An AI algorithm is an algorithm whose creator does not understand enough to modify in a way that produces predictable results.
Think for example about machine translation: an algorithm that takes a text in one language and searches every word in the dictionary to replace it with a word in the target language, would be a non-AI translator. Of course it would also not be very good, but we can develop it further and build complex linguistic rules into it; we can design complex algorithms to determine which words are nouns and which are verbs, and translate conjugations and declensions in a more suitable way to the target language. We can maintain a database of idioms the algorithm can search through to try to recognize them in the source text, and so on. With all these additions and complexities, it’s still not AI in my definition, because at all stages, the algorithm does what the programmer told it to, and the programmer understands it perfectly well. The programmer could just as well do the same things by themselves, it would just take an absurd amount of time.
On the other hand, an algorithm that constantly reads texts in the source language and their (human-made) translations to the target language and tries to figure out the rules for translation by itself, through some sort of machine learning process, would be actual AI. The programmer does not really understand how the algorithm translates a text; All they know is how it’s built and how it learns. The programmer would not be able to change anything in a reliable and predictable way – if they find out that for some reason the translation has a problem with some particular grammatical structure, they cannot easily fix it because they have no idea where and how the algorithm represents that grammatical structure. So that algorithm would be true AI.
I argue that this definition is useful, because algorithms that don’t count as AI by this definition are not only unable to turn into superintelligent dangers by themselves, but they are also “on our side” – they are tools we use in our own thought. Deep Blue, the famous computer that made history in defeating the world champion in chess, was a non-AI algorithm – it worked by using its large computation resources to try millions and millions of different possibilities, and checking which ones are beneficial according to rules explicitly defined by its programmers. The programmers understand how it works – they can’t defeat it using their own brains, but that’s just because their biological brains don’t have the ability to calculate so many things so quickly. So if we think about the level of AI versus humans in Chess right now, it would be unfair to ask if the best AI player can defeat the best human player – we should ask if the best AI player can defeat the best human player, using a supercomputer with a non-AI algorithm they designed to help them. Because if the AI apocalyptic scenario happens, and a malicious AI tries to destroy humans for whatever reason, we’re going to have supercomputers on our side, and we’re definitely going to use them. So if you let Garry Kasparov join forces with Deep Blue, or more interestingly – with some software Kasparov himself would design as a perfect assistant to a Chess player – would he still be defeated by the best AI player? I’m not sure at all[1].
Bottom Line:
The difference between humans and AGI, that makes us worry that an AGI will advance significantly more quickly than humans towards superintelligence, is described in being the superior hardware of the AGI. But humans have access to the same hardware; We can calculate and think at the exact same speed, the only difference is that one small (though important) part of that calculation is done in a slower, biological computer. So how is that a big enough difference to justify the worry of superintelligence?
(Move on to part II and part III)
[1] I offer this as a thought experiment, but I did hear Kasparov say he’s interested in the idea of human-computer teams playing together in Chess; I don’t know what exactly he meant by that, and could not find any information online.
Shai Shapira 
Pingback: Doubts about AI risk – Part II | Shai Shapira
Pingback: Doubts about AI risk – Part III | Shai Shapira
Quick comment: previous to last paragraph uses the term “malicious AI”. We don’t really think AI systems will be malicious (most likely they won’t be conscious at all), but we do think it will be possible to create dangerously unsafe systems: systems that follow design specifications, yet through unexpected side effects cause tremendous harm, including, possibly, human extinction.
You’re right, I meant to write “malignant”. That would be acceptable, right?
Response: This is a good point. While I’m not sure I’d define AI in exactly the same way, the notion that the risk scenario that matters is a showdown between an agent-like silicon-based system and a human (or all humans) is not really the relevant one. The closest this comes to scenarios of concern is if we have a system that is performing some unexpected behaviour (e.g. starting to turn schools into paperclip factories) and a human wants to shut it down. The AI risk argument says the paperclip maker will avoid being shut down because:
1) being shut down will lead to fewer paperclips being made and
2) the paperclip maker is more intelligent than humans and will figure out in advance ways to prevent it from being shut down.
Your counter move is to give the humans advanced computing powers.
Does this work?
1. Elon Musk seems to think so, as long as your bandwidth to the advanced computing powers is wide enough, which is why he wants to develop a neural lace [1]. Without the increased bandwidth the AI seems to be at a great advantage: it can pass megabits or gigabits per second between its policy-deciding subsystem(s) and the various computing systems it is connected to, while the humans are stuck typing or speaking to their systems at a few kilobits per second.
2. But even this might not be enough, because the decision processing speeds in silicon seem to be much higher than in brain wetware, so a silicon-based decision-making system will be able to explore plans that require shifting actions or policies at speeds not accessible by humans. Depending on the scenario, this may matter a great deal.
3. The main point of AI safety is to make sure we build systems such that we are never in an adversarial situation against them. If we do end up in one, and both sides are utilising advanced non-AI computing sources, these sources themselves become targets for the adversaries’ policies. In other words, if some futuristic agent-like AlphaGo is playing a future human Go master, and both have access to vast computing powers, and the most important thing in the world is to win the game, then the future AlphaGo will consider hacking the Go master’s computers as part of its plan for winning the game. A small difference in intelligence can be used to gain a resource advantage, which leads to greater decision-making ability and a bigger resource advantage, and so on.
* Usual caveats: we don’t know enough about how the brain works, we don’t know how AI technologies will develop, and we definitely don’t know how future technologies will be used.
[1] https://waitbutwhy.com/2017/04/neuralink.html
Well, first of all I’d say, this “showdown” is mostly a dramatic way I used for demonstrating it; More fundamentally, I think it’s not an issue of the humans having advanced computing powers during a showdown against the super AI, but rather a reason why super AI will not exist to begin with, and therefore no showdown will happen; Because we have no reason to expect an AGI to advance faster than a human. The AI argument lies in the concept of the intelligence explosion, the idea that a human-level intelligence armed with advanced computing powers will advance in an unprecedented speed. I argue that while this is true, it’s already happening right now, and we are this human-level intelligence with advanced computing powers; The intelligence explosion is exactly what has been happening ever since the information revolution started and computers were created.
1-2. Has anyone actually studied these assumptions scientifically? Because this “seems to be at a great advantage” does not necessarily seem to me. Good decision making depends on many things, and it’s not at all obvious that basic-level processing speed is such a big factor in it. Data availability and general processing speed seem much more significant to me. As I said in part II, we did not discover the physics required to change the world by sitting and thinking. We had to gather data, and the speed of gathering data did not depend on how fast our internal processor was.
3. First of all, I think you’re diving into “Hollywood hacking” here. Do you really think it’s impossible to guard your (presumably offline) supercomputer from being hacked by someone with “a small difference in intelligence”, if your life depended on it? But anyway, as I said in my clarification, my argument is less for what happens during a showdown, and more for the question of whether there is actually a reason to expect an AGI to advance faster than humanity. If the AI already starts working towards disabling human computing powers while it’s still only slightly more intelligent than us, it’s not likely to succeed; If it doesn’t, then we have no reason to assume it will advance much faster than us, since we are both human-level intelligences with advanced computing powers.
Look at the recent results from OpenAI on playing DOTA2 [1], especially the amount of progress from Aug 7th to Aug 11th. Would a human augmented with a non-learning automation assistant be able to increase in performance at this rate? We should both look up the relevant data on human learning rates, but I’m sceptical, unless a significant amount of the DOTA playing is done by the automated system, in which case loss of control becomes a problem again.
[1] https://blog.openai.com/more-on-dota-2/
Would a human augmented with a non-learning automation assistant be able to increase in performance at this rate? Absolutely. You talk about human learning rates, but I don’t think that’s relevant – I have no doubt that we’re already very close to reaching the full potential of the human part of the “human + non-machine-learning algorithm” team, so the main addition will come from the algorithm. In that case, the progress graphs would not even be continuous – in all likelyhood, they would include some work done behind the scenes, and then immediately jump to an extremely high level. How high? I have no idea because I’ve never tried to design a DOTA playing bot. But what I’d like to know is how much time they spent behind the scenes on designing this bot, and what would I be able to produce if I spent the same amount of time and resources designing a non-machine-learning bot.
Again I say – this AI is playing against humans with a huge handicap. Unless you have any data on the performance of non-machine-learning bots, and certainly ones that were created with as much investment as the OpenAI bot, then we can have a basis for discussion.
Just to clarify – your last comment seems to imply that if my system is indeed a bot, it will have a loss of control problem, but the whole point of our discussion until now was that a non-machine-learning algorithm cannot have a loss of control problem. So did I misunderstand you?
A recent video that discusses the bandwidth issue: https://www.youtube.com/watch?v=gP4ZNUHdwp8
Still, he’s talking about the speed in which you’d see one digit or type one digit into a calculator, compared to the speed in which an AGI would do the same. Indeed, the AGI is faster. But how quickly would the human input a billion digits into a calculator? How quickly would the human read a billion digits? It definitely won’t be “speed of reading/writing one digit times billion”. If we worry about our speed of doing nontrivial things, we cannot just extrapolate from our speed of doing trivial things. The calculation is not important by itself; It’s important as part of some bigger process we do. And a human would do that process on a computer properly built for it, just like the AGI will, and each calculation in it would take the same time. The question is just how long it would take to make the decision to start that process, and I would not expect that to be extremely big.