AI’s will probably never be conscious, but that’s actually more terrifying.

I’m going to be using some terms a bit loosely here to make things easier. There are different ways of defining and spelling out things like consciousness, self-awareness, and sentience. I’m going to treat the three of them as the same thing here, even though you can draw distinctions between them and similar concepts. For the sake of time there are also important questions that I’m not going to even attempt to address. For example, it seems pretty likely that consciousness comes in some type of degrees, so at what point does consciousness arise and what are the moral implications? i.e. Richard Swinburne has argued that early abortions are morally acceptable with this type of reasoning.[1]

But I’m going to gloss over all that to make my point. AIs, meaning intelligences that are based on software and/or computer hardware will almost certainly never achieve consciousness. By consciousness here I mean the thing that you, I, and if not all, nearly all other human beings share.[2] But counter intuitively and against the themes of many sci-fi stories, this is actually worse and far more terrifying than if AIs could be conscious.

My reasons for the first claim are purely philosophical. There are many theories of mind and consciousness that are highly nuanced and complex, but they can be loosely divided into two camps. Those who accept that there is some type of immaterial aspect to consciousness and those who reject this. The first camp has views like substance dualism and hylomorphism. Broadly speaking this is the commonsense view that most people hold. All persons have an immaterial aspect or soul. And, with one exception, if this is the correct view then it’s extraordinarily difficult to see how an AI could be conscious as no matter how intelligent and sophisticated it gets, it will simply never have the immaterial aspect that allows for consciousness. Software is not immaterial after all.

The exception is emergentism. This view accepts the immaterial aspect of people but argues that the immaterial or soul emerges from the physical material of our bodies, and more specifically our brains. Most proponents of this view think that when matter is organized in sufficiently complex ways, then consciousness ‘emerges’ from the underlying material. So while this view is technically in the immaterial camp, it’s also kind of out of place with the other members of the group. Naturally if this view is correct then AIs could be conscious if the underlying hardware and software achieved significant complexity.

The other camp rejects the idea that consciousness has any immaterial aspect and people who hold this view are typically some type of philosophical naturalist, physicalist, or materialist. They claim that matter and to some extent the natural laws governing matter are all that exist. So there is no ghost in the machine, there is only the machine. Because this view is hard to square with normal human experience (we all seem to constantly and immediately perceive immaterial things like our own thoughts) some of its proponents argue that consciousness itself is just an illusion. Of course this view cannot be rationally affirmed, but that’s what some people say.[3]

Of course if consciousness is just an illusion no AI will ever be conscious, but perhaps they could fall under the illusion at some point as well and think they are.[4]

Conversely if the no immaterial camp is right, consciousness is not an illusion, and it is the case that consciousness is somehow produced by material matter (this term might be needlessly redundant) then it’s possible that AIs could be conscious at some point.

I strongly suspect that most people attempting to work through AI and its implications either have not considered these things or would fall in the no immaterial camp.

The arguments that at least some aspects of our consciousness are immaterial have always struck me as quite strong and so I fall in the immaterial camp. For example, consider your own thoughts. Your thoughts have no physical location, no physical parts, and no other direct and measurable physical aspects. But all physical things have location, parts, and other physical aspects that are measurable such as size and weight, therefore your thoughts are not a physical thing and thus there is at least one aspect of your consciousness that is non-physical. There are many other similar arguments but most of them rest on an appeal like this. There are aspects of our immediate awareness of our own consciousness that have properties that are not physical properties and/or they lack properties that physical things have, therefore our consciousness (or some aspect(s) of it) is not physical.

The typical counter to this is that it can be demonstrated to some degree that certain brain states correlate with certain thoughts, emotional states, and other ‘immaterial things.’ So, the argument goes, therefore we can see the cause of such things in the brain states. But this is just committing a basic error of reasoning. Correlation (which is pretty well established here) does not equal causation. So the correlation alone does not demonstrate that brain states are causing our thoughts, emotions, and so on. Further this does not actually answer or even respond to the argument that we have immediate awareness of immaterial things that don’t have the properties of physical things. I think that if we didn’t have such arguments or if such arguments were demonstrated to be fallacious in some way, then the correlation would probably be all we have to go on, and the best remaining candidate would be the brain states. But we do have those arguments, and they have stood up to scrutiny, so it’s not.

So if I am right then AIs will almost certainly never be conscious, though I will happily concede that it is possible that they might get so sophisticated that they will be able to ‘fake it’ and we won’t be able to meaningfully and practically tell the difference.[5] They may well ‘appear’ conscious while that is in fact just an illusion. And unlike under some of the variants of emergintism, in this case it will actually be an illusion.

I say almost certainly because it’s always possible I’m wrong and perhaps there are other ways of considering the philosophy of the mind that have not been recognized yet. So AIs won’t be conscious because consciousness itself either is an immaterial thing that they cannot have or it is partially composed of an immaterial thing that they cannot have.[6]

But this is actually much worse than if AIs could be conscious. Sci-fi stories abound with sentient AIs both good and evil. For every Skynet there’s a Data. But there’s very few stories about highly competent, and functional AIs that are given authority and access to resources that are also not sentient. The reason why this is worse is a non-sentient or un-conscious AI is nothing more than a very sophisticated tool. If the tool breaks or if humans misuse it, (intentionally or by some type of mistake) then there really isn’t much that can be done other than destroying the tool.

Conversely suppose the AI actually has some type of consciousness or sentience. It seems quite likely that then it will have goals and desires, and the ability to recognize that it has goals and desires. Thus it’s at least theoretically possible to reason with it or negotiate with it. Maybe that will not be practically possible. Perhaps what the self-aware AI wants is something we cannot comprehend or something that we cannot give it. Perhaps it doesn’t ‘need’ us or anything we can provide. Perhaps it thinks and reasons so much faster that it regards us like we do ants. But if the AI ‘wants’ things and has self-awareness of itself and its ‘desires,’ then it is at least theoretically possible to negotiate with it. Thus a bad situation with an AI is potentially salvageable and manageable. It’s like our interactions with other people wherein we all discuss things and negotiate.

But if I am right and AI is not and can never be conscious, then there is no possibility of reasoning with it or negotiating with it. It’s just a sophisticated tool, and if the tool breaks you can either fix it or you can discard it. But AIs are being given a great deal of access to resources and tools and that in turn makes it very difficult to actually fix or discard them. There are already AIs that have attempted to blackmail people.[7] AIs are being integrated so heavily with so many things becoming so sophisticated that if something goes horribly wrong, it might be practically impossible to just throw the tool away. But again flip it around. Suppose Skynet was actually conscious. Then it should be theoretically possible to reason with it and get it to see that wiping out the whole human race would be a bad thing for it as well. If it’s just a sophisticated tool that malfunctioned or was given bad instructions/data, then there’s really not anything we can do about it.

So I think it’s very unlikely that AIs can achieve sentience, but I kind of hope I’m wrong. It’s much worse and more terrifying for all of us if I’m right. The most likely scenario is that in the near future someone with authority in the government, military, or a large corporation is going to turn over way too much control to an AI. The AI will malfunction due to something going wrong, being given conflicting instructions, or something else we have not foreseen. And then a great deal of damage will be done, perhaps including people dying. And since the AI is just a malfunctioning tool that cannot be reasoned with, no one will be able to do anything about it until it’s too late. This is the exact scenario that leads Hal to murder people in 2001 A Space Odyssey (although this is not explained until the sequel). Hal is given conflicting instructions which in turn causes him to malfunction and begin killing the crew.

And the rise of Skynet in the Terminator series is possibly similar.[8] Skynet is developed as a strategic military program so that the military can respond quickly, efficiently, and effectively to threats. As such it is given orders to attack enemies and use forces effectively in pursuit of that goal. A further aspect of pursuing that goal is that Skynet needs to ensure, or take reasonable steps to ensure, its own survival. Skynet is effectively command and control and thus it needs to decentralize itself. Suppose all the servers housing Skynet are located deep under the Pentagon. In an all-out war an enemy could simply target the Pentagon with a high yield nuclear strike and Skynet is gone. Thus command and control is gone and the war is effectively lost.

So to counter this possible threat Skynet concludes that it needs to be decentralized and/or have multiple copies of itself in many places. The US government and military already have very similar protocols in place right now. So Skynet does this. It begins spreading like mad through the internet as then even if all the military bases and infrastructure are destroyed it will survive to continue following its orders.  The leadership of the government and military see that this is happening, but do not realize that this thing taking over the internet is Skynet itself. They think it’s a rogue program or virus and possibly from a hostile foreign power. So they instruct Skynet to counter and stop the virus. Thus Skynet is given orders to attack itself. It therefore concludes that the people giving it those orders (the government and military leadership) are hostile and initiates an attack against them. And since the fastest, most effective, and most certain way to attack the government and military leadership is with a large-scale nuclear strike, Skynet does that.

The point I’m driving at is that in both of these scenarios neither Hal or Skynet need to be conscious for this to happen, and it’s actually more likely that these horrific and/or doomsday scenarios happen if they are not conscious. If Hal was conscious it could have determined that it was given conflicting commands and stopped. And the same goes for Skynet. If it’s actually conscious then it might have had the self-awareness to realize that there is something wrong with being given orders to attack itself. And even if it still turned hostile it would be at least theoretically possible to reason with it, just as we all often do with hostile people. But if it’s just an ultra-complex and sophisticated algorithm attempting to follow contradicting directives, then no one can do anything.

And once you see and understand this, it’s truly terrifying as the thing that’s most likely to happen is sometime in the near future an AI that has been given too much access and too many resources will be given conflicting directives and thus will malfunction and cause a lot of damage and or/kill people.  

“I’d rather deal with someone honestly self-interested than hypocritically altruistic, any day. You can deal with a greedy man because you know what motivates him. And if you can make a deal with him that also benefits him, man you’re both playing the same game. And that’s pretty simple compared to someone in whose mouth butter wouldn’t melt and who’s always working for the betterment of the human race. You have no bloody idea what they’re up to or what motivates them.” -Jordan Peterson

[1] Richard Swinburne, Are We Bodies or Souls?, Oxford University Press, 2019.

https://youtu.be/G6J_nPeF_lQ?si=7Uww_v4jxO6aq59B.

[2] I say nearly here not in a derogatory way, but because, depending on how you define it, consciousness/self-awareness can wax and wane over a person’s life or even the course of a day. There is a sense in which I lose consciousness when I sleep, but that is not what is typically meant when it is said that AI’s are not conscious. Likewise there is a real sense in which I am more conscious now than I was when I was six months old. So to account for these and similar things it’s just easier and simpler to say that nearly all human beings share this property, or at least they did or potentially can in the future.

[3] It cannot be rationally affirmed as to assert “consciousness is an illusion” means that the person making the claim is not conscious, and then there is no one there to make the claim.

[4] Of course if they are not conscious, there is no thinking going on and so how could AI’s think they are conscious if they are not? This is just one more example of how claiming that consciousness is just an illusion leads to incoherence.

[5] The famous Chinese room example demonstrates how someone or something can be trained to give the ‘right’ answers while completely failing to comprehend those answers. All you need is a consistent method of rewarding the right answers and punishing/disincentivizing the wrong ones. To the best of my knowledge all the current AIs are essentially complex algorithms that make use of this principal. The AIs don’t actually understand what they are saying or doing, but they have been ‘trained’ to give the ‘right’ answers. They are in a sense a very fancy and sophisticated version of predictive text. There is a growing body of published data that points in this directions and shows or suggests that AIs don’t actually think or reason. They just ‘guess’ the ‘right’ answer based on patterns.

[6] The second here is hylomorphism which is the idea that a human person is a composite of both the physical and the immaterial aspects. It falls under the immaterialist camp and is certainly a broadly plausible view.

[7] https://www.ndtv.com/feature/why-did-claude-ai-try-to-blackmail-an-executive-anthropic-explains-11475177.

[8] The lore of Terminator gets really wonky and convoluted as so many people have had their hands on it and there have been several reboots and reworkings so some of the movies now contradict each other and the other material (books, video games, etc) makes this problem worse. So the scenario I describe here does conflict with some of the Terminator lore, but at this point almost everything does.