Can AI Be Conscious? Some Claim It Already Is. An AI Sociologist Cuts Through The Hype

There is plenty of potential in the use of these machine learning techniques, with applications in many scientific disciplines, from finding a cure for cancer to reading an ancient scroll. But there is also a lot of hype about what these computer programs may do or may become. The biggest claim of all is that these programs can be – or already are – conscious.

We do not even know how consciousness arises in us, so it's a big claim to make, which is why FLScience recently spoke to Dr Alex Hanna, director of research at the Distributed AI Research Institute, to tackle the question of artificial consciousness and how humans can navigate the complexity surrounding these claims.

LLMs have been around since the 1990s, but what are they?

AH: A large language model is a type of statistical model that models the distribution of tokens. These are words or parts of words, and it models that distribution. Using that probability distribution, it can be used to predict the next token or next words in a sentence. 

I'm not a linguist, but I am familiar enough with these concepts to state this quite confidently, but in prior iterations, it was a model of some elements of language. Those elements can include syntax, semantics, or the structure of sentences.

In the case of large language models, however, the basic elements are tokens that go together and have some kind of distributional model based on training data. A large language model, in the current sense of the term, uses training data to do that. In the past, it did not require training data. You could model it to understand the structure of a grammar. This is how modern compilers work, for instance: where there is a structure or grammar to a programming language, parsing that language is one of its tasks. That could be a way to think about language, or understand how language operates, that is then put into a computational system.

A large language model, however, is based on these millions of texts on which the model is trained, and then it outputs or predicts what's next given some input prompt.

So basically it's using mathematical probability to work out what the likeliest next word is, not "thinking"?

Yes, yeah, exactly.

Why were LLMs developed, why are they useful?

We can think of lots of different types of potential uses for language models. One of them could be to reframe more complicated language into less complicated language or to do query completion or auto-complete. You can envision those types of uses to be helpful if you're building a search engine or if you are building a grammar device or a much more fancy spell checker. 

Those are much smaller-scale uses.  The difference [now] is the scale at which training data could be ingested and then retained within the architecture. That allows a larger context window, so instead of the window being a sentence, now there is a context window of a couple 100 words or a couple 1,000 words. Large language models really scaled all that up intensely.

What are the challenges and drawbacks that one might encounter with LLMs?

There are many different [challenges and drawbacks]. Some of them emerge from scale. Some of them emerge from what they're used for.

In terms of scale, one of the drawbacks we've seen is that the training data that are used are very hard to document. Because companies like OpenAI, Google, and Anthropic are scraping anything that's not nailed down on the internet, there's really no way to document what's in the data, what biases might be endemic to the data, and how that is going to affect model outputs.

We know from a lot of research that these outputs are biased in terms of social categories like race, gender, class, and religion. That's one function of scale.

The other function of scale is that these technologies are very expensive to train and they take a lot of energy. That has required this massive data center buildout that we've seen all over the world, especially in communities that are not benefiting from any of those outputs.

In terms of their outputs, there's nothing about an LLM that is going to guarantee correctness. This is called, in literature and in popular imagination, hallucination. We don't like to use that term because hallucination is an aspect of humans and human psychology. What the machines are doing is making stuff up: there's the regurgitation of prior training data into things that look like what you'd expect from the distribution of tokens within the training data set.

Then the other drawback is the societal element of it. When we see things that are mimicking human language and seem like they have acceptable responses to language and communication, there are parts of [society] that seem very apt for replacement by LLMs. We see this in social services, in government services, in employment, in healthcare.

Those are really dangerous places in which to have text that is not guaranteed to have any kind of sense of correctness.

Do you think any current LLM is conscious? Do you think they could ever be conscious?

The short answer is no and no.

The longer answer is no, and the reason why we think they're conscious stems from a few things. One of them is what our brains do with language; we are often thinking about how there is a mind behind text that can either be written or spoken, or can be signed. That is something that our brain does. It makes attributions to language.

This is because in non-chatbot environments our brains are apt to say, "Oh, there's coherent language; there must be some kind of subjectivity behind that." This is because when we see language that is written or spoken, we are imagining a mind behind it. This is why we think that there is some consciousness happening there.

The other element of it is that so many of the analogs that we use in the conversation around LLMs have narrow biological significance. We talk about models as being a type of neural network.

Many of the tasks used in machine learning became neural-network-based because that happened to do well enough on particular types of tasks and had the ability to scale in terms of training and in computing that made those types of models more attractive. So the neural network metaphor has gone quite far.

That is not based on how neurons work. I mean, we know little about neurons and how they work, and I think machine learning people have also run with it and are extending the metaphor. They are trying to almost backdate or make the claim that the computational neuron looks like the biological ones.

That's been the argument about why these things could be conscious, both the linguistic one as well as the language we use around neural networks. But they're not conscious. They're not going to be conscious. They are next-token-prediction machines.

There is an argument that while AIs are not and cannot be conscious, they might develop superintelligence and outthink humans.

What these machines are doing is not thinking, and they do not possess intelligence. The allusions to intelligence have a dark history. People like [evolutionary biologist] Stephen Jay Gould effectively say any kind of reification or quantification of intelligence is going to lead to this stack ordering of beings, and then a comparison of language models and humans. 

If what we're talking about is LLMs performing better on tests, this is typically because there's been some ingestion of training data, especially if it's something like a fact-based situation, that the machine could possibly do better than a person on a test. But that is not saying anything about intelligence; that's saying something about the way these systems are trained and what they do well with regard to matching particular outputs. That is not intelligence, and we ought not to quantify or characterize it as such.

What is something you'd like the general public to know about AI?

I would like the general public to first stop calling everything AI. There are lots of different technologies that get characterized as AI. It's better to think about what's being automated. These are not all "AI."

It's helpful to think about what the stakes in that automation are when something goes right and also when something goes wrong. When it goes wrong, that can lead to many types of errors. But when it's right, it can be used to justify the replacement of human discretion in many different situations. It's really important to know who should be accountable for decisions.