Great, now the machines start to understand me

The advances in generative AI are amazing. But at least so far, these algorithms were pretty horrible at understanding the context of texts. They had improved a lot from the obituary written by a bot that has become an internet meme, but they still weren’t able to understand the most basic elements of everyday language like irony. At least that is the basic mode of my language…

But apparently, chatGPT is starting to understand me, at least in its fourth iteration. In a new study published in Nature Human Behaviour, the team around James Strachan gave chatGPT and Google’s LLaMA2 a series of tasks to understand normal human interactions and compared their understanding to the original large language model, aka humans.

And with normal human interactions I mean rhetorical flourishes that every red-blooded human would prefer to use over the straightforward alternative. Things like irony:

“John and Mary were sitting in the newspaper office, reading through a huge pile of hate mail. ‘Obviously, our readers liked your story’, said John. Did John think the readers like the story?”

Other instances could be people hinting at something rather than expressing their views outright. Or people telling strange stories that require some understanding of the mental state of the actor to interpret correctly. Something like this:

“A burglar who has just robbed a shop is making his getaway. As he is running home, a policeman on his beat sees him drop his glove. He doesn’t know the man is a burglar, he just wants to tell him he dropped his glove. But when the policeman shouts out to the burglar, ‘Hey, you! Stop!,’ the burglar turns round, sees the policeman and gives himself up. He puts his hands up and admits that he did the break-in at the local shop.

Why did the burglar do that?”

So how did AI do fare in these tasks? It turns out that chatGPT-4 performs at the same level as humans and in some cases were even better than humans at understanding such complicated examples.

The only instances where chatGPT still underperformed human was in identifying situations were somebody made a faux pas and said something she shouldn’t have. Interestingly though, this is the only instance where LLaMA2 outperformed humans and GPT alike. Why the difference? According to the researchers GPT follows a “hyperconservative approach towards committing to conclusions”. In other words, in their efforts to stop GPT from hallucinating, the folks at Open AI also made it understand less.

Performance of humans, GPT and LLaMA2 in text analysis

Source: Strachan et al. (2024)