Great, now the machines start to understand me
The advances in generative AI are amazing. But at least so far, these algorithms were pretty horrible at understanding the context of texts. They had improved a lot from the obituary written by a bot that has become an internet meme, but they still weren’t able to understand the most basic elements of everyday language like irony. At least that is the basic mode of my language…
But apparently, chatGPT is starting to understand me, at least in its fourth iteration. In a new study published in Nature Human Behaviour, the team around James Strachan gave chatGPT and Google’s LLaMA2 a series of tasks to understand normal human interactions and compared their understanding to the original large language model, aka humans.
And with normal human interactions I mean rhetorical flourishes that every red-blooded human would prefer to use over the straightforward alternative. Things like irony:
“John and Mary were sitting in the newspaper office, reading through a huge pile of hate mail. ‘Obviously, our readers liked your story’, said John. Did John think the readers like the story?”
Other instances could be people hinting at something rather than expressing their views outright. Or people telling strange stories that require some understanding of the mental state of the actor to interpret correctly. Something like this:
“A burglar who has just robbed a shop is making his getaway. As he is running home, a policeman on his beat sees him drop his glove. He doesn’t know the man is a burglar, he just wants to tell him he dropped his glove. But when the policeman shouts out to the burglar, ‘Hey, you! Stop!,’ the burglar turns round, sees the policeman and gives himself up. He puts his hands up and admits that he did the break-in at the local shop.
Why did the burglar do that?”
So how did AI do fare in these tasks? It turns out that chatGPT-4 performs at the same level as humans and in some cases were even better than humans at understanding such complicated examples.
The only instances where chatGPT still underperformed human was in identifying situations were somebody made a faux pas and said something she shouldn’t have. Interestingly though, this is the only instance where LLaMA2 outperformed humans and GPT alike. Why the difference? According to the researchers GPT follows a “hyperconservative approach towards committing to conclusions”. In other words, in their efforts to stop GPT from hallucinating, the folks at Open AI also made it understand less.
Performance of humans, GPT and LLaMA2 in text analysis
Source: Strachan et al. (2024)
I have never seen that Brenda meme before, great one :) Of course I had to feed it to ChatGPT to fix it and freely fill in new information if needed. I find it remarkable that it kept the "horses" joke:
In Loving Memory of Brenda Tent
Brenda Tent, a beloved wife, mother, and friend, passed away peacefully at the age of 46, surrounded by her cherished family and comforted by their love. Born on June 12, 1978, Brenda brought light and laughter to everyone who knew her.
A dedicated librarian, Brenda's passion for books and learning was matched only by her curiosity about the world. She was known for her warm spirit, quick wit, and the gentle kindness she extended to all. Brenda had a knack for creating connections—whether helping students find the perfect book or lending a listening ear to those in need.
Brenda met her soulmate, Michael Tent, in high school, and they built a beautiful life together. She found her greatest joy in raising their two sons, Ethan and Lucas, guiding them with love, strength, and wisdom. She had a playful sense of humor and loved crafting, gardening, and trying her hand at new hobbies—her creativity knew no bounds.
Though Brenda joked that she “owned no horses but thought she did,” she cherished every adventure, big or small. A devoted member of her church, Brenda sang with a voice as pure and uplifting as her spirit, a true gift to the congregation. She also had a hidden talent for poetry, leaving behind words that will continue to inspire.
A celebration of Brenda’s life will take place on December 10, 2024, at Grace Community Church. In lieu of flowers, the family kindly asks for donations to the local library fund, a cause close to Brenda’s heart, or simply take a moment to share a story, a smile, or an act of kindness in her memory.
Brenda will be deeply missed but forever remembered for the love she gave and the light she shared.
The founder of machine learning who just won the Nobel prize predicted in 20 years time AI will be more intelligent than humans are. He explained that in 20 years they will be more rational and analytical. You can see it already: humans are becoming more like bots and AI are becoming more like humans; the machines are softening us up. He said AI companies need to invest far more in security NOW so in 20 years time they won't want to control us. But we need to start now. How can we trust AI if we don't trust the companies that are developing it? The CEOs of the digital dictators are like James Bond baddies...what can be done? Bring back the Baader Meinhof?