A long time ago I wrote about the Loebner Prize, and how it seemed like this competition isn’t so much a test of machine intelligence, but rather a test of how well the programmers can fool the judges into thinking they’re talking to a human. Not that anyone has actually done this successfully — none of the judges have ever been convinced that any of the chatbots were really a human, and the annual “winner” of the Prize is decided by sympathy points awarded by some very charitable judges.
In that previous post, I remember being struck by how little has changed in the quality of the chatbots that are entered into this competition: it hadn’t improved since the inception of the prize. So, today I thought I’d randomly “check in” on how the chatbots are doing, and read the chat transcripts from the competition from more recent years. (I encourage you to read the transcripts for yourself, to get a taste of the “state of the art” of today’s chatbots.)
And can you guess how they’re doing? Of course you can: they still haven’t improved by any margin, except perhaps a larger repertoire of cleverly crafted catch-all responses. Even the winner and close runner-up of the 2017 competition are comically robotic, and can be effortlessly pegged as artificial by a human judge.
My goal in this post, however, is not to bash the authors of these chatbots — I’m sure they’re competent developers doing the best they can. My goal is to discuss why chatbot technology hasn’t moved forward since the days of ELIZA. (Note: When I refer to chatbots, I’m not referring to today’s virtual assistants like Siri or Alexa, which have made big strides in a different way, but rather to human-approximating, Turing-test-passing machines, which these chatbots attempt to be.)
I think much of it has to do with a lack of corporate support. Chatbots have never really found a good business use case, so we haven’t seen any major companies devote significant resources to chatbot development. If someone like Google or Amazon had put their weight behind this technology, we might have seen an advancement or two by now. Instead, the competitors in the Loebner Prize still consist of individual hobbyists and hackers with no corporate backing.
Interest in chatbots seems to have peaked around 2012, when the cool thing was to add a customized bot to your website and let it talk to your users, but thankfully this died down very shortly thereafter, because apparently people prefer to communicate with other people, not lame attempts at imitation. We can theorize that someday we may hit upon an “uncanny valley” effect with chatbots, where the conversation is eerily close to seeming human (but not quite), which will cause a different kind of revulsion, but we’re still very far from that point.
Another thing to note is the actual technology behind today’s (and yesterday’s) chatbots. Most of these bots, and indeed all the bots that have won the Loebner Prize in recent years, are constructed using a language called AIML, which is a markup language based on XML. Now, there have been plenty of ways that people have abused and misused XML in our collective memory, but this has to be one of the worst! AIML attempts to augment XML with variables, stacks, macros, conditional statements, and even loops. And the result is an unwieldy mess that is completely unreadable and unmaintainable. If chatbot technology is to move forward, this has to be the first thing to throw out and replace with something more modern.
And finally, building a chatbot is one of those endeavors that seems tantalizingly simple on the surface: if you look at the past chat logs of the prize-winning bots, it’s easy to think to yourself, “I can build a better bot than that!” But, once you actually start to think seriously about building a bot that approximates human conversation, you quickly come up against research-level problems like natural language processing, context awareness, and of course human psychology, behavior, and consciousness in general. These are most definitely not problems that can be solved with XML markup. They likely can’t even be solved with today’s neural networks and “deep learning” algorithms. It will probably require a quantum leap in AI technology. That is, it will require building a machine that is truly intelligent in a more general way, such that its conversations with humans are a by-product of its intelligence, instead of its primary goal.
For now, however, the dream of chatbots has been laid to rest in the mid-2010s, and will probably not come back until the technology allows it, and until they’re actually wanted or needed.