Bot or Not?

Bot or Not was a verbal strategy web game I made inspired by (but not identical to) the Turing test. Players entered the game are paired with a real, human partner. Both players began chatting in one of two game modes: either players were chatting with each other, or they were both chatting with a learning chatbot. The first player to correctly guess which game mode they were in wins.


Here’s how it’s different (and easier) than the Turing test: you are simultaneously the judge, and the subject of judgement. So, if the bot is unhuman you cannot be sure if it is just bad AI or a person pretending to be a bot. These game dynamics are much more forgiving for the programmer than the actual Turing test, where the human has no incentive to be anything but human. Still, I think you’ll be surprised at how tricky the bot is, given its simplicity.

The bot stores every conversation you have in it’s database (MongoDB for those interested). If the conversation ends with you or your partner guessing “Not” (ie., they think it’s a human like conversation), then all exchanges in that conversation become fair game for the bot to use from then on. To come up with a response, the bot does a full text search of your prompt through all messages in its database (about 175,000 of them at the time of writing), to find the archived message that is most like the one you just typed. After doing some filtering and randomization, it narrows it’s search down to a single message, and it responds to you with the response to that similar message from its database. The bot has no knowledge of grammar, vocabulary, or semantics – it has the sole, powerful ability to recognize similarities between its stimulus and experience.

Bot or Not was released on May 23, 2014. I posted the site to Reddit and got a pretty good reaction – it was the #1 web game for two days and on the front page for six days. You can check out the Reddit thread here. I kept it active until April 2, 2015.

Here are the results after ~24k conversations:

confusion matrix

Note that the bot convinces players it’s human almost 44% of the time – pretty good!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s