Computer science class fails to notice their TA was actually an AI chatbot

With all this talk about chatbots from Facebook and Microsoft, teaching artificial intelligence to be smarter has become a central topic of the tech world. But what about what AI can teach us?

Ashok Goel, a computer science professor at Georgia Tech, put that question to the test when he added “Jill Watson” – a chatbot powered by IBM’s Watson technology –  to his list of of teaching assistants for an online course. The chatbot was so good at answering questions that students did not notice their TA was made of silicon until after they’d turned in their finals.

Now that’s what I call machine learning.

Jill came to be after Goel decided he and his teaching assistants were being spread thin. Goel’s class was a popular online course, and his teaching team receives over 10,000 online questions per semester. Jill was trained by reading questions and answers from previous semesters, and was set to only respond to new ones if it was 97 percent confident in its answer or higher.

More difficult questions were reserved for the human TAs, and some students did become suspicious:

Still, the experiment proved successful; one student even planned on nominating Jill as an outstanding TA for the school’s teacher survey. 

Jill will be renamed in future semesters in order to keep the guessing game going, but it’ll be interesting to see whether students can identify their synthetic TA now that they know to look for it.

Meanwhile, Goel plans on bringing the chatbot to more schools and classes. While he doesn’t see Jill completely replacing professors and assistants, he thinks giving more students the opportunity for one-on-one interactions – even if with an AI – will help keep them engaged in the coursework.