Imitation training could finally make robot butlers a reality

From The Jetsons to Bicentennial Man, a common threat running along popular culture is the idea that one day, robots would perform the household chores that we all find so deeply tedious.

Numerous barriers prevent this from becoming a reality, sadly. Perhaps the biggest is that robots lack the innate intuition that humans do. It’s this intuition that makes us expert problem solvers.

Let’s suppose we’re in an unfamiliar kitchen, and you ask me to make you a cup of tea. Even though I may have never used the particular kettle before, and I may not be entirely sure where the milk and teabags are kept, I’ll muddle through.

Give me enough time, and I’ll bring you a smashing cup of “builder’s tea.” Milk first, obviously. Anything else is just wrong.

Computers aren’t like that. They need more explicit instruction. You can’t just say “make me a brew.” You’ve got to spell it out, step by step.

Walk to the kitchen. Open the cabinet. Fetch a cup. Open the tea box. Grab a teabag. Put it in the cup. Open the fridge. Grab the milk. Open the milk. Pour an inch into the cup. Put the lid back on the milk. Return the milk to the fridge. Fill the kettle. Turn it on. Wait for it to boil. Pour the boiling water into the cup. Stir the cup. Let it stand for a minute. Remove the teabag. Drink. Aaaaaah.

Get the idea? Fortunately, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), McGill University, the University of Ljubljana, and the University of Toronto are working on a way to make machines more suited to homemaking.

The researchers built a Sims-like environment called VirtualHome. This system represents a typical home environment, allowed the researchers to simulate typical domestic tasks, which were executed by artificial “agents.”

While we’re years away from the world of Bicentennial Man, the ultimate goal of the VirtualHome project is to build the groundwork that could eventually allow real-world robots to perform these tasks.

The researchers trained VirtualHome with 3,000 household activities, which are further broken down for a computer to understand. These activities range from watching TV, turning a toaster on or off, or placing a pot on the stove.

“Describing actions as computer programs has the advantage of providing clear and unambiguous descriptions of all the steps needed to complete a task,” explained PhD student Xavier Puig, who was lead author on the paper.

“These programs can instruct a robot or a virtual character, and can also be used as a representation for complex tasks with simpler actions.”

The VirtualHome agent was able to deconstruct tasks based on natural language. So, you could tell it to “pour milk into a glass,” and it’d figure out the intermediate steps.

The system could also learn tasks, having been provided a video demonstration of the activity.

There’s a caveat here though. The video has to come from the same Sims-like environment used by the robots. However, the researchers hope that eventually, it’ll be able to learn from live-action YouTube videos.

So, you’d record yourself doing something in the house — like opening the oven and inserting a frozen pizza — and it’ll take it from there.

“This line of work could facilitate true robotic personal assistants in the future,” said Qiao Wang, an Arizona State University academic who was not involved in the research.

“Instead of each task programmed by the manufacturer, the robot can learn tasks just by listening to or watching the specific person it accompanies. This allows the robot to do tasks in a personalized way, or even some day invoke an emotional connection as a result of this personalized learning process.”

Personally, I can get behind any research that brings me a robot butler. You can read the full paper from MIT CSAIL here.