Robots Learn to Perform Tasks by Watching Videos

Published: February 3, 2015 9:39 PM /


umd robotics

A stunning development in artificial intelligence may pave the way for personal robot butlers to perform everyday tasks. Researchers from University of Maryland Institute for Advanced Computer Studies have developed an AI capable of learning to perform tasks by watching videos, and then performing those actions for itself. So far the robots have mainly been watching cooking videos, and learning how to grasp and manipulate cooking utensils, but the implications of this development go far beyond cooking.

The scientists originally chose cooking to test out this AI system, because cooking is a common task that ordinary people are familiar with, but still complex enough to test out the AI's capabilities. This AI is an improvement over previous AIs which simply copied movements exactly. It can break down cooking videos it watches into distinct steps, figure out what the goal of each step is, and then decides for itself the most efficient movements to accomplish that goal, which may differ in some way from what it is viewing. It can string together different actions it has learned in novel ways.

In addition to a powerful decision-making algorithm, they also designed systems to give the robot accurate vision, so that it could perceive and track the movement of objects. Also required was a powerful natural language processing system, which can understand spoken instructions. While these features certainly helped it understand the videos it was watching, the more long-term goal is for this type of AI to interact with humans, and being able to understand people through their words and movements will be invaluable.

However there are numerous other applications for this AI. Robots can be sent to perform dangerous tasks such as bomb disposal, rather than risking human life. These learning robots can essentially be trained to perform tasks in the same way humans are currently trained, and perform them just as well. The scientists believe they are ushering in a new industrial revolution, where robots no longer need to be carefully programmed by humans to perform a particular task, but can learn it on their own.

What do you think will be the long-term results of this development in AI? Leave your comments below.

Have a tip, or want to point out something we missed? Leave a Comment or e-mail us at

No author image supplied
| Senior Writer

I’m a technology reporter located near the Innovation District of Kitchener-Waterloo, Ontario.