The idea of having a helper robot at home that can make fold your laundry or bring you breakfast in bed is kind of awesome, but in order to get to that point, it must be properly trained. In order to make the orientation process easier, Ashutosh Saxena of Cornell University is developing robots that can be trained through verbal communication, without requiring a knowledge of computer programming. Saxena and his team will be presenting their work at the 2014 Robotics: Science and Systems Conference at UC Berkeley on July 12.
Giving directions can be tricky, because someone who is really familiar with a process may forget to specify minor, yet critical details. For instance, when instructing on how to boil pasta, you might specify to fill a pan with water, put it on the stove, and add the noodles once it reaches a rolling boil. However, you may have forgotten to specify how full the pan needs to be (you wouldn’t want the water all the way up to the edge), the fact that the stove needed to be turned on, or how high the heat needs to be. Those missing tidbits can be assumed later with experience, but a naïve robot (or even a child learning to cook) might sit there waiting for the water to miraculously begin boiling on its own.
Saxena has developed software that converts spoken English into computer commands which the robot can understand, and will use a 3D camera to search the environment for the necessary objects. The robot will react to commands based on prior experience, though it can become confused if commands are worded differently. The researchers are working on correcting this by teaching the robots that different commands will have the same action. The robots are exposed to video simulations of activities being performed while commanded by several different human speakers, allowing them to learn by seeing.
The robots are being trained to understand a wide range of functions for a variety of objects, such different pans being used for different things and the fact that things can be heated on the stove or in the microwave. Ultimately, the team hopes to create robots that are flexible enough to perform the same actions, such as cooking, even if placed in different kitchens or having the equipment and ingredients in different places.
In recent testing, a robot was asked to prepare ramen noodles and a dessert made of coffee and ice cream. The robot was able to correctly identify and carry out commands 64% of the time while inferring some missing information, which is about 3-4 times higher than previous trials. While that is fairly impressive, the researchers are constantly striving to improve.
If you would like to help train these robots, the researchers are attempting to crowdsource instructions for a variety of tasks on the Tell Me Dave site. This will expose the robots to a variety of instruction styles and expedite learning.
Check out the robot in action:
10