Robots are getting the idea: How a Cal lab is making dramatic advancements in machine learning and robotics
By Daniel Zou and Paul Grosen
For millions of years, animals have had a monopoly on learning — we pride ourselves on our ability to generalize from past experiences. But at the Berkeley Artificial Intelligence Research (BAIR) Lab, a team of researchers is succeeding in challenging that eons-long dominance with powerful computers and a fleet of robots.
The current state-of-the-art robot training technique, reinforcement learning, requires vast training datasets and enormous computational power to teach computers how to perform tasks. However, graduate student Abhishek Gupta and other researchers at the BAIR Lab are augmenting this conventional approach with two new technologies, meta-learning and imitation learning, to dramatically improve training efficiency. These advances enable BAIR robots to imitate complex, human-like tasks, and even apply their knowledge to previously unseen challenges.
Traditional reinforcement learning (RL) can be thought of as the computer equivalent of how animals have been trained for centuries. To start, the robot takes a random approach to a task (for example, by flailing its arm about wildly), on which it will likely perform poorly according to the so-called reward function that tells the robot how well it is doing. But by using the reward function to figure out what small changes will improve its score, the robot can progress toward the desired goal. Gupta likens it to the process in which a child adopts behaviors: “The way that young children learn is through this process of trial and error where they experiment with the world, get feedback on whether the thing they are doing is good or not, and, based on said feedback, they improve their behavior to be more and more directed.”
Reinforcement learning has become increasingly popular over the past decade due to its exceptional ability to perform “fuzzy” tasks that can’t be easily defined by a set of rules. In fact, it’s extremely prevalent in that domain today, spanning everything from speech-to-text to translation services to object recognition to self-driving cars.
But RL is not without drawbacks. Because of its brute-force nature, training a robot requires considerable computing power and can be extremely time-consuming. As a result, it is often infeasible to those with limited resources to perform it. Furthermore, the skills learned by the robot are startlingly specific: while it may grasp a particular cone without any problem, it could fail dramatically on a pyramid or a slightly taller cone.
In an effort to tackle these problems, Gupta’s team is researching a subfield known as meta-learning; literally, “learning how to learn.” With meta-learning, the objective is “not just to get the best performance on training tasks, but to get to the best performance as quickly as possible,” says Gupta. Meta-learning shifts the burden of learning a complex task to a number of more basic tasks, in the hope that parameters derived from the process of training, rather than the training results, can be reapplied. Then, the robot can use the knowledge it gained to very quickly solve the more complex task with relatively few iterations.
In addition to meta-learning, Gupta is exploring another variant of RL known as imitation learning, which incorporates a set of “demonstrations” that show the robot how to perform the task at hand. “You can think of demonstrations as a way of telling you what to do, and also how to do it,” Gupta says.
In previous work, researchers used complex setups that involved moving the robot arm manually or with a virtual reality (VR) controller in order to let the robot precisely measure what movements were being made. Obviously, this isn’t ideal; moving a piece of steel manually is cumbersome at best, and VR controllers aren’t designed with robot arms in mind.
BAIR Lab is attacking this problem by introducing video demonstrations that are simple recordings of a person performing the task. To do this, they’ve added an additional step to the robot’s processing pipeline: when it encounters a new scenario, it first transforms each of the demonstrations to look like its current view. Then, using this new data, it can try to match its own actions to the examples.
Currently, the usage of video-based demonstrations to train robots is still restricted by several factors. Gupta says that “[video demonstrations] are still very controlled — the scene can’t have clutter, the background must be white, the robot and the human is restricted in the workspace. It still may be a few years away from the public.” Nevertheless, he still envisions this relatively simple and cost-effective approach to imitation learning as a way forward.
Ultimately, these new techniques have wide-reaching applications for society. Unlike other machine learning research, Gupta says his work “has the potential to be applicable to our day-to-day lives instead of when we’re just at the computer.” He envisions novel, autonomous robots in a variety of scenarios: from machines that can restore dignity and independence for the elderly to heavyweights that can assist firefighters and search for survivors of a nuclear disaster, this tech shows promise for saving lives and helping humanity for generations to come.
The work that engineers do shapes the world around us. But given the technical nature of that work, non-engineers may not always realize the impact and reach of engineering research. In E185: The Art of STEM Communication, students learn about and practice written and verbal communication skills that can bring the world of engineering to a broader audience. They spend the semester researching projects within the College of Engineering, interviewing professors and graduate students, and ultimately writing about and presenting that work for a general audience. This piece is one of the outcomes of the Fall 2019 E185 course.
