If someone plops you down in the middle of a building you’ve never been in before, you might feel a bit lost. But you could look around, find a door and get out of the room quickly without bumping into anything or getting stuck. Most robots could not do that. Researcher Antoni Rosinol first got interested in this problem while working with drones. Usually, someone pilots a drone via remote control. That’s because most drones can’t fly very well by themselves. “They can barely go forward without colliding against a tree,” notes Rosinol. He is a PhD student studying computer vision at the Massachusetts Institute of Technology (MIT) in Cambridge.

Perception is a big problem for many navigating robots, just as it is for robots that grasp things. The robot needs to map the entire space around itself. It needs to understand the shape, size and distance of objects. It also has to identify the best paths to get where it needs to go.

Computer vision has gotten very good at detecting and even identifying the objects around a robot. Some robots, advanced drones and self-driving cars do a very good job at getting around obstacles. But too often, they still make mistakes. One of the trickiest things for a navigating robot to handle is a large, blank space such as a ceiling, wall or expanse of sky. “A machine has a lot of trouble understanding what’s going on there,” Rosinol explains.

Another issue is that robots don’t understand anything about human living spaces. A robot trying to get out of an unfamiliar room will usually circle around, looking for openings everywhere — including on the floor. If it finds a bathroom, it may go in, not realizing that this room won’t lead anywhere else.

The MIT team has developed a system that can help solve this problem. They call it Kimera. “It’s a group of maps for robots,” says Rosinol. Those maps are nested into layers. The bottom layer is what most robots already create. It’s a map of the shape of the three-dimensional space around them. This shape has no meaning to a robot, however. All it sees at this level is a bumpy mass, as if the world around it were all the same color. It can’t pick out walls, doorways, walking people, potted plants or other things.

Kimera’s other layers add meaning. The second layer divides up the bumpy mass into objects and agents. Objects are things like furniture that don’t move around. Agents, such as people and other robots, do move. A third layer identifies places and structures. Places are open areas that the robot can move through, such as doorways or corridors. Structures are walls, floors, pillars and other parts of the building itself. The final two map-layers identify rooms and the entire building that those rooms belong to. A robot equipped with Kimera can build all these maps at once as it moves through some space. This should help it more easily find a direct path.

Task 3: Understand people

No matter what a robot tries to do, one big thing holds it back. Robots lack common sense. This is the knowledge that people don’t have to think or talk about because it’s so obvious to us. Thanks to common sense, you know that you should grab a lollipop by the stick and not by the candy. You know that doors are almost never located on the floor or ceiling. And so on.

Computer scientists have tried teaching robots common-sense rules. But even huge databases of these rules don’t seem to help much. There are just too many rules and too many exceptions. And no computer or robot understands the world well enough to figure out these rules on its own.

This lack of understanding makes grasping and navigating tougher. It’s also the main reason that virtual assistants sometimes say ridiculous things. Without common sense, an AI model can’t understand people’s words or actions, or guess what they’ll do next. “We want to build AI that can live with humans,” says Tianmin Shu, a computer scientist at MIT. This type of AI must gain common sense.

But how?

We learn a lot of common sense from our experiences in the world. But some common sense is with us and many other animals from birth, or soon after.

Before they are 18 months old, human babies understand the difference between agents and objects. They also understand what it means to have a goal and how obstacles can get in the way of a goal. We know this because psychologists have run experiments to test babies’ common sense.

Based on these experiments, Shu decided to make a virtual world called AGENT. Like DexNet, AGENT is a training arena for AI models. The models practice by watching sets of videos that demonstrate a basic common-sense concept. For example, in one video, a brightly colored cube jumps over a pit to reach a purple cylinder on the other side. In the next two scenarios, that pit is gone. In one of them, the cube jumps anyway. In the other, it moves straight to the cylinder. Which of these decisions is more surprising?

Common sense says that you take the most direct route. Babies (and grown-ups) show surprise when they see an unnecessary jump. If an AI model gets rewarded when it shows surprise at the jump, it can learn to show surprise for similar scenarios. However, the model hasn’t learned the deeper concept — that people take the shortest path to a goal to save time and energy.

Shu knows this because he tried training a model on three types of common-sense scenarios. He then tested it on a fourth type. The first three scenarios contained all the concepts needed to understand the fourth. But still his model didn’t get it. “It’s still a grand challenge in AI to get a machine that has the common sense of an 18-month-old baby,” says Melanie Mitchell. She’s an AI expert at the Santa Fe Institute in New Mexico.

Task 4: Think of new ideas

In spite of these challenges, AI has come a very long way — especially in recent years. In 2019, AI models that had learned to recognize both dragons and elephants still couldn’t combine those into a new concept, an elephant dragon. Of course, a child can imagine and draw one quite easily. As of 2021, so can a computer.

Dall-E is a brand new AI model made by the company OpenAI. It turns a text description into a set of new, creative images. (The name “Dall-E” is a combination of the last name of Spanish artist Salvador Dali and the robot Wall-E, the star of a 2008 Disney movie.)

OpenAI released a website on which people can pick combinations of words from a drop-down list and see what images Dall-E comes up with. Possible combos include a kangaroo made of carrots, a confused jellyfish emoji and a cucumber in a wizard hat eating ice cream.

Dall-E doesn’t have common sense. So its imaginings aren’t always on target. Some of its carrot-kangaroos look more like awkward orange blobs. And if you tell it to draw a penguin wearing a green hat and a yellow shirt, all the pictures show penguins, although some sport yellow or red hats.

Sometimes though, Dall-E’s non-human way of seeing things is delightful and creative. For example, if you ask a person to imagine a shark playing chess, they’ll probably draw the shark using its fins as hands. In one of Dall-E’s images, a shark uses its tail instead. Someday, Dall-E or a similar model could help human artists and designers come up with a wider range of ideas. “So many things I’ve seen, I never would have thought of on my own,” says Ashley Pilipsizyn, who until recently was technical director at OpenAI.

It’s very likely that in the coming years someone will design a graceful robot or even an AI model with common sense. For now, though, if you want to beat a robot at chess, make it play on a real, physical chess board.