High-tech clues to facial cues
A new device helps people with Asperger's or autism interpret expressions of emotion
PROVIDENCE - TJ can usually tell when someone is happy. Happiness is easy, but negative emotions are trickier. Sorrow can look like laughter; sadness can pass for fatigue. The expressions that signify guilt and hunger, he said, "are the exact same thing."
TJ Conway, 16, has Asperger's Syndrome, a neurological condition related to autism, which makes it hard for him to interpret facial expressions. Is that smile sarcastic, apologetic, polite, angry, or just happy? TJ is not always sure. It's a disability that can strain his dealings with the world.
Today he is holding a small computer, about the size of a hardcover book, that, like TJ, is trying to solve the complex problems of recognizing facial expressions. Point the device at a person's face - as one would with a camera - and, ideally, it will identify the emotion they are feeling.
Although the computer prototype has many bugs to work out, it represents the next generation of technological assist that, if successful, could help people with Asperger's - including the half dozen or so teens in the Providence classroom with TJ - make sense of the emotional world around them. The device is a product of MIT's Media Lab, where computer scientists are studying ways to help machines think emotionally, like humans.
The prototype is called an iSet, which stands for "interactive social emotional toolkit." It's a tablet with a camera on one side and a screen on the other, like an oversized cellphone. TJ points the device at another teen, and holds it steady until a choppy black-and-white video image appears on the screen, along with such labels as "agreeing," "disagreeing," "concentrating," and "confused."
When the computer thinks it's identified a facial expression, a colored dot appears above the corresponding label. The dot grows larger as the computer becomes more confident it has identified the correct emotion.
The software behind the iSet combines commercially available face-recognition programs with machine-learning algorithms that allows the computer to compare new facial expressions to ones it's already seen, and to calculate probabilities that a certain facial expression might mean a certain thing. It was developed by MIT researcher Rana el Kaliouby.
Some social anomalies were apparent among the teens testing the device. One boy talks in blurted lists of facts. Another over-annunciates his words. Some are uncomfortable with direct eye contact. They call themselves "Aspies." Once a month they meet here, at the Groden Center, a school for children with autism and other disabilities. It's a rare opportunity for them to socialize on their own terms.
"There's a comfort level," said Virginia Law, TJ's mother. "They all know that they each have their own little quirks. Outside their group, it's harder."
"I think my awareness of other people's emotions is slightly less than other people's," said Stephen Shea, 16. "I tend to say the wrong thing more often than other people."
For all of them, the challenge is to consciously recognize cues, like a smile or a raised eyebrow, that most of us recognize by instinct.
Rosalind Picard, a computer scientist at MIT's Media Lab, has been studying those emotional cues since the mid-1990s. At first, Picard said, she was hesitant to tackle such a project in a field in which "feelings" were seen as the opposite of rational, cognitive thought. But research shows that emotions are essential to cognition, she said. Emotions help people to catalog and retrieve memories and to prioritize goals.
Picard wondered: What if computers had access to that power? Even better - what if they could recognize emotions in their human users, and react appropriately?
The idea was a huge technical challenge.
The complexity of problems such as solving moves in games such as chess or Go, Picard said, "are dwarfed by trying to solve all the possible moves in peoples' faces."
People can make around 10,000 different combinations of movements on their faces, she said, and change these in milliseconds.
When a friend suggested she research autism and Asperger's Syndrome, Picard realized that her research might have practical applications people with those conditions.
"We're helping computers be emotionally intelligent," Picard said. "Well, there are people who need that too."
That realization lead to the Media Lab's new Autism Communication and Technology Initiative: a group of researchers working on technology to help people with autism communicate socially.
"We're trying to meet people in the middle ground and build tools and technology for them," Picard said. Her philosophy: "Don't 'fix' them, or rework them into a completely different person, but help them to adapt themselves in any way they choose to adapt."
On a crisp fall day, the boys at the Groden Center are ready to test the initiative's newest prototype.
"You can tell us what you like about it, what you dislike about it, and if you have any ideas about how to improve it we definitely want to hear it," Matthew Goodwin tells them.
Goodwin has worked with Picard since his grad-student days at the University of Rhode Island; he's now director of clinical research at both the Media Lab and the Groden Center.
TJ frowns down at the device in his hand. The camera is picking up Goodwin's bespectacled face. Is Goodwin agreeing? Disagreeing? At that moment, the computer is drawing a blank.
The project has cost about $500,000 to develop so far, Picard said. But the device obviously has a long way to go before it could be practical.
A subject has to stand perfectly still in order for the computer to lock onto their features, and if the camera isn't held steady, the computer can get confused and think the subject is moving around.
"So you see this is not a perfect system," Goodwin says ruefully. "We still have to work some of these bugs out."
At this point in the iSet's evolution, the boys are better at recognizing facial expressions than the computer is. And none of them like the idea of carrying around a bulky tablet that, they say, would only stigmatize them more.
But they do like the idea of a small, discrete version - a pocket translator for emotions. And that is what they hope will evolve from the work-in-progress at hand.
"If it gets past the prototype stage, it would be really useful," TJ said. "To some level, everyone could benefit from it."
His friend Steven agreed, noting. "You could react to things in the moment more, instead of having to realize things retrospectively."
More importantly, he said, a device like the iSet could allow him to navigate other people's emotions without having to change who he is.
"A lot of times, what people do is they try to make the person that's 'different' normal," he said. "Shouldn't we make it so everyone can be understood, and understand things, on their own terms?"
Picard and Goodwin say they hope people with autism and Asperger's will become less like test subjects and more like co-designers.
That way, they and the computers can learn from each other.
"The partnership between a person with autism and a computer," Goodwin said, "is more powerful than either one alone."
S.I. Rosenbaum can be reached at sara.i.rosenbaum@gmail.com 