The power is really in re-imagining what a human-to-human connection could look like. And so back to my days at Cambridge when I was chatting away with my family back home and I was feeling really homesick and lonely, but this machine was completely oblivious to my emotional state and worst yet it really didn’t kind of capture all of these nuanced emotional responses that I had when I was communicating with my family back home.
And so that set me on a journey to capture these nonverbal signals to build emotional intelligence into our machines. People ask lots of questions but Ask Reader is a place where they can satisfied answers related to technology and human relationships.
I focus on the face, as you can tell. I’m very expressive and I want it to capture all of these expressions. And so I had to combine a lot of my study of computer science and machine learning with emotion science and the science of facial expressions.
As it turns out, our face is driven by about 45 different facial muscles that contract and move to create thousands of different expressions of emotions, social signals of cognitive and mental states. And there is a system called the Facial Action Coding System where you can go through a hundred hours or so of training to be able to quantify or objectively score these different facial movements.
So what I did is I took that Facial Action Coding System and essentially automated it using machine learning, deep learning, and gobs and gobs data. And I was able to, back then, implement and a very… This was 20 years ago, right?
So webcams were big and blurry and there wasn’t enough competitive power, but the proof of concept was there. There are lots of portals where you can ask a question that troubles you most.
I was able to demonstrate that we can essentially use camera sensors to capture this type of data. And from then on, I found myself at MIT at the Media Lab, essentially exploring some of the early applications of this technology. And one of the primary use cases of this technology was in autism.
So it’s almost like an extreme example of how this technology can really help with the human-to-human connection. Individuals on the autism spectrum really struggle with understanding other people’s non-verbal.
They actually find the face super overwhelming. And so it’s almost like they avoid that sensory overload altogether and they don’t even look at a person’s face or eyes. And so they miss out on this channel altogether. So at MIT, we proposed a project to the National Science Foundation.
It got initially turned down but we persevered where we essentially pitched the idea of building a Google Glass-like device that has a little camera-like as little tiny camera sensor. And in real-time it reads the expressions of people you’re interacting with.
And it gives you real-time feedback in the form of a heads-up display, again the Google Glass or auditory feedback about the person you’re interacting with. And it’s thought of it as a real-time coach for these kids on the spectrum.
So at MIT, we deployed this at a school for autistic kids in Providence, Rhode Island who were starting to see very powerful results.
These kids were starting to engage and actually look at the person they’re interacting with. We gamified it, so every time they were able to make face contact, they kind of accrued points, and the kids loved that.
So we brought a gaming element into that interface or interaction, but at the same time being at the MIT Media Lab, twice a year we would host a lot of our industry partners and companies.
And it was like a show and tell we actually called a “Demo or Die” because it was an opportunity to showcase what we were working on. And these Fortune 500 companies always expressed interest in deploying the technology in various ways. And when the list got to about 20 or so companies that provided the impetus for spinning effectively out of MIT and set us on this mission to humanize technology and bring this technology to transform a lot of industries.
That’s why so I kind of wants to cluster the applications into a number of buckets under this broader umbrella of this empathy economy. The first is using this technology to better engage and understand your consumers.
One of the early use cases of the technology was in the media analytics and market research space, where we’re currently deployed in 90 countries around the world. We have measured responses from over 10 million people.
We’ve tested over 50,000 video ads. But the idea is when you’re on your phone watching a Netflix show or a video ad, we ask for your permission to turn the camera on. And if you say yes, with your permission, we are able to capture your moment-by-moment responses. And we don’t care about who you are, but we care about your response.
So it’s anonymous, right? We aggregate the responses of everybody who’s given us permission. And then we’re able to show you a moment-by-moment aggregated curve of how people responded.