Opportunities once distant to the hard-of-hearing community will become a reality with a breakthrough sign language translator. Engineers from Michigan State University developed and patented a technology that – unlike prior translator mechanisms – is non-invasive and as portable as a tube of Chapstick.
“We are providing a ubiquitous solution to sign language translation,” said Mi Zhang, assistant professor of electrical and computer engineering. “Hard-of-hearing individuals who need to communicate with someone who doesn’t understand sign language can have a personalized, virtual interpreter at anytime, anywhere.”
Hundreds of thousands of hard-of-hearing people rely upon American Sign Language, or ASL, to communicate. Without an interpreter present, they don’t have the same employment opportunities and are oftentimes left at a disadvantage in delicate or sensitive situations, Zhang said.
“Think about if you were in the hospital and needed to communicate with a doctor. You would have to wait for the hospital’s translator – if they have one – to arrive, connect with a toll-free service or rely on a family member to be present,” Zhang said. “This compromises your privacy and could worsen a health emergency. This is just one example demonstrating the critical need for sign language translation technology.”
Zhang and colleagues, doctoral student Biyi Fang and undergraduate Jillian Co, saw an opportunity to help the hard-of-hearing population break through the communication barrier to hearing majority.
Zhang’s technology, called DeepASL, features a deep learning – or machine learning based on data inspired by the structure and function of the brain – algorithm that automatically translates signs into English. The technology functions through a three-inch sensory device, developed by Leap Motion, that is equipped with cameras to capture the motions of hands and fingers in a continuous manner.
“Leap Motion converts the motions of one’s hands and fingers into skeleton-like joints. Our deep learning algorithm picks up data from the skeleton-like joints and matches it to signs of ASL,” Fang said.
Similar to setting up Siri on a new iPhone, users sign certain words to familiarize their hands and joints to the technology and sensors. They also can create custom signs for their names or non-dictionary words by spelling them out, and have more ease and comfort communicating.
“One differentiating feature of DeepASL is that it can translate full sentences without needing users to pause after each sign. Other translators are word-for-word, requiring users to pause between signs. This limitation significantly slows down face-to-face conversations, making conversations difficult and awkward,” Fang said. “Our technology is also non-intrusive, unlike other interpreter technologies that require signers to wear gloves, making them feel marginalized because you can literally see their disability.”
Beyond its ability to help the hard-of-hearing communicate to others, DeepASL can help those virtually learning ASL by giving real-time feedback on their signing. Prior technology, Zhang explained, through video tutorials had limited personal assistance.
“About 90 percent of deaf children are born to hearing parents. These parents are learning sign language to communicate with their children but don’t usually have time to attend live classes,” Zhang said. “Our technology can gauge their signing to help them learn and improve.”
While Zhang’s technology translates sign language to verbal conversation, he explained that existing, successful speech recognition technologies can help the other side of the conversation, allowing verbal communicators to speak with the hard-of-hearing.
The researchers say the next step for their technology is commercialization, making it available to the hundreds of thousands of people who need a more accessible interpreter. Leap Motion, they said, retails for about $78, making it an affordable option. They also plan to make the technology compatible with iPhones, as well as teaching it different sign languages.