TECHNOLOGY/RIGHTS
The creation of hyper-realistic sign language avatars by a New Zealand start-up might not only open pathways to learning in the deaf community but could also mean the difference between life and death in an emergency.
When one of the co-founders of Kara Technologies, Arash Tayebi, developed Meniere’s disease, which led to deafness in one ear, he discovered there was a paucity of educational resources for deaf people. With friends, including Farmehr Farhour — a fellow engineering student at the University of Auckland — he began investigating the issue.
Farhour says their first step was to look at creating captioning software catering to the New Zealand accent. “But we soon realised sign language is the first language of any deaf person. Then when we created the company we said ‘Let’s make digital avatars’ because there is just so much content out there that human interpreters cannot keep up,” he says.
Kara Technologies was born in 2018, the year they finished university.
Farhour is quick to point out they’re not aiming to replace human interpreters. “For instance, in an emergency situation, we can deliver an accurate sign language translation in under a minute, whereas it would normally take a long time to find a human interpreter. However, our technology would not be ideal for long-form contextually-heavy information such as press conferences, which is where human interpreters are needed,” he notes.
Farhour says his team’s technology is a world-first and is able to generate sign language translations in real-time from a natural language input. “We have co-designed the technology with the deaf community from day one and have developed the most advanced sign language avatars in the world. We use the latest Large Language AI models for semantic translation and our own proprietary technology to kinematically drive the avatars based on a dictionary of signs.
“Our process is end-to-end reviewed by deaf sign language experts for quality assurance. That means we can ensure our content is 100% accurate while maintaining the speed of content production,” Farhour says.
The technology uses the best-in-class digital humans from Epic Games and builds on top of pre-existing AI models for content translation
. “We wanted to differentiate ourselves with the quality of the avatars’ sign language. Every country has its own languages, which are very, very focused on minute fingerspelling or specific facial expressions. We realised if you were in the context of storytelling, it’s all about that emotion and how you portray it and it doesn’t make a difference between languages. However, for systems where it is about getting the information out, such as an emergency situation, we wanted to make sure that the way we automate that generation of avatar sign language content is standardised to an extent where it’s not dialect-based,” he says.
Farhour says its deaf community advisory board also deals with questions of ethnicity and culture. For example, a Maori avatar cannot be derived from motion capture of a non-Maori person.
“There’s quite a lot of discussion in the community and there’s no consensus about this. What we do is make sure we standardise as much as possible, and we create avatars that are as neutral and diverse as possible. We try to represent all communities across different spectrums,” he says. “We are a deaf-led organisation, and co-create our technology with guidance from the community and stakeholders. We have conducted extensive surveys in Australia, New Zealand and the US, and the project received a positive reception in all three countries, highlighting the importance of providing equal access to information in sign language for the deaf community.”
To ensure Kara Technologies can expand, Farhour says its system is agnostic. While many countries have differences in their sign language, there are enough similarities that the team only needs to create a new data set of signs, like a vocabulary. The underlying backend of the systems remains the same.
For such a young team, there have been some steep learning curves.
“Our first challenge was keeping ourselves accountable by having the deaf community included in the design process. As engineers coming straight out of university all we want to do is build, but you have to step back and ensure that you do it the right way. You include the people who receive the service at the end of the day, and they need to make the decisions on how it looks, not us,” Farhour says.
Another challenge is to educate organisations as to what accessibility looks like as well as the importance of understanding the need for extra accessibility features and inclusivity.
All indications so far suggest they’re on the right track. The company’s sign language avatar version of The Very Hungry Caterpillar has close to 90,000 views on YouTube and Farhour says feedback, especially from teachers, has been extremely positive. The next challenge is to scale the approach across large chunks of digital content, such as educational resources. “We are in the middle of working on how we automate that process while having some human oversight,” Farhour adds.
The team is also working with the Australian Red Cross and emergency providers in the US to create avatars for emergency and public announcements.
TOP A sample of avatars
ABOVE Farmehr Farhour
PHOTOS Kara Technologies