Facebook adds spoken photo captions for iOS screen readers

Facebook today announced the launch of a feature called automatic alternative text. It’s a big step forward in accessibility because it will allow people who use screen readers — software that provides spoken descriptions of what’s onscreen — to receive summaries of what’s contained in photos that people post on Facebook.

Until this point, only the name of the person, the person’s share text, and the word “photo” was declared by a screen reader’s automated voice when a post with a photo appeared in the News Feed. Now, thanks to artificial intelligence (AI), certain people with disabilities will be able to get a better understanding of what their friends are trying to communicate.

The new captions are short and at times so simple that they’re poetic. For instance: “image may contain sky, tree, outdoor.” Facebook is initially testing out this feature on iOS for people who use the social networking app in English.

“We’re … making it possible for people to feel totally included in the social interaction and be able to feel part of it without having to feel awkward, without having to be annoying to all of your friends, being like, ‘What’s so funny in this photo?’ Nobody wants to do that,” Facebook accessibility specialist Matt King told VentureBeat in an interview.

The launch of the feature comes one week after Twitter announced that users could manually include captions of the photos they tweet out. And in the days since then, Apple and Microsoft have both hinted at their commitment to accessibility.

But the contrast between what Twitter did and what Facebook is doing lines up with the fact that Facebook is spending a lot of money on AI research, while Twitter only established its Cortex AI research lab last year.

Like other parts of Facebook, deep learning — a type of artificial intelligence that involves training artificial neural networks on lots of data, like photos, and then getting them to make predictions about new data — is at the core of the new feature. Facebook’s neural networks have certainly seen a lot of photos and have learned how to recognize what’s in photos. In fact, people now share more than 2 billion photos every day on Facebook, Instagram, Messenger, and WhatsApp.

Even so, Facebook doesn’t want to overextend its reach when telling people what’s in a photo. Automatic alternative text only presents users with a select few words to describe what’s in an image. It only mentions things that it’s very confident are actually in the image, Facebook head of accessibility Jeff Wieland told VentureBeat. As Wieland likes to say, less information is better than inaccurate information.

King doesn’t seem to mind that the feature is still evolving. Since he began using the new feature internally, he’s seen it get better. During the interview, held on Facebook’s campus in Menlo Park, California, King, who is blind, found himself impressed that the system informed him about the presence of cauliflower and deviled eggs in the photos attached to one Facebook post. “It recognizes those things — that’s sweet!” He said. “Oh, wow!”

For those who aren’t blind, this change to Facebook might sound like little more than a novelty. But for those who struggle to see exactly what’s onscreen, it could make a big difference in usability and usefulness. King, who spent more than 20 years working on accessibility and other things at IBM before joining Facebook’s accessibility team last year, remembers when using Facebook was “really frustrating.” It wasn’t that long ago. But the experience has improved since the company formed the accessibility team a few years ago, King said.

Facebook is a service for everyone, and the company isn’t selling software to government agencies, where employees will be required to use it (even if the accessibility bar there is low). People aren’t required to use Facebook; they’ll only use it if they enjoy it. And in this case, Facebook is trying to make things a little bit better for certain people.

“How to make software just as fun and enjoyable and easy to learn for people with a disability as it is for everybody else — that’s what I’ve always wanted to do,” King said. “I want to make the world as good for people with disabilities as it is for everybody else.”

When you look at the big picture, then, this addition to Facebook is actually pretty important. And it could help attract and retain more users — along with other company initiatives, like its Aquila drone for beaming down the Internet from the sky and it’s Free Basics program.

People can turn on automatic alternative text by enabling VoiceOver in the Accessibility menu on iOS devices. The technology is pretty much ready to go on Android, too, Wieland said. The feature will be come available in languages other than English later.

See Facebook’s blog post on the news for more detail.

Update on April 26: Facebook is rolling out automatic alternative text on Facebook for Android this week, a company spokesperson told VentureBeat in an email.