Facebook’s latest AI experiments: generating captions and recognizing faces in videos

At its F8 developer conference in San Francisco today, Facebook demonstrated its latest artificial intelligence (AI) research efforts. Not surprisingly, they are about video.

Video implies a whole bunch of individual images put together. So it logically flows from Facebook’s progress around object recognition and image caption generation using AI.

“You can imagine us building image search on steroids,” Joaquin Quiñonero Candela, Facebook’s director of Applied Machine Learning, said onstage today. Think of what Google Photos lets you do, but think of what Facebook’s implementation would look like — being able to search through all the pictures your Facebook friends have shared. But video is where scalability makes things more challenging and more interesting.

Quiñonero Candela showed off two specific efforts:

1) Generating captions for the things people say in videos.

2) Identifying the people who appear in videos so that they can be tagged, and even associated with specific times in the video, so that users can get right to the moment when a person first appears in the frame.

The company has been quite busy with artificial intelligence research in the two and a half years since tapping Yann LeCun to head up the Facebook Artificial Intelligence Research (FAIR) division.

Facebook is one of several Web companies engaged in the advancement of deep learning, a type of AI that generally involves training artificial neural networks on large sets of data and then getting them to make inferences about new data.

Facebook AI researchers have published many academic papers documenting their discoveries, and the company has open-sourced AI software and even shared designs for its specialty AI hardware. Earlier this year Facebook said it would send AI-oriented servers to academic labs around the world.

In the specific domain of video, Snapchat, among others, is already engaged in AI research. But Facebook’s AI research operation is more mature than Snapchat’s, and that company’s user base is not as large as Facebook’s.