Facebook VP: AI has a compute dependency problem

In one of his first public speaking appearances since joining Facebook to lead its AI initiatives, VP Jérôme Pesenti expressed concern about the growing amount of compute power needed to create powerful AI systems.

“I can tell you this is keeping me up at night,” Pesenti said. “The peak compute companies like Facebook and Google can afford for an experiment, we are reaching that already.”

More software innovation will be required if artificial intelligence is to grow unhindered, he said, and optimization of hardware and software — rather than brute force compute — may be critical to AI in years ahead.

Examples of systems less reliant on compute for innovative breakthroughs include Pluribus, an AI system developed by Facebook AI Research and Carnegie Mellon University and introduced today, that can take on world-class poker players. In an article in Science, researchers said Pluribus only required $150 in cloud computing to train.

The end of Moore’s Law means the compute needed to create the most advanced AI is going up.

In fact, Pesenti cited an OpenAI analysis that found the compute necessary to create state-of-the-art systems has gone up 10 times each year since 2012.

“We still see gains with increase of compute, but the pressure from the problem is just going to become bigger,” Pesenti said. “I think we will still continue to use more compute, you will still net, but it will go slower, because you cannot keep pace with 10 times a year. That’s just not possible.”

Analysis introduced last month found that the costs of training systems like OpenAI’s GPT-2 can exceed carbon emissions of the lifetime of five cars.

Speaking onstage at VentureBeat’s Transform 2019 event, Pesenti talked about the unique challenges Facebook encounters when deploying AI systems for 2.8 billion unique users around the world. These challenges include parsing nuance — like determining whether a post qualifies as hate speech or whether a video has simply been altered or is a deepfake.

Road blocks companies may encounter on their journey to deploy AI can be cultural, logistical, or just a failure to recognize that the AI stack isn’t the same as the typical engineering stack. 

AI plays a role in virtually every aspect of Facebook’s services, from determining which ads to display to making recommendations on Instagram to content moderation. AI also powers new customer experiences, such as Portal’s Smart Camera.

Many Facebook services are powered by Intel CPUs, Facebook engineering manager Kim Hazelwood said last year.

Pesenti — like executives from Google, Microsoft, and Airbnb in their Transform 2019 talks — also spoke about the importance of diversity in hiring and of making sure AI works the same for everyone.

He believes bias typically comes from data sets, rather than the creators of AI systems.

“We’re making progress. It’s still very far from where we ought to be,” he said. “We need to do everything we can to increase the diversity in the field.”

Facebook shared new statistics related to company diversity earlier this week, but did not break out statistics about race or gender diversity within divisions like Facebook AI Research that are devoted entirely to artificial intelligence.

Analysis by Data & Society fellow and Algorithmic Accountability Act coauthor Mutale Nkonde found that Facebook AI Research currently employs 146 people, none of whom are of African descent.

An Inclusive AI program created by Facebook AR/VR business lead Lade Obamehinti is currently being used internally to vet products for bias. Obamehinti created the program one year ago, after she found that Portal’s Smart Camera AI didn’t work on people with darker skin pigment, like herself.

Measurement of AI diversity by team may soon be outdated, however. Pesenti wants developers within Facebook to be part of every team and division in the company.

“My goal is to make every single engineer in the organization an ML engineer, and that number has increased 3 times in the last year, so you’re talking about thousands and thousands of engineers that are not on my team and are not actually ML engineers,” he said.