Inside Palantir's philanthropic data engineering team

REDWOOD CITY, Calif. — If you believe the rumors, Palantir helped a black ops initiative kill Osama Bin Laden and the National Security Agency in its massive surveillance operation.

The shadowy company, which is based in Palo Alto, Calif., is the go-to data-mining service for counter-terrorism and military agencies. Palantir has kept quiet about its customers for obvious reasons, but chief executive Alex Karp revealed that the tools been used to hunt down drug cartels in Mexico and sex traffickers in Eastern Europe.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":868597,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"D"}']

Palantir solves really hard, multifaceted technical problems using analytics. According to senior software engineer Ari Gesher, the company is building tools to arm organizations with data by putting it in a form they can understand. “We build software that encodes this symbiosis between humans and computers,” he said at DataBeat/Data Science Summit, VentureBeat’s data science conference.

Palantir is currently seeking a $9 billion valuation, according to recent reports, making it one of Silicon Valley’s most valuable private companies.

Data science and disaster relief

Few people realize that Palantir also works closely with a number of nonprofit organizations, assisting in various disaster relief efforts after Hurricane Katrina and Superstorm Sandy. The company is far more open about this work than its military efforts (and this isn’t a surprise, because of course a firm would rather talk about the good things than the unseemly), which provides us with some rare insight about how its technology actually works.

Gesher described how a small team of philanthropic developers is working closely with Team Rubicon, a nonprofit founded by former Marines. Team Rubicon is typically the first to arrive at disaster areas, like Tacloban, Philippines, where more than 5,200 people died after Typhoon Haiyan tore through in November.

“Team Rubicon has a problem with allocating resources, as there are far too many people who require services,” said Gesher. “We help them identify those most in need.” When Palantir first formed a partnership with Team Rubicon, its volunteers relied on paper-based methods. Today, the organization has adopted web-based and mobile systems to determine how and where to deploy foodstuffs, blankets, medical materials, labor, and other resources.

After a flood or hurricane, Team Rubincon’s volunteers are under pressure to dry out the affected homes within a few weeks. If the property remains wet too long, the walls will begin to rot. Those houses often need to be torn down, which disrupts people’s lives and increases costs for insurance providers. Palantir’s algorithms chart the level of mildew at each home and deploy volunteers to mitigate the damage.

“We sent our computer scientists to the field,” said Gesher. Often, volunteers are skeptical by the influx of twentysomething nerds in Palantir tees who arrive to the disaster zone in a blue bus. However, these developers typically exceed expectations, according to Gesher, and are now a welcome sight (by those that know them, of course).

Gesher told me about one Palantir employee, who was deployed to the Rockaways in the wake of Sandy. It was his first week on the job. This particular engineer had recently relocated to the United States to work for Palantir after he had been shot in the Tahrir Square protest in Egypt.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":868597,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"D"}']

“Within a week in the Rockaways, he sat in our bus and banged out a full geo-cluster algorithm to determine how best to spread people out throughout the area,” Gesher recalled.

Disaster relief is just the beginning for Palantir’s self-described philanthropic engineering group. Developers are also working to shine a light on the global human trafficking trade. This trade affects an estimated 21 million people, and its victims are often entangled in deep-rooted criminal networks. Palantir analyzes these networks to find points of connection, in an attempt to locate victims and their captors. The company also works with the Polaris Project to compile data about people who call the human trafficking hotline.

Palantir helps call center workers locate safe-houses anywhere in the world, so victims have a shot at escaping.

According to Gesher, the team is overloaded with requests for help from nonprofits, which signals a sea change in the perception of data science. The team can’t take on every case, but Gesher is open to initial conversations, particularly if there’s a strong data component.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":868597,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"D"}']

“We connect with people who care about the data, and let them use it without friction,” said Gesher.

For more on Palantir’s work with the Polaris project, check out the video below.