HealthCare.gov costs show that feds have literally no idea how to build a big web site

Could you build a working website for $300 million?

Apparently, the U.S. government is having difficulty doing that. HealthCare.gov, the website meant to help people find insurance under the Affordable Care Act, has cost hundreds of millions of dollars to build, and it still doesn’t work right.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

Cost estimates vary, but the Washington Post figures that the government has spent at least $170 million, and possibly as much as $292 million — so far.

The Sunlight Foundation examined payments made to the site’s chief contractor, Virginia-based CGI Federal (a subsidiary of a Canadian company, CGI Global), and estimated that CGI alone received $70 million in payments so far.

And yet the site has performed so badly since its launch a month ago that people can’t even create accounts. It’s frequently offline, and even when you can log on, it’s often so slow as to be unusable, according to reports. The most updates from the site’s official Twitter account illustrate the problem perfectly: Of the most recent five, four were about site outages, and one encouraged people to apply for insurance using paper forms. Paper!

That it got me to thinking: Just how much money does it take to build a website like this? In all the years that I’ve covered Internet technology, I’ve rarely run into a website — no matter how complex — that cost anywhere near that amount. Even if you add in costs for complex back-end systems integration, training, and marketing, $170 million to almost $300 million is way more than you could possibly need. I can’t even figure out where that much money would go.

Now, it’s not really fair to call this a web “site” — it’s more of a web application, as longtime Java guy Miko Matsumura, now a VP of developer relations at HazelCast, told me.

“A web application of this size is very problematic, especially when you have distributed accountability and you have a poor integration and testing paradigm in place,” Matsumura said.

In other words: There’s a lot of stuff going on in the background that needs to be tied together, and whenever you build a complex system made of many different, interconnected parts, it’s going to be expensive — and will require a lot of time to test (which the makers of HealthCare.gov didn’t have — the compressed launch timeline gave them just a week for testing, a ridiculously short period.) Fair enough.

---

Editor’s note: Developers! Think you’re good enough to fix HealthCare.gov? Check out our upcoming DevBeat conference, Nov. 12-13 in San Francisco, is a hands-on event for excellent developers, all aimed at boosting your code skills, security knowledge, hardware hacking, and career development. Register now.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

---

But let’s take a look at what else you could build for almost $300 million — like Facebook, for instance.

By the time Facebook had raised $280 million, in October 2007, it had about 50 million users, had recruited 100,000 businesses to create Facebook pages, and was just launching its developer platform. It was already looking like the dominant social network. And it was dead easy to create a new account.

Note that Facebook had raised almost $300 million by late 2007 — it hadn’t spent nearly that much. And yet it accomplished so much more than HealthCare.gov did with the same budget.

Other private companies have done as much by spending even less. Instagram’s founders turned $57.5 million of capital into a company worth over a billion dollars. Workday raised just $175 million, and LinkedIn $200 million, before their successful IPOs.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

But those aren’t health care-related companies. The data and regulatory complexity of health care makes it a lot harder to build a working web site, right?

Maybe not. Castlight Health, a company that collects data on health care providers to help consumers make informed choices, has raised just $160 million so far. Its web-based service seems to be working just fine. Oscar, a private health insurance exchange (limited to New York residents for now) has raised $40 million. eHealth and GetInsured have been offering health insurance comparison-shopping for years, and while I couldn’t find how much money those two companies have raised, it’s worth noting that they exist, and they work — so it’s not like the makers of HealthCare.gov were tackling an insoluble problem.

Other smart people are drawing the same conclusion as Matsumura: This was a complicated project, yes, but it was crippled by bad management.

“The real problems are with the back end of the software,” Dan Hanson wrote on Marginal Revolution, a blog run by economist Tyler Cowen. “When you try to get a quote for health insurance, the system has to connect to computers at the IRS, the VA, Medicaid/CHIP, various state agencies, Treasury, and HHS. They also have to connect to all the health plan carriers to get pre-subsidy pricing. All of these queries receive data that is then fed into the online calculator to give you a price. If any of these queries fails, the whole transaction fails.”

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

That gives you a peek at what is so complicated about the site, but it’s not an unheard-of level of complexity. The real problem seems to be that the project was run without effective oversight.

“The best way to do a project of this size is to iterate around a core of 4-5 extremely elite engineers who are building a platform based around something extremely scalable,” Matsumura said. “Once you have that nucleus in place, you can begin to integrate applications. Even with such a paradigm, you have issues about distributed application clustering and scalability.

“Sadly, there are web applications for less important things that have handled way more traffic.”

Let’s not even get into how many lines of code it took to build HealthCare.gov. One of the estimates being bandied about is that it contains 500 million lines of code — an absurd figure, as it translates to 10 times the amount of code in Windows Vista, or, if you figure 500 developers were banging away at their keyboards for a whole year, about 83,000 lines of code for every man-month. (The man-month is a mythical concept used by old-school tech developers and, perhaps, government contractors.)

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

Let’s just say that HealthCare.gov is fantastically complex and lacked the competent oversight that such a complicated tech project demands.

It’s telling that CGI was saying that testing the service was the government’s responsibility and wasn’t part of its contract. If so, then shame on the feds for offering CGI a contract without testing built in. You just can’t build and deploy a site like this without extensive testing, and that testing needs to be tightly integrated with the development of the site.

“None of these missteps would have occurred if the contractors had taken a gradual, agile approach,” Practice Fusion cofounder Matthew Douglass wrote this week on VentureBeat. Practice Fusion, by the way, has built a successful, nationwide platform for electronic health records now used by over 100,000 medical professionals. Its total fundraising to date? Just $140 million.

President Barack Obama took to the Rose Garden a week ago to talk about how disappointed he was in the site and to promise a “tech surge” aimed at fixing it. (As if that will work.) Secretary of Health and Human Services Kathleen Sebelius spent hours on Capitol Hill this week, offering her apologies and getting grilled by Congressmen about the site’s failure.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":851507,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,dev,","session":"C"}']

But any number of apologies won’t fix this basic fact: As with so many other government projects, we’ve thrown hundreds of millions of dollars at a problem that could have been solved with a fraction of the expenditure — if someone in charge actually knew what they were doing.