Who broke the Obamacare exchanges? Part 2: The federal bureaucracy

It’s a fact of life among software developers that most projects– especially big, complex ones– will have problems when they’re launched. One analysis found that over the past 10 years, 94% of software projects that cost over $10 million were either over budget, behind schedule, didn’t meet user expectations, or simply didn’t work and were abandoned or restarted from scratch. The trick is to to be able to fail smart (or at least without catastrophe)– so that the inevitable problems are the “glitches” everyone expected with the launch of the Obamacare websites, and not the total disaster we’ve seen.

In Part 2 of our series on who broke the Obamacare exchanges, we look at why the way government agencies typically approach big technology projects, like the exchanges, makes disastrous failures much more likely.

1. The government has trouble recruiting the best programmers

Good programmers are in high demand pretty much everywhere, and private companies can offer great salaries, stock options, and fringe benefits like free meals from professional chefs that the federal government simply can’t compete with. Government agencies can sort of get around this problem by hiring outside contractors on big projects like the Obamacare marketplaces, but, says Ed Felten, a computer science professor at Princeton and former chief technologist for the FTC, you still need “technically sharp people” in government to keep an eye on things:

One of the challenges in contracting is to know that the contractor is building what you need, and questions that the contractors will do the work they’re supposed to. [They also need to manage] the work of integration among contractors. Those are things you’d like employees to be on top of. That requires a set of technical skills, along with management and contracting expertise.

From the outside, it’s hard to say whether the Centers for Medicare and Medicaid Services, the agency responsible for the exchanges, had enough “technically sharp” people to manage this project. But if they didn’t, it might explain why the higher-ups seemed genuinely surprised that the site was in such bad shape after the launch.

2. Contracts often go to companies based on how well they navigate the federal bureaucracy

One question that’s come up a lot recently is why the Obama administration, which ran the most technologically savvy campaign in history, has been unable to apply that same expertise to government systems. Clay Johnson, CEO of the Department of Better Technology, a nonprofit that develops technology for governments, has an answer:

“The wizards from the campaign have no desire to contract with the federal government because it’s a pain in the butt.”

Tom Lee of the Sunlight Foundation, a nonpartisan group that advocates for more government transparency, sums up the problem this way:

“The episode [with the marketplaces] is all too typical of how government creates IT services. The procurement process tends to select for firms that are good at navigating the procurement process, not providing good IT services for the dollar.”

Johnson says the problem starts with the request for proposals (RFP) the government typically sends out, inviting companies to bid on IT projects:

They tend to be really long and hard to read. They begin with a page of abbreviations and pointers to clauses in the federal regulation. No normal, functional human being would read those clauses and understand them without years of experience and some legal help. This legal language and tone of authority really turns away people who might bid on these contracts. A lot of businesses look at these contracts, decide they can’t even understand them, and it makes it unattractive to them.

Of course, there’s a reason why there are so many regulations in federal contracts– in the past, politicians would simply hand these jobs to friends and campaign supporters. Still Johnson says there needs to be a balance between the rules necessary to avoid a corrupt patronage system and so many rules that only a handful of companies can navigate through them.

Another obstacle to getting the best company for a given IT job: risk averse government officials tend to only want to work with “known quantities”– contractors they’ve worked with in the past. So even if a newcomer does make it through all the bureaucratic hoops, they’re still at a disadvantage compared to more entrenched companies. For example, CGI, the lead contractor on the federal Obamacare exchanges, was one of 16 companies pre-vetted to do IT work for CMS, before it even bid on the exchange contract. Other companies might have had a better approach that would cost less, but they would have had to go through a much longer process to land the job.

3. Government agencies often have a backwards approach to managing tech projects

The healthcare.gov process was typical of the government’s approach to big IT projects– CMS gave the contractors thousands of pages of requirements up front, telling contractors in great detail what the finished product had to look like, and then waited for the finished products, with no “end-to-end” testing until two weeks before the launch. That’s pretty much the opposite of how most software development is done these days, says Lydia DePillis at The Washington Post:

The better way to do things is a school of software development called Agile — it’s been around since the 1950s, was basically codified in the early 2000s, now has a whole non-profit devoted to it, and is the dominant form of software design in teams. Rather than moving from one static stage to the next, it emphasizes constant iteration and testing, with prototypes building on prototypes so the endpoint is something that works. The only problem, from a government perspective, is that you need to be comfortable with not knowing exactly that they will look like.

“If you’re going to take the time to write a 1,000 page specification, you’re going to miss the point,” Larry Fitzpatrick [who used to run IT systems for the Financial Industry Regulatory Authority] says. “They should have fairly quickly, within a quarter, been able to put together a skeleton for what the system had to do, what systems had to talk to each other, and get it up and running. This is really the only way to build systems of this complexity.”

The individual contractors probably used agile methods on their own pieces of the site, but it sounds like CMS, who was in charge of making sure all the pieces worked together, approached the project like many federal agencies: asking for a specific thing and then walking away, instead of working intensely with the development team.

4. Secrecy in the development process

More and more companies (and some other governments, like Brazil, Spain, France, and Malaysia ) are posting their code online, to sites like GitHub, where other programmers can help them spot bugs and make suggestions. Healthcare.gov, like many U.S. government projects, was developed in near total secrecy– partly because some people still see open development as risky, and partly because they were using a lot of commercial software that they couldn’t share for legal/copyright reasons.

Programmer Paul Ford, writing for Bloomberg Business Week, writes:

Regardless of your opinions on the health-care law, this is the wrong way to make software. We may never be able to stop websites from crashing or keep bugs from creeping into code. But opening the process would expose what went wrong and why. People can still be paid large amounts of money to code for the government, if that’s what it takes. They just shouldn’t do it in secret. The U.S. needs to say that all development will take place in the open, using a tool like GitHub, or GitHub itself. If some things can’t be open, for reasons of security or copyright, fine; say so and get an exemption. […]

It may sound odd, but some people will file bug reports and track down problems for the joy of it. Nurturing this openness helps everyone and brings people into a new, weird, and exciting civic process.

Mark Wellenweg, co-developer of WordPress (the open-source platform that five states are using to run their exchange websites) agrees:

“How cool would it have been if the site launched, didn’t work, and some passionate coders came in and said, ‘Oh, here’s a problem, and here, and here?’ But the people who want to help aren’t able to.”