In the United States, we long ago reached the point where government agencies rely on technology to achieve their missions. Unfortunately, those big technology projects fail as a matter of course. When custom software projects have a budget of over $10 million, they have a 6% success rate, and the average cost overrun on a government IT project is 210%. A lot has to go wrong for such terrible results, but one significant source of trouble is that vendors’ work is often just bad. Until we correct that, governments will continue to struggle to deliver on their missions. But it can be fixed.
Let’s go deeper into the problem before we talk about the solution
When these big software projects fail, they generally don’t fail partway through—often, they are discovered to have failed only after all of the money is spent. Why doesn’t government understand a project is failing while work is still underway? Simple: because agencies rarely employ people who understand how software is built. Decades of outsourcing anything IT-related has eliminated even the ability to know if a vendor’s work is any good. As a matter of course, agencies hire independent validation and verification (IV&V) vendors to check on the work of the primary vendor, but that’s the approach that has resulted in our status quo, so we can see that’s not doing any good. Agencies often bring in U.S. Digital Response after these projects fail, to help chart a path through the wreckage, because we’re vendor-agnostic, non-conflicted, trusted partners, so we’ve had a front-row seat to the mess that these failures leave behind.
So how do you start to fix this?
First, you’ve got to check vendors’ abilities at the time that they bid on projects. To examine past performance, you have to actually look at their past performance! We promote the best practice of requiring that bidding vendors provide a code repository from an actual past project done for a customer, so that agency technical staff can perform a full code review. We also recommend that the vendors submit user research artifacts from that project, to determine whether they’re properly coupling their code development with continuous user research. As you’d want to look at other homes built by a homebuilder or other landscaping projects by a landscape architect, it’s essential to look at the past work of software development vendors. We’ve helped many agencies with this process, and it’s always revealing. There are a lot of software development vendors who talk a good game and share some pretty-looking screenshots, but their actual software development skills are awful. At USDR we’ve found that we need to support partners in this review process while we help them to develop the in-house ability to perform this review on their own.
The second part of fixing this is to routinely evaluate the quality of vendors’ work. The vendor should be delivering completed code every two weeks, and that needs to be inspected to ensure that it meets the quality standards laid out in the contract. Of course, evaluating code requires a programmer who understands it, so this requires employing a technical lead whose job it is to perform that review. USDR has also supported partners in this, both performing that evaluation as they hired a programmer and helping them to recruit and hire a programmer.
One partner who we performed both of these services for is the Council of State Governments (CSG). As described in this recent case study, when CSG issued an RFP for custom software development services, we brought on a pair of USDR volunteers to perform technical evaluations of the vendors’ proposals, both in terms of the quality of the programming and the quality of the user research. They examined what the vendor proposed to do for CSG as well as the code samples that the vendors submitted, and provided their evaluations to CSG to factor into their vendor selection process. When the selected vendor started working, USDR served as the interim technical lead while helping CSG to recruit and hire a permanent technical lead. This is a perfect representation of the USDR approach to being the capacity for our partners while building their capacity.
How can you move toward this improved state?
First, modify your solicitations to measure past performance based on actual past performance, using the model custom software development solicitation published by the General Service Administration’s 18F. Second, identify software developers within the ranks of your IT organization who can participate in this process, helping to review proposals and then to review the work performed by selected vendors. If your IT organization doesn’t employ such people, then that’s an important change to make. It is imperative to employ the staff required to actually oversee the work that vendors are performing. This is the change that is needed to hold vendors to account, to ensure that government and the public are getting their money’s worth.
Vendors will continue to deliver poor quality software so long as government selects poorly performing vendors and doesn’t demand better of them. It is only by requiring better work from vendors that we can transform government technology into the powerful tool for public policy that we need it to be.
Footnote 1: As a rule of thumb, it requires a quarter of a full-time employee’s time to review the work of a single “scrum team” of 4–9 people, including coders, user researchers, designers, etc.
Photo by Eliott Reyna on Unsplash
