A Data-Driven Look at the GAO Protest System
WHAT: GAO issued its Annual Report to Congress last Thursday providing a high level overview of key protest statistics such as the number of protests filed and closed during FY16 as well as a summary of the most relevant grounds for sustaining protests. We’ve also independently analyzed the data GAO provides on its public docket.
WHEN: GAO’s report, issued December 15, 2016, covers all protests filed or decided during FY16.
WHAT DOES IT MEAN FOR INDUSTRY: For more information on the key findings in GAO’s Report as well as our independent analysis of the data GAO provides on its public docket, read our full analysis below. Some of the key takeaways include:
- GAO reports that the number of protests sustained this year doubled, while the number of protests filed increased from 2,496 to 2,621—the largest year-over-year increase since FY10, when the number of protests filed increased by 16%. GAO’s use of Alternative Dispute Resolution procedures also dropped significantly.
- Protesters that filed at least one supplemental protest had an 80% greater chance of their protest being sustained compared to other protests resolved in a final merits decision without any supplemental protests.
- Roughly half (50.2%) of all protests decided in FY16 were resolved in 30 days or less, before agencies are required to provide a formal response to the protest.
- The sustain rate, as well as the litigation strategies, varied significantly between agencies. Of agencies with at least 10 merits decisions, the Department of Homeland Security had the lowest sustain rate and the General Services Administration had the highest.
GAO’s Annual Report
GAO’s Annual Report shows four interesting changes in the protest landscape this year. First, the number of protests has continued to increase this year, from 2,496 filed in FY15 to 2,621 filed in FY16. This increase (6% over FY15) is the largest year-over-year percentage increase since FY10, when the number of protests filed increased by 16%.
Second, the number of protests GAO sustained jumped even more significantly, from 12% in FY15 (and an average of approximately 15% over the past four fiscal years) to just over 22% for FY16. This is highest GAO’s annual sustain rate has been since FY07, when it hit 27% (in FY06, the sustain rate was even higher, at 29%).
Third, although the number of protests GAO sustained doubled over the past year (from 68 to 139), GAO’s effectiveness rate rose by only one percentage point from FY15 to 46%. The effectiveness rate reflects the percentage of cases in which a protester obtains some form of agency relief, as reported to GAO, either as a result of voluntary agency corrective action or a sustained protest. These numbers show that although GAO sustained a greater number of protests this year, agencies were less willing to take corrective action without being directed to do so by GAO.
Fourth, GAO used Alternative Dispute Resolution (ADR) in only 69 cases. This is a significant drop: GAO has used ADR in a higher number of cases in each of the last 16 years it has reported its ADR utilization except FY07. But in FY07, where GAO used ADR procedures in only 62 cases, GAO decided only 1,394 cases—nearly half its current volume.
There has been considerable debate over whether the protest process ensures the integrity of competition and protects taxpayer dollars, or just unnecessarily delays procurements. We’ve crunched the numbers available through GAO’s docket to get behind the high-level statistics. This article provides a glimpse into the insights that can be gleaned from the data that GAO makes available on its online docket.
Our Methodology: What Counts as a “Protest”?
To analyze the data, we re-aligned our definition of what counts as a “protest filed” to better match how we and our clients generally view and discuss protests. When GAO counts protests, it counts the number of docket entries (or “B” numbers) assigned. As GAO acknowledges, this measurement has its shortcomings because it “may overstate the number of procurements with sustained protests, as a single procurement can have multiple protests sustained.” Congressional Research Service, GAO Bid Protests: Trends and Analysis, 4 (July 21, 2015). That’s not to say counting docket entries is wrong. In fact, GAO’s method is important because it allows GAO to report protests on a consistent basis each year. Because we’re starting from scratch we have more freedom.
As a result, we considered what counts as a protest from three angles based on all the entries on GAO’s public online docket:
- All Filings (GAO’s Method): Each docket entry (or “B” number counts as a separate protest). Despite its acknowledged shortcomings, this method still produces a valuable metric because it allows the procurement community to track changes based on a methodology that has not changed for decades.
- Initial Filings (Initial Protests Only): Only the initial protest filed by each company against a particular award is counted. We determined which protests relate to the same award by looking to three data points: (1) the first six digits of the file number, (2) the name of the protester, and (3) the date of decision. If all three points match, then we count the earliest filing and disregard the additional filings as supplemental protests. This is our preferred metric because it aligns with the protester’s mindset and the question our clients ask: “What are my chances of winning?”
- Procurement (Simultaneous Challenges to Same Procurement): This last method is similar to the Initial Filing method except that when two or more companies file simultaneous protests of the same procurement, only the first one counts. For this method we looked at two data points: (1) the first six digits of the file number and (2) the date of decision. If these two points match, then we counted only one of the protests and disregarded the others as related protests. This metric aligns with the agency’s (and awardee’s) viewpoint, because the number of protesters and quantity of supplemental protests are often secondary to the ultimate question: “Was the award protested?” [1]
Here’s an example to put these methods in context. On August 3, 2016, GAO issued a decision recommending that the Defense Information Systems Agency revise its solicitation for the ENCORE III program. Two prospective offerors (Booz Allen Hamilton Inc. and CACI, Inc.—Federal) protested the terms of the solicitation, and Booz Allen Hamilton also filed a supplemental protest. Using GAO’s All Filings method, GAO sustained three protests; using our Initial Filings method, GAO sustained two; and using the Procurement method, GAO sustained only one. As the graph below shows, the most significant takeaway is that GAO’s All Filings method produces a much higher sustain rate compared to the other two methods. We’ll show you why later when we discuss supplemental protests.
The difference in these methodologies is also important for measuring other protest statistics. For example, to measure the time it takes to resolve a protest, it’s important to distinguish and measure the time it takes to resolve the initial protest. By including the time it takes to resolve supplemental protests, which are filed later in the process, the average protest resolution time would be artificially shortened. For purposes of our analysis, we used the Initial Filings method. A more detailed chart comparing all three methods is included at the end.
One other important caveat regarding this data: throughout this article, we avoid referring to voluntary corrective action because GAO does not publish that data as part of its online docket entries. The outcomes identified in our charts—withdrawn, dismissed, sustained, and denied—are the only outcomes that GAO identifies on its online docket. Although many often associate a dismissed protest with one where the agency takes voluntary corrective action, there is no reliable way to measure how often dismissed decisions are the result of corrective action. As the chart above shows, the percentage of dismissed decisions alone (59.9 to 64.5%) significantly exceeds GAO’s published effectiveness rate (46%), which includes both sustained protests and protests where agencies took voluntary corrective action. We also note that, although less common, we have occasionally seen protests officially withdrawn, rather than dismissed, as a result of an agency’s voluntary corrective action.
What Matters More, Initial or Supplemental Protests?
Here’s why GAO’s All Filings method results in a higher sustain rate than either the Initial Filings or Procurement method: protesters are more likely to succeed (by having their protests sustained) when they file at least one supplemental protest. In fact, as shown in the chart below, the odds of a protest being sustained in FY16 (the dotted green line) increased by more than 80% if the protester filed at least one supplemental protest.
The increase in GAO’s sustain rate is even more substantial if you look at all protests, not just those with merits decisions. But GAO’s denial rate also increases as well because protests involving supplemental protests are more likely to go to a final merits decision. This is partially because protesters often file supplemental protests after receiving the Agency Report, which they obviously cannot do if GAO dismisses the protest (or if the protester withdraws it) before an Agency Report is produced.
Without reviewing the merits of each protest, we can’t say for certain why supplemental protests are more likely to be sustained, but we have a few leading theories based on our experience. In most cases, agencies provide offerors with debriefings that give only the slightest peek into their evaluation processes and findings. Even when agencies provide robust debriefings, they still cannot disclose their full evaluation record, especially as it relates to the other competitors. As a result, most initial protests rely on limited information as to how the agency evaluated the offerors’ proposals (and how the agency erred). After all, the very documents that most offerors rely on for their initial protests—the debriefing notes and slides—are irrelevant to GAO’s ultimate decision, which is based on the contemporaneous evaluation record. Thus, in preparing initial protests, companies do so in significant part so that they can obtain more information and then, with the advice of their outside counsel under the protective order, make a more informed decision about whether to continue to pursue the original protest grounds and any new supplemental protest grounds. By contrast, when protesters prepare supplemental protests, their counsel does so with the benefit of the contemporaneous evaluation record. It should not surprise anyone to learn that protesters tend to be more successful when they raise challenges based on what they discover in the contemporaneous evaluation record (generally supplemental protests) than when they raise challenges to an evaluation record that they hope to gain access to during the protest process (initial protests).
How Long Do Most Protests Take?
Critics frequently point to long delays created by protests, so we examined how long the protest process normally takes. As shown in the chart below, the vast majority of protests decided during FY16 are resolved in far fewer than the full 100 days. The majority (50.2%) were resolved within the first 30 days—before agencies are even required to produce the relevant documents and their formal response to the protest. In fact, 12.5% were even resolved in the first 10 days. In addition, 73.2% were resolved within 60 days. Ultimately, while we generally think of protests based on the 100-day CICA timeframe, very few protests take that long. This chart also highlights just how few protests actually go all the way to a written decision.
How Do Agencies Differ?
In looking at these statistics, it’s important to remember that not all agencies are the same. In the chart below, we highlight the sustain and deny rates for FY16 merits decisions for all agencies with at least 10 such decisions. The Department of Homeland Security leads the pack with only 8.3% of its merits decisions sustained. GSA follows the rest with 25% of its merits decisions sustained.
There’s also much more to these differences than this chart shows. For example, while the Air Force has a higher sustain rate than the Army, that does not necessarily mean that more of its procurements have problems. For the Air Force, a higher percentage of its protests go all the way to a final merits decision (22.4% versus 15.8% for the Army). Thus the difference in sustain rate might indicate only that the Air Force is more willing to litigate its protests to the end than the Army. In evaluating the merits of the protest process, it is important to account for these and other differences.
Appendix
The table below provides a more detailed comparison of the three methodologies we have discussed. In addition, while our numbers are close to the numbers reported by GAO, we have identified a few discrepancies. For all protests decided in FY16, GAO reported one additional protest, for a total of 2,586. There are also a few differences within the subcategories. We recorded six additional merits decisions (all of which were denied) above the 616 reported by GAO. Because our total number differed by only one, we assume that GAO recorded these decisions as either withdrawn or dismissed. For all protests decided in FY16, we recorded one additional protest above the 2,621 reported by GAO.
As an added bonus, for the protests filed, we’ve broken out the protests filed by Latvian Connection, the company that GAO took the rare step of suspending from protesting for an entire year because of its abuse of the protest process. See Latvian Connection LLC, B-413442, Aug. 18, 2016, 2016 CPD ¶ 194.
__________________________________________
1 In both the Initial Filings and Procurement methods, the count starts over when the agency takes corrective action. For example, if three companies protest an agency’s initial award, and then three companies protest again after the agency takes corrective action, we would count a total of six protests using the Initial Filings method (two for each of the three companies) and two under the Procurement method.
2 In this table, as well as in the chart above, we have excluded the protests where more than two supplemental protests were filed.
