Top 10 Findings from DOD’s Independent Study on Bid Protests

January 17, 2018

WHAT: The RAND Corporation issued its long-awaited independent study on Department of Defense bid protests, Assessing Bid Protests of U.S. Department of Defense Procurements: Identifying Issues, Trends, and Drivers, as required under Section 885 of the FY17 National Defense Authorization Act.  

WHEN: The RAND Corporation sent the study to Congress on December 21, 2017 and made it publicly available shortly thereafter. 

WHAT DOES IT MEAN FOR INDUSTRY: In the last few years, there has been no shortage of opinions for improving (or curtailing) the bid protest process. For the most part, however, these ideas have been based on anecdotes about the protest process from one side or the other. To chart a more objective, data-driven course, Congress directed the Department of Defense to conduct an independent study into the protest process. The RAND study fulfills this directive and provides a thorough, comprehensive foundation for discussing the bid protest process and potential reform. Continue reading below for what we view as the top 10 findings from the RAND study and its key recommendations for policymakers. 

Top 10 Findings

RAND dove deep into data provided by the Government Accountability Office (GAO) and the Court of Federal Claims (COFC) on protests resolved over the last eight years as well as additional federal spending data. The RAND study includes many notable findings based on their comprehensive analysis, and we’ve highlighted the top 10 below.

10.  Protests Are Unique and Complex: Although RAND’s “statistical modeling indicates differences between types of cases (categorization of cases), it is not possible to predict the outcome of any case based on its general characteristics. Each case is different and its details affect the outcome.” Although this finding should not surprise anyone, it’s an important reminder that protests are distinct, often complex cases that must be decided based on the facts in each individual case.

9.  GAO and COFC Follow Different Cycles: At GAO, RAND found that protest filings generally peak in October, following the end of the government fiscal year. At the COFC, by contrast, RAND found no consistent month-to-month trend. 

8.  Stay Overrides Are Rare: RAND found that agencies overrode the automatic stay in only 1.4% of all protests. RAND recommended further research in this area to evaluate the cause behind this figure—whether the override standard is too difficult, whether agencies are overly conservative, or whether most procurements are just not that urgent.  

7.  Task-Order Protests Are More Successful: RAND found that 10.6% of protests relate to a task order issued under an Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract. RAND also found a slightly higher effectiveness rate for task-order protests than for other procurements, demonstrating that companies are more likely to succeed when protesting a task order.

6.  Small Business Protesters Are More Frequent, Less Successful: RAND found that more than half of protests filed (53% at GAO and 58% at COFC) were filed by companies that identified themselves as small businesses. At GAO, these small businesses were also less successful in securing relief (measured by GAO’s effectiveness rate) than larger businesses. RAND attributes this lack of success at GAO in part to the fact that small businesses are less likely to retain counsel to represent them at GAO. At COFC, where all parties are represented by counsel, there is no meaningful difference between small businesses and other protesters.

5.  Top DOD Contractors Are More Successful: The top 11 DOD contractors (measured by FY16 spending) were more likely to prevail at GAO (measured by sustain and effectiveness rates) than all other businesses. RAND suggested this distinction could suggest that the largest 11 companies are “more selective in the protests they file” and they spend more resources developing their cases when they do file.

4.  GAO Is Quick: RAND found that GAO resolved 50% of all protests within 30 days and 70% within 60 days. Of the protests that proceeded beyond 60 days, however, the majority were decided between days 90 and 100. This quick pace is consistent with what we’ve reported based on our own independent research.

3.  Protests Are Exceedingly Uncommon: RAND found that only 0.3% of all DOD contracts are protested. As RAND thus concluded, “bid protests are exceedingly uncommon for DOD procurements.”

2.  Inadequate Debriefings Lead to Protests: RAND noted significant concerns about the quality of post-award debriefings, recounting the worst debriefings as “skimpy, adversarial, evasive, or failing to provide required reasonable responses to relevant questions.” RAND reached the same conclusion we’ve emphasized and that many in industry have been stressing for years: “too little information or debriefings that are evasive or adversarial will lead to a bid protest in most cases.”

1.  GAO’s Stable Effectiveness Rate Suggests Healthy System: RAND found that the effectiveness rate has remained stable over the past eight years. We’ve discussed before GAO’s stable effectiveness rate, which you can observe by comparing GAO’s annual reports over time, but what’s critical here is the conclusion that RAND draws in its independent study: this “suggests that firms are not likely to protest without merit.” 

Key Recommendations

The RAND study concluded with several substantive recommendations. We’ve highlighted the top three below, two of which suggest caution rather than change.

• Enhance Post-Award Debriefings: RAND recommended that DOD adopt a debriefing process like the U.S. Air Force’s extended debriefing process. Under this process, agencies would provide each unsuccessful offerors’ outside counsel with source-selection-sensitive information so that companies can obtain a separate opinion from their own counsel on the fairness and impartiality of the evaluation and better evaluate whether a protest is warranted. The FY18 NDAA already implements some aspects of this process for DOD awards above $100 million for all contractors and above $10 million for small businesses and nontraditional contractors.
• Be Careful with GAO’s Decision Timeline: RAND cautioned against recent proposals to shorten GAO’s deadline for resolving protests from 100 days to 65 days. RAND stressed that most cases are already decided within that timeframe but that the ones that take longer are “typically more complex” and “not easily resolved.” Reducing the timeframe for these protests “might not leave enough time for [GAO] to develop these more complex decisions.” RAND also noted that the end-of-year filing pattern could also strain GAO’s workload and require additional staff.
• Be Careful with Task Order Jurisdiction: RAND acknowledged that reducing task order protests could reduce GAO’s overall docket but emphasized that this might not be the area worth cutting because “task-order protests are generally more likely to be sustained or have corrective action.” In other words, “task-order protests fill an important role in improving the fairness of DOD procurements.”