oversight

Department of Defense's Waiver of Competitive Prototyping Requirement for Enhanced Polar System Program

Published by the Government Accountability Office on 2012-08-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

United States Government Accountability Office
Washington, DC 20548




   August 23, 2012

   Congressional Committees

   Subject: Department of Defense’s Waiver of Competitive Prototyping Requirement for
   Enhanced Polar System Program

   The Weapon Systems Acquisition Reform Act of 2009, as amended (WSARA), requires that the
   acquisition strategy for each major defense acquisition program provides for competitive
   prototypes before Milestone B approval—which authorizes entry into system development—
   unless the Milestone Decision Authority waives the requirement. 1 Competitive prototyping,
   which involves commercial, government, or academic sources producing early prototypes of
   weapon systems or critical subsystems, can help Department of Defense (DOD) programs
   reduce technical risk, refine requirements, validate designs and cost estimates, and evaluate
   manufacturing processes prior to making major commitments of resources. WSARA states that
   the Milestone Decision Authority may waive the competitive prototyping requirement only on the
   basis that (1) the cost of producing competitive prototypes exceeds the expected life-cycle
   benefits (in constant dollars) of producing such prototypes, including the benefits of improved
   performance and increased technological and design maturity that may be achieved through
   competitive prototyping; or (2) but for such a waiver, DOD would be unable to meet critical
   national security objectives. WSARA also directed us to review and assess DOD’s rationale for
   competitive prototyping waivers. 2

   On June 18, 2012, DOD notified us that it had waived the competitive prototyping requirement
   for the Control and Planning Segment (CAPS), a subsystem of the Air Force’s Enhanced Polar




   1
    Pub. L. No. 111-23, § 203(a), as amended by the Ike Skelton National Defense Authorization Act for Fiscal Year
   2011, Pub. L. No. 111-383, § 813. Specifically, WSARA required DOD to modify its guidance relating to the operation
   of its acquisition system to incorporate these competitive prototyping provisions. DOD did so through Directive-Type
   Memorandum (DTM) 09-027, “Implementation of Weapon System Acquisition Reform Act of 2009 (Dec. 4, 2009,
   incorporating Change 3, Dec. 9, 2011). Major defense acquisition programs are those estimated by DOD to require
   an eventual total expenditure for research, development, test, and evaluation of more than $365 million, or for
   procurement, including all planned increments or spirals, of more than $2.19 billion in fiscal year 2000 constant
   dollars. The Milestone Decision Authority for major defense acquisition programs is the Under Secretary of Defense
   for Acquisition, Technology and Logistics, the head of a DOD component, or if delegated the component acquisition
   executive.
   2
    Pub. L. No. 111-23 § 203(b)(2). Specifically, WSARA provides that whenever a Milestone Decision Authority
   authorizes a waiver of the competitive prototyping requirement on the basis of what WSARA describes as “excessive
   cost,” the Milestone Decision Authority is required to submit notification of the waiver, together with the rationale, to
   the Comptroller General of the United States at the same time it is submitted to the congressional defense
   committees. WSARA further provides that we are to review the rationale for the waiver and submit a written
   assessment of that rationale to the congressional defense committees.




   Page 1                                                                      GAO-12-983R EPS Prototyping Waiver
System (EPS). 3 EPS, which consists of two payloads hosted on classified satellites, a gateway
to connect user terminals to other communication systems, and a control and planning segment
to control the payloads and manage communications, will provide extremely high frequency,
jam-resistant, and secure satellite communications to strategic and tactical forces in the polar
region. According to DOD, this is the first waiver of WSARA’s competitive prototyping
requirement since it was enacted in 2009.

In this report, we assess (1) DOD’s rationale for waiving the competitive prototyping requirement
for CAPS and the analysis used to support it and (2) the acquisition strategy for CAPS in the
context of the waiver. To conduct our assessment, we compared the rationale in the waiver to
the WSARA requirement to determine the extent to which the waiver is consistent with the
statute. In addition, we reviewed the Air Force’s business case analysis, which provides the
data and assumptions on which the waiver is based, the acquisition strategy, and other relevant
documentation. We also submitted written questions to DOD, Air Force, and EPS program
officials to clarify information in this documentation, as necessary.

We conducted this performance audit from June 2012 to August 2012 in accordance with
generally accepted government auditing standards. These standards require that we plan and
perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that the evidence obtained
provides a reasonable basis for our findings and conclusions based on our audit objectives.


Results in Brief

DOD’s rationale for waiving WSARA’s competitive prototyping requirement for CAPS covered
both bases provided in the statute; however, DOD did not provide complete information about
the potential benefits of competitive prototyping or support for its conclusion that prototyping
would result in schedule delays. In the waiver, DOD found reasonable the Air Force’s
conclusion that the additional $49 million cost of producing competing prototypes exceeded the
negligible expected life-cycle benefits because minimal opportunities exist to increase CAPS
technological and design maturity through competitive prototyping. The Air Force’s conclusions
about CAPS technical and design risk are supported by its market research, but its cost-benefit
analysis was incomplete because neither the waiver nor the business case analysis supporting
it provided an estimated dollar value for the expected benefits. Further, the Air Force’s cost
estimate of competitively prototyping CAPS was based on a program office estimate, which has
not been independently reviewed by DOD’s Office of Cost Assessment and Performance
Evaluation. DOD also did not provide support in the waiver for its conclusion that implementing
competitive prototyping will delay EPS’s initial operational capability and result in DOD being
unable to meet critical national security objectives. Finally, the CAPS acquisition strategy
anticipates the use of a cost-reimbursement contract for designing and building the system;
however it may be difficult for DOD to meet the requirements for justifying the use of a cost-type
contract for development, given the low design and development risk described in the waiver.




3
 DOD reported in the waiver that the program has produced a single prototype. Pub. L. No. 111-23 § 203(a)(3)(A)
provides that whenever a Milestone Decision Authority authorizes a competitive prototyping waiver, the program is
still required to produce a prototype prior to Milestone B approval if the expected life-cycle benefits (in constant
dollars) of producing such prototype exceed its cost and its production is consistent with achieving critical national
security objectives.



Page 2                                                                      GAO-12-983R EPS Prototyping Waiver
Waiver Rationale Is Consistent with WSARA, but Gaps Exist in the Analysis

DOD’s rationale for waiving WSARA’s competitive prototyping requirement for CAPS addressed
the two bases provided for a waiver in the statute, which focus on costs and benefits and critical
national security objectives. In the waiver, DOD accepted the Air Force’s conclusion that
competitively prototyping CAPS would increase development costs by approximately $49 million
(base year 2012 constant dollars). 4 Based on market research and other factors, such as
previous efforts to reduce CAPS requirements and risk reduction work that demonstrated these
requirements can be met with a low risk design and primarily commercial off-the-shelf hardware
and software, the Air Force concluded that the life-cycle benefits of competitive prototyping were
negligible and the level of cost savings needed to recoup the additional investment was
unrealistic. 5

The Air Force’s conclusions about CAPS technical and design risk are supported by its market
research, but its cost-benefit analysis was incomplete. Neither the DOD waiver nor the Air Force
business case analysis included an estimate of the life-cycle benefits that may be achieved
through competitive prototyping. In the absence of a specific dollar value or range of values for
the estimated life-cycle benefits of CAPS competitive prototyping, the Air Force cannot
conclusively determine whether the costs outweigh the benefits. According to the EPS program
office, it did not perform a more detailed life-cycle cost-benefit analysis because the additional
cost and burden of producing such an analysis was not warranted based on its business case
analysis. Further, the Air Force’s estimated $49 million cost for competitively prototyping was
based on a program office estimate and has not been independently estimated or reviewed by
DOD’s cost estimating organization, the Office of Cost Assessment and Performance
Evaluation. According to our best practices criteria, an independent cost estimate is considered
one of the best and most reliable resource estimate validation methods because it provides an
independent view of expected program costs that tests the program office estimate for
reasonableness. 6

DOD’s conclusion that it cannot meet critical national security objectives without the CAPS
competitive prototyping waiver was not well supported in the waiver. The waiver states that
implementing competitive prototypes for CAPS will further delay the EPS’s initial operational
capability, but it does not explain why or how it adds to the schedule risks the program already
faces. We noted during our review that the Air Force’s acquisition strategy concludes that
competitive prototyping could delay initial operational capability by 6 months to 2 years because

4
 According to the EPS program office, $49 million includes not only the cost of producing an additional prototype, but
also the cost of funding a second contractor through preliminary design review.

5
 During our review, we found errors in the program costs and percentages cited in the waiver. The waiver states that
the Air Force estimate for remaining CAPS development following prototyping is $177 million and for CAPS
operations and sustainment is $134 million (base year 2012 constant dollars). By comparing these costs to the cost
of competitive prototyping, DOD concluded that a cost savings of more than 16 percent would have to be achieved to
make competitive prototyping viable. However, the waiver incorrectly stated that these cost estimates were in base
year 2012 constant dollars. We found and the Air Force acknowledged that the dollar amounts were in then-year
dollars. During our review, we requested and received these dollars amounts in 2012 base year constant dollars from
the EPS program office. It also provided us an updated estimate of operations and sustainment costs. According to
this data, the estimated cost of CAPS development work following prototyping is $158 million and CAPS operations
and sustainment is $71 million (base year 2012 constant dollars).
6
 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program
Costs, GAO-09-3SP (Washington, D.C.: March 2009).




Page 3                                                                    GAO-12-983R EPS Prototyping Waiver
the program would have to not only award contracts for competitive prototyping, but also go
through another source selection to award a CAPS design and development contract. On the
other hand, according to the EPS program office, alternatives exist if CAPS is not ready by the
time the payloads, which have almost completed production, are launched. According to the
program office, ground test assets have been procured to enable payload calibration and check-
out prior to CAPS operational readiness, though these assets could not sustain CAPS
operations. In addition, there are currently two Interim Polar System payloads in operation that
the program office estimates will continue to meet operational requirements while the EPS is
deployed and readied for operations.


Contract Type in CAPS Acquisition Strategy May Be Difficult to Justify

The CAPS acquisition strategy anticipates the use of a cost-reimbursement contract for
designing and building the system; however, it may be difficult for DOD to meet the
requirements for justifying the use of a cost-type contract for development, given the description
of risk in the waiver. The waiver concluded that, based on market research, there is no
significant technology development anticipated in the contractors’ proposals for CAPS and
design risks are expected to be low. However, DOD acquisition regulations state that if a cost-
type contract is selected for a development program at Milestone B, the contract file shall
include the Milestone Decision Authority’s written determination that the program is so complex
and technically challenging that it would not be practicable to reduce program risk to a level that
would permit the use of a fixed-price type contract. 7 As shown in figure 1, the EPS program’s
Milestone B is planned for the first quarter of fiscal year 2014.

Figure 1: EPS and CAPS Program Schedule




Notes: CDR       critical design review
       FY        fiscal year
       PDR       preliminary design review
       Q         quarter

According to the acquisition strategy, a cost-reimbursement contract was chosen for the basic
CAPS contract because of the nature of the design work, which includes software design and
development. This contract type was also chosen for the option to develop and build the system
because of the difficulty in pricing the development of a system that has not yet completed

7
 Subpart 16.1 of the FAR and DFARS §§ 216.104-70 and § 234.004(2)(iii) outline a variety of factors to consider in
selecting contract types.



Page 4                                                                   GAO-12-983R EPS Prototyping Waiver
preliminary design review. According to the acquisition strategy, using a cost-plus-incentive fee
contract will reduce the level of risk for the competing contractors and allow them to deliver
more reasonably priced proposals up front; however, it will result in the government assuming a
greater portion of the cost and performance risk. Decisions on the specific contract type remain
and in making those decisions, DOD will need to appropriately address the expected level of
risk to both the government and contractors.


Agency Comments and Our Evaluation

We provided a draft of this report to DOD for comment. In its written comments, reproduced in
the enclosure, DOD agreed that it could have provided additional documentation in support of
the waiver and stated that it appreciated our clarification of the types of information and data
that should be included in the future waivers for best practices purposes. We believe that if DOD
takes note of our findings on the CAPS waiver, in particular the lack of a specific estimate of the
expected life-cycle cost benefits of competitive prototyping, it would improve future waiver
requests. DOD also noted that WSARA does not specifically ask us to review waivers based on
national security objectives and considers our comments on its national security rationale
outside the scope of the statute. We recognize that WSARA specifically provides for our review
of competitive prototyping waivers after receipt of a notification of a waiver on the basis of
excessive cost, i.e., the cost of producing competitive prototypes exceeds the expected life-
cycle benefits of producing such prototypes. However, in this instance, because DOD’s rationale
for the waiver included both excessive cost and national security considerations, our
assessment included both bases. Our finding that DOD’s national security rationale was not well
supported still stands; and we believe that providing adequate support for the national security
rationale in competitive prototyping waivers is important for congressional oversight. Finally,
DOD noted that we questioned the justification for using a cost-reimbursement contract for
CAPS. For clarification, we did not question DOD’s decision, which is ultimately one for the
agency, on the appropriate contract type. Instead, we highlighted that the low design risks
described in the waiver seem inconsistent with the requirements in DOD’s acquisition
regulations for justifying the use of a cost-reimbursement contract for development.


                                           - - - - - -


We are sending copies of this report to interested congressional committees, the Secretary of
Defense, and Secretary of the Air Force. In addition, the report will be available at no charge on
the GAO website at http://www.gao.gov.




Page 5                                                        GAO-12-983R EPS Prototyping Waiver
If you or your staff have any questions, please contact me at (202) 512-4841 or
sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made key contributions to this
report were Ron Schwenn, Assistant Director; Jacob Leon Beier; Marie Ahearn; Kenneth E.
Patton; and Carol Petersen.




Michael J. Sullivan
Director, Acquisition and Sourcing Management

Enclosure




Page 6                                                     GAO-12-983R EPS Prototyping Waiver
List of Committees



The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Member
Committee on Armed Services
United States Senate

The Honorable Daniel K. Inouye
Chairman
The Honorable Thad Cochran
Ranking Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Howard P. “Buck” McKeon
Chairman
The Honorable Adam Smith
Ranking Member
Committee on Armed Services
House of Representatives

The Honorable C.W. Bill Young
Chairman
The Honorable Norman D. Dicks
Ranking Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 7                                  GAO-12-983R EPS Prototyping Waiver
Enclosure

            Comments from the Department of Defense




Page 8                                   GAO-12-983R EPS Prototyping Waiver
(121085)



Page 9     GAO-12-983R EPS Prototyping Waiver
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.