oversight

Defense Logistics: Improvements Needed to Enhance Oversight of Estimated Long-term Costs for Operating and Supporting Major Weapon Systems

Published by the Government Accountability Office on 2012-02-02.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to the Subcommittee on
                Readiness, Committee on Armed
                Services, House of Representatives


February 2012
                DEFENSE LOGISTICS

                Improvements Needed
                to Enhance Oversight
                of Estimated Long-
                term Costs for
                Operating and
                Supporting Major
                Weapon Systems




GAO-12-340
                                               February 2012

                                               DEFENSE LOGISTICS
                                               Improvements Needed to Enhance Oversight of
                                               Estimated Long-term Costs for Operating and
                                               Supporting Major Weapon Systems
Highlights of GAO-12-340, a report to the
Subcommittee on Readiness, Committee on
Armed Services, House of Representatives




Why GAO Did This Study                         What GAO Found
With the nation facing fiscal challenges       DOD’s reports to Congress on estimated weapon system O&S costs are often
and the potential for tighter defense          inconsistent and sometimes unreliable, limiting visibility needed for effective
budgets, Congress and the                      oversight of these costs. The SAR statute requires that life-cycle cost reporting
Department of Defense (DOD) have               for major weapon systems be uniform, to the extent practicable, across the
placed more attention on controlling           department, but GAO found a number of inconsistent practices in how program
the billions of dollars spent annually on      offices were reporting life-cycle O&S cost estimates in the SAR. Program offices
weapon system operating and support            were inconsistent in (1) the explanatory information they included with the cost
(O&S) costs. These costs include,              estimates; (2) the source of the cost estimate they cited as the basis for the
costs for repair parts, maintenance,           reported costs; (3) the unit of measure they used to portray average costs;
and personnel, and account for about           (4) the frequency with which they updated reported costs; and (5) the reporting of
70 percent of the total costs of a             costs for an antecedent system being replaced by the new weapon system. For
weapon system over its life cycle. The         example, 35 (42 percent) of the 84 programs that reported O&S costs in the 2010
selected acquisition report (SAR) is           SAR did not cite a source of these data, contrary to DOD’s guidance, and 57 (68
DOD’s key recurring status report on
                                               percent) of the programs did not report O&S costs for an antecedent system.
the cost, schedule, and performance of
                                               Also, O&S cost submissions in the SAR did not always incorporate best practices
major defense acquisition programs
and is intended to provide authoritative
                                               for presenting cost estimates, such as tracking cost changes over time and
information for congressional oversight        identifying cost drivers. In addition, 11 systems did not provide O&S cost
of these programs. Oversight of O&S            estimates in the 2010 SAR.
costs is important because many of the
                                               Although SARs are intended to provide Congress with authoritative program
key decisions affecting these life-cycle
costs are made during the acquisition          information on major weapon systems, 7 of the 15 sample programs GAO
process. GAO reviewed weapon                   reviewed submitted unreliable O&S cost estimate data in the 2007, 2009, or 2010
system O&S cost estimates that DOD             SARs. For example, an Air Force program underreported O&S costs by $2.1
submits in the SAR. Specifically, GAO          billion (fiscal year 2002 dollars), or 18 percent. While some of the program offices
determined the extent to which the             did not provide an explanation for the errors in the submitted data, others cited
SARs provide consistent and reliable           specific reasons. For example, one Navy program office underreported O&S
O&S cost estimate information that             costs in the SAR and explained that it excluded certain costs that were not under
enables effective oversight of these           its control, such as externally funded spare parts and military personnel.
weapon system costs. To conduct its            However, excluding such costs is contrary to the SAR statute. An Air Force
review, GAO analyzed SAR data for 84           program reported current and projected funding for the program rather than
major systems that submitted O&S               estimated life-cycle O&S costs. This practice also had the effect of
cost estimates in the 2010 SAR and             underreporting these costs.
selected a nonprobability sample of 15
systems for more in-depth review.              DOD’s reports to Congress on estimated weapon system O&S costs were often
                                               inconsistent and sometimes unreliable due to a lack of (1) detailed
What GAO Recommends                            implementation guidance for reporting these costs and (2) an effective process
To enhance visibility of weapon system
                                               for reviewing the O&S cost sections of the SAR before final submission to
O&S costs during acquisition, GAO              Congress. DOD’s guidance collectively provides minimal instructions for O&S
recommends that DOD improve its                cost reporting. The guidance also does not incorporate some of the best
guidance to program offices on cost            practices GAO has identified for presenting cost estimates. Further, although the
reporting and also improve its process         SAR data submitted by program offices are subject to multiple reviews within the
for reviewing these costs prior to final       military services and by the Office of the Secretary of Defense, this review
submission of the SAR to Congress.             process has not provided assurance that O&S costs are reported consistently
DOD concurred with GAO’s                       and reliably. In the absence of improvements to the SAR guidance and to the
recommendations.                               review process, deficiencies in reporting O&S costs are likely to continue.
                                               Improved reporting of O&S costs in the SAR could help to place more emphasis
View GAO-12-340. For more information,
contact Cary B. Russell at (404) 679-1808 or   on assessing, managing, and controlling long-term weapon system O&S costs.
russellc@gao.gov.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                   1
               Background                                                                5
               DOD’s Reports to Congress on Estimated Weapon System O&S
                 Costs Were Inconsistent and Sometimes Unreliable                       9
               Conclusions                                                             28
               Recommendations for Executive Action                                    28
               Agency Comments and Our Evaluation                                      30

Appendix I     Scope and Methodology                                                   33



Appendix II    DOD’s Guidance for Implementing the Statutory SAR Requirements          36



Appendix III   Comments from the Department of Defense                                 39



Appendix IV    GAO Contact and Staff Acknowledgments                                   43



Tables
               Table 1: Source of Cost Estimate Cited for 84 Programs Reporting
                        O&S Costs in the 2010 SAR                                      12
               Table 2: Major Weapon System Programs That Did Not Identify
                        O&S Costs in Their 2010 SARs                                   21
               Table 3: Major Weapon Systems in GAO’s Nonprobability Sample            35




               Page i                                          GAO-12-340 Defense Logistics
Abbreviations

CAPE              Cost Assessment and Program Evaluation
DOD               Department of Defense
FBCB2             Force XXI Battle Command Brigade and Below
GPS               Global Positioning System
JTRS              Joint Tactical Radio System
MRAP              Mine Resistant Ambush Protected
O&S               operating and support
OSD               Office of the Secretary of Defense
SAR               Selected Acquisition Report



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                                      GAO-12-340 Defense Logistics
United States Government Accountability Office
Washington, DC 20548




                                   February 2, 2012

                                   The Honorable J. Randy Forbes
                                   Chairman
                                   The Honorable Madeleine Bordallo
                                   Ranking Member
                                   Subcommittee on Readiness
                                   Committee on Armed Services
                                   House of Representatives

                                   With the nation facing fiscal challenges and the potential for tighter
                                   defense budgets, both Congress and the Department of Defense (DOD)
                                   in recent years have placed more attention on controlling the billions of
                                   dollars spent annually on weapon system operating and support (O&S)
                                   costs. These costs include, among other things, costs for repair parts,
                                   maintenance, and personnel, and historically have accounted for about
                                   70 percent of a weapon system’s total costs. 1 In short, the acquisition of a
                                   weapon system today involves a significant financial commitment to that
                                   system over its entire life cycle, a period that may last several decades
                                   from the system’s development to the time it is removed from DOD’s
                                   inventory. For example, the program office for the F-35 Joint Strike
                                   Fighter, the newest aircraft being acquired for the Air Force, Navy, and
                                   Marines, estimated in 2010 that life-cycle O&S costs were about $1.0
                                   trillion, in addition to an estimated $379.4 billion in total acquisition costs. 2

                                   Since 1969, the selected acquisition report (SAR) has been the key
                                   recurring summary status report to Congress on the cost, schedule, and




                                   1
                                     According to DOD officials, O&S costs generally range from 60 to 80 percent of a
                                   weapon system’s total costs, depending on the weapon system type. DOD previously
                                   estimated that weapon system product support costs in fiscal year 2008 were at least
                                   $132 billion. According to DOD, product support encompasses materiel management,
                                   distribution, technical data management, maintenance, training, cataloging, configuration
                                   management, engineering support, repair parts management, failure reporting and
                                   analysis, and reliability growth. Under this definition, product support does not include all
                                   the costs categorized as O&S costs.
                                   2
                                    These costs are expressed in then-year dollars. Estimated O&S costs in fiscal year 2002
                                   dollars were about $420.3 billion.




                                   Page 1                                                         GAO-12-340 Defense Logistics
performance of DOD’s major defense acquisition programs. 3 (We also
use the term major weapon systems to refer to these programs, although
some programs involve other types of defense systems.) The SAR is
intended to provide authoritative information needed for congressional
oversight of these programs. Although much of the data reported within
the SAR is acquisition related, in 1985 Congress amended the SAR
statute to require that a full life-cycle cost analysis also be included in the
SAR, 4 and subsequently specified that this life-cycle cost analysis include
estimated O&S costs. 5 This information is important for effective oversight
because many of the key decisions affecting a weapon system’s O&S
costs over its life cycle are made while the system is still in the acquisition
process. Acquisition-related decisions about the design, materials, and
technology for the system affect the logistics support that is eventually
needed to keep the system available and ready after it enters into DOD’s
inventory. Further, investments to improve reliability, availability, and
maintainability during acquisition can reduce the future O&S costs of
DOD’s weapon systems. For example, higher reliability can be designed
into the weapon system during its development by reducing technical
requirements, using highly reliable and proven components, or investing
more in early testing, thus improving prospects for readiness and
reducing O&S costs across the life cycle. As the visibility and
management of O&S costs have become more of a focus in the
department, the Office of the Secretary of Defense (OSD) has recognized
the SAR as a source of O&S cost information for oversight of weapon
system acquisitions. Therefore, the O&S costs reported in the SARs are




3
  Major defense acquisition programs are those estimated by the Under Secretary of
Defense for Acquisition, Technology and Logistics to require an eventual total
expenditure, including all planned increments, of more than $365 million for research,
development, test, and evaluation, based on fiscal year 2000 constant dollars
(approximately $509 million in fiscal year 2010 dollars), $2.190 billion for procurement,
based on fiscal year 2000 constant dollars (approximately $3.054 billion in fiscal year
2010 dollars), or are designated as a major defense acquisition program by the Milestone
Decision Authority. 10 U.S.C. § 2430; Under Secretary of Defense for Acquisition,
Technology and Logistics Directive-Type Memorandum (DTM) 09-027-Implementation of
the Weapon Systems Acquisition Reform Act of 2009, attachment 1, para. 13 (Dec. 4,
2009).
4
    Department of Defense Authorization Act, 1986, Pub. L. No. 99-145, § 1201 (1985).
5
 See National Defense Authorization Act for Fiscal Year 1991, Pub. L. No. 101-510, §
1407(b) (1990); Federal Acquisition Streamlining Act of 1994, Pub. L. No. 103-355, §
3002(c) (1994).




Page 2                                                       GAO-12-340 Defense Logistics
not only reported to Congress, but they are also included in internal OSD
management reports as well.

In response to the Weapon Systems Acquisition Reform Act of 2009, 6 we
previously reviewed the growth in O&S costs of major weapon systems
and reported that DOD lacked key information needed to effectively
manage and reduce O&S costs—including life-cycle O&S cost estimates
and complete historical data on actual O&S costs. 7 DOD generally
concurred with the recommendations in that report and stated that it was
taking corrective actions. More recently, the National Defense
Authorization Act for Fiscal Year 2012 directs DOD to take a number of
actions consistent with our recommendations, with the aim of better
assessing, managing, and controlling weapon system O&S costs. 8 In light
of the findings from our prior work, you requested that we review the O&S
cost estimate information for major weapon systems that DOD submits to
Congress within the annual SARs. Specifically, our objective was to
determine the extent to which the SARs provide consistent and reliable
O&S cost estimate information to enable effective oversight of these
weapon system costs.

To meet our objective, we reviewed statutory requirements and DOD
guidance for reporting weapon system O&S cost estimates in the SARs.
We also reviewed DOD cost-estimating guidance, which identifies the
specific categories of O&S costs for weapon systems, and GAO-identified
cost-estimating best practices to identify the scope and nature of cost
estimate information needed for effective program management and
oversight. We interviewed DOD and military service officials responsible
for weapon system acquisition, logistics, and cost analysis to understand
DOD’s approach and process for reporting O&S cost estimates in the
SARs.

We obtained SARs from the Defense Acquisition Management
Information Retrieval system, which is a web-based system used within
DOD to collect and maintain SAR information submitted by the program


6
    Pub. L. No. 111-23 (2009).
7
 GAO, Defense Management: DOD Needs Better Information and Guidance to More
Effectively Manage and Reduce Operating and Support Costs of Major Weapon Systems,
GAO-10-717 (Washington, D.C.: July 20, 2010).
8
    Pub. L. No. 112-81, § 832 (2011).




Page 3                                                 GAO-12-340 Defense Logistics
offices. We determined that the data in this system accurately reflected
information submitted by weapon system program offices and therefore
were sufficiently reliable for the purposes of our analysis. We obtained
and analyzed annual SAR data for all 84 major weapon systems that
reported O&S costs in their annual SAR for 2010. This analysis covered
the time period from 2005 through 2010 with the exception of 2008. 9 In
addition, from these 84 weapon systems, we selected a sample of 15
programs for further analysis. 10 Through a data collection instrument,
interviews, and other contacts, we obtained information from the program
offices on the O&S cost reports they had submitted in the SAR, including
cost estimates, where available, that formed the basis for their cost
reports. We designed the sample to ensure that a range of weapon
systems were represented based on commodity type 11 and the military
service under which the system acquisition was being managed. The
results from this nonprobability sample cannot be used to make
inferences about all major weapon systems because the sample may not
reflect all characteristics of the population. We also obtained SARs for 11
weapon systems that reported an annual SAR for 2010 but did not
identify O&S costs in the SAR. We obtained information from the program
offices about the reasons these costs were not reported. Our scope and
methodology is discussed further in appendix I.

We conducted this performance audit from February 2011 to February
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




9
 DOD did not report annual SARs for 2008. In addition, because some of the weapon
systems began reporting SARs at some point during this time period, our analysis for such
systems only dates back to their initial SAR report.
10
   Our original sample included 16 weapon systems. However, one system, the Army’s
Increment 1 Early-Infantry Brigade Combat Team, was subsequently canceled.
11
  We categorized systems as aircraft, ship, ground, or “other” (e.g., missile programs and
command and control systems).




Page 4                                                      GAO-12-340 Defense Logistics
Background
Statutory Requirements for   Requirements for submitting SARs to Congress, including the timing of
Submitting SARs to           these reports and the types of information to be included, are established
Congress                     in statute. Under 10 U.S.C. § 2432, the Secretary of Defense shall submit
                             to Congress at the end of each fiscal-year quarter a report on current
                             major defense acquisition programs. 12 Each SAR for the first quarter of a
                             fiscal year (also known as the comprehensive annual SAR) shall be
                             designed to provide to the Committee on Armed Services of the Senate
                             and the Committee on Armed Services of the House of Representatives
                             the information these committees need to perform their oversight
                             functions. The comprehensive annual SAR shall be submitted within 60
                             days after the date on which the President’s Budget is sent to Congress
                             for the following fiscal year.

                             The statute also requires that the annual SAR include a full life-cycle cost
                             analysis for each major defense acquisition program and each designated
                             major subprogram included in the report that is in the system
                             development and demonstration stage or has completed that stage. 13
                             Further, the Secretary of Defense must ensure that this requirement is
                             implemented in a uniform manner, to the extent practicable, throughout
                             DOD. The term full life-cycle cost, with respect to a major weapon
                             system, means all costs of development, procurement, military
                             construction, and operations and support, without regard to funding
                             source or management control. If the major weapon system has an
                             antecedent system, a full life-cycle cost analysis for that system must also
                             be reported. The SAR reporting requirement ceases after 90 percent of
                             the items are delivered or 90 percent of planned expenditures under the
                             program are made.




                             12
                                SARs for the second, third, and fourth quarters of a fiscal year, known as quarterly
                             SARs, are not required for a program if a report on the program was included in a
                             previous SAR for that fiscal year and during the period since that report the program has
                             not experienced cost increases or schedule delays beyond certain thresholds.
                             13
                                  This stage of acquisition is now called engineering and manufacturing development.




                             Page 5                                                       GAO-12-340 Defense Logistics
DOD’s Guidance for   DOD has issued various guidance documents that implement the
Implementing the     statutory SAR requirements. This guidance is contained in an acquisition
Statutory SAR        instruction, 14 a guidebook on defense acquisition best practices, 15 a draft
                     SAR policy, 16 and an annual memorandum on preparing SARs. 17 DOD
Requirements
                     also has developed instructions 18 and training for entering SAR data into
                     the Defense Acquisition Management Information Retrieval system. 19
                     According to officials, program offices rely on DOD’s implementation
                     guidance because the services do not have their own formal SAR
                     reporting guidance. DOD’s guidance is summarized below and discussed
                     more fully in appendix II.

                     DOD’s implementation guidance states that SARs should be submitted at
                     program initiation or at the time that the program is designated as a major
                     defense acquisition program, and then at least annually thereafter. The
                     guidance requires the submission of a quarterly SAR after a program is
                     rebaselined based on a major milestone decision. 20 With respect to
                     reporting O&S costs in the SAR, DOD’s implementation guidance states



                     14
                          DOD Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008).
                     15
                          DOD, Defense Acquisition Guidebook (2011).
                     16
                       OSD, Selected Acquisition Report (draft) (2006). Although issued in draft, the policy
                     was never issued in final form.
                     17
                        At the time we conducted our review, the most recent annual guidance was issued in
                     January 2011. Under Secretary of Defense for Acquisition, Technology and Logistics,
                     Memorandum for Assistant Secretaries of the Military Services, December 2010 Selected
                     Acquisition Reports (SARs) Guidance (Jan. 14, 2011).
                     18
                        Under Secretary of Defense for Acquisition, Technology and Logistics, SAR Data Entry
                     Instructions (draft) (Nov. 5, 2010).
                     19
                        Throughout the report, we collectively refer to all of DOD’s SAR guidance as DOD’s
                     implementation guidance, including the draft SAR policy.
                     20
                        DOD’s acquisition process includes a series of decision milestones as the systems
                     enter different stages of development and production. As part of the process, the DOD
                     component or joint program office responsible for the acquisition program is required to
                     prepare life-cycle cost estimates, which include O&S costs, to support these decision
                     milestones and other reviews. Key decision milestones include milestone B, which
                     approves entry into the engineering and manufacturing development phase, and
                     milestone C, which approves entry into the production and deployment phase, including
                     low-rate initial production. Continuation into full-rate production occurs after the full-rate
                     production decision review is held. In conjunction with a milestone decision, a program
                     may be rebaselined, which means that the cost, quantity, schedule, and performance
                     goals are changed to reflect the current status.




                     Page 6                                                           GAO-12-340 Defense Logistics
that program offices should provide explanatory information such as the
source and date of the cost estimate, assumptions underlying the
estimate, the antecedent system used for comparison purposes, and an
explanation of how average annual costs were calculated. DOD officials
stated that programs should report the cost estimate that was developed
for the latest acquisition milestone decision. 21 According to the guidance,
programs should report total estimated O&S costs and should also report
average O&S costs by a unit of measure (e.g., average annual cost per
squadron, average annual cost per system). 22 DOD’s guidance states that
if a program has an antecedent system, 23 then O&S costs and
assumptions should be submitted for the antecedent system.

In addition to its SAR implementation guidance, DOD has issued
guidance for developing weapon system O&S cost estimates, which
provide the basis for the O&S cost section of each SAR. Specifically, the
OSD Cost Analysis Improvement Group, now known as the Cost
Assessment and Program Evaluation (CAPE) office, has established
guidance for preparing and presenting life-cycle O&S cost estimates at
acquisition milestone reviews. 24 O&S cost elements, for example, are to
be grouped into six major areas—unit-level personnel, unit operations,
maintenance, sustaining support, continuing system improvements, and
indirect support—which are further broken down into 23 subelements. In
addition, we have identified federal government best practices for
preparing and presenting cost estimates. These practices include tracking
cost estimates over time; identifying the major cost drivers; identifying the




21
   Training materials from the Defense Acquisition University similarly reflect that the
estimate developed for the latest acquisition milestone should be reported in the SAR.
However, the draft SAR policy differs in that it calls for reporting the “most recent” estimate
of life-cycle O&S costs.
22
   The guidance directs program offices to report total estimated O&S costs in constant
dollars as well as then-year dollars, and to report average costs in constant dollars.
Constant dollars measure the value of purchased goods and services at price levels which
are the same as the base year and, unlike then-year dollars, do not contain any
adjustments for inflationary changes that have occurred or are forecasted to occur outside
the base year.
23
   An “antecedent system” is defined by DOD as one that has been replaced by another
due to obsolescence (technical or otherwise).
24
     OSD, Operating and Support Cost Estimating Guide (October 2007).




Page 7                                                         GAO-12-340 Defense Logistics
                         method and process for estimating each cost element; and comparing the
                         program-developed cost estimate to an independent cost estimate. 25


DOD’s Process and        When required, a comprehensive annual SAR is prepared for each major
Schedule for Preparing   weapon system by the program office that is managing its acquisition.
SARs                     Program offices are responsible for weapon systems throughout the life
                         cycle, to include implementing, managing, and/or overseeing their
                         development, production, fielding, sustainment, and disposal. The
                         reporting time frame for the annual SAR is linked to the issuance of the
                         President’s Budget, which occurs early in the calendar year, and the cost,
                         schedule, and performance data reported in the annual SAR should
                         reflect this budget request. 26

                         The Office of the Under Secretary of Defense for Acquisition, Technology
                         and Logistics begins the process by sending out its annual memorandum
                         guidance in mid-January. Program offices then enter data into the
                         Defense Acquisition Management Information Retrieval system and
                         submit the SARs to OSD acquisition officials, generally after some level of
                         internal review by the program office, the Program Executive Officer, 27
                         and the military service under which the program is organized. OSD
                         officials review the SAR submissions, and officials within the Office of the
                         Assistant Secretary of Defense (Logistics and Materiel Readiness) focus
                         on the O&S section of the reports. OSD officials then hold a series of
                         meetings with the services and program office representatives to discuss
                         the SAR submissions and any recommended changes. Consistent with
                         the statutory requirement, the final annual SAR is typically submitted to
                         Congress in April, 60 days after the President’s Budget has been
                         submitted in February.




                         25
                           GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
                         Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).
                         26
                            The annual SAR is dated at the end of the calendar year that just ended because it is
                         the first quarterly SAR of the fiscal year. For example, the annual SAR prepared and
                         reported in early calendar year 2011 is dated December 2010.
                         27
                            A Program Executive Officer is the military or civilian official who has responsibility for
                         directing the acquisition of several major weapon systems, as well as other acquisition
                         programs. This individual reports to and receives guidance and direction from the service
                         acquisition executive.




                         Page 8                                                          GAO-12-340 Defense Logistics
                             Program offices reporting life-cycle O&S cost estimates in the SAR were
DOD’s Reports to             often inconsistent in their cost reporting and also did not follow best
Congress on                  practices for presenting cost estimates. In addition, some programs did
                             not provide any O&S cost estimates in the 2010 SAR. Further, several of
Estimated Weapon             the programs we reviewed in more depth reported unreliable O&S cost
System O&S Costs             data. The main cause for these deficiencies was a lack of detailed SAR
Were Inconsistent and        implementation guidance for reporting O&S costs. In addition, DOD’s
                             process for reviewing the O&S cost sections of the SAR prior to their final
Sometimes Unreliable         submission did not provide assurance that the program offices reported
                             costs uniformly, to the extent practicable, and that these reported costs
                             were reliable. In the absence of improvements to the SAR guidance and
                             to DOD’s review process, deficiencies in reporting estimated life-cycle
                             O&S costs are likely to continue. Such deficiencies may limit visibility
                             needed for effective oversight of long-term weapon system O&S costs
                             during the acquisition process.


Program Offices Were         The SAR statute requires that life-cycle cost reporting for major weapon
Inconsistent in Reporting    systems be uniform, to the extent practicable, across the department, but
Estimated Life-Cycle O&S     we found a number of inconsistent practices in how program offices were
                             reporting life-cycle O&S cost estimates in the SAR. Based on the SAR
Costs and Did Not            submissions we reviewed, program offices were inconsistent in (1) the
Incorporate Best Practices   explanatory information they included with the cost estimates, (2) the
for Presenting These Cost    source of the cost estimate they cited as the basis for the reported costs,
Estimates                    (3) the unit of measure they used to portray average costs, (4) the
                             frequency with which they updated reported costs, and (5) the reporting of
                             antecedent system costs. In addition to these inconsistencies, we found
                             that SAR submissions also did not incorporate best practices for
                             presenting cost estimates, such as tracking cost changes over time and
                             identifying cost drivers. In addition, 11 systems did not provide O&S cost
                             estimates in the 2010 SAR.

                             Submitting more consistent cost reports and incorporating best practices
                             for presenting cost estimates would improve visibility of estimated life-
                             cycle O&S costs in the SAR, as decision makers will have more
                             information with which to evaluate the reported data. For example, the
                             inclusion of the date and the source of the reported estimate provides
                             context about the currency of the reported costs and the level of review
                             (that is, whether the cost estimate was prepared by the program office, by
                             the military service, or by CAPE). Likewise, the inclusion of significant
                             assumptions underlying the cost estimate, an explanation of changes in
                             the cost estimate from the prior year, and information on major cost
                             drivers provides insight into the cost challenges facing the program. In


                             Page 9                                            GAO-12-340 Defense Logistics
                              addition, showing average costs using a common unit of measure allows
                              for easier comparison of program costs to the costs of similar
                              commodities (such as other aircraft programs).

Explanatory Information Was   DOD’s implementation guidance for the SAR directs programs to include
Often Minimal                 explanatory information in the narrative accompanying the O&S cost
                              estimates, such as the source and date of the cost estimate, assumptions
                              underlying the estimate (such as operating tempo, expected reliability and
                              maintainability of the system, maintenance concept, and manning and
                              logistics policies), the antecedent program used for comparison purposes,
                              and an explanation of how average costs were calculated. Although
                              explanatory information can provide context and background for
                              understanding reported costs, we found that the explanatory information
                              included in the O&S narrative was often minimal.

                              Of the 84 programs that reported O&S costs in the 2010 SAR, we found
                              that 35 (42 percent) did not include the source of the estimate and 12 (14
                              percent) did not include the date of the estimate in the O&S narrative.
                              Additionally, for the 15 programs in our sample, we found that beyond
                              providing a few basic details such as the number of units that were to be
                              acquired, their expected service life, and operating tempo, where
                              applicable, the O&S narrative contained minimal explanation of reported
                              cost estimates and the assumptions underlying these estimates, as the
                              following examples illustrate:

                              •   The program office for the Army’s High Mobility Artillery Rocket
                                  System included several assumptions, such as the number of
                                  launchers and the service life. However, instead of reporting
                                  additional O&S cost estimate assumptions (such as operating tempo
                                  and expected reliability/maintainability) in the SAR narrative, the
                                  program stated that this information was available in the service cost
                                  estimate.

                              •   The Joint Mine Resistant Ambush Protected (MRAP) vehicle program
                                  office noted a few specific assumptions, such as the expected service
                                  life of the fleet and the cost per mile for replenishing spare parts.
                                  However, the remaining O&S narrative for the program generically
                                  explained that the estimate included personnel, training, facilities,
                                  vehicle and component repair, and sustainment overhauls, but
                                  provided no other specifics on these areas.

                              •   Only three programs—the Army’s High Mobility Artillery Rocket
                                  System, the Air Force Joint Primary Aircraft Training System, and the



                              Page 10                                           GAO-12-340 Defense Logistics
      Army’s Force XXI Battle Command Brigade and Below (FBCB2)—
      included the maintenance concept planned for that system in their
      O&S narratives. However, even in these cases, the explanatory
      information for O&S costs was very limited.

•     None of the 15 programs included assumptions on the reliability and
      maintainability of the weapon system in their O&S narrative.

While not required by DOD’s implementation guidance, 1 of the 15
programs in our sample included explanatory information on cost drivers
in the SAR O&S narrative. The V-22’s SAR submission for 2009 provided
an explanation of the significant O&S cost increase from the prior SAR in
2007. In the 2007 SAR, the total O&S costs reported were $48.8 billion
(fiscal year 2005 dollars). In the 2009 SAR, the program reported that this
amount had grown to $75 billion (fiscal year 2005 dollars), and that the
O&S cost category showing the greatest increase was unit-level
consumption. 28 In the O&S narrative, the program office attributed the
majority of the cost increase to changes in the methodology used to
estimate unit-level consumption costs. Specifically, the estimate was
updated with the actual costs of parts from fiscal year 2009 and with
projected future cost growth for parts higher than OSD’s inflation
indices. 29 The program office also noted actions being taken to reduce
unit-level consumption costs, such as changes to contracting strategy and
accelerated timelines for repair capabilities.

GAO-identified best practices for presenting cost estimates include
identifying the largest cost elements and cost drivers, and providing
enough information for informed decision making. In addition, we have
previously reported that leading companies identify major drivers of O&S
costs and work with manufacturers to reduce these costs. 30 During our



28
  Unit-level consumption is an O&S cost category that includes a number of subelements
such as the cost of fuel and energy resources; operations, maintenance, and support
materials consumed at the unit level; depot-level reparable spares (individual parts,
subassemblies, or assemblies) required for recurring repair; operational munitions
expended in training; transportation in support of deployed unit training; temporary
additional duty/temporary duty pay; and other unit-level consumption costs, such as
purchased services (e.g., equipment leases and service contracts).
29
     OSD’s inflation indices are published annually and are used to develop cost estimates.
30
  GAO, Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems’ Total Ownership Costs, GAO-03-57 (Washington, D.C.: Feb. 11, 2003).




Page 11                                                       GAO-12-340 Defense Logistics
                           current review, we found that programs typically use CAPE’s O&S cost
                           element structure in reporting O&S costs, but their presentation is limited
                           to the six major elements (e.g., unit-level personnel, maintenance, indirect
                           support). Since each major O&S cost element includes various costs, this
                           information is not sufficient to identify specific cost drivers. Using lower-
                           level cost elements, as provided for in CAPE’s cost element structure,
                           could provide greater visibility of O&S costs for oversight by decision
                           makers. For example, as noted in the case of the V-22 discussed above,
                           the unit-level consumption cost element consists of a number of
                           subelements that can provide additional insight into the discrete factors
                           driving a change in the estimated life-cycle O&S costs for that system.

Sources of Reported Cost   Various cost estimates may be developed over the life cycle of a weapon
Estimates Varied among     system, and DOD officials stated that programs should report the cost
Programs                   estimate developed for the latest acquisition milestone decision. We
                           found that program offices—when a source was cited—cited several
                           different sources as the basis for their reported O&S cost information in
                           the 2010 SAR, and they did not provide an explanation for selecting the
                           source that was used rather than another source that may have been
                           available. As shown in table 1, for the 84 programs that included O&S
                           costs in the 2010 SAR, 42 (50 percent) of the programs cited a specific
                           cost estimate as the source of reported O&S costs. These sources were
                           either a program office cost estimate, service cost estimate, or CAPE
                           independent cost estimate. Another 35 programs (42 percent) did not cite
                           a source, as previously noted. The remaining 7 programs (8 percent)
                           cited a source other than a specific cost estimate.

                           Table 1: Source of Cost Estimate Cited for 84 Programs Reporting O&S Costs in the
                           2010 SAR

                            Source cited in SAR                                        Number of programs
                            No source cited                                                               35
                            Program office cost estimate                                                  23
                            Service cost estimate                                                         15
                            Other                                                                          7
                            CAPE independent cost estimate                                                 4
                            Total                                                                         84
                           Source: GAO analysis of 2010 SAR data.



                           Five programs in the “other” category in table 1 referred to cost estimates
                           but did not provide enough detail to determine what type of cost estimate
                           was used. For example, one program cited a “validated cost estimate”



                           Page 12                                               GAO-12-340 Defense Logistics
without additional specificity about this estimate. Similarly, four programs
stated only that the source of their SAR O&S costs was a cost estimate
prepared for an acquisition decision, but they did not provide additional
information to identify a specific cost estimate. The remaining two
programs in the “other” category in table 1, both of which were included in
our sample, cited a source other than a cost estimate. One of these
programs, the Navy Multiband Terminal, reported total costs from the
milestone C acquisition program baseline, 31 despite the existence of a
service cost estimate prepared for the acquisition decision in July 2010.
The other program, the Air Force’s Navstar Global Positioning System
(GPS), reported using current and future funding data instead of a cost
estimate.

Overall, six programs in our sample were among those that did not cite
the source of the estimate used to report O&S costs. When we asked
these six programs what source was used, five stated that the O&S cost
estimate data in the 2010 SAR were derived from program office cost
estimates, and the remaining program office stated that the source was a
CAPE independent cost estimate. The other nine programs in our sample
had cited a source, with five citing a program office cost estimate; two
citing a service cost estimate; and two, as noted above, citing either an
acquisition program baseline or funding data as the source of their O&S
costs.

As shown in table 1, some programs cited a CAPE independent cost
estimate as the source of the O&S costs reported in the 2010 SAR.
However, we found that one program in our sample, the LHA 6 America
Class, cited a program office cost estimate even though CAPE had
developed an independent cost estimate. Further, while not required, the
program did not mention in the SAR that an independent cost estimate
had been developed. Since 2005 the LHA 6 America Class program has
reported total O&S costs of $4.45 billion (fiscal year 2006 dollars) in its
SAR submissions, reflecting a 2005 program office cost estimate.
However, CAPE’s 2006 independent cost estimate of the program’s O&S
costs was about $300 million (7 percent) higher. According to a CAPE
memorandum, this higher estimate was also not adjusted for cost growth
above inflation. CAPE noted that O&S costs for the LHA 1, an antecedent



31
  An acquisition program baseline quantifies key parameters for the performance, cost,
and schedule of a program throughout the acquisition phase.




Page 13                                                    GAO-12-340 Defense Logistics
                                 system, had increased 4 percent annually since 1990 due to increased
                                 mission personnel and overhaul costs. According to CAPE, adjusting for
                                 this same rate of cost growth above inflation in its LHA 6 estimate would
                                 result in an additional $530 million throughout the system’s life cycle, or
                                 total O&S costs of $5.29 billion. Additionally, F-35 officials told us that
                                 they plan to continue using the program office’s cost estimate to report
                                 O&S costs in the SAR although CAPE is preparing an independent cost
                                 estimate for the program’s next acquisition milestone.

                                 GAO-identified best practices for presenting cost estimates include
                                 providing a comparison of the program estimate to an independent cost
                                 estimate, with an explanation of results and differences. Such a
                                 comparison is beneficial because an independent cost estimate should
                                 provide an objective and unbiased assessment of expected program
                                 costs that tests the program’s estimate for reasonableness. History has
                                 shown a pattern of higher, more accurate cost estimates the further away
                                 from the program office the independent cost estimate is prepared. In the
                                 2009 Weapon Systems Acquisition Reform Act, Congress placed greater
                                 emphasis on independent review of program cost estimates 32 by requiring
                                 that CAPE review cost estimates prepared in connection with all major
                                 weapon systems, and conduct independent cost estimates for certain
                                 systems prior to the milestone A, milestone B, low-rate initial production,
                                 and full-rate production acquisition decisions. Prior to the Act, CAPE was
                                 required to conduct independent cost estimates for some programs, but
                                 was not required to review cost estimates prepared for all major weapon
                                 systems.

Units of Measure for Reporting   DOD’s implementation guidance for the SAR states that programs should
Average Costs Were Not           report average O&S costs in a unit of measure determined by the military
Consistent                       service under which the system’s acquisition is being managed.
                                 Programs are to report these average costs using CAPE’s major cost
                                 elements. We found that several program offices had changed the unit of
                                 measure they reported in the SAR from that used in previous SARs. In
                                 addition, we found that the units of measure that were being reported
                                 varied, particularly among aircraft programs. These inconsistencies make
                                 it difficult to compare a program’s current and prior-year costs, or to
                                 compare costs of similar programs.


                                 32
                                     According to our Cost Estimating and Assessment Guide, an independent cost estimate
                                 is the most rigorous form of independent review; however, other independent reviews are
                                 also useful for decision making.




                                 Page 14                                                   GAO-12-340 Defense Logistics
Of the 84 programs that reported O&S costs in the 2010 SAR, 5 (6
percent) changed the average unit of measure reported from that used
the prior year. Specifically, two aircraft programs went from reporting
costs per squadron in the 2009 SAR to reporting costs per aircraft in their
2010 SAR, a missile program went from reporting costs per unit in the
2009 SAR to reporting total program costs in the 2010 SAR, and two
programs for communications systems went from reporting total program
costs in the 2009 SAR to reporting costs per radio in the 2010 SAR.
These last two programs— Joint Tactical Radio System (JTRS) Ground
Mobile Radios and JTRS Handheld, Manpack, and Small Form Fit—were
included in our sample. When we asked why they changed the unit of
measure, program officials responded that the decision was made based
on feedback they received from OSD when their 2010 SAR submissions
were undergoing review. Of the 5 programs, only the two aircraft
programs disclosed in the SAR that the unit of measure for that system
had changed from the prior year. These two programs reported that they
changed the unit of measure in order to standardize the calculation and
increase the comparability of programs within the same major command.

Also, based on analysis of the 84 systems, we found the most variation in
the unit of measure among aircraft systems. Different programs reported
the average cost per flying hour 33 or the average annual cost per aircraft,
per squadron of aircraft, or per the entire fleet. This issue was also
evident among the programs in our sample that we analyzed in more
depth. For example, the F-35 program reported average cost per flying
hour, the V-22 program reported average cost per aircraft, the F-22
program reported average cost per squadron, and the Joint Primary
Aircraft Training System (a training aircraft) reported average cost for the
whole fleet. Ship costs, in contrast, were generally reported as average
cost per ship or hull, although one ship program reported average annual
cost per fleet. O&S costs for ground and other types of weapon systems
were usually reported as either cost per weapon system unit or total cost
for all weapon system units. However, a few other metrics were reported
by these programs, such as average annual cost per battalion or per
brigade combat team.



33
   With regard to cost per flying hour, DOD officials explained that there are numerous
formulas used throughout DOD. Thus, while several aircraft system program offices
reported average cost per flying hour, they may not have used comparable data and
calculation methodologies.




Page 15                                                      GAO-12-340 Defense Logistics
                              Although portraying average costs with a unit of measure could be useful
                              for tracking cost changes over time, we found that it was generally not
                              possible to identify changes in estimated O&S costs based on the
                              information reported in a single, annual SAR, since programs do not
                              report costs from the prior SAR. Although major weapon system
                              programs are required to identify and reconcile changes to estimated
                              acquisition costs from the prior SAR, and to provide an explanation for
                              each change, this is not required for O&S costs. Even though two of our
                              sample programs, the V-22 and the Navstar GPS, included a statement in
                              the SAR narrative that their O&S costs had changed, it was not possible
                              to tell how much without the prior year’s cost data. Our year-to-year
                              comparisons of reported costs in the SARs showed that cost changes
                              were occurring. For example, we found that the total estimated O&S costs
                              for the JTRS Handheld, Manpack, and Small Form Fit program
                              decreased from $25.5 billion (fiscal year 2004 dollars) in 2009 to $10.2
                              billion in 2010 (fiscal year 2004 dollars). This $15.3 billion decrease
                              occurred despite an increase in acquisition quantity of about 5,000 radios,
                              from around 216,000 to around 221,000. This change, as well as the
                              reasons for the change, was not identified in the SAR narrative. Similarly,
                              we found that the total estimated O&S costs for the F-35 program
                              increased $50 billion (fiscal year 2002 dollars) from 2009 to 2010. The
                              reason for this increase was not explained in the O&S narrative in the
                              SAR.

                              According to GAO-identified best practices for presenting cost estimates,
                              cost estimates should be tracked over time. Specifically, after an estimate
                              is updated, a comparison of the current and prior estimate should be
                              routinely performed and documented, and the results reported to decision
                              makers. A documented comparison allows cost estimators to see how
                              well they are estimating and how the program is changing over time. It
                              also allows others to track the estimates and to identify when, by how
                              much, and why the program cost more or less than planned.

Cost Estimates Were Updated   Updated cost estimates can help to ensure that decision makers have the
at Different Intervals        most current data available on a program. The SAR statute requires
                              major defense acquisition programs to begin reporting when the program
                              is approved to begin the development phase of the acquisition process at
                              milestone B, and DOD’s implementation guidance similarly states that a
                              SAR should first be submitted when a program is initiated, normally at
                              milestone B, or designated as a major defense acquisition program, and




                              Page 16                                           GAO-12-340 Defense Logistics
also when the program is rebaselined after a major milestone decision.
DOD officials stated that programs should report the cost estimate
developed for the latest acquisition milestone decision. 34 Our analysis for
the 84 major weapon system programs that included O&S costs in the
2010 SAR showed that program offices were inconsistent in the
frequency of their O&S cost updates between 2005 and 2010. 35 In many
cases, programs provided more frequent updates than required by DOD’s
guidance, sometimes annually. However, 8 (13 percent) of the 61
programs that were included in the SAR every year during the 2005 to
2010 period did not update their O&S costs at any time during that period.
In contrast, 47 programs (56 percent) of the 84 programs in the 2010
SAR reported using a cost estimate that was prepared in 2010 or 2011 as
the source of their O&S costs. 36 These included 7 programs that began
reporting SARs in 2009 or 2010.

Of the 15 programs in our sample, 3 did not update their SAR O&S costs
during the period between 2005 and 2010, 5 updated their costs once, 5
updated their costs 2 or 3 times, and 2 updated their O&S costs 4 times
during the period. 37 For example, the Navy’s LHA 6 America Class
program office has consistently reported the O&S costs estimated for
milestone B, the program’s only acquisition milestone while under SAR
reporting requirements, in the annual SARs since 2005. Program officials
told us that they were in the process of developing a new cost estimate
for the LHA 7, the next ship in the America Class, and planned to use the
new estimate as the source to report O&S costs in the program’s 2011
SAR submission, if complete. Also, the Army’s FBCB2 program has not



34
   Training materials from the Defense Acquisition University similarly reflect that the
estimate developed for the latest acquisition milestone should be reported in the SAR.
However, as noted earlier, the draft SAR policy differs in that it calls for reporting the “most
recent” estimate of program O&S costs.
35
  As noted earlier, a number of programs did not include the date of the cost estimate
used in the December 2010 SAR.
36
   Because the annual SAR reporting process occurs in the calendar year after the date of
the SAR, a cost estimate that was completed in early 2011 could be used to report costs
in the 2010 SAR.
37
   One of the programs that did not update O&S costs between 2005 and 2010 only
started reporting annual SARs in 2009 and did not update the costs in 2010. Also, one of
the programs that updated annually—the Navstar GPS—reported the program’s current
funding and projected requirements for an 8-year period instead of a life-cycle O&S cost
estimate.




Page 17                                                         GAO-12-340 Defense Logistics
updated its O&S SAR costs and is reporting costs estimated in 2004,
even though the program’s production quantity has quadrupled since
then. 38 FBCB2 program officials told us that since its full-rate production
decision in 2004, the program has experienced nearly continuous
changes to its production quantity requirement, resulting in a significant
effort to maintain and update the acquisition portion of the cost estimates
and little time to research and update the O&S portion of the cost
estimates.

In contrast, several of our sample programs updated their O&S costs
annually. The F-35 program has updated the reported SAR O&S costs
annually since 2006, the beginning of the period we reviewed. According
to F-35 program officials, they chose to do this because the F-35 is a
high-visibility, high-interest program. Further, estimating O&S costs
annually helps inform DOD leadership and keeps partner countries
updated, program officials noted. Additionally, the Joint MRAP program
office has updated its SAR O&S costs annually since the program began
reporting these costs in 2009 and plans to do so until the services
assume responsibility for the system around 2013. Program officials said
they are incorporating actual cost data from the field as it becomes
available and updating O&S costs annually in order to give the services
the best data once the transfer takes place. Finally, the Army’s MQ-1C
program has updated its SAR O&S costs annually since 2009. Although
these costs were updated in 2010 for several reasons, including an
increase in the number of systems to be acquired, program officials said
they do not plan to update the program’s O&S costs annually.

Officials for the remaining programs in our sample, which updated their
O&S costs intermittently, gave various reasons for updating their
program’s SAR O&S costs when they did. While one program updated
the SAR as required to reflect the O&S costs estimated for an acquisition
decision, other programs in our sample chose to update the costs after
they developed estimates to reflect changes to the acquisition program
(e.g., changes in production quantity or schedule), to incorporate actual
O&S costs that are considerably different than previously estimated, or to
comply with guidance not related to the SAR. For example, the Navy’s



38
   The O&S costs reported in the FBCB2’s 2006 SAR are different than those reported in
the 2005 SAR. However, program officials told us that this was a mistake and the costs
had not actually been updated. This case is discussed further later in our report.




Page 18                                                   GAO-12-340 Defense Logistics
                           V-22 program office updated the O&S costs in the 2009 SAR because
                           actual O&S costs incurred after the program’s initial operational capability
                           in 2007 for the Marine Corps and 2009 for the Air Force were significantly
                           higher than had been anticipated in the program’s most recent cost
                           estimate. Prior to the 2009 update, the V-22 was reporting costs based on
                           the estimate completed for an acquisition decision in 2005. The V-22
                           program office, in conjunction with U.S. Naval Air Systems Command,
                           plans to review the program’s O&S costs annually and update the SAR as
                           necessary until the program stops reporting SARs. According to officials,
                           the final deliveries of the V-22 are scheduled for 2020. As another
                           example, the Joint Primary Aircraft Training System program updated
                           O&S costs in the 2010 SAR after reporting the same costs since 2001.
                           According to officials, an updated program office cost estimate was
                           developed to comply with a policy from the program’s major command
                           that cost estimates be updated annually.

                           DOD acquisition best practices and GAO-identified cost-estimating best
                           practices call for maintaining updated estimates of program costs.
                           According to the Defense Acquisition Guidebook, although a DOD or
                           service cost estimate is required at milestone reviews, it is a good
                           practice for this estimate, or at least its underlying program office cost
                           estimate, to be updated more frequently, usually annually. Updated
                           estimates should be useful in program management and financial
                           management throughout the life of the program. GAO-identified best
                           practices call for continual updates of cost estimates to keep them
                           relevant and current, as most programs do not remain static, especially
                           those in development. Routine updates that incorporate actual data result
                           in higher-quality estimates as the program matures. Further, updating the
                           cost estimate provides an accuracy check, defense of the estimate over
                           time, shorter estimate preparation times, and archived cost and technical
                           data for use in future estimates.

Antecedent Program Costs   In accordance with the SAR statute, DOD’s implementation guidance
Were Often Not Reported    states that if a program has an antecedent system, then O&S costs and
                           assumptions should be submitted for the antecedent system. We found
                           that program offices, however, were inconsistent in reporting on
                           antecedent system costs, with many not reporting any O&S cost data.
                           Specifically, 57 (68 percent) of the 84 programs reporting O&S costs in
                           the 2010 SAR did not report O&S costs for an antecedent system. It was
                           unclear from the SARs how program offices had identified an antecedent
                           system or whether, in cases where no antecedent system costs were
                           included, the program offices had determined that an antecedent system
                           did not exist.


                           Page 19                                            GAO-12-340 Defense Logistics
                               Nine of the 15 programs in our sample did not report O&S costs for an
                               antecedent system in the 2010 SAR. Officials from these program offices
                               provided various reasons for not reporting antecedent system costs,
                               including that the system was the first of its type or not intended to
                               replace any other system, that the system had advanced capabilities or
                               no other system was similar enough for comparison, and that the system
                               was replacing several legacy systems. As an example, Joint MRAP
                               program officials said other systems, such as the High Mobility
                               Multipurpose Wheeled Vehicle, were too different for cost comparisons.
                               As another example, the Navy Multiband Terminal program began
                               reporting in the 2006 SAR and has never reported antecedent O&S costs.
                               According to program officials, an antecedent system was not identified
                               because the system was replacing several legacy weapon systems.
                               However, during a joint OSD/Navy SAR review meeting in March 2011,
                               the program office was instructed to list two systems as antecedent
                               systems in the 2010 SAR. While the program identified the Super High
                               Frequency and Navy Extremely High Frequency Satellite programs as
                               antecedent systems in the O&S section of its SAR, it also stated that
                               program costs for these systems were not readily available.

Some Programs Did Not Report   The SAR statute requires that all program costs be reported, regardless
Any O&S Costs                  of funding source or management control. However, we found that of the
                               95 major weapon systems that had passed milestone B and reported
                               costs in the 2010 SAR, 39 11 (12 percent) did not identify any O&S costs in
                               their SARs. The 11 programs, as of December 2010, accounted for a total
                               estimated investment of $56.7 billion (fiscal year 2011 dollars) for
                               research and development, procurement, military construction, and
                               acquisition-related operation and maintenance (see table 2). Most of the
                               programs that did not report O&S costs were modifications to other
                               weapon systems but qualify as major defense acquisition programs
                               based on their procurement or research and development costs.




                               39
                                    One program submitted a 2010 SAR but had not yet passed acquisition milestone B.




                               Page 20                                                    GAO-12-340 Defense Logistics
Table 2: Major Weapon System Programs That Did Not Identify O&S Costs in Their 2010 SARs

Dollars in billions
Major weapon system                                                                 Service                Estimated investment costsa
C-5 Avionics Modernization Program                                                  Air Force                                            $1.3
C-5 Reliability Enhancement and Reengineering Program                               Air Force                                            $7.3
GPS III                                                                             Air Force                                            $4.3
Multi-Platform Radar Technology Insertion Program                                   Air Force                                            $1.4
Airborne Signals Intelligence Payload                                               Air Force                                            $0.5
Large Aircraft Infrared Countermeasures                                             Air Force                                            $0.5
B-2 Radar Modernization Program                                                     Air Force                                            $1.3
B-2 Extremely High Frequency Satellite Communications                               Air Force                                            $0.6
Apache Block IIIB                                                                   Army                                                 $2.2
Chemical Demilitarization-Assembled Chemical Weapons Alternatives                   DOD                                                $10.0
Chemical Demilitarization-U.S. Army Chemical Materials Agency                       Army                                               $27.4
                                          Source: GAO analysis of 2010 SAR data.
                                          a
                                           Costs are in fiscal year 2011 dollars and include costs associated with research and development,
                                          procurement, military construction, and acquisition-related operation and maintenance.


                                          Eight of the programs that did not report O&S costs are major
                                          modifications to, or subsystems of, Air Force weapon systems. 40 When
                                          we asked why O&S costs were not reported, officials from six Air Force
                                          programs said they did not report O&S costs in the 2010 SAR because
                                          they do not fund or track these costs. For example, officials for two
                                          programs associated with the C-5 aircraft explained that all O&S fleet
                                          costs are the responsibility of another entity, the System Program
                                          Manager at Warner Robins Air Logistics Center in Georgia. Program
                                          officials for the other two Air Force programs, the B-2 Radar
                                          Modernization Program and B-2 Extremely High Frequency Satellite
                                          Communications program, told us that these modification programs were
                                          expected to reduce O&S costs and they could not input cost reductions
                                          into DOD’s Defense Acquisition Management Information Retrieval
                                          system, the database that maintains SAR data. In contrast to these
                                          modification programs, the Air Force’s C-130 Avionics Modernization
                                          Program did report total estimated O&S costs in the 2010 SAR.




                                          40
                                            These modifications are considered major defense acquisition programs in
                                          development and are therefore required to report submit SARs.




                                          Page 21                                                            GAO-12-340 Defense Logistics
                            According to officials, one of the remaining three programs—the Army’s
                            Apache Block IIIB—was not required to report O&S costs in the SAR, as
                            approved by the Defense Acquisition Executive. 41 The other two
                            programs are the Chemical Demilitarization-Assembled Chemical
                            Weapons Alternatives, and the Chemical Demilitarization-U.S. Army
                            Chemical Materials Agency. According to the SAR for each program,
                            O&S costs are reported in other sections of the reports. For example,
                            program officials told us that O&S costs for the Assembled Chemical
                            Weapons Alternatives program are captured in research, development,
                            test, and evaluation costs. According to program officials, the Chemical
                            Demilitarization program is a one-of-a-kind national environmental and
                            safety program that is unlike weapon systems that report SARs. Further,
                            officials said that the two programs have not separately reported any O&S
                            costs since they were designated major defense acquisition programs in
                            1994.


Program Offices             SARs are intended to provide Congress with authoritative program
Sometimes Provided          information on the cost, schedule, and performance of major weapon
Unreliable Life-Cycle O&S   systems, but we found that some programs submitted unreliable O&S
                            cost data. More specifically, our review of SAR reports for the 15
Cost Estimates in Their     programs in our sample identified inaccurate cost estimates and other
SARs                        errors in SARs submitted in 2007, 2009, and 2010. (As noted earlier,
                            DOD did not submit SARs in 2008.) While some of the program offices
                            told us specific reasons for the errors, others did not provide an
                            explanation.

                            Based on our analysis of O&S cost data reported in the SAR compared
                            with the underlying cost estimates and other information provided by the
                            program offices, we found that 7 of the 15 programs reported inaccurate
                            O&S costs in one or more of the three annual SARs.




                            41
                               The O&S costs of the Apache Block IIIB are captured in the SAR for the Apache Block
                            IIIA Remanufacture program. According to program officials, the programs were originally
                            a single program but at milestone C the decision was made to split them into two separate
                            programs. The Defense Acquisition Executive approved the reporting of O&S costs in the
                            Apache Block IIIA Remanufacture SAR only. SARs for both programs note that the O&S
                            costs are reported in the Apache Block IIIA Remanufacture SAR, but the costs are not
                            broken out between the two programs.




                            Page 22                                                    GAO-12-340 Defense Logistics
•   The F-35 Joint Strike Fighter program office underreported the
    average cost per flying hour for the aircraft in the 2010 SAR. The
    average, steady-state O&S cost per flying hour was reported as
    $16,425 (fiscal year 2002 dollars). Program officials told us that the
    number of aircraft used in the estimate for the Air Force’s inventory
    was not accurate and the estimate also did not project for future cost
    growth above inflation. The estimate included approximately 528 extra
    aircraft that when calculating the average cost per flying hour, resulted
    in higher flight hours and lower average costs per hour. Further,
    according to the SAR, some of the F-35’s O&S costs were
    intentionally excluded from the estimate to enable comparison with
    the antecedent system, the F-16 C/D. Costs for support equipment
    replacement, modifications, and indirect costs were removed from the
    F-35’s cost per flying hour since they were not available for the F-16
    C/D. Officials calculated that the revised cost per flying hour for the F-
    35 was $23,557 (fiscal year 2002 dollars), or 43 percent higher, after
    including the excluded costs, projecting for future cost growth above
    inflation, and correcting the number of aircraft. However, they noted
    that the total O&S life-cycle cost reported in the SAR for the F-35 was
    accurate because it was calculated separately from the average cost
    per flying hour.

•   The Navy Multiband Terminal program office underreported estimated
    life-cycle O&S costs in the 2010 SAR. The program reported $219.1
    million in total O&S costs but excluded an additional $591.3 million for
    externally funded depot-level repairables ($148.4 million) and military
    personnel ($442.9 million), which were included in a 2010 service cost
    estimate. 42 Therefore, only 27 percent of the program’s estimated
    total O&S costs were reported in the 2010 SAR. Program officials
    stated that these costs are not under the control of the program office
    and should not be reported in the SARs. However, the SAR statute
    states that full life-cycle costs, including O&S costs, should be
    reported without regard to funding source or management control.

•   The Air Force Joint Primary Aircraft Training System program office
    underreported O&S costs in the 2007 and 2009 SARs, both of which
    were based on a 2001 service cost estimate. The program, which
    includes the T-6 aircraft and a ground-based training system, reported


42
   The excluded costs in the service cost estimate were expressed only in then-year
dollars, rather than constant dollars. Therefore, all cost figures in this example are
presented in then-year dollars.




Page 23                                                       GAO-12-340 Defense Logistics
    total O&S costs of $9.4 billion (fiscal year 2002 dollars) in both SARs
    but excluded $2.1 billion (fiscal year 2002 dollars)—or about 18
    percent—of O&S costs for the program’s ground-based training
    system. Program officials have reported the same O&S costs since
    the annual 2002 SAR. The program, which updated its O&S estimate
    in 2011, included these costs in the total O&S costs reported in the
    2010 SAR.

•   The Army’s High Mobility Artillery Rocket System program office
    overstated O&S costs in the program’s 2007, 2009, and 2010 SARs.
    Although program office estimates were provided to us for the 3
    years, the estimates did not match the costs reported in the SARs.
    The O&S costs reported in 2007 were higher than the estimate by
    $11.1 million (fiscal year 2003 dollars), and the $988 million (fiscal
    year 2003 dollars), reported in both 2009 and 2010, was higher than
    the estimates by about $300 million (fiscal year 2003 dollars), or about
    43 percent. Program officials told us that the costs had been reported
    incorrectly in each year.

•   The JTRS Handheld, Manpack, and Small Form Fit program
    underreported total O&S costs in the annual 2007 SAR. The SAR
    stated that the O&S costs had been updated, but the O&S costs were
    unchanged from prior annual SARs. Program officials also provided
    us with an estimate that matched the numbers reported in the 2007
    SAR. When asked why the costs had not changed, program officials
    stated that while the costs for procurement and research,
    development, test, and evaluation were correctly updated in 2007, the
    O&S costs were not. They explained that the reported costs of $4.9
    billion (fiscal year 2004 dollars) should have been higher by $120
    million (fiscal year 2004 dollars), but they did not provide us the
    estimate on which that higher amount was based.

•   The Air Force’s Navstar GPS program, as noted earlier, did not report
    a life-cycle cost estimate in the annual SARs from 2007 through 2010.
    For example, according to the 2010 SAR, the O&S costs reported
    were based on funding for fiscal years 2008 through 2016. Program
    officials confirmed that the O&S amounts reported included actual
    funding for the current year and funding from the Air Force’s budget
    system for the remaining years. However, even this amount—about
    $469 million (fiscal year 2000 dollars) in 2010, for example—was
    significantly understated. According to program officials, the amount
    reported in the SAR is only 60 percent of the program’s actual
    requirements of approximately $782 million—a difference of $313
    million—and the program has historically been funded to 90 percent of


Page 24                                            GAO-12-340 Defense Logistics
    requirements with supplemental funds. However, this was not noted in
    the SARs.

•   The FBCB2 program underreported total O&S costs in the annual
    2007 through 2010 SARs. As explained earlier, reported O&S costs
    were estimated for the program’s final acquisition milestone, full-rate
    production, in 2004. In subsequent years, however, the program’s
    procurement quantities increased and were about 305 percent higher
    in the 2010 SAR than the amount used to develop the estimate.
    Further, total O&S costs of $468 million (fiscal year 2005 dollars)
    reported in the SARs were $129 million less than the $596.2 million
    estimated in 2004. Officials initially indicated that some of the
    estimated O&S costs were likely included with the program’s
    acquisition costs in the SAR, but they were unable to reconcile the
    costs in the two documents.

We also found examples of inaccuracies in other data reported in the
O&S cost section of the SARs. For example, the 2010 SAR for the Joint
MRAP states that the program’s O&S costs were reviewed by CAPE in
2010, but program officials and prior-year SARs stated that the review
actually occurred in 2008. Further, neither CAPE nor the program office
was able to provide any record of the 2008 review. As another example,
the 2010 SAR for the F-22 indicates that the reported O&S costs were
based on a 2004 acquisition decision estimate that was updated with
analyses from 2010 to bring the estimate in line with the current approved
F-22 production program and operational concepts. However, the O&S
costs reported are identical to those reported in the 2009 SAR, which
states it was updated based on analyses from 2009.

Implementation of the GAO-identified best practices already discussed
could improve the reliability of O&S costs reported in the SARs. Together,
the best practices work to provide more assurance that the correct
information is reported. For example, routinely updating O&S cost
estimates—and related SAR data—will likely require more frequent
changes to the reported cost data. Therefore, it is less probable that an
error or omission will be regularly reported. In addition, as noted earlier,
comparing a program’s cost estimate with an independent cost estimate,
and explaining any significant differences, could help decision makers
monitor the reasonableness of the reported data. Finally, tracking O&S
costs over time, by presenting the current year and prior-year program
cost estimates and explaining significant differences, would also help to
test the reasonableness of reported costs.




Page 25                                            GAO-12-340 Defense Logistics
DOD Could Improve O&S     DOD’s reports to Congress on estimated weapon system O&S costs were
Cost Reporting with       often inconsistent and sometimes unreliable due to a lack of detailed
Detailed Implementation   implementation guidance for reporting these costs. In addition, DOD’s
                          process for reviewing the O&S cost sections of the SAR prior to final
Guidance and a More       submission did not provide assurance that the program offices reported
Effective SAR Review      costs uniformly, to the extent practicable, and that these reported costs
Process                   were reliable. In the absence of improvements to the SAR guidance and
                          to DOD’s review process, deficiencies in reporting estimated life-cycle
                          O&S costs are likely to continue.

                          DOD’s existing implementation guidance collectively provides minimal,
                          and in some areas conflicting, instructions for O&S cost reporting. For
                          example, the guidance does not identify which cost estimate or estimates
                          should be used to report O&S costs when more than one estimate is
                          available. Often multiple cost estimates are prepared by the program
                          office, the service, and CAPE to support acquisition decisions. Further,
                          DOD officials stated that O&S costs reported in the SAR should be
                          updated only at acquisition milestones. Because many years may pass
                          between these milestones, however, reported O&S costs may become
                          outdated, no longer reflecting the status of the current acquisition
                          program.

                          DOD’s guidance also provides very little detail on how program offices
                          should discuss assumptions underlying the cost estimate. DOD’s draft
                          SAR policy, for example, only mentions several assumptions for
                          consideration, such as operating tempo, expected reliability and
                          maintainability of the system, the maintenance concept, and manning and
                          logistics policies, and does not provide specific examples. In addition, the
                          statutory SAR requirement to report all program costs, regardless of
                          funding source or management control, is reflected in none of DOD’s
                          SAR implementation guidance; it is reflected in training course materials
                          on using the Defense Acquisition Management Information Retrieval
                          system. Finally, DOD’s draft SAR policy provides conflicting instructions
                          on cost reporting for antecedent systems. The draft policy states that
                          antecedent costs should be reported “whenever those costs have
                          previously been developed.” However, in the appendix, the draft guidance
                          states that O&S costs will be reported for antecedent systems “when the
                          replacement system is required to report O&S costs.” DOD officials could
                          not explain the reason for this variance in the guidance.

                          While some program offices we contacted indicated that DOD’s
                          implementation guidance on reporting O&S costs in the SAR was
                          sufficient, officials from several program offices in our sample indicated


                          Page 26                                            GAO-12-340 Defense Logistics
that more detailed guidance would be helpful when they prepare their
annual SAR submissions. These officials stated that there was minimal
guidance provided on what should be included in the O&S narrative and
that there needed to be more consistency in SAR O&S reporting.
Additionally, they explained that the current guidance is vague, unclear,
open to interpretation, and does not provide useful information or
examples for how programs should be reporting these costs. Officials
from one program also stated that there is no direction on the comparison
of program costs to the antecedent system’s costs, so the approach to
making this comparison is open to interpretation. They noted that the
guidance does not specify whether the program office should alter the
weapon system’s O&S costs to enable a true comparison with the costs
for the antecedent system, or whether the weapon system’s O&S costs
should be reported without modification. Finally, while several program
offices told us that the Defense Acquisition University provides useful
training on acquisition reporting in general, they said that the materials
should be more readily available as program representatives could not
always attend the class and that the O&S section of the SAR was not
covered sufficiently.

The SAR data submitted by program offices are subject to multiple
reviews within the military services and by OSD, but this review process
has not provided assurance that O&S costs are reported consistently and
reliably. Although our review did not include a full evaluation of DOD’s
SAR review process, OSD officials explained that once they receive the
SAR submissions, there is a relatively short amount of time to review the
SAR O&S data. For example, according to the SAR review schedule, the
Office of the Assistant Secretary of Defense (Logistics and Materiel
Readiness) usually has about a week to review the O&S cost
submissions. We also noted that “SAR review guidance” that is included
with the annual memorandum on preparing SARs does not provide
additional direction to the program offices on what to include in their O&S
cost submissions. In some cases, the annual memorandum is less
specific than the draft SAR policy. The deficiencies in DOD’s
implementation guidance likely hinder the effective review of SAR O&S
cost information at all levels.

The department’s emphasis on weapon system O&S costs has been
increasing in recent years, 43 but the primary focus continues to be on


43
     This increased emphasis is discussed in GAO-10-717.




Page 27                                                    GAO-12-340 Defense Logistics
                      acquisition costs. According to OSD acquisition officials, the SAR started
                      as—and is still often viewed as—primarily an acquisition report. This
                      perspective was reflected in comments from some program officials. For
                      example, officials at one program office told us that, due to a constantly
                      changing acquisition program, their time was largely spent on estimating
                      acquisition costs. Another program office noted that the focus of the SAR
                      statute was acquisition costs and that O&S costs will vary based on
                      emerging needs. Several other programs indicated that O&S cost
                      estimating was not particularly useful, as their systems had not yet
                      entered into production or sustainment, and actual cost data were either
                      not yet available or could not be obtained by the program office. Finally,
                      other program offices stated that since they do not fund the support of the
                      weapon system, the O&S cost estimates should be done by the
                      organizations responsible for providing this funding.


                      Without more consistent and reliable reporting of estimated weapon
Conclusions           system O&S costs, Congress and senior DOD officials may have limited
                      visibility of information needed to effectively oversee the full life-cycle
                      costs associated with weapon system acquisitions. Improvements in the
                      reporting of these data could provide a more complete picture of the
                      potential total financial commitment being made to these systems over a
                      period lasting many decades. SAR cost estimates are reported early
                      during acquisition, when there is the greatest chance for managing or
                      reducing future O&S costs. By facilitating inquiries on changes from prior
                      cost estimates and cost drivers, such information could affect acquisition
                      investment decisions and result in tradeoffs that otherwise might not be
                      considered. Furthermore, improvements to SAR reporting would be
                      consistent with a provision in the National Defense Authorization Act for
                      Fiscal Year 2012 directing DOD to take actions aimed at better
                      assessing, managing, and controlling weapon system O&S costs.




                      To improve visibility over estimated life-cycle O&S costs during weapon
Recommendations for   system acquisition, we recommend that the Secretary of Defense take the
Executive Action      following two actions.

                      First, we recommend that the Secretary of Defense direct the Under
                      Secretary of Defense for Acquisition, Technology and Logistics to revise
                      DOD’s guidance for implementing statutory SAR requirements. The



                      Page 28                                            GAO-12-340 Defense Logistics
revisions, at a minimum, should provide additional detail on the following
areas:

•   the explanatory information that should be included in the O&S
    narrative, including the specific assumptions underlying the cost
    estimate;

•   the source to be used as the basis for reported O&S cost estimate
    information, especially when more than one source is available (such
    as a program office cost estimate, service cost estimate, and CAPE
    independent cost estimate);

•   a consistent unit of measure for reporting average costs over time by
    commodity type—or other designated weapon system group—as
    agreed to by OSD and the services;

•   criteria for identifying an antecedent system and reporting on the
    results of the cost comparison in the SAR; and

•   reporting O&S costs for major modifications to existing weapon
    systems.

In revising the guidance, the Under Secretary of Defense should
incorporate best practices for preparing and presenting cost estimates,
including:

•   a comparison of current-year to prior-year O&S cost estimates; the
    identification of cost drivers that resulted in changes in these
    estimates, if significant; and the level of detail that should be reported;

•   a comparison of the reported cost estimate with the most recent
    independent cost estimate, along with an explanation of any
    significant differences between the two estimates; and

•   The frequency with which O&S costs reported in the SAR should be
    updated, including guidance on what changes in the program’s status
    should trigger an update.

Second, we also recommend that the Secretary of Defense direct that the
Under Secretary of Defense for Acquisition, Technology and Logistics, in
conjunction with the Secretaries of the Army, the Air Force, and the Navy,
evaluate the current review process, identify any weaknesses, and
institute corrective actions as needed to provide greater assurance that



Page 29                                              GAO-12-340 Defense Logistics
                     estimated life-cycle O&S costs included in the SAR reports submitted by
                     program offices consistently follow the implementation guidance,
                     including any revisions to the guidance as described above, and report
                     reliable cost data. As part of this evaluation, DOD should consider
                     whether additional steps are necessary for the department to enhance the
                     emphasis placed on reporting estimated life-cycle O&S costs in the SAR.


                     DOD provided comments on a draft of this report. In its comments, DOD
Agency Comments      agreed with both of our recommendations. The department’s written
and Our Evaluation   comments are reprinted in appendix III. DOD also provided technical
                     comments that we have incorporated into this report where appropriate.

                     In concurring with our first recommendation to revise DOD’s guidance for
                     implementing statutory SAR requirements, DOD noted that the focus of
                     the SAR has always been primarily on acquisition rather than
                     sustainment. DOD further stated that Congress, in requiring DOD to add
                     O&S costs to the SAR report, did not intend for DOD to develop O&S
                     costs for each submission but to report the latest available estimate for
                     the program. Our report recognizes that the development of new O&S
                     cost estimates is not required for each annual SAR submission. However,
                     these costs represent a significant proportion of a system’s total costs
                     over its life cycle. Moreover, we found that the timing of updates to the
                     O&S costs reported in the SAR varied widely, as DOD has not identified
                     what changes in a program's status—other than established acquisition
                     milestones, which can be many years apart—should trigger such
                     updates. We also continue to believe that DOD needs to clearly identify
                     the source and date of the O&S cost estimate data reported in the SAR.
                     Our recommendations reflect these and other weaknesses in the current
                     reporting of O&S costs.

                     DOD’s comments identified actions it plans to take to implement our
                     recommendations. DOD stated that it will expand and update its current
                     guidance for the O&S cost section of the SAR, as contained in the
                     Defense Acquisition Guidebook. DOD plans to make revisions specifically
                     with regard to assumptions and ground rules (e.g., the source and date of
                     the estimate reported); a consistent unit of measure for reporting O&S
                     costs for each type of commodity; identifying, and reporting on,
                     antecedent systems; and reporting O&S costs for major modifications.
                     These planned revisions to the guidance are positive steps. We plan to
                     monitor DOD’s actions as part of our recommendation follow-up process.




                     Page 30                                          GAO-12-340 Defense Logistics
Regarding other revisions to the guidance that we recommend to
incorporate best practices for O&S cost reporting, DOD stated that the
department is not yet in a position to add a credible O&S cost variance
analysis. Although DOD does not define what it means by "cost variance
analysis," it is reasonable to expect that such analysis would involve
comparing changes from a previous cost estimate and identifying any
significant cost drivers. DOD noted that it is implementing new O&S-
related requirements from the National Defense Authorization Act for
Fiscal Year 2012, as well as previous requirements from the Weapon
Systems Acquisition Reform Act of 2009, including requirements that deal
with cost variance analysis. DOD stated that it is premature to determine
to what extent DOD's implementation of these requirements will affect the
reporting of O&S costs in the SAR. With these and other ongoing
activities related to the management and control of O&S costs, DOD
would prefer to defer these additional reporting requirements for the SAR
for now.

We are aware that DOD has a number of ongoing activities to improve
the management and control of O&S costs and must respond to several
new requirements, as stated in DOD's comments. For example, the O&S-
related guidance required by the National Defense Authorization Act for
Fiscal Year 2012 must be issued within 180 days from the date the Act
was enacted, which was December 31, 2011. If such activities result in
improved visibility of O&S costs within the department, and DOD
coordinates these activities with efforts to improve O&S cost reporting in
the SAR, then we agree that it may be preferable to delay implementation
of the best practices we recommend in our report. However, we continue
to believe that these best practices, when implemented, will provide better
information on the current status and direction of long-term O&S costs
and thus can improve congressional oversight of weapon system costs.
Therefore, these elements of our recommendation remain valid.

DOD also concurred with our second recommendation to evaluate and
make any changes needed to strengthen its current process for reviewing
O&S cost reporting prior to submission of SARs to Congress. In its
comments, DOD cited actions it would take in the short term to improve
the review of O&S costs prior to submission of SAR reports at the end of
March 2012. DOD stated that the O&S cost section will be given
additional emphasis during this reporting period. Subsequently, DOD will
convene a joint OSD/DOD component working group that will evaluate
the current SAR review process, identify any weaknesses, and institute
corrective actions as needed to improve the data quality for the estimated



Page 31                                           GAO-12-340 Defense Logistics
life-cycle O&S costs reported in the SAR. We believe these actions, when
implemented, will meet the intent of our recommendation.


We are sending copies of this report to interested congressional
committees; the Secretary of Defense; the Secretaries of the Army, the
Navy, and the Air Force; the Acting Under Secretary of Defense for
Acquisition, Technology and Logistics; and the Director, Office of
Management and Budget. In addition, the report will be available at no
charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please
contact me at (404) 679-1808 or russellc@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors are listed in appendix IV.




Cary B. Russell
Acting Director, Defense Capabilities and Management




Page 32                                          GAO-12-340 Defense Logistics
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             To determine the extent to which the selected acquisition reports (SAR)
             provide consistent and reliable operating and support (O&S) cost
             estimate information that enables effective oversight of major weapon
             system costs, we reviewed statutory requirements in 10 U.S.C. § 2432 for
             reporting weapon system life-cycle costs in the SARs, as well as
             Department of Defense (DOD) implementation guidance for the SAR. We
             also reviewed DOD guidance for preparing weapon system O&S costs
             and GAO-identified cost-estimating best practices to identify the scope
             and nature of cost estimate information needed for effective program
             management and oversight.

             We interviewed and obtained documentation from DOD and military
             service officials responsible for weapon system acquisition, logistics, and
             cost analysis to understand DOD’s approach and process for reporting
             O&S cost estimates in the SARs. Offices we contacted included the
             following:

             •   Office of the Under Secretary of Defense for Acquisition, Technology
                 and Logistics
                 •   Office of the Director, Acquisition Resources and Analysis
                 •   Office of the Deputy Assistant Secretary of Defense for Materiel
                     Readiness
             •   Office of the Director, Cost Assessment and Program Evaluation
             •   Office of the Deputy Assistant Secretary of the Army, Cost and
                 Economics
             •   Naval Center for Cost Analysis
             •   Air Force Cost Analysis Agency

             We obtained SARs for all 95 weapon systems that reported a December
             2010 SAR. 1 These reports were contained in the Defense Acquisition
             Management Information Retrieval system, which is a web-based system
             used within DOD to collect and maintain SAR information submitted by
             program offices. We determined that the data in this system accurately
             reflected information submitted by weapon system program offices and
             therefore were sufficiently reliable for the purposes of our analysis. After
             determining that a total of 84 of the 95 weapon systems included O&S
             costs in their December 2010 SARs, we analyzed the annual SARs that


             1
              Although a total of 96 programs submitted a SAR in 2010, the Ballistic Missile Defense
             Program had not yet reached milestone B and thus was not required to report O&S costs.
             For this reason, we did not include this program in our analyses.




             Page 33                                                    GAO-12-340 Defense Logistics
Appendix I: Scope and Methodology




were submitted for these systems between 2005 through 2010. 2
Specifically, we analyzed the SARs to determine the types and scope of
explanatory information included in the O&S narrative accompanying the
cost estimate data; the source of the O&S cost estimate cited as the basis
for the reported costs; the units of measure used to present O&S costs;
the frequency that O&S costs were updated from year to year; and the
extent to which O&S costs for antecedent systems were reported. We
compared the SARs across each of these categories to determine the
extent to which information was reported consistently across all 84
weapon systems.

From the population of 84 weapon systems that included O&S cost
estimates in the 2010 SARs, we selected a sample of 15 weapon
systems for further analysis. 3 We designed the sample to ensure that a
range of weapon systems were represented based on commodity type
and service responsible for managing the program. We selected three or
four weapon systems per service and at least one commodity type within
each service for a total sample size of 15. 4 We also examined the
distribution of weapon systems’ total costs across our sample selection in
terms of both dollars and the upper and lower 50 percent of weapon
systems that reported O&S costs in the 2010 SAR. We determined that
the sample contained an adequate mix of high- and low-dollar weapon
systems for our purposes. The results from this nonprobability sample
cannot be used to make inferences about all major weapon systems
because the sample may not reflect all characteristics of the population.
The 15 programs in our sample are shown in table 3.




2
 Annual SARs are submitted to Congress within 60 days after the date on which the
President’s Budget is submitted to Congress for the following fiscal year. We analyzed the
December SARs for the years 2005, 2006, 2007, 2009, and 2010. Annual SARs were not
submitted in 2008.
3
  Our original sample included 16 weapon systems. However, one system, the Army’s
Increment 1 Early-Infantry Brigade Combat Team program, was subsequently canceled.
4
 We categorized systems as aircraft, ship, ground, or “other” (e.g., missile programs and
command and control systems).




Page 34                                                     GAO-12-340 Defense Logistics
                                         Appendix I: Scope and Methodology




Table 3: Major Weapon Systems in GAO’s Nonprobability Sample

Service           Major weapon system                                                         Commodity type
Army              1.   Force XXI Battle Command Brigade and Below                             Other
                  2.   High Mobility Artillery Rocket System                                  Other
                  3.   MQ-1C Unmanned Aircraft System Gray Eagle                              Aircraft
Navy              4.   Joint Mine Resistant Ambush Protected                                  Land
                  5.   LHA 6 America Class                                                    Ship
                  6.   Navy Multiband Terminal                                                Other
                  7.   V-22 Joint Services Advanced Vertical Lift Aircraft (Osprey)           Aircraft
Air Force         8.   F-22 Raptor                                                            Aircraft
                  9.   Joint Primary Aircraft Training System                                 Aircraft
                  10. Navstar Global Positioning System                                       Other
                  11. National Polar-orbiting Operational Environmental Satellite System      Other
DOD               12. Airborne and Maritime/Fixed Station Joint Tactical Radio System         Other
                  13. F-35 Joint Strike Fighter                                               Aircraft
                  14. Joint Tactical Radio System Ground Mobile Radios                        Other
                  15. Joint Tactical Radio System Handheld, Manpack, and Small Form Fit       Other
                                         Source: GAO analysis of 2010 SAR data.



                                         Through a data collection instrument, interviews, and other contacts, we
                                         obtained information from the 15 program offices on the O&S costs they
                                         had reported in the SAR, including the cost estimates, where available,
                                         that formed the basis for their reported costs. We compared the cost
                                         estimates used to develop SAR O&S costs, as well as any additional cost
                                         estimates that had been developed for the weapon systems, to the O&S
                                         costs reported in the SARs. We interviewed program officials about the
                                         reasons for any discrepancies we identified between the SAR O&S costs
                                         and the cost estimates. For those weapon systems that did not include
                                         O&S cost estimates in the 2010 SARs, we contacted the individual
                                         program offices to obtain information about the reasons why these costs
                                         were not included. These programs are listed in table 2 of the report.

                                         We conducted this performance audit from February 2011 to February
                                         2012 in accordance with generally accepted government auditing
                                         standards. Those standards require that we plan and perform the audit to
                                         obtain sufficient, appropriate evidence to provide a reasonable basis for
                                         our findings and conclusions based on our audit objectives. We believe
                                         that the evidence obtained provides a reasonable basis for our findings
                                         and conclusions based on our audit objectives.




                                         Page 35                                           GAO-12-340 Defense Logistics
Appendix II: DOD’s Guidance Implementing
             Appendix II: DOD’s Guidance Implementing the
             Statutory SAR Requirements



the Statutory SAR Requirements

             This appendix provides additional information on DOD’s guidance that
             implements the statutory SAR requirements in 10 U.S.C. § 2432.

             DOD has issued various guidance documents that implement the
             statutory SAR requirements. DOD Instruction 5000.02, 1 which addresses
             the operation of the defense acquisition system, includes guidance on
             SARs that is similar to the basic statutory requirements in 10 U.S.C. §
             2432. The guidance, for example, states that SARs should be submitted
             at program initiation (normally milestone B except for some ship
             programs) or at the time that the program is designated as a major
             defense acquisition program. It reiterates that programs shall report
             annually, with the exception of quarterly reports that are required when
             acquisition costs increase or schedules slip. Further, the instruction
             requires the submission of quarterly SARs after the program rebaselining
             that occurs after a major milestone decision (i.e., milestone C or
             milestones B and C for some ship programs).

             Another source of guidance on SAR reporting is the Defense Acquisition
             Guidebook, 2 which describes discretionary best practices for acquisition
             professionals to consider while meeting various reporting requirements
             throughout the acquisition process. The guidebook contains a section
             summarizing the statutory requirements for SAR content and submission
             and reiterates that a full life-cycle analysis of costs should be reported for
             programs, including each evolutionary increment, as available, and for
             antecedent programs, if applicable.

             DOD’s SAR policy, 3 issued in draft in 2006 and never finalized, further
             identifies information that should be included in the O&S cost section of
             the SAR. According to the draft policy, programs that have reached
             milestone B should report the most recent estimate of O&S costs, such as
             those developed for service cost reviews and acquisition milestone
             decisions, and should include both the source and date of this estimate in
             the SAR. Average annual O&S costs should be reported in constant,
             base-year dollars in a unit of measure (for example, cost per aircraft,
             squadron, or wing) determined by the service. The draft policy further


             1
                 DOD Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008).
             2
                 DOD, Defense Acquisition Guidebook (2011).
             3
              OSD, Selected Acquisition Report (draft) (2006). Although issued in draft, the policy was
             never issued in final form




             Page 36                                                      GAO-12-340 Defense Logistics
Appendix II: DOD’s Guidance Implementing the
Statutory SAR Requirements




states that assumptions underlying the estimate should be included.
Operating tempo, expected reliability and maintainability of the system,
maintenance concept, and manning and logistics policies are provided as
examples of the estimate assumptions that should be included in the
SAR. Finally, the draft policy states that programs should report the total
estimated O&S costs, and estimate assumptions, for an antecedent
system if one has been identified and these costs were previously
developed for that system.

Each year the Under Secretary of Defense for Acquisition, Technology
and Logistics issues a memorandum to the military services that provides
guidance for preparing the annual SARs, including instructions for
programs that have reached milestone B and are required to report O&S
costs. 4 For fiscal years 2007, 2009, and 2010, this annual guidance
states that programs should report total estimated O&S costs in both
constant and then-year dollars, and that the assumptions that formed the
basis of the estimate and the date of the estimate should be included.
Further, programs should report an average unit of measure (e.g.,
average annual cost per squadron, average annual cost per system) for
the O&S costs of both the current weapon system and the antecedent
system in constant dollars. If there is no antecedent system, this should
be stated in the narrative of the O&S cost section. If there is an
antecedent system but the data are not currently available, the
antecedent system should be identified in the narrative, along with a
statement that the required data are not available (e.g., “the O&S costs
for the antecedent system are not currently available, but will be provided
in the next SAR”). Finally, programs should explain in the narrative how
the average annual costs were calculated using the estimated O&S cost
total.

Finally, DOD has developed instructions 5 and training for entering SAR
data into the Defense Acquisition Management Information Retrieval
system. DOD instructions for entering SAR data into the system generally
align with the 2006 draft policy. The Defense Acquisition University offers


4
 At the time we conducted our review, the most recent annual guidance was issued in
January 2011. Under Secretary of Defense for Acquisition, Technology and Logistics,
Memorandum for Assistant Secretaries of the Military Services, December 2010 Selected
Acquisition Reports (SARs) Guidance (Jan. 14, 2011).
5
  Under Secretary of Defense for Acquisition, Technology and Logistics, SAR Data Entry
Instructions (draft) (Nov. 5, 2010).




Page 37                                                   GAO-12-340 Defense Logistics
Appendix II: DOD’s Guidance Implementing the
Statutory SAR Requirements




SAR training on using this system. According to Office of the Secretary of
Defense (OSD) officials, the primary class, Acquisition Reporting for
Major Defense Acquisition Programs and Major Automated Information
Systems, is usually offered in January and October. During the 4-day
class, participants receive step-by-step instruction on report preparation
using the system’s web application. The training materials include basic
SAR O&S cost reporting information. For example, estimate assumptions
should be reported, calculation of average costs from total O&S costs
should be provided, and costs should always be updated at major
acquisition milestones. The training materials reiterate that costs should
include both direct and indirect costs, regardless of funding source or
management control.




Page 38                                           GAO-12-340 Defense Logistics
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Defense



Department of Defense




              Page 39                                      GAO-12-340 Defense Logistics
Appendix III: Comments from the Department
of Defense




Page 40                                      GAO-12-340 Defense Logistics
Appendix III: Comments from the Department
of Defense




Page 41                                      GAO-12-340 Defense Logistics
Appendix III: Comments from the Department
of Defense




Page 42                                      GAO-12-340 Defense Logistics
Appendix IV: GAO Contact and Staff
                  Appendix IV: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Cary B. Russell, (404) 679-1808 or russellc@gao.gov
GAO Contact
                  In addition to the contact name above, the following staff members made
Staff             key contributions to this report: Tom Gosling, Assistant Director; Kristine
Acknowledgments   Hassinger; Susannah Hawthorne; Charles Perdue; Janine Prybyla;
                  William M. Solis; and Erik Wilkins-McKee.




(351582)
                  Page 43                                            GAO-12-340 Defense Logistics
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.