oversight

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs

Published by the Government Accountability Office on 2003-02-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Subcommittee on
                Readiness and Management Support,
                Committee on Armed Services,
                U.S. Senate

February 2003
                BEST PRACTICES

                Setting Requirements
                Differently Could
                Reduce Weapon
                Systems’ Total
                Ownership Costs




GAO-03-57
                                               February 2003


                                               BEST PRACTICES

                                               Setting Requirements Differently
Highlights of GAO-03-57, a report to the       Could Reduce Weapon Systems’
Subcommittee on Readiness and
Management Support, Committee on               Total Ownership Costs
Armed Services, U.S. Senate




For fiscal year 2003, the                      Even though DOD has implemented several initiatives to reduce total
Department of Defense (DOD)                    ownership costs, some systems, such as the Apache helicopter or the
asked for about $185 billion to                Abrams tank, have experienced costly maintenance problems and
develop, procure, operate, and                 low readiness rates, which persisted even after the systems were fielded.
maintain its weapon systems. This              We found several reasons for these problems. First, DOD based
request represents an increase of
18 percent since 2001 for the total
                                               requirements for weapon systems in product development almost
ownership costs of DOD weapon                  exclusively on technical performance, with little attention to operating
systems. Often, DOD systems need               and support costs and readiness at the beginning of development when
expensive spare parts and support              there is the greatest chance of affecting those costs positively. Second,
systems after they are fielded to              using immature technologies to meet performance goals weakened DOD’s
meet required readiness levels.                ability to design weapon systems with high reliability. Finally, DOD’s
DOD has been increasingly                      organizational structure is linear and limits collaboration and feedback
concerned that the high cost of                among organizations charged with requirements setting, product
maintaining systems has limited its            development, and maintenance.
ability to modernize and invest in
new weapons. This report                       In contrast, commercial companies that we visited considered operating
examines the best practices of
leading commercial firms to
                                               and support costs to be integral to their new product development decisions.
manage a product’s total ownership             Studies have shown that by the time a product is ready for development,
costs and determines if those                  over 90 percent of the operating and support costs have been determined.
practices can be applied to DOD.               As a result, these companies required their equipment be easy to maintain,
                                               ready when needed, and reliable at a low cost. These requirements were of
                                               equal importance to other performance characteristics. After setting
                                               requirements, product developers then designed products to meet
GAO recommends DOD (1) revise                  established reliability rates, using technologies that were proven through
its guidance for setting                       past use or testing. At all of the companies we visited, customers and
requirements to include total                  product developers alike, had very collaborative processes and practices
ownership cost goals and readiness             that draw extensively on data from past operations to influence the design
rates for any major weapons
                                               of new products.
system as performance parameters
equal to any others; (2) revise
acquisition regulations to require a           Percent of Life Cycle Costs Determined at Various Points in the Acquisition Process
firm estimate of component and
subsystem reliability by the
systems integration phase and an
estimate of system reliability at
the production decision; and
(3) structure contracts to ensure
proper trade-offs between
reliability and performance.




www.gao.gov/cgi-bin/getrpt?GAO-03-57.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Katherine
Schinasi at (202) 512-4841.
Contents


Letter                                                                                        1


Executive Summary                                                                             3
                    Purpose                                                                  3
                    Background                                                               3
                    Results in Brief                                                         4
                    Principal Findings                                                       7
                    Recommendations for Executive Action                                    12
                    Agency Comments                                                         12

Chapter 1           Introduction                                                            13
                    Total Ownership Cost Is the Cost to Ensure Readiness                    13
                    Commercial Best Practices                                               15
                    Objectives, Scope, and Methodology                                      16

Chapter 2           DOD’S Requirements-Setting and Development
                    Practices Yield Higher Total Ownership Costs                            22
                    DOD’s Weapon System Programs Encounter Cost Growth in
                      Achieving Readiness Rates                                             22
                    DOD’s Linear Acquisition Approach Makes It Difficult to Control
                      Operations and Support Costs                                          24

Chapter 3           Commercial Companies Deliberately Manage
                    Ownership Costs through Product Requirements
                    and Design                                                              35
                    A Best Practices Model                                                  35
                    Leading Companies Treat Readiness and Operating and Support
                      Cost as Critical Product Requirements                                 37
                    Knowledge-Based Product Development Is Critical to Achieving
                      Desired Reliability and Managing Operating and Support Costs          43
                    Leading Commercial Firms Use Feedback from Operations to
                      Better Understand Customer Needs, Product Deficiencies, and
                      Operating and Support Costs                                           46




                    Page i                                             GAO-03-57 Best Practices
Chapter 4              Stressing Operating and Support Cost at the Outset
                       of an Acquisition Could Help DOD Reduce Total
                       Ownership Costs                                                          49
                       Differences in Practices Explain Different Outcomes for
                         Commercial Companies and DOD in Controlling Total
                         Ownership Costs                                                        50
                       Several DOD Efforts Underway to Reduce Total Ownership Costs             51
                       DOD’s Current Environment Does Not Provide Incentives to
                         Reduce Total Ownership Cost Early                                      59

Chapter 5              Conclusions and Recommendations                                          61
                       Recommendations for Executive Action                                     62
                       Agency Comments and Our Response                                         63

Appendix I             Comments from the Department of Defense                                  66



Appendix II            GAO Staff Acknowledgments                                                69



Related GAO Products                                                                            70



Tables
                       Table 1: Readiness and Operating and Support Costs for Selected
                                Weapons                                                         24
                       Table 2: DOD and Commercial Practices for Controlling Operating
                                and Support Costs                                               50


Figures
                       Figure 1: Nominal Life-Cycle Cost of Typical DOD Acquisition
                                Program with a 30-Year Service Life                             14
                       Figure 2: Percent of Operating and Support Costs Determined at
                                Various Points in the Acquisition Process                       15
                       Figure 3: Readiness, Reliability, and Operating and Support Costs        23
                       Figure 4: DOD’s Linear Acquisition Process                               26



                       Page ii                                             GAO-03-57 Best Practices
Figure 5: System Readiness Comes at High Operating and Support
         Costs When Reliability Is Not Ensured                                            27
Figure 6: Apache Helicopter                                                               30
Figure 7: Abrams Tank                                                                     32
Figure 8: Commercial Model for Reducing Operating and Support
         Costs                                                                            36
Figure 9: Benefits of Ensuring High Reliability Rates During
         Product Development                                                              37
Figure 10: Polar Tanker’s Polar Endeavor                                                  38
Figure 11: United Airlines/Boeing 777                                                     40
Figure 12: FedEx Express Delivery Van                                                     42
Figure 13: Joint Strike Fighter                                                           53
Figure 14: Advanced Amphibious Assault Vehicle                                            55




This is a work of the U.S. Government and is not subject to copyright protection in
the United States. It may be reproduced and distributed in its entirety without further
permission from GAO. It may contain copyrighted graphics, images or other materials.
Permission from the copyright holder may be necessary should you wish to reproduce
copyrighted materials separately from GAO’s product.




Page iii                                                        GAO-03-57 Best Practices
United States General Accounting Office
Washington, DC 20548




                                   February 11, 2003

                                   The Honorable John Ensign
                                   Chairman
                                   The Honorable Daniel Akaka
                                   Ranking Minority Member
                                   Subcommittee on Readiness and Management Support
                                   Committee on Armed Services
                                   United States Senate

                                   As you requested, this report examines how best practices offer
                                   improvements to the way the Department of Defense develops new
                                   weapon systems to reduce their total ownership cost, especially the
                                   operating and support costs, during design. It examines how the
                                   department currently designs for operating and support costs and how
                                   best practices could improve outcomes and reduce costs. We make
                                   recommendations to the Secretary of Defense for improvements to
                                   weapon system requirements policy, including establishing operating and
                                   support cost and readiness goals as performance parameters equal to any
                                   other performance parameters; revising acquisition policies to require a
                                   firm estimate of reliability during product development; and providing
                                   contract incentives for product developers to make appropriate trades
                                   between reliability and performance before production.

                                   We are sending copies of this report to the Secretary of Defense; the
                                   Secretary of the Army; the Secretary of the Navy; the Secretary of the Air
                                   Force; the Director of the Office of Management and Budget; the Director,
                                   Missile Defense Agency; and interested congressional committees. We will
                                   also make copies available to others upon request. In addition, the report
                                   will be available at no charge on the GAO Web site at http://www.gao.gov.




                                   Page 1                                              GAO-03-57 Best Practices
If you have any questions regarding this report, please call me at
(202) 512-4841. Other contacts are listed in appendix II.




Katherine V. Schinasi
Director
Acquisition and Sourcing Management




Page 2                                                GAO-03-57 Best Practices
             Executive Summary
Executive Summary


             For fiscal year 2003, the Department of Defense (DOD) asked for about
Purpose      $184.9 billion to develop, procure, operate, and maintain weapon systems.
             With this funding, DOD will have received increases totaling about
             18 percent since 2001 for what it defines as total ownership costs of its
             equipment. DOD’s budget for operations and maintenance increased by
             about 5.6 percent during the same period—from about $59 billion to about
             $62.3 billion. DOD has been increasingly concerned that the high cost of
             maintaining weapon systems to meet required readiness levels is depleting
             modernization accounts and denying the department the flexibility to
             invest in new weapons. In recognition of this concern, DOD has
             established goals to reduce operating and support costs of weapon
             systems already in the field as well as those currently in development. In
             order to provide another perspective on this problem, GAO has continued
             a body of work to identify best practices used by leading commercial
             companies to manage new products’ total ownership costs to see if these
             practices can be applied to DOD’s weapon system acquisitions.

             This report addresses how DOD can use best practices from commercial
             companies during its acquisition process to reduce total ownership costs
             of its major weapon systems. It presents a model of the process
             commercial companies use to incorporate reasonable and manageable
             operating and support cost into their product development process. In
             response to a request from the Chairman and Ranking Minority Member,
             Subcommittee on Readiness and Management Support, Senate Committee
             on Armed Services, GAO (1) determined the practices, processes and
             metrics DOD has historically used to manage and control operating and
             support costs of its major weapon systems; (2) determined the practices,
             processes and metrics commercial companies use to manage and control
             operating and support costs; and (3) analyzed the extent to which
             opportunities exist to apply best practices to reduce operating and support
             costs during product development.


             Readiness is a critical parameter of all DOD’s weapon systems. If a system
Background   is not ready, its performance characteristics are of no use. Each weapon
             system has an expected readiness rate, usually expressed in some
             percentage of available units, that it is expected to maintain for our
             national security. Readiness can be achieved by building highly reliable
             weapon systems or, if the systems are not highly reliable, supporting them
             with an extensive logistics system that can ensure spare parts and other
             support items are available when needed. In essence, the cost of a
             product’s readiness is the cost to develop, produce, and operate and
             maintain that system.


             Page 3                                              GAO-03-57 Best Practices
                   Executive Summary




                   DOD recognizes that the total cost of a weapon system includes more than
                   development and procurement costs. Traditionally, development and
                   procurement have accounted for about 28 percent of a weapon’s total
                   ownership cost, while costs to operate, maintain, and dispose of the
                   weapon system account for about 72 percent of the total. For a number of
                   years, the department’s goal has been to spend less on supporting systems
                   and to devote more funds to development and procurement in order to
                   modernize weapon systems. But, in fact, growth in operating and support
                   costs has limited the department’s buying power. DOD officials have cited
                   shortages of spare parts and unreliable equipment as reasons for low
                   mission-capable rates for some weapons. As a result, some modernization
                   has been postponed in order to pay high and unexpected operating and
                   maintenance costs.

                   GAO has issued a series of reports on best practices that commercial firms
                   use to manage and control the acquisition cost of their products.
                   Commercial firms attain knowledge early in the development process
                   about technology that they plan to incorporate; they make sure the design
                   is mature before production; and they have production processes under
                   control before production begins. The building of knowledge in these
                   areas can also contribute to the reduction of a product’s operating costs
                   over its life cycle, thereby reducing a major portion of its total ownership
                   costs. While those reports focused on best practices for reducing the cost
                   and cycle time for acquiring weapon systems, this report will focus on
                   what DOD can do during the acquisition of a weapon system, prior to and
                   during product development, to ensure that products are available when
                   needed, more cost effective to operate and support and more reliable once
                   fielded, thereby reducing their total ownership cost.


                   DOD weapons acquisition processes do not consider operations and
Results in Brief   support costs and readiness as key performance requirements for new
                   weapon systems, and DOD places less emphasis on establishing
                   operations and support cost and readiness as key nontradable goals early
                   in product development. Generally, the department settles for lower
                   reliability in its new weapon systems’ designs. In our review of data for
                   five fielded weapon systems, we found none had established operating and
                   support cost or readiness as key requirements. Although recent readiness
                   levels were acceptable to the services for the most part, the five systems
                   had experienced growth in their operating and support cost estimates of
                   between 16 and 48 percent within the last 12 years and problems with
                   reliability once in the field.



                   Page 4                                               GAO-03-57 Best Practices
Executive Summary




We found several reasons for cost growth. DOD’s acquisition process is
linear, progressing from one organization to the next with little interaction
among the groups. Requirements-setting by the war-fighting community
focuses on system performance. DOD policy does not require inclusion of
readiness or operating and support cost goals as key performance
requirements equal in importance to other performance requirements.
None of the systems we reviewed had a readiness or an operating and
support cost goal as a key requirement. Further, during product
development, the use of immature technologies and components to meet
performance goals worked against designing weapon systems with high
reliability. Using immature technologies also acts as a barrier to
manufacturing techniques such as open systems1 or designing for fewer
parts, practices that typically help reduce maintenance costs of the system
and increase its reliability. DOD’s systems for accumulating data to
analyze operating and maintenance actions on weapon systems already in
the field do not provide adequate or reliable information, thus making it
difficult for DOD to understand the total cost of operations and support.
The outcome of these practices at DOD has been an inability to stem the
continuous growth in total ownership cost, with actual operating costs
continuing to exceed initial estimates. As a result, DOD continues to
request more operating and support funding to sustain its systems or to
reprogram funds from other accounts to pay the bills.

In contrast, commercial companies consider reasonable operating and
support costs and the readiness or availability of their equipment as
requirements equal in importance to other performance characteristics,
thereby ensuring that the developer places proper emphasis on achieving
reliability and operating and support cost goals during product
development. Commercial companies have a collaborative process for
setting requirements, developing the product, and collecting and sharing
data on maintaining and supporting the product once it is delivered. Both
the customer and the developer have a voice in the process. During
product development, especially during the design process, the maintainer
has an active voice and is armed with information about operating and
support cost drivers in the previous product. Commercial product
developers maintain high standards for reliability, using proven
technologies to achieve critical performance requirements. They find an



1
 An open system is one that is designed with interfaces to accept upgrades easily without
redesign of the total unit. Replacements in an open system only have to meet interface
requirements to be accepted.




Page 5                                                          GAO-03-57 Best Practices
Executive Summary




evolutionary development process is critical to reducing operating and
support costs and achieving high readiness. Emphasis is placed on
reducing the number of parts in a design so there is less to maintain, using
standardized parts that are readily available in the industrial base and
using open systems to maintain competition. Once the product is delivered
to the customer, maintainers keep detailed records on its reliability and
the cost of its maintenance and support. Importantly, information on the
product’s performance is communicated back to the developer to be used
to improve the product.

DOD has implemented initiatives to reduce the total ownership cost of
its weapon systems. It has modified acquisition policies, established
programs to reduce operating and maintenance costs in existing systems,
and selected several acquisition programs to test different approaches to
reduce life-cycle costs during development. However, these steps do not
incorporate many of the practices used by commercial companies during
requirements determination, product development, and fielding. In
comparing DOD’s practices to those found at leading commercial firms,
we have identified several differences. Because companies operate in an
environment where operating costs and readiness are critical to their
survival, commercial customers establish low operating and support cost
and high readiness requirements when purchasing a product. This forces
product developers to design reliable systems that are easy and relatively
cheap to operate and maintain. The collaborative relationship between the
customer and the product developer is essential to driving down operating
and support costs. Further, companies understand that accurate operating
and support cost data from current products are also necessary to make
good decisions related to the purchase of a product, facilitate
cost/performance trade-offs, and provide feedback to the manufacturer for
continuous improvement.

GAO is making recommendations to the Secretary of Defense on ways to
improve DOD’s management of operating and support costs. We are
recommending that DOD revise its requirements generation process to
include total ownership cost, especially operating and support cost, and
weapon system readiness rates as performance parameters equal to any
others. We also recommend that any revision of the current policy
governing acquisition processes require a firm estimate of the systems’
reliability based on demonstrated reliability rates at component and
subsystem levels no later than the end of the system integration phase and
at the system level no later than the production decision. Finally, we
recommend that DOD structure contracts by Milestone B, the start of the
system development and demonstration phase, to ensure that proper


Page 6                                               GAO-03-57 Best Practices
                           Executive Summary




                           trades are made between reliability and performance before the
                           production decision.



Principal Findings
DOD’S Current Practices    DOD is spending more on operating and support costs for its weapon
for Setting Requirements   systems than it planned. We found three primary reasons for the high cost
and Developing New         of operating and supporting DOD’s fielded weapon systems. These were
                           (1) little or no attention to the trade-offs between readiness goals and the
Weapon Systems Continue    cost of achieving them when setting the key parameters for weapon
to Yield Higher Total      systems;2 (2) the use of immature technologies during product
Ownership Costs            development and delays in acquiring knowledge about the design and its
                           reliability until late in development, or in some cases, production; and
                           (3) insufficient data on the operations and maintenance costs and actions
                           for fielded systems that would allow improvements in products currently
                           in development. The outcome of these practices in DOD has been an
                           inability to stem continuous growth in total ownership cost, with actual
                           operating costs continuing to exceed initial estimates. As a result, DOD
                           continues to request more operating and support funding to sustain its
                           systems or reprogram funds from other accounts to pay the bills.

                           Even though operating and support costs are the largest factor in a
                           weapon system’s total ownership cost, they do not receive the same
                           attention when requirements are set for a weapon system as other
                           performance characteristics. In our review of data for five fielded weapon
                           systems, we found that none had established an operating and support
                           cost or a readiness goal as a key requirement prior to product
                           development. In fact, operating and support cost estimates were not
                           available in the Selected Acquisition Reports until at least 5 years into
                           product development on these programs. Most of the fielded systems we
                           reviewed were near or achieving readiness goals, but had experienced
                           significant cost growth in operations and support cost estimates within the
                           last 12 years to do so. Two of these systems, the Apache and the Abrams,
                           were designated as the Army’s most expensive weapons to support. The
                           C-17 reported a cost increase of almost 25 percent, and program officials



                           2
                            A key performance parameter represents a capability that is so significant that failure to
                           meet the minimum value could be a reason for DOD or the services to reevaluate the
                           concept or system or terminate the program.




                           Page 7                                                           GAO-03-57 Best Practices
                           Executive Summary




                           stated that they would not have a firm estimate of operating and support
                           cost until 2010—more than 25 years after the start of development.

                           We found practices in three areas—requirements-setting, product
                           development, and operations and maintenance—that contributed to this
                           condition. DOD’ s acquisition process is linear and serially involves several
                           organizations whose responsibilities in the process have differing
                           objectives. Communication among the different organizations is
                           fragmented. Requirements focused on the weapon system’s performance
                           characteristics. Once the weapon system’s requirements were set and the
                           development of the system began, product development focused on
                           achieving the program’s acquisition cost, schedule and performance goals,
                           rather than on increasing reliability in order to reduce its total life-cycle
                           costs. We found that the maintainers had limited involvement in making
                           design trades for lower operating and support costs during development
                           and that best practices such as designing for open systems or ease of
                           maintenance were not used by the developer. We also found that once a
                           system is fielded, the services’ systems for tracking operating and support
                           costs were suspect, providing inadequate feedback to suppliers and
                           requirements setters.


Commercial Companies       We found that commercial companies that use capital equipment
Deliberately Manage        considered operating and support costs integral to their new product
Ownership Costs through    development decisions. Companies such as United Airlines, FedEx
                           Express, and Polar Tanker employ practices to maintain the readiness of
Product Requirements and   their fleets at as low an operating cost as possible. Reducing these costs
Design Process             translates into revenues, profits, and market growth. Increases to these
                           costs can mean market failure. As customers, they have established
                           operating and support costs and product readiness as key system
                           requirements before development begins for a new product that are equal
                           in importance to requirements for its performance and the cost to develop
                           and produce the product. For example, United Airlines requires that new
                           aircraft maintain a readiness rate of 98.5 percent or the manufacturer must
                           reimburse it for lost revenue. Polar Tanker established a requirement that
                           its Endeavor Class tanker operate at least 330 days a year at a reduced
                           operating cost per tanker. These dual requirements drove trades during
                           design, sometimes increasing development costs to achieve lower
                           operating costs. Before FedEx Express agreed to a new design for
                           its delivery trucks, it required that the new design last for at least
                           300,000 miles over a specified number of years and at a specific cost per
                           mile. In gaining agreement with product developers on these requirements
                           prior to product development, these companies sometimes had to trade


                           Page 8                                                GAO-03-57 Best Practices
Executive Summary




performance or spend more in development, but they received more
reliable products, reduced total ownership costs, and made those costs
more predictable.

To meet their customers’ supportability requirements, we found that
commercial product developers focused on designing a product that was
easy to maintain, would be ready when needed, and reliable at low cost.
They used an evolutionary development process. Consequently, they did
not allow components or subsystems into a product’s design unless the
technology had been proven reliable through past use or testing. For
example, Boeing told us that it defers use of immature technology to later
evolutions of design and makes the reliability ratings of its components
available to the airlines before it begins product development. Maytag
completed reliability testing on every new product prior to going to
production. These companies also emphasized product designs with fewer
parts and open systems. Maytag has established a parts reduction program
as part of its development process, and Boeing built its 777 so that any
of three engines—GE, Pratt & Whitney, or Rolls Royce—would fit.
Developers also gained insight into design features their customers valued
through regular communication with them. For example, the design for
Boeing’s latest generation 737—geared toward reducing operating costs—
was inspired by the airlines. Boeing emphasized open systems,
standardized parts, and reduced parts from one generation to the next to
satisfy the airlines’ need for reduced operating costs.

All of the companies we visited, customers and product developers alike,
had very collaborative processes and practices for drawing extensively on
data from past operations to influence the design of new products. This
information was used as a baseline for new product designs and was used
to estimate the operating costs of new products. United Airlines officials
told us that the airlines and the manufacturers both keep meticulous
reliability and cost records at all levels of an airplane--components,
subsystems, and at the system level. Major operating costs drivers are
tracked on a daily basis by the airlines, and the manufacturers usually
have personnel residing with the airlines’ maintenance crew to help solve
problems on the spot and, just as importantly, to feed information back to
the manufacturer so that the next product can be improved. FedEx
Express and Polar Tanker both emphasized extensive data collection
from current operations. In fact, FedEx Express sets annual targets for
operating and support cost reductions based on data gathered on the road.
Polar Tanker gathered maintenance data from past operations and
established a team made up of its own maintenance personnel and outside



Page 9                                              GAO-03-57 Best Practices
                         Executive Summary




                         consultants to determine areas that could result in higher reliability and
                         lower maintenance costs in designing the new Endeavor Class tanker.


Greater Emphasis on      DOD and the commercial companies we visited have policy goals of
Operating and Support    developing products that will meet customers’ needs at the lowest possible
Cost at the Outset and   cost to build and operate. The difference between them is in how each
                         implements its policies. Leading commercial companies follow an
during an Acquisition    integrated, collaborative process of setting requirements, developing the
Program Could Help DOD   product, and ensuring that the product can be supported at an acceptable
Reduce Total Ownership   cost. DOD’s process is composed of disparate practices carried out by
Costs                    separate organizations with differing objectives and little communication
                         among them about how to support fielded systems. While commercial
                         firms focus on total ownership costs at the outset, DOD focuses mostly on
                         technical performance. One cause of this is that in DOD the accountability
                         and responsibility for total ownership costs are spread across many
                         organizations with separate goals. Another cause lies in motivation for
                         low costs. The commercial companies we visited are driven by the need
                         to be as profitable as possible to survive, and low total ownership costs
                         translate to higher profitability. DOD’s environment does not provide such
                         incentives, and the organizations charged with acquiring and operating
                         weapon systems are unconstrained by a need to lower costs since they
                         can request additional operations and maintenance funding to keep
                         systems working.

                         Some of the practices used by commercial companies to reduce a new
                         product’s operating costs during its development may be helpful to DOD.
                         In setting requirements, commercial customers make readiness and
                         operating cost requirements and collaborate directly with the product
                         developer. Product developers establish sound cost estimates early;
                         designs are simplified by using open systems and reducing parts; reliability
                         testing is done early; and a reliability growth curve is established before
                         production begins. Once a product is fielded, operating costs are managed
                         to established targets; operating cost data is collected, analyzed, and used
                         by the developer and the customer to develop more reliable products in
                         the future; and continuous improvements are made to future products.
                         The commercial practice of establishing readiness and operating cost as
                         key requirements for a new product necessitates substantive input from
                         operators and developers before and during product development. The
                         commercial practices used during product development to design reliable
                         systems that are easy and less costly to operate and maintain depend on
                         the use of good product development practices including the use of
                         mature technologies to meet requirements. Commercial firms use


                         Page 10                                               GAO-03-57 Best Practices
Executive Summary




incremental product development processes and depend on a strong
relationship between the manufacturer and the customer’s operators and
maintainers to continue throughout product development.

DOD does not focus on operating and support costs to the degree
commercial companies do. In setting requirements, DOD does not make
readiness and operating cost key parameters, performance is rarely
reduced in favor of reliability or reduced operating cost, and there is no
direct relationship between the requirement setters and the product
developer. During product development, firm estimates of operating costs
are not required, little attention is paid to reliability rates, and open
systems or design for manufacturing techniques are rarely used. Once a
weapon system is fielded, there is a lack of complete and reliable data
available from the field, and there is little collaboration between
maintainers and product developers to improve new systems. DOD’s
acquisitions usually begin with critical technologies that are immature,
with unproven reliability. This makes it difficult to implement best
practices such as design for manufacturing during product development.
Accurate operating and support cost data are not available for helping
management make good decisions, facilitating cost/performance trade-off
decisions, and providing feedback to the manufacturer for continuous
improvement. On the weapon systems we reviewed, we found that the
programs had poor initial estimates of operating and support costs for
weapon systems, partly because they do not have reliable systems in place
to track those costs per weapon system.

DOD has taken some steps to lower its weapon systems’ total ownership
costs. Those actions include concurring with and implementing
recommendations concerning the use of technology readiness levels,
indicators of design maturity, and controlled production processes.
Further, the department initiated pilot programs with 30 acquisition
programs to develop methods for reducing total ownership costs.
However, DOD’s current environment—both culturally and
organizationally—is not presently conducive to applying them. Currently,
its acquisition policies do not provide specific guidance for controlling
total ownership cost and its requirements-generation policies provide no
guidance for establishing readiness or cost goals for weapon systems once
they are fielded.




Page 11                                             GAO-03-57 Best Practices
                      Executive Summary




                      GAO recommends that the Secretary of Defense
Recommendations for
Executive Action      •   revise the Chairman of the Joint Chiefs of Staff Instruction 3170.01B on
                          the requirements generation process to include total ownership cost,
                          especially operating and support cost, and weapon system readiness
                          rates as performance parameters equal in priority to any other
                          performance parameters for any major weapon system prior to
                          beginning the acquisition program;
                      •   revise the current policy governing the operation of the defense
                          acquisition system (currently under revision) to require that the
                          product developer establish a firm estimate of a weapon system’s
                          reliability based on demonstrated reliability rates at the component and
                          subsystem level no later than the end of the system integration phase,
                          coinciding with the system-level critical design review, before
                          proceeding into the system demonstration phase of product
                          development; and at the system level no later than the full-rate
                          production decision; and
                      •   structure DOD contracts for major systems acquisitions so that at
                          Milestone B the product developer has incentives to ensure that proper
                          trades are made between reliability and performance prior to the
                          production decision. One option is to provide specific clauses in the
                          development contract to address reliability growth.


                      DOD partially concurred with all of our recommendations but, for the
Agency Comments       most part, found no further action was needed to lower total ownership
                      cost. We disagree. We believe that if DOD takes no further action in
                      implementing these recommendations, it would ignore significant
                      opportunities to improve readiness by lowering total ownership cost in a
                      budget environment that demands more effort to reduce these costs.

                      A detailed discussion of DOD’s comments appears in Chapter 5 and the
                      full text of DOD’s comments is in appendix I.




                      Page 12                                              GAO-03-57 Best Practices
                        Chapter 1: Introduction
Chapter 1: Introduction


                        For fiscal year 2003, DOD asked for $184.9 billion to fund research and
                        development, procurement and direct operations and maintenance costs
                        of its weapon systems. These elements along with disposal costs are
                        defined as the total ownership cost of a weapon system. The budget has
                        increased by about 18 percent for these activities since 2001—with direct
                        cost for operations and maintenance of weapon systems increasing by
                        about 5.6 percent from $59 billion to about $62.3 billion. Since the late
                        1990’s, DOD has been increasingly concerned that the cost of operating
                        and supporting weapon systems to meet required readiness levels is
                        depleting its modernization accounts and denying the department the
                        flexibility to invest in new weapons.


                        Commercial companies and DOD both use readiness as a key indicator of
Total Ownership Cost    a product’s success. Readiness is usually expressed as the percentage of
Is the Cost to Ensure   total units available and capable of performing a mission at any given time.
                        If a weapon system is not ready when it is needed, its performance
Readiness               characteristics are of no use. In general, readiness can be achieved either
                        by building highly reliable weapon systems or, if the systems are not highly
                        reliable, supporting them with an extensive logistics system that can
                        ensure spare parts and other support items are available when needed. In
                        essence, the cost to ensure a product’s readiness is the cost to develop,
                        produce, operate, and maintain that product through its life cycle.

                        The development and production cost of a weapon system, also known
                        as acquisition cost, usually represents about 28 percent of the weapon
                        system’s total ownership costs. The acquisition cost is funded through
                        DOD’s research, development, test and evaluation, and procurement
                        accounts. These funds are used to mature new technology and design and
                        manufacture new weapon systems. Operating and support costs1 are
                        typically the highest portion of a weapon system’s total ownership cost
                        because they represent the cost to operate the system and keep it ready
                        for action over many years, sometimes more than 30 years. These costs are
                        about 72 percent of the total ownership cost of a weapon system and are
                        funded through DOD’s operations and maintenance account. Operating
                        and support costs reflect the purchases of fuel, lubricants, and repair parts


                        1
                         Operations and support of weapons systems is a part of the Operations and Maintenance
                        budget, which also includes amounts for health care, base and facilities support, and other
                        activities for the well-being and operations of the military forces. Costs for operations and
                        support of weapon systems were about 48 percent of the Operations and Maintenance
                        budget in fiscal year 2002.




                        Page 13                                                           GAO-03-57 Best Practices
Chapter 1: Introduction




and their associated maintenance as well as modification kit procurement
and installation. Figure 1 depicts the typical distribution of total ownership
costs of DOD weapon systems over a 30-year life cycle.

Figure 1: Nominal Life-Cycle Cost of Typical DOD Acquisition Program with a
30-Year Service Life




Figure 1 shows that the greater part of a weapon system’s total ownership
cost is made up of its operating and support cost. While 72 percent of
the life-cycle cost of a weapon system is realized only after it is fielded,
the decisions made during its acquisition—when its performance
requirements are being established and its design is being matured—will
dictate operating and support costs very early. In fact, studies show that
about 85 percent of the operating and support costs of a weapon system
will be determined as soon as requirements are set, while less than
10 percent of the life-cycle cost have been spent. By the time a product is
ready for production, over 90 percent of the operating and support costs
have been determined, and about 28 percent of the total life-cycle costs
have been spent. Figure 2 illustrates this phenomenon.




Page 14                                                  GAO-03-57 Best Practices
                  Chapter 1: Introduction




                  Figure 2: Percent of Operating and Support Costs Determined at Various Points in
                  the Acquisition Process




                  Because so much of the eventual cost to support and maintain a weapon
                  system is decided very early, it makes sense that more attention should be
                  paid to supportability when the product’s requirements are being set and
                  its design is being finalized. World-class commercial companies that either
                  use or develop high-performing products know this and set requirements
                  and designs accordingly.


                  GAO has undertaken an extensive body of work that examines DOD’s
Commercial Best   acquisition issues from a different, more cross-cutting perspective—one
Practices         which draws upon the lessons learned from the best commercial practices
                  to see if they are applicable for DOD’s acquisition processes. Previous
                  GAO best practices reports focused on what DOD could do to control
                  product development costs that represent about 28 percent of total
                  ownership costs. This report will focus on best practices for reducing the
                  largest segment of total ownership costs—the operating and support costs.
                  The concepts discussed build on our previous reports that looked at



                  Page 15                                                  GAO-03-57 Best Practices
                         Chapter 1: Introduction




                         earlier phases of an acquisition, including matching customer needs with
                         available resources and designing and manufacturing products within cost,
                         schedule, and performance goals. A complete list of best practices reports
                         is at the end of this report

                         Leading commercial companies expect to obtain high-quality products that
                         meet their expectations in terms of performance, price, and reliability. To
                         ensure they make prudent buying decisions, they use a structured product
                         development process that ensures a high level of knowledge exists about a
                         product at key junctures during its development. Such a knowledge-based
                         process enables decision makers to be reasonably certain that product
                         quality, reliability, and timeliness are assured.

                     •   Knowledge point 1 occurs when a match is made between the customer’s
                         needs and the available resources—technology, design, time, and funding.
                         Commercial companies use this knowledge to meet essential product
                         requirements, such as low operating and support costs. To ensure that the
                         knowledge is attained, private companies require the product be
                         demonstrated in its intended environment. In addition, the product
                         developer must complete a preliminary product design, using systems
                         engineering to balance customer desires with available resources.

                     •   Knowledge point 2 occurs when the product’s design demonstrates its
                         ability to meet performance requirements. Program officials are confident
                         that the design is stable and will perform acceptably when at least 90
                         percent of engineering drawings are complete. To obtain this knowledge,
                         commercial companies use simulations and testing to fully understand
                         how the product should be built.

                     •   Knowledge point 3 occurs when the product can be manufactured within
                         cost, schedule, and quality targets and is reliable. Important indicators of
                         this are when critical manufacturing processes are in control and
                         consistently producing items within quality standards and tolerances.
                         Private companies demand these critical manufacturing processes be in
                         control because they could affect the product reliability.


                         The Chairman and the Ranking Minority Member, Subcommittee on
Objectives, Scope,       Readiness and Management Support, Senate Committee on Armed
and Methodology          Services, requested that we examine best practices for reducing total
                         ownership cost of DOD’s weapon systems. This report primarily covers
                         the operating and support cost portion of total ownership costs. Our
                         overall objective was to determine whether commercial best practices for
                         reducing total ownership costs, particularly operating and support costs,


                         Page 16                                                GAO-03-57 Best Practices
Chapter 1: Introduction




prior to and during the acquisition of weapon systems offer opportunities
to improve outcomes in DOD’s acquisitions and its efforts to manage and
reduce total ownership costs. Specifically, we (1) determined the
practices, processes, and metrics DOD uses to manage and control total
ownership costs of its major weapon systems; (2) determined the
practices, processes, and metrics commercial companies use to manage
and control total ownership cost; and (3) analyzed the extent to which
opportunities exist for DOD to apply best practices to reduce operating
and maintenance costs during product development.

We used case studies of the following six DOD weapons, chosen to reflect
all of the services across time, to examine DOD’s practices, processes
and metrics:

The Advanced Amphibious Assault Vehicle: This vehicle is the U.S. Marine
Corps’ replacement for its presently fielded amphibious assault vehicle.
The new development vehicle is equipped with a 30mm automatic cannon
and provides the capability to transport a Marine rifle squad at a speed of
20-25 knots in the water, and cross country with the agility and mobility
equal to or greater than that of the M-1 tank. The contract for the Product
Definition/Risk Reduction Phase was awarded in 1996. The Marine Corps
expects to buy 1,013 of these vehicles. Acquisition costs total $9.6 billion;
operating and support costs, $16.0 billion.

The Joint Strike Fighter program: This aircraft is the next-generation
aircraft for the Navy, Air Force, Marine Corps, and Allies. There are three
variants: a carrier variant will provide the Navy a multi-role, stealthy
aircraft to complement the F/A-18E/F. The Air Force variant will be a
multi-role aircraft, but primarily used in an air-to-ground role to replace
the F-16 and the A-10 and complement the F-22. The Marine Corps variant
will be a short-takeoff and vertical-landing aircraft to replace the Sea
Harrier and GR-7 for the United Kingdom Royal Navy and Royal Air Force.
The program is currently in system development and demonstration.
Acquisition costs total $226.5 billion; operating and support costs,
$387.6 billion.

The Landing Platform Dock 17 ship program: These ships are used for
transporting amphibious assault vehicles and other cargo. They
incorporate both a flight deck for helicopters and a well deck to
support landing craft. The contract for the detail design, integration and
construction of the ship was awarded in 1996. The Navy expects to buy
12 ships to replace 27 older amphibious ships. Acquisition costs total
$15.4 billion; operating and support costs, $56.5 billion.


Page 17                                               GAO-03-57 Best Practices
Chapter 1: Introduction




The Apache helicopter program: This Army helicopter’s mission is to find
tanks and other armored targets and destroy them with its laser-guided
Hellfire missiles, its 2.75-inch rockets, or its 30-millimeter gun. Apache
development began in 1973 and the helicopter entered production in 1982
and was fielded in 1984. The Longbow Apache is a remanufactured and
upgraded version of the Apache, which includes improved radar, engine
and Hellfire missiles. The Army currently fields 741 Apache and Longbow
Apache helicopters. Acquisition costs total $ 19 billion; operating and
support costs for the Longbow Apache are $11.1 billion.2

The C-17 cargo aircraft: The C-17 is a multi-engine, cargo aircraft expected
to improve Air Force capability to rapidly deploy, reinforce, and sustain
combat forces worldwide. The C-17 is capable of carrying outsized cargo
over extended distances into unimproved airfields. The C-17 introduces a
direct deployment capability that significantly improves airlift
responsiveness. Development began in 1982 and the aircraft entered
production in 1988 and was fielded in 1993. As of December 2002, the Air
Force fielded 96 of 180 aircraft. Acquisition costs total $58.2 billion;
operating and support costs, $144.9 billion.

The M-1 series Abrams tank program: This tank is the Army’s main battle
tank. The M-1’s development began in November 1972, entered production
in 1979, and was fielded in 1986. The M1A2 is an improved version of the
M-1, with improved armor, digital electronics, and an improved
commander’s weapon station. The Army has fielded over 8,800 M-1 and
M1A1 tanks, with about 1,000 upgraded to the M1A2 versions. Acquisition
costs total $29 billion; operating and support costs, $16 billion.3

For each of the six programs, we interviewed key managers and logistics
representatives to discuss how operating and support costs were being
managed and controlled during design. For the Joint Strike Fighter
program we also visited the prime contractor, Lockheed Martin Aircraft
Company, in Ft. Worth, Texas, and interviewed key designers and
engineers. We analyzed operating and support cost data contained in
Selected Acquisition Reports for the Apache Longbow helicopter, the
C-17 cargo aircraft, the Abrams tank, the Advanced Amphibious Assault
vehicle, Joint Strike Fighter and Landing Platform Dock 17. Information
obtained from the Selected Acquisition Reports was not always consistent


2
    The operating and support cost for the Apache is not available before 1993.
3
    The operating and support cost for the M-1 series Abram is not available before 1993.




Page 18                                                            GAO-03-57 Best Practices
Chapter 1: Introduction




because three of the systems—the Advanced Amphibious Assault Vehicle,
Joint Strike Fighter and Landing Platform Dock 17— are still in
development. We supplemented information for the development systems
with other program cost estimates. We also conducted limited analysis of
the B-1, B-2, and F-22 operating and support cost and readiness data,
based on information provided by their respective program offices and
previous GAO reports.

In addition to the case studies, we reviewed DOD policy, describing the
roles and responsibilities of various organizations in the requirements
development process. We discussed the implementation of these policies,
particularly the role of the logistics community in the requirements
determination process, with officials from each of the six program offices
listed above, as well as appropriate officials from the

Under Secretary of Defense (Acquisition, Technology, and Logistics);
Air Mobility Command, Scott Air Force Base, Ill.;
437th Airlift Wing, Charleston Air Force Base, S.C.;
Office of the Assistant Secretary of the Army, Washington, D.C.; and
U.S. Army Aviation Center, Ft. Rucker, Ala.

To determine the best practices, processes, and metrics commercial
companies use to manage and control operating and support costs, we
used a case study methodology by judgmentally selecting companies
based upon general literature searches and discussions with experts. On
this basis, we identified a number of commercial companies that have a
structured and defined process for managing and controlling their
operating and support costs. The following are descriptions of the six
commercial companies and one quasi-governmental agency we visited:

Boeing Commercial Aircraft designs and manufactures commercial
airplanes. In 2001, it reported revenues totaling $58.2 billion. We visited its
offices in Seattle, Washington, and discussed the development of the 737,
the 767, and the 777 aircraft.

Chicago Transit Authority is a quasi-governmental agency that operates
the nation’s second largest public transportation system and covers the
City of Chicago and 38 surrounding suburbs. In 2002, it reported an
operating budget totaling $915 million. We visited its offices in
Chicago, Illinois, and discussed the requirements determination process
for acquiring buses.




Page 19                                                 GAO-03-57 Best Practices
Chapter 1: Introduction




Allison Transmission, a division of General Motors, designs and
manufactures transmissions for medium and large vehicles as well as
military vehicles. In 2001, General Motors reported sales and revenues
totaling $177.3 billion. We visited its offices in Indianapolis, Indiana, and
discussed the development process for new transmissions.

FedEx Express delivers packages, freight, and information to its
customers worldwide. In 2001, it reported sales and revenues totaling
$15.5 billion. We visited its offices in Memphis, Tennessee and discussed
their process for setting requirements for a new package delivery vehicle.

Maytag designs and manufactures major home appliances. In 2001, it
reported sales and revenues totaling $4.1 billion. We visited its offices in
Newton, Iowa, and discussed the development process for new
appliances, particularly the Neptune washer and the Wide-By-Side
refrigerator.

Polar Tanker, a shipping division of ConocoPhillips Marine, manages the
marine transportation of ConocoPhillips’ Alaska North Slope crude oil
production. In 2001, it reported sales and revenues totaling $26.9 billion.
We visited its offices in Avondale, Louisiana, and discussed the
development process for the new Endeavor Class tanker, the Polar
Endeavor.

United Airlines, a commercial airline division for the UAL Corporation, is a
major commercial air transportation company, engaged in the
transportation of persons, property, and mail throughout the U.S. and
abroad. In 2001, it reported sales and revenues totaling $16.1 billion. We
visited its offices in Seattle, Washington; and San Francisco, California;
and discussed the product development process and maintenance
activities for the Boeing 777 aircraft.

At each of these organizations, we conducted structured interviews with
representatives to gather uniform and consistent information about their
processes, practices, and metrics for controlling operating and support
costs. During these meetings, we obtained a detailed description of the
practices and processes they believe are necessary and vital to control and
reduce operating and support costs. We met with managers of reliability,
maintainability, and new aircraft development; general directors of
operations; controllers; directors of configuration and integration; and
principal engineers. We did not use examples from Chicago Transit or
from Allison Transmissions, but discussions with those firms helped to



Page 20                                                 GAO-03-57 Best Practices
Chapter 1: Introduction




refine the commercial model and inform our analysis of commercial
best practices.

During the past 5 years, we have also gathered information on operating
and support costs from such companies as 3M, Chrysler Corporation,
Caterpillar, Bombardier Aerospace, Ford Motor Company, Hughes Space
and Communications, and Motorola Corporation. This information
enabled us to develop an overall model to describe the general practices,
processes, and metrics leading commercial companies take to control
operating and support costs.

This report highlights several best practices in controlling operating and
support costs based on our fieldwork. As such, they are not intended to
describe all practices or suggest that commercial companies are without
flaws. Representatives from the commercial companies visited told us that
the practices and processes, which we considered best practices, evolved
over many years and that the practices continue to be improved based on
lessons learned and new ideas and information. They admit that the
application and the use of these practices have not always been consistent
or without error. However, they strongly suggested that the probability of
success in controlling operating and support costs is greatly enhanced by
the use of these practices and processes.

We conducted our review between August 2001 and February 2003 in
accordance with generally accepted government auditing standards.




Page 21                                             GAO-03-57 Best Practices
                      Chapter 2: DOD’S Requirements-Setting and
Chapter 2: DOD’S Requirements-Setting and
                      Development Practices Yield Higher Total
                      Ownership Costs


Development Practices Yield Higher Total
Ownership Costs
                      DOD is spending more on operating and support costs for its weapon
                      systems than it planned. We found three primary reasons for the high cost
                      of operating and supporting DOD’s fielded weapon systems. These were
                      (1) little or no attention to the trade-offs between readiness goals and the
                      cost of achieving them when setting the key parameters for weapon
                      systems; (2) the use of immature technologies during product
                      development and delays in acquiring knowledge about the design and its
                      reliability until late in development, or in some cases, production; and
                      (3) insufficient data on the operations and maintenance costs and actions
                      for fielded systems that would allow improvements in products currently
                      in development. The outcome of these practices in DOD has been an
                      inability to stem continuous growth in total ownership cost, with actual
                      operating costs continuing to exceed initial estimates. As a result, DOD
                      continues to request more operating and support funding to sustain its
                      systems or reprogram funds from other accounts to pay the bills.

                      Three key groups are involved in DOD’s process to get a weapon system
                      to the war fighter. First, the war fighter’s service-based requirements
                      community establishes requirements for a new system. Second, the
                      service-based acquisition organizations design and produce a product.
                      Finally, after the product is developed and produced, it is turned over to
                      the war fighter’s operating and maintenance communities, who have the
                      responsibility to operate and maintain it. Decisions made in setting
                      requirements very early in product development have the most impact on
                      the subsequent costs of supporting a system. Trade-offs during the design
                      process can also be significant. In DOD, the focus in the requirements and
                      development process is to establish and meet technical war-fighting
                      performance capabilities, and when trade-offs are made, they are usually
                      to optimize those capabilities. The maintainers often come into this
                      process very late and have little influence. At the end, DOD has no
                      alternative but to pay the operating and support bills that accrue in order
                      to maintain readiness.


                      A weapon system’s operating and support cost will depend to a great
DOD’s Weapon          extent on its performance characteristics, expected readiness rate, and the
System Programs       overall reliability of its design. If a weapon system has a very high
                      expected-readiness rate but its design is not reliable, its operating and
Encounter Cost        support costs will be high and unpredictable. Conversely, if the design has
Growth in Achieving   been thoroughly tested for reliability and is robust, the cost to operate and
                      support it will be lower and more predictable. Ideally, there is a balance
Readiness Rates       that customers and product developers can strike between readiness and
                      operating cost. Figure 3 illustrates this balance.


                      Page 22                                               GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




Figure 3: Readiness, Reliability, and Operating and Support Costs




A product developer can opt to drive higher reliability into the product
during its development by reducing technical requirements, using highly
reliable and proven components, or investing more in early testing. Those
decisions would increase the product’s reliability and consequently
improve prospects for readiness and reduce operating costs across the life
cycle of the product.

We reviewed five weapon system programs currently in the field and
found that most had experienced significant growth in their estimated
operating and support cost. We also found that none of the programs
established goals for readiness or operating and support cost as key
parameters for the weapon system prior to launching the acquisition
program. In addition, we found that once fielded, some systems were not
achieving the readiness rates that program officials thought were possible
during development. Table 1 shows whether systems had specified
readiness and operating and support cost goals as key requirements as
well as the growth in operating and support costs that the systems have
experienced within the last 12 years.




Page 23                                                   GAO-03-57 Best Practices
                                        Chapter 2: DOD’S Requirements-Setting and
                                        Development Practices Yield Higher Total
                                        Ownership Costs




Table 1: Readiness and Operating and Support Costs for Selected Weapons

                  System readiness     Operating and support cost              Readiness        Percentage growth for
 Weapon           as a key parameter   goals as a key parameter            planned/actual   operating and support cost
 Apache           No                   No                                        62/73 %                         48 %
 Abrams           No                   No                                        90/93                            24
 C-17             No                   No                                        92/84                            25
 B-2              No                   No                                        70/42                            35
 B-1              No                   No                                        67/64                            16
Source: DOD (data), GAO (analysis).



                                        While some systems have maintained their expected readiness rates,
                                        they have experienced between 16 and 48 percent growth in estimated
                                        operating and support cost. It is reasonable to conclude that the systems
                                        have not achieved the reliability rates that were needed to meet their
                                        expected readiness goals and, as a result, had to expend more funds on
                                        parts and labor in the field than were planned for maintenance. Two of the
                                        systems, the Apache and the Abrams, were designated as the Army’s most
                                        expensive weapons to support. The C-17 has already experienced a cost
                                        growth of 25 percent in its operating costs, but program officials stated
                                        that they would not have a firm estimate of operating and support costs
                                        until 2010—more than 25 years after the start of development.


                                        Traditionally, DOD has used a linear approach to setting requirements and
DOD’s Linear                            developing a product. It focuses attention during product development on
Acquisition Approach                    achieving revolutionary performance goals while trying to keep acquisition
                                        costs for a program as low as possible. Often, it is not until the system is
Makes It Difficult to                   fielded and responsibility shifts to other agencies or the services that the
Control Operations                      operating and support costs become an overriding concern. By this time,
                                        there is no alternative but to pay the bills that accrue to maintain
and Support Costs                       readiness, no matter the cost.

                                        Three key groups are involved in DOD’s process to get a weapon system to
                                        the war fighter. First, requirements representatives from the war-fighting
                                        community establish a need for a new system, and the combat developers
                                        formulate a set of operating performance requirements to address the
                                        need. Requirements concerning how available the system must be and the
                                        cost to operate and support it are not considered key performance
                                        parameters. Second, DOD establishes an acquisition program office to
                                        begin product development and coordinate design development and
                                        production with the defense contractor. Often, the technologies and
                                        components needed to achieve tough performance requirements are new



                                        Page 24                                                GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




and unreliable; however, the program manager is responsible for
developing and producing the weapon system within certain acquisition
costs and schedule guidelines. Third, after the product is developed and
produced, the war fighter has the responsibility to operate and maintain it.
Although decisions made in setting requirements very early in product
development have the most impact on the cost to support the system, the
personnel who maintain it have less influence on the product development
process because the focus is on achieving difficult performance
requirements.

Figure 4 briefly describes DOD’s process for managing a new weapon
system’s requirements development, acquisition, and fielding. The most
notable aspect of the model is that there is little communication or
input from the maintenance community early in the process—during
requirement setting and product development—when decisions will be
made that will significantly influence the cost to maintain and support the
weapon system. The model shows that, traditionally, DOD’s processes are
separate and independent of each other. First, requirements are set
independently of the maintainers and the product developers. Second,
once the product development process begins, the focus is on maturing
technologies and achieving a producible design that will meet the
technical performance requirements for the weapon system. Finally, the
operator is tasked to use and maintain the system that has been developed
and produced.




Page 25                                              GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




Figure 4: DOD’s Linear Acquisition Process




The goal of this process is to field a high-performing weapon system that
will satisfy the needs of the war fighter better than any other weapon
system available. We previously reported that competition for limited
funding both within and among the services leads to performance
requirements that will make the particular weapon system stand out from
existing or alternative systems. 1 Those that provide the greatest leap
forward in promised performance stand the best chance of winning the
funding. As a result, the design for the weapon system is usually based on
undemonstrated components and subsystems that, when integrated into a
weapon system, have low reliability and, ultimately, high operating and
maintenance costs over their lifetime. Figure 5 illustrates this
phenomenon. The demonstrated reliability of the new system is lower,
causing an imbalance in the relationship between readiness and operating
cost toward the need for higher costs to maintain readiness.




1
 U.S. General Accounting Office, Best Practices: Better Matching of Needs and
Resources Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.:
Mar. 8, 2001).




Page 26                                                    GAO-03-57 Best Practices
                          Chapter 2: DOD’S Requirements-Setting and
                          Development Practices Yield Higher Total
                          Ownership Costs




                          Figure 5: System Readiness Comes at High Operating and Support Costs When
                          Reliability Is Not Ensured




                          Logistics officials at Air Mobility Command told us that even though
                          they are represented in the requirements determination process for new
                          weapon systems, they view their role as providing input on how the
                          logistics community could support performance requirements of a new
                          system, not on forcing trade-offs that would reduce operating and support
                          costs. The officials also said that the logistics community leads many
                          reliability improvement and cost reduction initiatives once aircraft are
                          fielded. However, this approach is harder and more costly to implement
                          than if reliability and operating and support costs had been considered
                          when requirements were set and product development began. Further, the
                          program must compete against other programs for operating and support
                          funds to implement the upgrades and improvement initiatives.


Operating and Support     Supportability aspects of system performance, such as operating the
Cost Estimates Are        system at the lowest possible cost and the percentage of time the system
Provided Later in DOD’s   has to be available for operations, are not given the same high priority as
                          enhanced performance. Operating and support costs are not estimated
Process                   until much later in development. For example, the B-2 bomber, Apache
                          helicopter, and Abrams tank programs did not publish an initial estimate
                          for the operating and support costs for those weapon systems in the
                          Selected Acquisition Reports until more than 15 years after the start of
                          development. The C-17 program did not publish an estimate until more
                          than 7 years after development began. These estimates were not in the
                          form of goals or key performance parameters. Throughout product
                          development, then, design goals for reliability are not supported by the
                          war fighters’ need to meet operating cost and availability goals.




                          Page 27                                               GAO-03-57 Best Practices
                             Chapter 2: DOD’S Requirements-Setting and
                             Development Practices Yield Higher Total
                             Ownership Costs




Maturing the Technology Is   In DOD, requirements for new weapon systems are usually based on the
the Focus of Development     enhanced performance deemed necessary to achieve a certain war-fighting
                             mission with little hesitancy in using new technology or what the cost to
                             support it may be. The user representatives define system performance
                             with limited input from the product developers and maintainers.

                             A case in point is the B-2 bomber. The low-observable requirement for the
                             B-2 bomber could only be met with technology that was immature at
                             program launch, and this technology continued to cause problems
                             throughout development, production, and fielding. The B-2 program began
                             full-scale development in 1981, and the Air Force began low-rate initial
                             production concurrently with development in November 1987. By 1991,
                             problems with the B-2’s low-observable material were still being reported,
                             causing delays in delivery and cost increases because the material was
                             difficult to manufacture. Once fielded, the low-observable materials
                             required very high maintenance. The processes to repair them were time
                             consuming and required an environmentally controlled repair facility. Poor
                             durability and extensive maintenance kept the aircraft from achieving its
                             planned availability. All of these factors are due to decisions to proceed
                             with product development without understanding this technology.

                             A more recent example is the F-22 fighter. The requirements for the
                             F-22 were very demanding. Performance characteristics included low
                             observability, super cruise speed, and fused avionics. These requirements
                             caused the product design to include many new and unproven
                             technologies. During product development, the program planned to
                             achieve a system-level reliability rate for the F-22 of 3 flying hours mean
                             time between maintenance actions when fully mature. The Air Force had
                             estimated that in late 2001, when the F-22 entered limited production, it
                             should have been able to demonstrate almost 2 flying hours between
                             maintenance actions. However, when the F-22 actually began limited
                             production, it could only fly an average of .44 hours between maintenance
                             actions. In other words, the F-22 is requiring significantly more
                             maintenance actions than planned. To date, the program has identified
                             about 260 types of failures, such as main landing gears wearing out more
                             quickly than planned, fasteners being damaged, and canopy delaminating,
                             all leading to unanticipated operating and support costs.




                             Page 28                                              GAO-03-57 Best Practices
                              Chapter 2: DOD’S Requirements-Setting and
                              Development Practices Yield Higher Total
                              Ownership Costs




Product Development in        Our previous work identifying best practices during product development
DOD Goes Forward in           concluded that during this phase, the tasks are to ensure the stability of
Spite of Poor Prospects for   the design and to ensure that the product can be produced.2 In DOD, the
                              product developer frequently is trying to catch up to design and
Reliability                   production tasks because product developments begin with immature
                              technology. Schedule concerns override the need to capture knowledge
                              about the design and production processes, and programs often proceed
                              through development and into production before the reliability of the
                              subsystems and systems has been demonstrated. Design features such as
                              open architectures that allow systems to receive upgrades as technology
                              advances or reductions to the number of parts in a design that reduce the
                              need for spare parts and maintenance time are not given due
                              consideration, even though they could lower operating and support costs
                              of the system.

                              As the schedule tightens, the lack of knowledge becomes more acceptable,
                              even preferred. Reliability testing is often pushed closer to fielding,
                              resulting in supportability problems being identified during operational
                              testing. The Office of the Director, Operational Test and Evaluation has
                              commented that operational testers identified reliability as a problem in
                              almost every program, because the product developer places more
                              emphasis on performance requirements than suitability. According to the
                              operational testers, many systems enter operational test and evaluation
                              with known, but unresolved reliability problems.

                              Once a system is fielded, unless the reliability and ease of maintenance
                              have been incorporated into the design already, there are limited
                              opportunities to improve these metrics without costly redesign or retrofit.
                              As the product’s design becomes firm and the system is produced and
                              fielded, the opportunities to influence these costs diminish. For example,
                              the Army is currently attempting to reduce operating and support costs on
                              its Apache helicopter and its Abrams tank. These efforts should have a
                              favorable impact on the systems’ operating and support costs. However,
                              the Army is retrofitting and replacing components that were identified as
                              problems much earlier in the programs. Examples from the Apache and
                              Abrams development illustrate the problems.




                              2
                               U.S. General Accounting Office, Best Practices: Capturing Design and Manufacturing
                              Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 15,
                              2002).




                              Page 29                                                    GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




Figure 6: Apache Helicopter




The Apache program focused on acquisition costs, schedule, and
performance during product development, even when problems were
identified that would impact reliability and maintainability. Today, the
Apache helicopter is the Army’s most costly system in terms of operating
and support costs with those costs rising over the years. For example, the
target acquisition and designation system enables the Apache to find


Page 30                                              GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




targets and guide its weapons. It is the helicopter’s most sophisticated
system, involving 26 major electrical, optical and mechanical components.
However, the sight requires frequent maintenance, and its complexity
reduces its reliability. The pilot’s night vision system is also a highly
complex system that experienced problems in development. Because its
target acquisition and designation system and the pilot’s night vision
system proved unreliable, the Apache incurred higher than expected costs
to maintain the helicopter’s availability.

During development, problems identified with the Apache foreshadowed
future support problems. These problems included excessive vibration and
excessive aircraft weight in addition to difficulty in achieving reliability
rates for the aircraft’s target acquisition and designation system. During
development, problems with the Apache’s fault detection system led to
questions concerning whether it could operate safely and reliably during
operations. Further, Army test and evaluation agencies warned that these
problems could cause serious supportability issues since they would result
in frequent need for maintenance and repair. By 1990, the majority of the
Apache helicopters had been produced and fielded, but Apache could
achieve only 50 percent availability rates, well short of their 70 percent
goal. Tests showed that the Apache required maintenance actions to
correct significant problems every 2.5 flying hours while the Army’s goal
was 4 hours between failures. Maintenance units were physically unable to
handle the repairs required to keep the helicopter flying. Subsequently, the
Army added 18 maintainers to the Apache battalion, constructed more
contractor repair facilities, and hired more contractors. During Operation
Desert Storm, although the system was effective in destroying tanks and
other targets, it continued to experience reliability and logistical support
problems that grounded some aircraft.




Page 31                                              GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




Figure 7: Abrams Tank




Likewise, the Army produced the Abrams tank without first resolving
reliability problems. The tank provided a major improvement in speed,
agility and lethality over the older M-60 tank and was also supposed to
decrease the operating and support cost burden. However, during
development, the tank experienced serious failures with the track, the
engine, fuel filters, the fuel and water separator, and the fuel pump. In
particular, the durability and reliability of the Army’s turbine engine was a
major concern. Tests performed up to the time of the initial production
decision showed that the tank generally met its performance requirements.
However, frequent breakdowns and component failures raised serious
questions about approving the tank’s production. In a 1993 operational test
report from the Office of the Director, Operational Test and Evaluation,
the reviewer concluded that the tank was operationally effective, but not
operationally suitable because of its many support problems such as those
mentioned above. Today, the Abrams is the second most costly Army
system in terms of operating and support costs and accounts for half the
repair parts costs by the entire ground combat fleet. The tank engine is a
major contributor to the tank’s high support costs. In recent years, the
operating and support costs for the Abrams have remained steady at about



Page 32                                               GAO-03-57 Best Practices
                        Chapter 2: DOD’S Requirements-Setting and
                        Development Practices Yield Higher Total
                        Ownership Costs




                        $2 billion per year, although the Army has reduced the number of tanks to
                        be supported by more than 300.


Data on Operating and   When estimating operating and support costs for a new weapon system or
Support Costs Is Not    trying to establish maintenance trends, maintenance data on the current
Reliable                system are an important source of information. However, the three
                        services’ operating and support data collection systems do not provide
                        accurate and reliable information about the cost to operate and maintain
                        the systems because they do not collect and maintain data on all elements
                        of the weapon system’s operating and support cost. Without reliable
                        information, the services cannot assess trends or identify top cost drivers
                        and take corrective action.

                        The Army’s Operating and Support Management Information System
                        provides historical cost data on Army weapon systems and is the primary
                        source of operating and support data used by the program managers to
                        project costs of new systems, forecast spare parts budgets, and generally
                        manage their programs. The Army uses this data system to develop its
                        operating and support cost budget for weapon systems for consumable
                        items such as repair parts, petroleum, oil, lubricant, fuel, and ammunition
                        as well as for some maintenance. However, the system does not provide a
                        complete and reliable basis for developing and reporting the costs of
                        weapon system support. The data are often incomplete and 12 to 18
                        months out-of-date. Several important cost elements used to establish the
                        Army’s operating and support budget estimates—such as contractor
                        logistics support, supply maintenance and software support—are not
                        included in the database.

                        The Navy’s central tracking system for operating and support costs, like
                        the Army data collection system just described, provides historical cost
                        data on weapon systems. Navy program officials told us that this data is
                        often inaccurate and cannot be relied upon to pinpoint causes of failure or
                        maintenance actions.

                        Until recently, tracking Air Force operating and support costs was difficult
                        because the cost data were unavailable in a usable format or were of poor
                        quality. Poor cost data weakened operating and support cost estimates. In
                        1998, the Air Force set up a total ownership cost database, with the
                        objective of providing accurate and reliable data to identify cost drivers
                        and support decision makers in making improvements to fielded aircraft.
                        However, the new system is not available at all aircraft maintenance
                        locations, and therefore data may not be complete. In any case, the Air


                        Page 33                                              GAO-03-57 Best Practices
Chapter 2: DOD’S Requirements-Setting and
Development Practices Yield Higher Total
Ownership Costs




Force will need several years of cost data before it can evaluate the
effectiveness of the system.




Page 34                                               GAO-03-57 Best Practices
                   Chapter 3: Commercial Companies
Chapter 3: Commercial Companies
                   Deliberately Manage Ownership Costs
                   through Product Requirements and Design


Deliberately Manage Ownership Costs
through Product Requirements and Design
                   Leading commercial companies consider the total ownership cost of a
                   new product, including its operating and support costs, integral to their
                   new product development decisions. Companies that use airplanes, trucks,
                   and ships to deliver goods and people such as United Airlines, FedEx
                   Express, and Polar Tanker understand the importance of maintaining the
                   readiness of their fleets at as low an operating cost as possible. Reducing
                   these costs results in increased revenues, profits, and market growth.
                   Increases in these costs, on the other hand, can result in failure. The
                   companies also understand that unless reliability is designed into a
                   product, there are limited opportunities to improve readiness and reduce
                   cost without costly retrofit or redesign. They have been successful in
                   reducing these costs because they developed a collaborative process with
                   companies that develop those products, such as Boeing, for setting the
                   product’s requirements, developing the product with operating and
                   support costs in mind, and capturing accurate operations and support data
                   once it is delivered.

                   The companies we visited that bought products all established the
                   product’s operating cost and its availability as requirements equal to its
                   performance characteristics prior to product development, thereby
                   ensuring that the developer placed priority on those goals during product
                   development. They were amenable to reducing the product’s performance
                   features to reduce its operating cost, as long as performance was within
                   acceptable limits for achieving market objectives. They also considered
                   bearing additional cost for the product’s design if it resulted in a net
                   benefit from reduced operating costs. Once product development began,
                   maintainers had a strong voice in the product’s design and leading product
                   developers set high reliability standards for components they chose, using
                   proven technologies to achieve performance requirements. Companies,
                   such as Boeing and Maytag, chose an evolutionary approach to product
                   development to achieve goals for life-cycle costs, testing extensively and
                   early for reliability. These companies also emphasized other practices to
                   reduce operating costs such as reducing the number of parts in a product’s
                   design, using standardized parts, and using open systems to ease
                   maintenance and maintain competition. Once the product is delivered,
                   maintainers keep detailed records on its reliability and maintenance and
                   provide that information to the developer to improve future products.


                   Figure 8 represents a model of the best practices that were most helpful to
A Best Practices   the companies we visited in achieving high reliability and reducing a
Model              product’s operating and support costs. Notably, the most critical events—
                   those that have the most impact on a product’s operating and support


                   Page 35                                              GAO-03-57 Best Practices
                                       Chapter 3: Commercial Companies
                                       Deliberately Manage Ownership Costs
                                       through Product Requirements and Design




                                       costs—take place very early, either prior to product development when
                                       the product’s key requirements are established, or very early in product
                                       development before the design is finalized. Another notable aspect of the
                                       model is that each of the activities—requirements-setting, product
                                       development, and operations—depend on clear and constant
                                       communication and collaboration among the customer, the product
                                       developer, and the maintainer from the time a product is conceived until
                                       the operator disposes of it.

Figure 8: Commercial Model for Reducing Operating and Support Costs




                                       The goal of this process is to develop and field a product that will perform
                                       in accordance with the customer’s needs and will be ready when needed
                                       within cost targets. As illustrated above, decisions about the product’s
                                       performance and cost are finalized prior to beginning the product’s



                                       Page 36                                               GAO-03-57 Best Practices
                      Chapter 3: Commercial Companies
                      Deliberately Manage Ownership Costs
                      through Product Requirements and Design




                      development, and the costs to operate and support the product are key
                      considerations. Once the requirements are set, the product developer can
                      focus as much on achieving a reliable and robust design for the product as
                      on achieving its performance goals. Finally, once the product is fielded,
                      those responsible for its operation and maintenance continue to feed
                      information back to the developer to improve future designs. Figure 9
                      illustrates what happens when the commercial firms we visited set
                      requirements that force the product developer to consider reliability rates
                      during design. When both operating cost and readiness are key
                      requirements for the product developer, the developer focuses on using as
                      many components and subsystems as possible that have demonstrated
                      reliability rates. The customer, by demanding readiness at a certain cost as
                      a hard requirement upfront has raised the importance of achieving it from
                      the very start.

                      Figure 9: Benefits of Ensuring High Reliability Rates During Product Development




                      We visited three companies—Polar Tanker, an operator of large oil
Leading Companies     tankers; United Airlines; and FedEx Express—to determine their
Treat Readiness and   practices for ensuring low operating and support costs from the
                      equipment they purchased from product developers. All three companies
Operating and         believe that understanding and controlling the cost to operate and support
Support Cost as       a product while it is being designed is essential to driving down the total
                      ownership cost of a product. To do this, they set requirements for a new
Critical Product      product’s availability and its operating and support costs equal in
Requirements          importance to requirements for its performance and acquisition cost. We
                      found these companies set requirements before development begins that
                      their products be ready almost 100 percent of the time at the lowest
                      operating cost possible. They typically set maintenance goals that drive
                      operating and support cost decisions, such as maintenance cost per mile



                      Page 37                                                  GAO-03-57 Best Practices
               Chapter 3: Commercial Companies
               Deliberately Manage Ownership Costs
               through Product Requirements and Design




               over a product’s lifetime. The following summaries illustrate the
               commercial processes.

               Figure 10: Polar Tanker’s Polar Endeavor




Polar Tanker   Polar Tanker is a commercial oil-transporting firm that recently decided
               that a new oil tanker was necessary to haul oil between Prince William
               Sound and Puget Sound. The company’s critical requirements for the new
               Endeavor tanker, which Polar Tanker believed would reduce the cost of
               delivering oil, were

               •   less expensive operations and maintenance over a 30-year life cycle
                   (versus the industry standard of a 20-year life cycle); and
               •   availability for operations at least 330 days a year.

               Polar Tanker teamed its maintenance engineers with industry consultants
               to ensure these requirements were met. The procurement team relied on
               its archived maintenance data from previous Alaskan operations to
               develop its double-hulled tankers. It documented locations, lengths and
               types of fractures, and stresses in the structures of its existing inventory of
               ships. This documentation from past operations was used to determine
               structural changes required to reduce maintenance on the new Endeavor
               class. As a result of their record keeping and the constant communication
               with the product developer, Northrop Grumman’s Avondale Shipyard,



               Page 38                                                GAO-03-57 Best Practices
Chapter 3: Commercial Companies
Deliberately Manage Ownership Costs
through Product Requirements and Design




Polar Tanker’s owners were successful in redesigning their ships and
meeting their requirements for improved performance, reliability, and
lower operations and maintenance costs.

The dual requirement of reduced operating and support cost coupled with
high readiness rates drove design trades that increased development costs
but improved reliability. For example, once Polar Tanker’s procurement
team identified ballast tank maintenance as one of the most significant
maintenance burdens, it directed Northrop Grumman to use the best and
most expensive epoxy coatings and specialized paints to protect the tanks
from corrosion. Another design trade—adding additional structure to the
ship’s hull to reduce the impacts of fatigue cracking—increased the
acquisition cost of the tanker but improved reliability and reduced the
need for maintenance. Polar Tanker accumulated data from its current
fleet and conducted extensive modeling of the hull design to understand
where the most critical cracking occurred and to identify operating and
support cost drivers. It hired an engineering consulting company to
conduct further analysis. As a result, Polar Tanker and Northrop
Grumman utilized the most current design tools to optimize the ship’s
structure. Polar Tanker also developed a list of equipment and suppliers
based on reliability analysis and incorporated that list into the design
contract with Northrop Grumman. Polar Tanker estimated that these
design trades cost about $25 million in additional design costs, but they
believe the changes will ensure its tankers will be able to operate more
reliably over 30 years.

Polar Tanker’s procurement team also required that the new Endeavor
Class design use open systems when possible. It worked closely with
Northrop Grumman, its contractor, to ensure requirements for an
integrated bridge system that consisted of commercial-off-the shelf
components with open systems to provide the capability to modernize the
system much less expensively as technology improved. For example,
systems in the tanker’s wheelhouse including the autopilot, marine radars,
bridge control console, and satellite communication equipment used the
open systems concept. The design also included easily accessible decks to
minimize delays and disruption during maintenance. Polar Tanker has
already upgraded its electronics since putting its first ship, the Polar
Endeavor, into service and experienced minimal disruption in operations.




Page 39                                             GAO-03-57 Best Practices
                  Chapter 3: Commercial Companies
                  Deliberately Manage Ownership Costs
                  through Product Requirements and Design




                  Figure 11: United Airlines/Boeing 777




United Airlines   As the launch customer for the new Boeing 777, United Airlines
                  established stringent requirements for aircraft readiness and operating
                  costs, thereby ensuring that reliability would be an important design
                  element. When United and Boeing negotiated the requirements, United
                  stated that it wanted a twin-engine airplane that could fly extended ranges
                  from any airport in the United States. In addition, United required that the
                  777 be available at the gate for departure within 15 minutes of scheduled
                  departure 98.5 percent of the time. Boeing guaranteed United that the
                  777 would meet the departure requirement by the third year of operation
                  or Boeing would pay a financial penalty. According to United officials, the
                  98.5 percent rate was achieved by the third year. United also specified that
                  operating and support costs for the 777 be no higher than on past
                  airplanes. The agreement reached with Boeing was that Boeing would
                  reimburse United for revenues lost as a result of airplanes being
                  unavailable. By setting requirements for operating cost and readiness,


                  Page 40                                              GAO-03-57 Best Practices
Chapter 3: Commercial Companies
Deliberately Manage Ownership Costs
through Product Requirements and Design




United ensured that Boeing would build reliability into the design of
the 777.

Boeing brought together a working group of customers—the leading
commercial airlines—to discuss requirements for the new design. Boeing
officials told us that during those requirements meetings the participating
airlines defined major design initiatives for the Boeing 777 based on
estimates of life-cycle costs. Initially there were differences among the
airlines as to what exactly was needed on the new aircraft—from wider
fuselages to additional electronics—but from the very first meeting, the
airlines were all equally concerned with operating and maintenance costs.
They were focused on designing an aircraft that would be easy for
mechanics to repair. The airlines unanimously agreed that an airline
maintenance representative was needed to adequately address operations
and maintenance requirements. Boeing named a chief mechanic who had
previously worked for United to the working group that was influential in
defining the maintenance requirements. Although Boeing provided its
customers engineering estimates for the operating cost of its new aircraft
in comparison to the older model, United officials said they developed
their own estimates based on its historical experience with Boeing
aircraft. United officials said that having two perspectives from which to
consider its purchase was helpful.




Page 41                                              GAO-03-57 Best Practices
                Chapter 3: Commercial Companies
                Deliberately Manage Ownership Costs
                through Product Requirements and Design




                Figure 12: FedEx Express Delivery Van




FedEx Express   The FedEx Express mission is to provide global air and ground
                transportation of high-priority goods and documents that require rapid,
                time-certain delivery. This mission demands high availability and reliability
                from its delivery equipment. In purchasing its newly designed fleet of
                delivery trucks, FedEx Express considered reliability, maintainability, and
                low operating and support costs to be critical measures of a successful
                acquisition. For example, improving availability and reliability were the
                key drivers in its acquisition of the new 700 cubic foot truck. FedEx
                Express collaborated with a product developer, Freightliner, to set the
                requirements for a newly designed vehicle with high reliability and
                endurance to withstand frequent stops, short travel distances between
                stops, and demanding use of the brakes. During the requirements-setting
                process, they also established cost and reliability requirements for the new
                truck that estimated an assumed number of stops per day, a certain
                number of miles per year at an assumed cost per mile.

                To make sure all costs, particularly operating and support costs were
                considered during product development, FedEx Express had a logistics



                Page 42                                               GAO-03-57 Best Practices
                      Chapter 3: Commercial Companies
                      Deliberately Manage Ownership Costs
                      through Product Requirements and Design




                      manager lead their discussions with Freightliner for the development of
                      the new truck. The FedEx Express logistics managers told us that they are
                      required to reduce the company’s operating and support costs, and the
                      company believes that it is essential to give these managers an integral
                      role during these discussions. The new delivery truck has been successful
                      in meeting its reliability and maintenance goals. The truck currently
                      averages 70,000 miles between breakdowns and is operating within the
                      cost per mile of service that was set as a requirement by FedEx Express at
                      the beginning of development.


                      To meet the readiness and operating and support requirements that their
Knowledge-Based       customers demanded, we found that product developers focused on
Product Development   designing a product that was durable, easy to maintain, ready when
                      needed, and reliable at low cost. They used an evolutionary development
Is Critical to        process to meet these requirements and did not allow components or
Achieving Desired     subsystems into a product’s design unless their reliability had been proven
                      through past use or testing. Boeing told us that it makes the reliability
Reliability and       ratings of its components available to the airlines before it begins product
Managing Operating    development, and Maytag did extensive reliability testing on every new
and Support Costs     product prior to going to production. The companies also emphasized
                      product designs with fewer parts and used open systems architectures as
                      much as possible. Developers also gained valuable insight into design
                      features their customers valued that drove down operating and support
                      costs. For example, the design for Boeing’s latest generation of the 737—
                      geared toward reducing operating costs—was inspired by the airlines’
                      request for more affordable operations and maintenance.

                      Once defined, the functional and operating and support requirements are
                      tightly managed and controlled to minimize scope increase during product
                      development. Companies work within a common framework to provide
                      management oversight and control. To move a product successfully from
                      concept to operations, the companies we visited used a “gated” product
                      development process and firm criteria to dictate when a product is ready
                      to exit a stage. The design reviews address all operating and support cost
                      requirements of the products. Senior management review teams grant
                      product approval at each gate only after determining that business cases
                      adequately address major drivers of operating and support cost as well as
                      reliability goals.


Boeing                Boeing uses a structured, gated product development process to define,
                      evaluate, and approve projects and to integrate new technology into its


                      Page 43                                              GAO-03-57 Best Practices
Chapter 3: Commercial Companies
Deliberately Manage Ownership Costs
through Product Requirements and Design




aircraft. This process separates technology development from product
development programs. In fact, Boeing keeps immature or untested
technologies in a research and development phase until they have been
tested for reliability in a realistic environment. The process forces the
company to obtain purchase agreements from customers and build a
business case that shows the expected profitability of a product line
before detailed designs are developed and a large dollar investment is
made in manufacturing. In order to get these purchase agreements, Boeing
product teams work collaboratively with potential customers to set
requirements for readiness, reliability, performance, acquisition costs,
and operating and support costs. Because aircraft are large capital
investments, customers are working with other developers to make sure
that the competitive forces of the market will help them get the best
aircraft at the lowest cost.

Boeing officials told us that to remain competitive in this environment,
they focus on meeting customer requirements at the highest reliability
leading to the lowest operating and support cost possible. For example,
when it was designing the new 737, Boeing used maintenance records to
prove to the airline that they could redesign the 737 with high reliability
and reduced operating and support costs, a key market requirement for
that aircraft. According to Boeing officials, they prepared a comparative
analysis of the acquisition and operating and support costs for older 737
aircraft with the estimated costs for the 737 Next Generation. This analysis
showed that operating and support costs were significantly lower for the
proposed next generation 737 and resulted in big savings to the customer.
Boeing was able to develop good operating and support cost estimates for
potential customers because it used an evolutionary approach for
developing new aircraft. This approach allowed Boeing to improve
performance and to insert new, reliable, and mature technology.

Once a new product line is approved, Boeing continues to work with its
customers to identify improvements that can be made to an aircraft in
terms of parts reduction, parts standardization, and ease of maintenance.
For example, Boeing formed four airline working groups with
representatives from 21 airlines around the world to focus on
maintainability, interiors, power plant, auxiliary power units, and common
display system issues on the 737. Their objective was to collect feedback
from operators on design changes that would reduce maintenance costs.
Examples of airline-driven design changes include




Page 44                                              GAO-03-57 Best Practices
         Chapter 3: Commercial Companies
         Deliberately Manage Ownership Costs
         through Product Requirements and Design




         •   simplifying the wing flaps by eliminating one third of the parts,
             designing a simpler flap mechanism, and making parts removable with
             common grip length fasteners;

         •   reducing engine removal and installation time by increasing the on-
             wing life of the engine by 40 percent and reducing the predicted change
             time from 12 hours to 6 hours; and

         •   improving the reliability and standardizing parts of the fuel system by
             using a fuel shutoff valve that is common with other aircraft.

         To facilitate and improve communication with its customers, Boeing
         oftentimes collocates customer representatives at its production facilities.
         Boeing has found that involving the customer in early design decisions
         improves their ability to design a reliable aircraft that is easy and less
         costly to maintain. For example, when it began developing the 777 aircraft,
         airline maintenance workers offered over 5,000 suggestions for changes to
         the design based upon their experiences with other Boeing aircraft. These
         suggestions helped reduce parts and improve reliability of the 777,
         resulting in increased time between maintenance actions. Boeing also
         utilized the concept of open systems to reduce the total ownership cost of
         the 777 by allowing customers to choose from three different types of
         engines—GE, Pratt & Whitney, or Rolls Royce—depending on their needs.


Maytag   Maytag follows a similar product development process. For Maytag, the
         most critical phase is when design specifications for product reliability
         and manufacturing feasibility are fully defined and understood. Early in its
         product development, an integrated product team takes all new product
         features through rigorous reliability growth analysis to determine what
         can be expected from the design and whether it will be able to meet the
         requirements. Maytag uses failure analyses, mock-ups, and other
         simulation tools to focus testing on the most critical elements of the new
         design and reduce the number of design/build/test iterations of the
         product. Even though these analyses require more upfront planning,
         Maytag officials stated that they still cut testing times in half and yet
         improved reliability and lower costs.

         During this phase, Maytag employs two project leaders, both a
         marketing manager and technical systems engineer, to conduct cost
         performance trades and co-chair subsystem design reviews. As a result
         of this co-leadership, most design issues are resolved immediately, and
         80 percent of all reliability testing and cost reductions take place early in



         Page 45                                                GAO-03-57 Best Practices
                        Chapter 3: Commercial Companies
                        Deliberately Manage Ownership Costs
                        through Product Requirements and Design




                        the process. Also, during this phase the project leaders present their
                        strongest business case, fully disclosing feasibility data, product
                        definitions, and estimates of life-cycle costs for team approval. The team
                        uses this information to make the critical product development decision to
                        commit further to product development or kill a product idea. Maytag
                        officials stated that their company has a quality image to uphold and that
                        consumers expect the highest reliability and quality from their products.

                        Maytag officials were also conservative in their use of new technologies
                        during product development. Decisions to incorporate new technology
                        were made in the earliest phases of product development and were based
                        on assessments of the adaptability and maturity of the technology and
                        associated risks to achieving established reliability targets. Even though its
                        Neptune washer incorporated over 90 percent new technologies in its
                        development, Maytag officials stated that they spent time with suppliers
                        and developers, maturing technologies to an acceptable level of reliability
                        before launching the new development.


                        The collection and analysis of the operating and support costs for
Leading Commercial      delivered products was considered essential by leading commercial
Firms Use Feedback      companies. Once a product is fielded, leading companies track actual
                        operating cost, reliability of parts, and readiness of the product against
from Operations to      what was estimated during product development to make sure the
Better Understand       company is getting what it paid for. We found companies are always
                        identifying the top drivers of operating and support cost and working with
Customer Needs,         the manufacturers to reduce these costs. When there are part failures, the
Product Deficiencies,   companies can quickly identify whether or not they are under warranty
and Operating and       and get the part replaced.

Support Costs           All of the companies we visited, customers and product developers alike,
                        had very collaborative processes and practices for drawing extensively on
                        data from current and past operations to improve the reliability of existing
                        products or influence the design of new products. United Airlines officials
                        told us that the airlines and Boeing both keep meticulous reliability and
                        cost records at all levels of the 777: components, subsystems, and the
                        system level. Major operating costs drivers are tracked on a daily basis by
                        the airlines. The manufacturers usually have personnel residing with the
                        airlines’ maintenance crews to help solve problems on the spot and, just as
                        importantly, feed information back to the manufacturer so that the next
                        product can be improved. FedEx Express and Polar Tanker both
                        emphasized extensive collection of data from current operations. In
                        fact, FedEx Express sets annual targets for operating and support


                        Page 46                                               GAO-03-57 Best Practices
Chapter 3: Commercial Companies
Deliberately Manage Ownership Costs
through Product Requirements and Design




cost reductions based on reliability data gathered on the road. Polar
Tanker gathered maintenance data from past operations to determine
areas that could result in higher reliability and lower maintenance costs in
designing the new tanker.

United conducts quarterly meetings for each of its fleets to discuss open
issues and short-term and long-term solutions to current problems with
operational aircraft. Attendees at the meeting include maintenance
mangers, financial representatives, representatives of the manufacturers,
and executives with authority to resolve issues. United’s practice is to
resolve problems as expeditiously as possible, no matter how small.
United also monitors flight movements on a real time basis through a
computer system that tracks each aircraft by tail number. The monitoring
system provides the maintenance history of the aircraft, reports problems
on a current flight that require maintenance upon landing, and alerts for
other required maintenance based upon the number of flying hours on the
aircraft. If a specific part is broken, the system also indicates whether or
not it is under warranty. United archives information on parts that break,
when they break, and whether they are still under warranties.

Manufacturers like Boeing and other major suppliers like this type of
feedback because it provides useful information to them on how to
improve the product for future iterations. They also believe that quick
responses to customer problems will help them get repeat business. They
use this feedback to develop preventive maintenance schedules, better
estimate operating and support costs, and refine reliability requirements to
be used in preparing budgets and cost estimates for future products.
Feedback mechanisms also accumulated operations and maintenance data
and “lessons learned” that highlighted reliability problems and other
maintenance issues. United officials stated that taking lessons learned
from data gathered from current products is an effective tool for
improving product reliability and maintainability or developing
requirements for new products.

FedEx Express takes any failure to meet its on-time delivery goal very
seriously and holds daily failure analysis meetings every morning to
analyze and review each delivery failure from the night before. This
constant feedback allows them to take immediate corrective action on
individual vehicles and to identify trends that may result in larger
problems and costlier maintenance requirements. FedEx Express
maintained a metric for vehicle miles between road calls and maintenance
costs per asset. In addition to tracking these costs per asset, FedEx
Express has established a performance goal with its managers to reduce


Page 47                                               GAO-03-57 Best Practices
Chapter 3: Commercial Companies
Deliberately Manage Ownership Costs
through Product Requirements and Design




the overall costs of maintenance each year. FedEx Express managers set
their annual targets for reductions in operating and support cost based on
reliability data gathered on the road.




Page 48                                             GAO-03-57 Best Practices
              Chapter 4: Stressing Operating and Support
Chapter 4: Stressing Operating and Support
              Cost at the Outset of an Acquisition Could
              Help DOD Reduce Total Ownership Costs


Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs
              DOD and the commercial companies we visited have policy goals of
              developing products that will meet customers’ needs at the lowest possible
              cost to build and operate. The difference between them is in how each
              implements its policies. Leading commercial companies follow an
              integrated, collaborative process of setting requirements, developing the
              product, and ensuring that the product can be supported at an acceptable
              cost. DOD’s process is composed of disparate practices carried out by
              separate organizations with differing objectives and little communication
              between them about how to support fielded systems. While commercial
              firms focus on total ownership costs at the outset, DOD focuses mostly on
              technical performance. One cause of this is that in DOD the accountability
              and responsibility for total ownership costs are spread across many
              organizations with separate goals. Another cause lies in motivation for low
              costs. The commercial companies we visited are driven by the need to be
              as profitable as possible to survive, and low total ownership costs
              translate to higher profitability. DOD’s environment does not provide such
              incentives. The organizations charged with acquiring and operating
              weapon systems are unconstrained by a need to lower costs since they can
              request additional operations and maintenance funding to keep systems
              working.

              From time to time, DOD stated the need to lower its total ownership costs
              in policy documents and in annual budget statements; however, it has not
              been successful because it does not have an environment that demands
              collaboration and accountability in setting requirements and developing
              products with operating costs in mind. DOD has some efforts underway to
              improve. First, it has rewritten its acquisition and requirements generation
              policies. Second, a few programs now in development established an early
              estimate of operating and support costs and are working to gain
              knowledge of the impacts of requirements and design on those costs.
              Third, information from pilots on fielded systems, if disseminated
              throughout the acquisition community, could be used to lower costs.
              Results are pending.

              What DOD’s efforts do not do is provide incentives to make investments
              for more reliable, less-costly-to-maintain systems at the beginning of an
              acquisition. Instead, DOD provides incentives to field systems with
              unknown reliability by allowing whatever funding necessary to operate
              and maintain the systems once they are fielded.




              Page 49                                              GAO-03-57 Best Practices
                                              Chapter 4: Stressing Operating and Support
                                              Cost at the Outset of an Acquisition Could
                                              Help DOD Reduce Total Ownership Costs




                                              In Chapter 3, we discussed our findings that leading commercial
Differences in                                companies set specific requirements for readiness and operating and
Practices Explain                             support costs prior to initiating product development that forced
                                              developers to design products with a high degree of reliability. In Chapter
Different Outcomes                            2, we noted that DOD, on the other hand, typically focused its
for Commercial                                requirements on revolutionary performance that often forced developers
                                              to mature technologies at the same time they were completing detailed
Companies and DOD                             design work. As a consequence, system reliability often suffered, forcing
in Controlling Total                          the department to spend a great deal of money to maintain and repair
Ownership Costs                               fielded systems in an effort to achieve desired readiness levels. The
                                              following table provides a comparison of the specific practices used by
                                              commercial companies we visited and DOD programs we reviewed to
                                              address operating and support costs early in a new product’s life cycle.

Table 2: DOD and Commercial Practices for Controlling Operating and Support Costs

 Commercial prevailing practice                                          DOD prevailing practice
                                              Practices used to set initial product requirements
 Operating and support cost goals as a key requirement.                  Operating and support cost goals are not established as key
                                                                         parameters.
 Readiness a key requirement.                                            Readiness is not a key parameter.
 Trade performance for reduced operating and support costs, if           Technical performance is sometimes traded using cost as an
 appropriate; sometimes results in increased costs.                      independent variable, but cost is usually production cost or
                                                                         development cost, and the trades occur during the design phase.
 Direct relationship during requirements-setting between the user        User and product developer separated by user representative and
 and the product developer.                                              government program office.
                                                 Practices used during product development
 Provide detailed operating and support cost estimates early in          Operating and support cost estimates not required until product
 product development.                                                    development launch.
 User and developer focus on ways to reduce product parts and            Product developer has responsibility of focusing on ways to
 standardize parts across product lines.                                 reduce parts counts or use standardized parts with little input from
                                                                         the user (operators or maintainers).
 Use open systems architecture approach to improve the cost              Open systems approach is mandated but implementation is
 effectiveness and installation efficiency of future upgrades to the     limited.
 product.
 Set realistic reliability growth goals for the product.                 Reliability goals set, but they are tradable or not met.
 Conduct reliability testing early.                                      Reliability testing sporadically performed.
                                                      Practices used during operations
 Collect and analyze operations and support data.                        Data is often incomplete or unreliable.
 Manage operations and support costs to targets.                         Do not manage to operations and support targets.
 Identify areas for continuous improvement.                              Lack of complete and reliable data makes identifying areas for
                                                                         improvement difficult; some areas that are identified are not
                                                                         funded for improvement.
 Feedback to developer on product performance.                           Limited feedback to the developer. The maintainer does not have
                                                                         a direct relationship with the product developer.
Source: GAO.




                                              Page 50                                                            GAO-03-57 Best Practices
                             Chapter 4: Stressing Operating and Support
                             Cost at the Outset of an Acquisition Could
                             Help DOD Reduce Total Ownership Costs




                             Clearly, the practices used by the commercial companies we visited before
                             product development when requirements are set, early in product
                             development when the design is finalized, and during the new product’s
                             operating life focus as much on providing a reliable product as on
                             providing a high-performance product. The companies make operating
                             cost and readiness key requirements, they perform extensive reliability
                             testing, and they aim toward continuous improvement once the product is
                             in the field. In DOD, because performance is the overriding concern of the
                             requirement setters, none of these practices are in place.


                             The changes to policies and the investment in improving systems’
Several DOD Efforts          reliability are encouraging indicators that DOD has focused its attention
Underway to Reduce           on reducing costs to support weapons. DOD has revised acquisition
                             policies, tested new approaches for reducing costs in a few systems,
Total Ownership              explored differing approaches, and created initiatives to reduce costs of
Costs                        legacy systems. Each of these efforts had some initial success, but most
                             are aimed at reducing costs after fielding when over 90 percent of the
                             costs have been determined.


DOD’s Requirements           DOD has revised its 5000 series acquisition policies several times over the
Generation and               past 10 years with the intent of defining an acquisition environment that
Acquisition Policies Could   makes DOD a smart and responsive buyer. During this time, the policy has
                             not substantively changed with regard to how acquisition programs can
Be More Specific in          best control total ownership costs. The department is striving for an
Addressing Supportability    integrated acquisition and logistics process that is characterized by, among
Issues                       other things, a stronger focus on using supportability as a key design and
                             performance factor. However, rules for total ownership cost goals at the
                             outset of an acquisition program are defined by the Chairman of the Joint
                             Chiefs of Staff’s Instruction 3170.01B1 on requirements generation. This
                             guidance states that cost should be addressed in the operational
                             requirements document for a new weapon system, if an estimate is
                             available at that time. However, policy does not require the services to set
                             requirements for operating and support costs or readiness. Instead, it
                             allows them to identify system capabilities or characteristics they consider
                             essential for successfully completing the mission. It states that the DOD




                             1
                              Chairman of the Joint Chiefs of Staff Instruction, Requirements Generation System,
                             (Washington, D.C.: Apr. 15, 2001).




                             Page 51                                                       GAO-03-57 Best Practices
                          Chapter 4: Stressing Operating and Support
                          Cost at the Outset of an Acquisition Could
                          Help DOD Reduce Total Ownership Costs




                          sponsor may make cost a key requirement if it desires and identify the cost
                          it wishes to evaluate.

                          We previously reported that DOD officials believe they must promise new,
                          revolutionary weapon systems with significantly better performance
                          capabilities than the ones they are replacing in order to obtain funding.2
                          Therefore, key parameters are usually focused on performance rather than
                          supportability.

                          In order to effectively minimize total ownership costs of its systems, the
                          Department of the Navy recently issued its own guidance that establishes
                          specific supportability and affordability thresholds and objectives3 for all
                          requirements documents. The Navy believes that by establishing readiness
                          and operating and support cost as required parameters, there is assurance
                          that major drivers of total ownership costs will be addressed and
                          minimized throughout the acquisition process. Specifically, the new
                          guidance states that requirements documents must include goals for
                          operating and support costs. It also states that operational availability be
                          included as a key performance parameter, except when logistics delays
                          are not an issue or if the requirements are for a major aircraft or ship
                          platform. In those cases, mission capable rates or full mission capable
                          rates focused on the platform’s primary mission areas will be used as key
                          requirements. The Army is discussing a similar change in its guidance.


Three New Acquisition     We found three DOD programs still in development—the Joint Strike
Programs Are Placing      Fighter, the Advanced Amphibious Assault Vehicle, and the Landing
Greater Emphasis on       Platform Dock 17—that appear to be using good practices to reduce
                          operating and support costs during product development. Each, in its own
Readiness and Operating   way, has had a powerful internal incentive to establish more collaborative
Cost Goals                practices or to focus attention on operating and support costs and product
                          reliability.




                          2
                              GAO-01-288.
                          3
                           A threshold is the minimum acceptable operational combat capability required to meet
                          war-fighter minimum requirements. An objective is the capability desired of the system
                          beyond minimum requirements.




                          Page 52                                                        GAO-03-57 Best Practices
Chapter 4: Stressing Operating and Support
Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs




Figure 13: Joint Strike Fighter




The Joint Strike Fighter program is intended to produce an affordable
next-generation aircraft to replace DOD’s aging aircraft inventory. The
program is structured to use a common production line to produce three
aircraft variants that meet conventional flight requirements for the Air
Force, short take-off and vertical landing characteristics for the Marine
Corps, and carrier operation suitability needs for the Navy. The program
will also provide aircraft to the British Royal Navy and Air Force. A key
objective of the acquisition strategy is affordability—reducing the
development, production, and operating costs of the program relative to
prior fighter aircraft it will replace. The program’s latest stated estimate
for operating and support cost savings compared to legacy systems is
$135 billion, or a 56 percent reduction in cost.

To achieve this affordability objective, the program office has
incorporated various DOD and commercial initiatives into the acquisition
strategy. For example, two key provisions in its operational requirements
document—mission reliability and logistics footprint— will have a direct
impact on operating and support costs. Specifically, all variants of the
fighter are expected to achieve a mission reliability rate of over 90 percent
and meet numeric goals of cargo aircraft or ship space needed to support a
30-day self-sustained deployment. These two requirements, along with



Page 53                                                GAO-03-57 Best Practices
Chapter 4: Stressing Operating and Support
Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs




other reliability and maintainability goals, demonstrate DOD’s desire to
reduce total ownership costs. The product developer currently estimates
the Joint Strike Fighter will be able to reduce operating and support cost
primarily through efforts to improve

•   reliability and durability of materials,
•   accessibility of parts or systems that need to be inspected or replaced,
•   supportability of low observable materials,
•   ability of on-board systems to predict impending flight critical failures,
    and
•   training materials and systems.

For example, DOD expects to save about $39 billion over the life of the
fighter through reduced maintenance on low observable materials. The
developer estimates that 99 percent of the maintenance actions will
require no low observable restoration because they are using high
durability materials, parts, or systems that are easier to access and harder
to damage. In order to reach this level of savings, the developer spent a
great deal of time evaluating previous DOD maintenance experience with
the B-2A bomber and the F-117 fighter aircraft and used an evolutionary
approach for upgrading these materials. Operating and support costs for
the B-2A bomber, for example, were significantly increased by the
decision to use an immature technology for low observability.

However, the Joint Strike Fighter program must be careful not to
overestimate the total ownership cost savings it can achieve over legacy
systems it will be replacing because the program is also depending on new
technology for on-board systems to predict failure—prognostics and
health management technology—that is not yet ready for product
development. In October 2001, GAO reported that this technology was not
at an acceptable readiness level for inclusion in product development, but
DOD and the contractor decided to include it in order to meet total
ownership cost objectives. 4 Program officials stated that about
$16 billion—12 percent of the estimated $135 billion in total ownership
cost savings—is expected to come from that technology. Since then, the
officials have allowed for the possibility that the technology may not be
included on initial production lots for the Joint Strike Fighter if it is
not ready.



4
 U.S. General Accounting Office, Joint Strike Fighter Acquisition: Mature Critical
Technologies Needed to Reduce Risks, GAO-02-39 (Washington, D.C.: Oct. 19, 2001).




Page 54                                                        GAO-03-57 Best Practices
Chapter 4: Stressing Operating and Support
Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs




Figure 14: Advanced Amphibious Assault Vehicle




The Advanced Amphibious Assault Vehicle is a Marine Corps program to
improve its amphibious landing vehicle. The new development promises
faster sea and land speeds, better protection, and more lethality. The
development program has focused on maturing technology and paying
attention to operating and support costs early in development. The
program has used some of the best practices of commercial companies
during development. Some of those include collocating the program office
at the contractor’s facility and making extensive use of Marine Corps war
fighters and maintainers to provide a “hands on” assessment of how
effective the vehicle would be in operations as well as how supportable it
would be during operations. The Marines developed an early estimate of
total ownership costs and included a reliability metric as one of its key
requirements. The Marines developed three vehicle prototypes to mature
the design and have conducted extensive reliability testing. The vehicle
will have parts that are common with other weapons such as a gun that
will be common with the Landing Platform Dock 17. Advanced
Amphibious Assault Vehicle officials estimated that they will save
$29 million in operating and support costs.

The Navy program office for the Landing Platform Dock 17 has adopted a
total ownership cost approach. The program office established a process
for suggesting and evaluating design trades that could reduce operating
and support costs. Some of the design changes that the Navy made include
enclosing the mast to reduce exposure to weather and salt water, and
investing in high performance covering for the deck and well deck to


Page 55                                             GAO-03-57 Best Practices
                          Chapter 4: Stressing Operating and Support
                          Cost at the Outset of an Acquisition Could
                          Help DOD Reduce Total Ownership Costs




                          mitigate corrosion. Other practices include involving users to complete
                          tasks using the virtual software to test special design elements in the ship,
                          making greater use of sensors and automated processes to reduce
                          maintenance and to reduce crew requirements.

                          Both the Landing Platform Dock -17’s and the Advanced Amphibious
                          Assault Vehicle’s estimate for operating and support costs have recently
                          increased. The Navy raised its estimate of Landing Platform Dock 17’s
                          requirements for spare parts, fuel, and software maintenance. The
                          Advanced Amphibious Assault Vehicle program office attributed its
                          increase in the cost estimate to funding additional prototypes to improve
                          reliability. Information like this allows options for decision makers, while
                          the system is still in development, to accept the costs or re-examine the
                          performance characteristics to see if they can be relaxed in order to
                          improve reliability and, thereby, reduce operating and support costs.


DOD Pilot Programs        In 1999, the Defense Systems Affordability Council implemented a
Attempt to Reduce Total   program to explore ways to reduce the total ownership cost of its weapon
Ownership Costs           systems. The Council—chaired by the Undersecretary of Defense,
                          Acquisition, Technology, and Logistics—set a goal of reducing logistics
                          costs for selected fielded weapon systems by 20 percent by fiscal year
                          2005. The Council also set a goal for selected systems still in development
                          to achieve total ownership costs targets that are 20 percent to 50 percent
                          below historical norms.

                          DOD selected 30 programs (10 from each military department) to test
                          various approaches for reducing total ownership costs, such as using
                          commercial items or technology to reduce costs of legacy systems and
                          using industry standards when developing systems to make upgrades
                          easier and less costly to complete.

                          The Air Force’s C-17 program is one of the pilot programs. It is a fielded
                          system whose operating and support costs increased by about 25 percent
                          between 1995 and 1999. One C-17 initiative is an engine upgrade to extend
                          the time between removals, reduce unexpected shop visits and spares
                          purchases, and reduce the number of engine overhauls. The Air Force
                          believes the C-17 could avoid $724.5 million in support costs if the upgrade
                          is completed.

                          The Apache recapitalization program, another of DOD’s pilots, integrates a
                          number of selected upgrades that taken together are expected to achieve a
                          30 percent reduction in operating and support costs by 2010. Most of the


                          Page 56                                               GAO-03-57 Best Practices
                         Chapter 4: Stressing Operating and Support
                         Cost at the Outset of an Acquisition Could
                         Help DOD Reduce Total Ownership Costs




                         Apache helicopters will be refurbished and modified to the Apache
                         Longbow configuration. The target acquisition and designation system, the
                         top cost driver, is a focus of these improvements along with improvements
                         in the drive train, the rotor, and the propulsion system. The current
                         average cost per flying hour for the Apache fleet is $3,348. The Army’s
                         projected cost per flight hour after the modifications is $2,230.

                         The Abrams tank is also undergoing a major upgrade estimated to cost
                         about $5 billion. The top cost driver on the tank is the power pack, which
                         includes the engine and transmission, followed by the auxiliary
                         automotive system, hull and frame, fire control system, armament, and
                         track. Army officials believe upgrading and replacing the engine is the
                         most effective way to reduce operating and support costs for the tank. The
                         current cost per mile for the Abrams fleet is $181 per mile, including repair
                         parts and fuel but excluding most personnel cost. The recapitalization
                         program’s goal is to reduce the cost to $107 per mile and to improve
                         reliability.


Other Initiatives Show   We found three other initiatives—the change in acquisition policy toward
Promise, but             evolutionary acquisition, an open systems approach for weapon systems
Implementation Is Slow   and the Commercial Operations and Support Savings Initiative—that could
                         help DOD reduce total ownership cost. However, implementation has been
                         limited in the latter two initiatives because consistent high-level support is
                         lacking.

                         DOD defines evolutionary acquisition as an approach for delivering
                         capability in increments, recognizing the need for future capability
                         improvements. DOD allows two processes to achieve an evolutionary
                         acquisition, both of which include requirements for collaboration between
                         the user, the tester, and the developer. The first process is referred to as
                         incremental development. In an incremental development process, a
                         desired capability is identified, an end-state requirement is established,
                         and the requirement is met over time by the development of several
                         increments of the product, each dependent on available, mature
                         technology. The second process is referred to as spiral development. In a
                         spiral development process, the end-state requirement is not known, and
                         each increment of the product is based on feedback from the user. Each
                         increment yields the best possible capability for the user. The movement
                         toward evolutionary acquisition and time-phased requirements bodes well
                         for the potential to understand reliability, readiness, and implications for
                         total ownership cost early, because an evolutionary process allows an
                         acquisition program to design a weapon system to requirements based


                         Page 57                                               GAO-03-57 Best Practices
Chapter 4: Stressing Operating and Support
Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs




only on demonstrated technologies. This is very similar to commercial
practices.

DOD chartered an open-systems joint task force to implement an open
systems approach in weapon systems acquisitions. Open systems can
reduce cost through use of widely accepted standard products from
multiple suppliers, allowing DOD to benefit from the commercial market
place and take advantage of the competitive pressures that motivate
commercial companies to improve products and reduce prices. DOD
expected open systems to reduce the cost of ownership of weapon
systems, delay system obsolescence, and allow fielding superior war-
fighting capability more quickly. The DOD Inspector General recently
reported that the DOD acquisition community has not fully applied the use
of open systems objectives in the acquisition planning and review
process.5 The report recommended DOD enforce the use of an open
systems approach as part of the acquisition milestone review process.

Another initiative that showed promise but lacks high level support is the
Commercial Operations and Support Savings Initiative introduced to
improve weapon system readiness and reduce operating and support costs
by inserting existing commercial items or technology into military legacy
systems. It emphasizes the rapid development of prototypes and fielding of
production items based on current commercial technology. According to a
2001 report by an independent assessment team, the initiative’s three
objectives of reducing operations and support costs for legacy systems,
simplifying prototype development, and attracting commercial firms to the
defense marketplace are being met. But, the initiative lacks high-level
support. The Under Secretary of Defense (Acquisition, Technology, and
Logistics) recently directed that funding for the program be terminated.




5
 Department of Defense, Office of the Inspector General, Use of An Open Systems
Approach for Weapon Systems, Report No. D-2000-149 (Washington, D.C.: June 14, 2000).




Page 58                                                      GAO-03-57 Best Practices
                       Chapter 4: Stressing Operating and Support
                       Cost at the Outset of an Acquisition Could
                       Help DOD Reduce Total Ownership Costs




                       While initiatives for acquisition programs and potential reductions for the
DOD’s Current          fielded systems are welcomed, the department has not institutionalized the
Environment Does       practices used in the initiatives by demanding them on all acquisition
                       programs. As we discussed in Chapter 1, 90 percent of the operating and
Not Provide            support costs are determined before fielding, and these initiatives do not
Incentives to Reduce   attack the causes of higher operating and support costs. Those are: the
                       division of responsibility among the requirements community, the
Total Ownership Cost   acquisition community, and the maintenance community for controlling
Early                  costs; the lack of focused attention on reliability early in development; and
                       the lack of accountability for total ownership cost when setting
                       requirements that is caused by the division of responsibilities across these
                       communities.

                       Companies we visited have incentives to make operating cost and product
                       readiness equal to technical performance when setting requirements for
                       new products because these factors largely determine their profitability
                       and, therefore, survival in the market place. Lower operating costs
                       translate to higher profits and increased sales. Customers cannot afford to
                       have large amounts of capital tied up in extra equipment, spare parts, or
                       personnel to ensure their equipment is ready to perform when needed.
                       They cannot afford to have equipment fail during operations, because
                       failure precludes accomplishment of the company’s mission and loss of
                       revenue. These companies are constrained by a finite amount of funding to
                       acquire and operate their equipment, and, therefore, they hold the people
                       setting the product requirements accountable for total ownership cost.
                       Many of the companies we visited use one integrated product team to
                       identify needs, set requirements, and monitor product development. Most
                       importantly, the organization that will be responsible for supporting the
                       equipment in the field sets requirements for new products. There is also a
                       direct relationship between the requirements-setting team and the product
                       developer while the product’s requirements are being set, during
                       development, and after products are put into service. Information flows
                       throughout the integrated process, with each new phase in the process
                       being informed by knowledge from the phase just ending.

                       DOD’s current acquisition environment does not provide the same
                       incentives or practices. Traditionally, DOD does not constrain its
                       requirement setters in the same way. Requirement setters in DOD have
                       demanded weapon systems that, due to their performance features,
                       consistently cost more to operate and support than anticipated to achieve
                       necessary readiness levels. This has been accepted because a large
                       logistics organization—separate from the requirement setting
                       organization—is charged with supporting these weapon systems and uses


                       Page 59                                               GAO-03-57 Best Practices
Chapter 4: Stressing Operating and Support
Cost at the Outset of an Acquisition Could
Help DOD Reduce Total Ownership Costs




monies from a different budget to do so. In essence, DOD’s environment
frees the requirements community to insist on technical requirements that
cannot be made into reliable products, are costly to support, and cannot
be maintained cost effectively. Accountability for operating and support
costs does not rest with the requirement setters, or, for that matter, with
the acquisition community. Eventually, maintenance organizations have no
choice but to request sufficient funding to keep weapon systems operating
once they are fielded. DOD has identified this division of responsibility as
a key cause of higher weapon system operating and support costs. In this
current environment, there is no incentive for collaboration and
accountability in setting requirements and developing products with
operating costs in mind. Instead, it provides incentives to field systems
with unknown reliability by allowing whatever funding necessary to
operate and maintain the systems once they are fielded.




Page 60                                              GAO-03-57 Best Practices
              Chapter 5: Conclusions and Recommendations
Chapter 5: Conclusions and
Recommendations

              Acceptable readiness levels are a function of having platforms available
              when required. Such levels can be achieved by having highly reliable
              platforms, by spending whatever is necessary on ongoing maintenance, or
              by having excess capacity. The high cost of maintaining weapon systems
              to meet required readiness levels is depleting DOD’s modernization
              accounts and denying DOD the flexibility to invest in new weapons. DOD
              must find ways to reduce total ownership cost while maintaining needed
              readiness rates. Readiness is a critical component of all DOD weapons
              systems. If a system is not ready, its performance capabilities are of no
              use. The decision on whether readiness will be achieved by spending
              additional funds on operations or by designing high reliability into the
              weapon system must be made while requirements are being set and early
              in product development.

              DOD’s prevailing practices run counter to achieving high reliability. Often,
              DOD does not make readiness or operating cost performance parameters
              equal in importance to others when it establishes requirements for
              weapons systems. Further, reliability growth during product development
              is hampered by immature technologies and delays in gaining knowledge
              about the product’s design. Finally, DOD does not have sufficient
              knowledge about its fielded systems to inform its product development
              process for new systems. DOD is at a crossroad in this regard. It has made
              positive changes to acquisition policy in order to change its environment.
              Requiring higher readiness at lower cost will enable DOD to take the next
              step, ensuring lower total ownership cost.

              In contrast, commercial companies that are in the market for new capital
              equipment understand that they must specify and control the readiness
              and total ownership cost of a product, especially the operating and
              support costs early in development. Therefore, they specify how available
              or ready the products must be in order to carry out the company’s mission.
              Further, they set goals for operating costs when acquiring new equipment;
              they make sure they understand their own operating costs from data they
              have collected and analyzed on equipment they are now using. Those two
              goals—how available the product must be and how much the customer
              wants to spend per operating unit to support the equipment—are key
              requirements equal in importance to other performance characteristics
              that the commercial customer demands from the companies that develop
              the products. Bounded by the twin requirements of specific operating
              costs and availability, the product developer sets reliability goals for the
              components, subsystems, and the full system once it is integrated into a
              product that will satisfy the customer’s requirements. Product developers
              remain focused on good product development practices with mature


              Page 61                                               GAO-03-57 Best Practices
                      Chapter 5: Conclusions and Recommendations




                      technologies, stable designs, and production processes that are in control.
                      During operations they collect data from their customers on reliability and
                      performance and use that data to predict operating and support costs for
                      subsequent developments or upgrades.

                      DOD has initiatives underway that partially address the issue of
                      controlling operating and support costs. However, without significant
                      emphasis on providing a better framework for decision-making, these
                      initiatives will not yield sufficient improvements. The department has
                      encouraged the services to include key performance parameters in its
                      newer developments such as the Joint Strike Fighter that indicate how
                      long a system must perform between maintenance actions. It has moved to
                      follow best practices for reducing risk from technology and achieving
                      more stable designs in the Advanced Amphibious Assault Vehicle and the
                      Joint Strike Fighter. However, these programs are early in development
                      and it will take some time to see how reliably they perform. We believe
                      that practices found at the commercial companies we visited to make
                      operating and support costs and product readiness requirements equal in
                      priority to other performance characteristics forces developers to focus on
                      achieving high reliability and that adopting these practices will help DOD
                      achieve high readiness and control total ownership cost.


                      DOD should take steps to make the cost to operate and support weapon
Recommendations for   systems at required readiness rates a priority when setting weapon system
Executive Action      requirements for an affordable weapon system and finalizing the design of
                      the selected system. To do this, its requirements and acquisition
                      communities must collaborate to fully understand and control the costs to
                      operate and support a weapon system prior to and early in product
                      development, when it is possible to have significant impact on those costs.
                      In establishing requirements for a weapon system, the requirements
                      community should include the costs to operate and support the weapon
                      system over its life cycle and the readiness rate for the weapon system. To
                      establish an affordable design for the weapon system, the acquisition
                      community and acquisition programs should be required to accurately
                      estimate—based on demonstrated component and subsystem reliability
                      testing—that portion of the costs that DOD plans to spend on operations
                      and support of the weapon system throughout its life cycle before the
                      design is finalized.




                      Page 62                                              GAO-03-57 Best Practices
                   Chapter 5: Conclusions and Recommendations




                   With this in mind, to ensure that the user’s requirements for a weapon
                   system can be met with a reliable design, we recommend that the
                   Secretary of Defense

                   •   revise the Chairman of the Joint Chiefs of Staff Instruction 3170.01B on
                       the requirements generation process to include total ownership cost,
                       especially operating and support cost, and weapon system readiness
                       rates as performance parameters equal in priority to any other
                       performance parameters for any major weapon system before
                       beginning the acquisition program;
                   •   revise the current policy governing the operation of the defense
                       acquisition system (currently under revision) to require that the
                       product developer establish a firm estimate of a weapon system’s
                       reliability based on demonstrated reliability rates at the component and
                       subsystem level no later than the end of the system integration phase,
                       coinciding with the system-level critical design review, before
                       proceeding into the system demonstration phase of product
                       development; and at the system level no later than the full-rate
                       production decision; and
                   •   structure its contracts for major systems acquisitions so that at
                       Milestone B the product developer has incentives to ensure that proper
                       trades are made between reliability and performance prior to the
                       production decision. One option is to provide specific clauses in the
                       development contract to address reliability growth.



                   DOD partially concurred with all of our recommendations; however, for
Agency Comments    the most part, it found no further action was needed to lower total
and Our Response   ownership cost. We disagree. We believe that if DOD takes no further
                   action in implementing these recommendations, it ignores significant
                   opportunities to improve readiness and lower the total ownership cost of
                   its major weapon systems. The current budget environment demands more
                   effort in reducing these costs.

                   The performance of weapon systems as described in this report is
                   evidence that they demand much more money than planned to remain
                   ready. DOD should consider each of these recommendations as parts of a
                   whole solution for its “death spiral”—that is, the inability to modernize its
                   forces because the cost to operate and maintain unreliable weapon
                   systems at needed readiness rates constantly impinges on its
                   modernization budget. Taken as a whole, our recommendations encourage
                   DOD’s requirement setters to demand readiness at an affordable cost as a



                   Page 63                                               GAO-03-57 Best Practices
Chapter 5: Conclusions and Recommendations




part of a system’s performance, provide a mechanism to hold the product
developer accountable for determining the reliability needed to satisfy
DOD’s requirements, and provide contractual incentives for the product
developer to build reliability into a weapon system very early in its
development. The details of DOD’s response to each recommendation are
summarized below along with our rebuttal.

In a response prepared by the Office of the Joint Chiefs of Staff, DOD
partially concurred with our recommendation to include readiness and
total ownership cost as performance parameters equal in priority to any
others before beginning an acquisition program, commenting that they are
currently equal in priority to “non-key performance parameters.” We are
concerned that DOD does not recognize the importance of requiring
targets for a system’s readiness and its total ownership cost before
beginning product development. These targets are critical to providing a
realistic goal for the product developer to deliver reliable, cost-efficient
weapon systems.

We examined five deployed weapon systems for this report. None had key
requirements for readiness rates or operating costs. All had significant
problems with reliability and, therefore, readiness and total ownership
cost. We also reviewed several commercial products that were developed
with readiness and total ownership cost as critical requirements. In each
case, the products were ready to perform when needed at affordable and
predictable cost. Unless these requirements are equal in importance to any
others, they will not withstand the pressures of an acquisition program.
The Joint Chiefs stated that its requirements generation policy is currently
under revision and that no decision has been made about the priority
readiness and cost should have as requirements. We believe DOD has an
excellent opportunity to finally lower the total ownership cost of current
and future weapon acquisitions, thereby freeing significant funds for
modernization if it implements this recommendation.

In a response prepared by the Office of the Undersecretary of Defense for
Acquisition, Technology, and Logistics, DOD partially concurred with our
second recommendation to establish estimates of a weapon system’s
reliability—first, based on demonstrated reliability rates at the component
and subsystem level by the end of system integration, coincident with the
critical design review and next, at the system level at the time the
production decision is made—however, DOD found no need to revise the
policy governing the defense acquisition system to achieve this. We
disagree. Demonstrating reliability during product development has not
received the priority it requires if DOD is to have a realistic opportunity to


Page 64                                                GAO-03-57 Best Practices
Chapter 5: Conclusions and Recommendations




reduce the total ownership cost of its weapons while maintaining required
readiness levels. The current policy does not provide a mechanism to
ensure consistent application of reliability estimates based on
demonstrated performance. We believe that the disparity between the
actual costs to operate and maintain weapon systems and what DOD had
estimated those costs to be during the weapon systems’ development, as
described in the report, provides strong evidence that product developers
do not understand reliability under DOD’s current process.
Implementation of this recommendation will assist DOD in requiring
reliability estimates on a consistent basis.

In a response prepared by Undersecretary of Defense, Acquisition,
Technology, and Logistics to our recommendation that DOD structure its
development contracts to include requirements to provide incentives for
product developers to trade performance for reliability when it makes
sense, DOD found no need for additional incentives to contractors beyond
giving them total system performance responsibility. We have yet to see
evidence that total system performance responsibility has provided an
incentive for any product developer to trade performance for reliability in
order to reduce total ownership cost. Further, we believe a contractual
agreement similar to those we found in commercial cases—such as
financial penalties for readiness below certain specified rates—would
provide an excellent incentive for product developers to gain the
knowledge required to meet reliability rates early in a weapon system’s
design before committing to production.

DOD provided some technical comments in attachment 2. We have
addressed those in the report as necessary. The full text of the
department’s response to the recommendations is provided in appendix I.




Page 65                                              GAO-03-57 Best Practices
                   Appendix I: Comments from the Department
Appendix I: Comments from the
                   of Defense



Department of Defense




         Page 66                                              GAO-03-57 Best Practices
          Appendix I: Comments from the Department
          of Defense




Page 67                                              GAO-03-57 Best Practices
          Appendix I: Comments from the Department
          of Defense




Page 68                                              GAO-03-57 Best Practices
                  Appendix II: GAO Staff Acknowledgments
Appendix II: GAO Staff Acknowledgments


                  Cheryl Andrew, Beverly Breen, Belinda LaValle, Carol Mebane, Gary
Acknowledgments   Middleton, Michael Sullivan, Adam Vodraska, and Earl C. Woodard




                  Page 69                                           GAO-03-57 Best Practices
             Related GAO Products
Related GAO Products


             Best Practices: Capturing Design and Manufacturing Knowledge
             Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:
             July 15, 2002.

             Defense Acquisitions: DOD Faces Challenges in Implementing Best
             Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

             Best Practices: Better Matching of Needs and Resources Will Lead
             to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
             March 8, 2001.

             Defense Acquisitions: Higher Priority Needed for Army Operating and
             Support Cost Reduction Efforts. GAO/NSIAD-00-197. Washington, D. C.:
             September 29, 2000.

             Defense Acquisitions: Air Force Operating and Support Costs
             Reductions Need Higher Priority. GAO/NSIAD-00-165. Washington, D. C.:
             August 29, 2000.

             Best Practices: A More Constructive Test Approach Is Key to
             Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:
             July 31, 2000.

             Defense Logistics: Actions Needed to Enhance Success of Reengineering
             Initiatives. GAO/NSIAD-00-89. Washington, D. C.: June 23, 2000.

             Defense Acquisition: Employing Best Practices Can Shape Better
             Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:
             April 26, 2000.

             Best Practices: DOD Training Can Do More to Help Weapon
             System Programs Implement Best Practices. GAO/NSIAD-99-206.
             Washington, D.C.: August16, 1999.

             Best Practices: Better Management of Technology Development
             Can Improve Weapon System Outcomes. GAO/NSIAD-99-162.
             Washington, D.C.: July 30, 1999.

             Defense Acquisitions: Best Commercial Practices Can Improve Program
             Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

             Defense Acquisition: Improved Program Outcomes Are Possible.
             GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.


             Page 70                                          GAO-03-57 Best Practices
           Related GAO Products




           Best Practices: DOD Can Help Suppliers Contribute More to Weapon
           System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

           Best Practices: Successful Application to Weapon Acquisition Requires
           Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:
           February 24, 1998.

           Major Acquisitions: Significant Changes Underway in DOD’s
           Earned Value Management Process. GAO/NSIAD-97-108. Washington,
           D.C.: May 5, 1997.

           Best Practices: Commercial Quality Assurance Practices Offer
           Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:
           August 26, 1996.




(120092)
           Page 71                                           GAO-03-57 Best Practices
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548