oversight

Best Practices: Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from F/A-22 Program

Published by the Government Accountability Office on 2003-04-11.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                             United States General Accounting Office

GAO                          Testimony
                             Before the Subcommittee on National
                             Security, Emerging Threats, and
                             International Relations, Government Reform
                             Committee, House of Representatives
For Release on Delivery
Expected at 10:00 a.m. EDT
Friday, April 11, 2003       BEST PRACTICES
                             Better Acquisition
                             Outcomes Are Possible If
                             DOD Can Apply Lessons
                             from F/A-22 Program
                             Statement of David M. Walker,
                             Comptroller General of the United States




GAO-03-645T
                                               April 2003


                                               BEST PRACTICES

                                               Better Acquisition Outcomes Are
 Highlights of GAO-03-645T, a testimony
 before the Subcommittee on National
 Security, Emerging Threats, and
                                               Possible If DOD Can Apply Lessons From
 International Relations, Government
 Reform Committee, House of                    F/A-22 Program
 Representatives




Over the next 5 years, DOD’s                   GAO’s reviews of commercial best practices have identified key enablers to
overall investments are expected to            the success of product development programs and focused on how DOD can
average $150 billion a year to                 better leverage its investments by shortening the time it takes to field new
modernize and transition our                   capabilities at a more predictable cost and schedule. First, commercial firms
forces. In addition, DOD must                  use an approach that evolves a product to its ultimate capabilities on the
modernize its forces amid
competing demands for federal
                                               basis of mature technologies and available resources. This approach allows
funds, such as health care and                 only the product features and capabilities achievable with available
homeland security. Therefore, it is            resources in the initial development. Further product enhancements are
critical that DOD manage its                   planned for subsequent development efforts when technologies are proven
acquisitions in the most cost                  to be mature and other resources are available. Second, commercial firms
efficient and effective manner                 ensure that a high level of knowledge exists at key junctures during a
possible.                                      product’s development. The knowledge-based process includes three points:

DOD’s newest acquisition policy                    •    Before a program is launched, successful programs match customer
emphasizes the use of evolutionary,                     needs and available resources—technology, engineering knowledge,
knowledge-based concepts that
                                                        time, and funding.
have proven to produce more
effective and efficient weapon
systems outcomes. However, most                    •    About midway through development, the ability of the product’s
DOD programs currently do not                           design is demonstrated to be stable and meet performance
employ these practices and, as a                        requirements.
result, experience cost increases,
schedule delays, and poor product                  •    Before production begins, programs must show that a product can
quality and reliability.                                be manufactured within cost, schedule, and quality targets.
This testimony compares the best
practices for developing new
                                               In contrast, the F/A-22 program illustrates what can happen when a
products with the experiences of               major acquisition program is not guided by the principles of evolutionary,
the F/A-22 program.                            knowledge-based acquisition. When the program was started, several
                                               key technologies were not mature. Program managers proceeded
                                               through development without the requisite knowledge to effectively
                                               manage program risk and, at the start of production, key manufacturing
GAO is not making
recommendations in this                        processes were not under control. The F/A-22 program has undergone
testimony. However, in a number                significant cost increases. Instead of fielding early capabilities to the war
of prior reports, GAO has                      fighter, the development cycle has extended to 19 years, so far, and
recommended that DOD adopt                     original quantities have been significantly reduced, raising concerns
policies with metrics for                      about the capability the program will eventually deliver.
technology, design, and
manufacturing maturity to support              DOD recognizes the need to get better weapon system outcomes, and its
knowledge-based decision making.
These policies should apply when
                                               newest acquisition policy emphasizes the use of evolutionary,
making decisions on individual                 knowledge-based acquisition concepts proven to produce better
weapons programs.                              outcomes in developing new products. However, policy changes alone
                                               are not enough. Leadership commitment and attention to putting the
www.gao.gov/cgi-bin/getrpt?GAO-03-645T.        policy into practice for individual programs is needed to avoid the
To view the full report, including the scope   problems of the past. DOD will have many opportunities to do so over
and methodology, click on the link above.      the next several years with its force modernization investments.
For more information, contact Katherine
Schinasi at (202) 512-4841 or
schinasik@gao.gov.
Mr. Chairman and Members of the Subcommittee:

I am pleased to be here today to participate in the Subcommittee’s hearing
on how the Department of Defense (DOD) can—and must—get better
outcomes from its weapon system investments. DOD is on the threshold of
several major investments in acquisition programs that are likely to
dominate budget and doctrinal debates well into the next decade. These
programs include such systems as the Missile Defense Agency’s suite of
land, sea, air, and space defense systems; the Army’s Future Combat
Systems; and the Air Force’s and Navy’s Joint Strike Fighter. Over the next
5 years, DOD’s overall investments are expected to average $150 billion a
year as DOD works to keep legacy systems as well as modernize and
transform our national defense capabilities for the future. Therefore, to
meet these challenges, it is essential that sound foundations for these and
other weapon system investments be laid now so that the resulting
programs can be executed within estimates of available resources.

Any discussion of improvements to DOD’s modernization efforts must be
set in the context of overall expected budget availability. There are
important competing priorities. Health care costs are growing at double-
digit rates, and spending on homeland security will likely grow as we seek
to defeat terrorism worldwide. We face an oncoming demographic tidal
wave, and by 2035 the number of people who are 65 or older will have
doubled, creating much larger demands on the federal budget. The
demand of funding for entitlement programs continues to grow, creating
increasing pressures on discretionary funding for other federal priorities
like education and defense. Therefore, it is critical that DOD manage its
acquisitions in the most cost efficient and effective manner possible.

My testimony today is about improving the outcomes of major weapon
system acquisitions by using best practices to capture and use the right
product knowledge at the right time for better decision making during
product development. As per your request, I will compare acquisition
practices and decisions made for the F/A-22 with these best practices for
developing new products. The divergence between F/A-22 experiences and
best product development practices, we believe, largely explains why the
F/A-22 has been in development for over 16 years and its cost has grown
substantially. It is also a primary contributor to other performance issues
that are currently faced by the program. My testimony will also include
observations on what can be done at this time to limit further negative
outcomes in the F/A-22 program. Lastly, I will discuss the need for
enforcing DOD’s newest acquisition policy, which on paper embraces best



Page 1                                            GAO-03-645T Best Practices
                          practices but in operation does not always do so, if DOD really expects to
                          get improved outcomes in its major weapon system acquisitions.


                          Clearly, the acquisition process has produced superior weapons, but it
Improving Major           does so at a high price. Weapon systems routinely take much longer time
Weapon System             to field, cost more to buy, and require more support than investment plans
                          provide for. These consequences reduce the buying power of the defense
Acquisition Outcomes      dollar, delay capabilities for the war fighter, and force unplanned—and
                          possibly unnecessary—trade-offs in desired acquisition quantities and an
                          adverse ripple effect among other weapons programs or defense needs.
                          Because of the lengthy time to develop new weapons, many enter the field
                          with outdated technologies and a diminished supply base needed for
                          system support. Frequently, this requires upgrades to the capability as
                          soon as the new system is fielded. As previously noted, these inefficiencies
                          have often led to reduced quantities of new systems. In turn, legacy
                          systems remain in the inventory for longer periods, requiring greater
                          operations and support cost that pull funds from other accounts, including
                          modernization. DOD is facing these problems with its tactical air force
                          assets now. We believe DOD can learn lessons from the experiences with
                          the F/A-22 program as it frames the acquisition environment for its many
                          transformational investments.

                          DOD recognizes the need to get better weapon system outcomes, and its
                          newest acquisition policy emphasizes the use of evolutionary, knowledge-
                          based acquisition concepts proven to produce more effective and efficient
                          outcomes in developing new products. It incorporates the elements of a
                          knowledge-based acquisition model for developing new products, which
                          we have recommended in our reviews of commercial best practices. Our
                          body of work focuses on how DOD can better leverage its investments by
                          shortening the time it takes to field new capabilities at a more predictable
                          cost and schedule. However, policy changes alone will not guarantee
                          success. Unless written policies are consistently implemented in practice
                          through timely and informed decisions on individual programs, outcomes
                          will not change. This requires sustained leadership and commitment and
                          attention to the capture and use of key product knowledge at critical
                          decision points to avoid the problems of the past.


The Case for an           A key enabler to the success of commercial firms is using an approach that
Evolutionary Product      evolves a product to its ultimate capabilities on the basis of mature
Development Environment   technologies and available resources. This approach allows commercial
                          companies to develop and produce more sophisticated products faster and

                          Page 2                                             GAO-03-645T Best Practices
less expensively than their predecessors. Commercial companies have
found that trying to capture the knowledge required to stabilize the design
of a product that requires significant amounts of new technical content is
an unmanageable task, especially if the goal is to reduce development
cycle times and get the product to the marketplace as quickly as possible.
Therefore, product features and capabilities not achievable in the initial
development are planned for subsequent development efforts in future
generations of the product, but only when technologies are proven to be
mature and other resources are available. DOD’s new policy embraces the
idea of evolutionary acquisition. Figure 1 compares evolutionary and
single step (“big bang”) acquisitions.




Page 3                                             GAO-03-645T Best Practices
Figure 1: Comparison of Evolutionary and Big Bang Acquisition Approaches




                                       Page 4                              GAO-03-645T Best Practices
                          An evolutionary environment for developing and delivering new products
                          reduces risks and makes cost more predictable. While the customer may
                          not receive an ultimate capability initially, the product is available sooner,
                          with higher quality and reliability, and at lower, more predictable cost.
                          Improvements are planned for future generations of the product.


The Case for Knowledge-   Leading commercial firms expect that their program managers will deliver
Based Product             high-quality products on time and within budgets. Doing otherwise could
Development Process       result in losing a customer in the short term and losing the company in the
                          longer term. Thus, in addition to creating an evolutionary product
                          development environment that brings risk in control, these firms have
                          adopted practices that put their individual program managers in a good
                          position to succeed in meeting these expectations on individual products.
                          Collectively, these practices ensure that a high level of knowledge exists
                          about critical facets of the product at key junctures during its
                          development. Such a knowledge-based process enables decision makers to
                          be reasonably certain about critical facets of the product under
                          development when they need to be.

                          The knowledge-based process followed by leading firms is shown in detail
                          in table 1, but in general can be broken down into three knowledge points.
                          First, a match must be made between the customer’s needs and the
                          available resources—technology, engineering knowledge, time, and
                          funding—before a program is launched. Second, a product’s design must
                          demonstrate its ability to meet performance requirements and be stable
                          about midway through development. Third, the developer must show that
                          the product can be manufactured within cost, schedule, and quality targets
                          and is demonstrated to be reliable before production begins. The following
                          table illustrates more specifically what we have learned about how
                          successful programs gather knowledge as they move through product
                          development.




                          Page 5                                               GAO-03-645T Best Practices
Table 1: Highlights of Specific Best Practices for Acquisitions

 Knowledge Point 1 (Should occur before program launch)
 Separate technology from product development.

 Have clear measures and high standards for assessing technology maturity—technology readiness levels.

 Use a disciplined systems engineering process for translating and balancing customer’s desires with product developer’s technology,
 design, and production limitations; in other words, bring the right knowledge to the table when laying down a program’s foundation.

 Identify the mismatches between desired product features and the product developer’s knowledge and either (1) delay the start of the
 new product development until knowledge deficit can be made up or (2) reduce product features to lessen their dependence on areas
 where knowledge is insufficient (evolutionary acquisition). The main opportunities for trading off design features to save time and
 money occur here, before a program is started.

 When do you know you have achieved this knowledge point? When technologies needed to meet essential product requirements
 have been demonstrated to work in their intended environment and the producer has completed a preliminary design of the product.

 Knowledge Point 2 (Should occur midway between system integration and demonstration)
 Hold a major decision review between system integration and system demonstration that determines that the product design is stable
 and includes specific criteria to move into the system demonstration phase.

 Use integrated engineering prototypes to demonstrate design stability and prove with testing that the design meets the customer’s
 requirements. It is important that this happen before initial manufacturing begins—a point when investments are increased to produce
 an item.

 Identify critical manufacturing processes and establish a plan to bring these under statistical control by the start of production; also
 establish reliability goals and a growth plan to achieve these by production. This facilitates the achievement of process control and
 reliability goals at the completion of knowledge point 3.

 When do you know you have achieved this knowledge point? When 90 percent of engineering drawings are releasable to
 manufacturing organizations. Drawings are the language used by engineers to communicate to the manufacturers the details of the
 new product—what it looks like, how its components interface, how to build it and the critical materials and processes needed to
 fabricate it. This makes drawings a key measure of whether the design is stable or not.

 Knowledge Point 3 (Should occur before production)
 Demonstrate that all critical manufacturing processes are under statistical control and consistently producing items within the quality
 standards and tolerances for the overall product before production begins. This is important, since variation in one process can
 reverberate to others and result in defective parts that need to be repaired or reworked.

 Demonstrate product reliability before the start of production. This requires testing to identify the problems, design corrections, and
 retest the new design. Commercial firms consider reliability important and its achievement a measure of design maturity.

 When do you know you have achieved this knowledge point? When all key manufacturing processes have come under statistical
 control and product reliability has been demonstrated.




                                               Page 6                                                          GAO-03-645T Best Practices
DOD programs often do not employ these practices. We found that if the
evolutionary, knowledge-based acquisition concepts were not applied, a
cascade of negative effects became magnified in the product development
and production phases of an acquisition program. These led to acquisition
outcomes that included significant cost increases and schedule delays,
poor product quality and reliability, and delays in getting new capability to
the war fighter. This is often the case in DOD programs as shown in our
past work on systems like F/A-22 fighter, C-17 airlifter, V-22 tiltrotor
aircraft, PAC-3 missile, BAT antitank munition, and others. We did find
some DOD programs that employed best practice concepts and have had
more successful program outcomes to date. These included the Global
Hawk unmanned vehicle, AIM-9X missile, and Joint Direct Attack
Munitions guided bomb. Figure 3 shows a notional illustration of the
different paths and effects of a product development.




Page 7                                              GAO-03-645T Best Practices
Figure 2: Different Paths That A Product’s Development Can Take




                                        It is clear that knowledge about the product’s technology, design, and
                                        processes captured at the right time can reduce development cycle times
                                        and deliver a more cost effective, reliable product to the customer sooner
                                        than programs that do not capture this knowledge.

                                        In applying the knowledge-based approach, the most leveraged decision
                                        point of the three, is matching the customer’s needs with the developer’s
                                        resources—technology, design, timing, and funding. This initial decision


                                        Page 8                                             GAO-03-645T Best Practices
sets the stage for the eventual outcome—desirable or problematic. The
match is ultimately achieved in every development program, but in
successful development programs, it occurs prior to program launch. In
successful programs, negotiations and trade-offs occur before a product
development is launched to ensure that a match exists between customer
expectations and developer resources. The results achieved from this
match are balanced and achievable requirements, sufficient investment to
complete the development, and a firm commitment to deliver the product.
Commercial companies we have visited usually limit product development
cycle-time to less than 5 years.

In DOD, this match is seldom achieved. It is not unusual for DOD to
bypass early trade-offs and negotiations, instead planning to develop a
product based on a rigid set of requirements that are unachievable within
a reasonable development time frame. This results in cost and schedule
commitments that are unrealistic. Although a program can take as long as
15 years in DOD, the program manager is expected to develop and be
accountable for precise cost and schedule estimates made at the start of
the program. Because of their short tenures, it normally takes several
program managers to complete product development. Consequently, the
program manager that commits to the cost and schedule estimate at the
beginning of the program is not the same person responsible for achieving
it. Therefore, program accountability is problematic. Ironically, this
outcome is rational in the traditional acquisition environment. The
pressures put on program managers to get programs approved encourage
promising more than can be delivered for the time and money allotted.
They are not put in a position to succeed.

The differences in the practices employed by successful commercial firms
and DOD reflect the different demands imposed on programs by the
environments in which they are managed. Specific practices take root and
are sustained because they help a program succeed in its environment.
The way success and failure are defined for commercial and defense
product developments differs considerably, which creates a different set
of incentives and evokes different behaviors from managers. Attempts at
reforming weapon system acquisitions have not succeeded because they
did not change these incentives. All of the participants in the acquisition
process play a part in creating incentives. The F/A-22 program, advertised
as a flagship of acquisition reform in its early days, failed to establish this
match before program launch and today we are discussing the resulting
outcomes to-date.




Page 9                                                GAO-03-645T Best Practices
                          The F/A-22 provides an excellent example of what can happen when a
F/A-22 Did Not            major acquisition program is not guided by the principles of evolutionary,
Employ Evolutionary       knowledge-based acquisition. The program failed to match requirements
                          with resources and make early trade-offs and took on a number of new
or Knowledge-Based        and unproven technologies. Instead of fielding early capability and then
Process                   evolving the product to get new capabilities to the war fighter sooner, the
                          Air Force chose a “big bang” product development approach that is now
                          planned to take about 19 years. This created a challenging and risky
                          acquisition environment that has delayed the war fighter the capabilities
                          expected from this new aircraft. Program leaders did not capture the
                          specific knowledge identified as key for each of the three critical
                          knowledge points in product development. Instead, program managers
                          proceeded through the F/A-22’s development without the requisite
                          knowledge necessary for reducing program risk and achieving more
                          successful program outcomes. Now the optimism underlying these
                          decisions has resulted in significant cost increases, schedule delays, trade-
                          offs—making do with less than half the number of originally desired
                          aircraft—and concerns about the capability to be delivered.


F/A-22 Program Outcomes   Since the F/A-22 acquisition program was started in October 1986, the F/A-
                          22 cost and schedule estimates have grown significantly to where, today,
                          the Air Force estimates the total acquisition unit cost of a single aircraft is
                          $257.5 million.1 This represents a 74 percent increase from the estimate at
                          the start of development and a commensurate loss in the buying power of
                          the defense dollar. Intended to replace the aging F-15 fighter, the F/A-22
                          program is now scheduled to reach its initial operational capability in
                          December 2005—making its development cycle about 19 years. During this
                          cycle, the planned buy quantity has been reduced 63 percent from 750 to
                          276 aircraft2. In addition, since fiscal year 2001, funding for F/A-22
                          upgrades has dramatically increased from $166 million to $3.0 billion, most
                          of which is to provide increased ground attack capability, a requirement
                          that was added late in the development program.




                          1
                           All references to F/A-22 costs in this testimony are in then-year dollars in order to
                          maintain consistent reporting with our prior reports on the F/A-22.
                          2
                           Between 1986 and the start of engineering and manufacturing development in 1991, the
                          quantity was reduced from 750 to 648 aircraft.



                          Page 10                                                         GAO-03-645T Best Practices
F/A-22 Did Not Use                       The F/A-22 acquisition strategy from the outset was to achieve full
Evolutionary Acquisition                 capability in a “big bang” approach. By not using an evolutionary
or Capture Knowledge                     approach, the F/A-22 took on significant risk and onerous technological
                                         challenges. While the big bang approach may have allowed the Air Force
Required at Key Decision                 to more successfully compete for early funding, it hamstrung the program
Junctures                                with many new undemonstrated technologies, preventing the program
                                         from knowing cost and schedule ramifications throughout development.
                                         Cost, schedule, and performance problems resulted. The following table
                                         summarizes the F/A-22 program’s attainment of critical knowledge and key
                                         decision junctures during the development program and the changes in
                                         development cost and cycle time at each point.

Table 2: Knowledge Attainment in the F/A-22 Program

                         Program start—1986                  Design review—1995                  Production start—2001

Best practice            Attain knowledge point 1.           Attain knowledge point 2. 90        Attain knowledge point 3. 100% of
                         Separate technology and product     percent of systems and              critical manufacturing processes in
                         development, deliver mature         structures engineering drawings     statistical control and reliability
                         technology, and have preliminary    releasable and subsystem            goals demonstrated.
                         design.                             design reviews completed.


F/A-22 practice          Knowledge point 1 not attained.     Knowledge points 1 and 2 not        Knowledge point 3 not attained.
                         Failed to separate technology       attained. Only 26 percent of        Less than 50 percent of critical
                         and product development. Three      drawings released at the critical   manufacturing processes in control.
                         critical technologies immature:     design review in February 1995.     Only 22 percent of reliability goal
                         Low-observable materials,           Knowledge point 2 not attained      demonstrated with many
                         propulsion, and integrated          until September 1998, after         outstanding deficiencies.
                         avionics. Knowledge point 1 not     delivery of second test aircraft.
                         attained until September 2000.

F/A-22’s estimated
                 a
development cost         $12.6 billion                       $21.2 billion                       $28.7 billion
                                                             (68 percent increase)               (128 percent increase)

Estimated development
cycle time               9.4 years                           18.1 years                          19.2 years
                                                             (93 percent increase)               (104 percent increase)
                                         a
                                         The development estimate includes all F/A-22 RDT&E costs.


                                         Technology—The F/A-22 did not have mature technology at the start of
                                         the acquisition program. The program included new low-observable
                                         (stealth) materials, integrated avionics, and propulsion technology that
                                         were not mature at this time. The Air Force did not complete an evaluation
                                         of stealth technology on a full-scale model of the aircraft until several
                                         years into development. It was not until September 2000, or 9 years into

                                         Page 11                                                       GAO-03-645T Best Practices
development, that the integrated avionics reached a maturity level
acceptable to begin product development. During development, the
integrated avionics was a source of schedule delays and cost growth.
Since 1997, avionics software development and flight-testing have been
delayed, and the cost of avionics development has increased by over $980
million dollars. Today, the avionics still has problems affecting the ability
to complete developmental testing and begin operational testing, and the
Air Force cannot predict when a solution will be found.

Design—The effects of immature technologies cascaded into the F/A-22
development program, making it more difficult to achieve a stable design
at the right time. The standard measure of design stability is 90 percent of
design drawings releasable by the critical design review. The F/A-22
achieved only 26 percent by this review, taking an additional 43 months to
achieve the standard. Moving ahead in development, the program
experienced several design and manufacturing problems described by the
F/A-22 program office as a “rolling wave” effect throughout system
integration and final assembly. These effects included numerous design
changes, labor inefficiencies, parts shortages, out of sequence work, cost
increases, and schedule delays.

Production—At the start of production, the F/A-22 did not have
manufacturing processes under control and was only beginning testing
and demonstration efforts for system reliability. Initially, the F/A-22 had
taken steps to use statistical process control data to gain control of critical
manufacturing processes by full rate production. However, the program
abandoned this best practice approach in 2000 with less than 50 percent of
its critical manufacturing processes in control. In March 2002,3 we
recommended that the F/A-22 program office monitor the status of critical
manufacturing process as the program proceeds toward high rate
production.

The reliability goal for the F/A-22 is 3 hours of flying time between
maintenance actions. The Air Force estimated that in late 2001, when it
entered production, it should have been able to demonstrate almost 2
flying hours between maintenance actions. Instead, it could fly an average
of only 0.44 hours between maintenance actions. Since then there has



3
 U.S. General Accounting Office, Tactical Aircraft: F-22 Delays Indicate Initial
Production Rates Should Be Lower to Reduce Risks, GAO-02-298 (Washington, D.C.: Mar.
5, 2002).



Page 12                                                   GAO-03-645T Best Practices
                                  been a decrease in reliability. As of November 2002, development test
                                  aircraft have been completing only 0.29 hours between maintenance
                                  actions. Additionally, the program was slow to fix and correct problems
                                  that had affected reliability. At the time of our review in July 2002,
                                  program officials had identified about 260 different types of failures and
                                  had identified fixes for less than 50 percent of the failures. To achieve
                                  reliability goals will require additional design changes, testing, and
                                  modifications. Therefore, additional problems and costs can be expected if
                                  the system is fielded with the level of reliability achieved to date.


It Is Too Late for the F/A-       The F/A-22 did not take advantage of evolutionary, knowledge-based
22 Program to Gain Full           concepts up front and now, the best it can hope for is to limit cost
Benefit of a Knowledge-           increases and performance problems by not significantly increasing its
                                  production until development is complete—signified by developmental
Based Process                     and operational testing and reliability demonstrations. To that end, we
                                  have recommended that the Air Force reconsider its decision to increase
                                  the aircraft production rate beyond 16 aircraft per year.4 The program is
                                  nearing the end of developmental testing and plans to start initial
                                  operational testing in October 2003. If developmental testing goes as
                                  planned, which is not guaranteed, operational testing is expected to be
                                  completed around September 2004. By the end of this fiscal year, 51 F/A-
                                  22s will be on contract as low rate production began in 2001.

                                  Our March 2003 report identifies various problems still outstanding that
                                  could have further impacts on cost, schedule, and delivered performance
                                  that are in addition to undemonstrated reliability goals. The problems
                                  identified are of particular concern, given Air Force plans to increase
                                  production rates and make a full rate production decision in 2004. The
                                  problems include:

                              •   unexpected shutdowns (instability) of the avionics,
                              •   excessive movement of the vertical tails,
                              •   overheating in rear portions of the aircraft,
                              •   separations of the horizontal tail material,
                              •   inability to meet airlift support requirements, and
                              •   excessive ground maintenance actions.



                                  4
                                   U.S. General Accounting Office, Tactical Aircraft: DOD Should Reconsider Decision to
                                  Increase F/A-22 Production Rates While Development Risks Continue, GAO-03-431
                                  (Washington, D.C.: Mar. 14, 2003).



                                  Page 13                                                    GAO-03-645T Best Practices
These problems are still being addressed, and not all of them have been
solved as yet. For example, Air Force officials stated they do not yet
understand the problems associated with the avionics instability well
enough to predict when they would be able to resolve them, and certain
tests to better understand the vertical tail problem have not yet begun.
Despite remaining testing and outstanding problems, the Air Force plans
to continue acquiring production aircraft at increasing annual rates and
make the full rate production decision in 2004. This is a very risky strategy,
given outstanding issues in the test program and the system’s less than
expected reliability. The Air Force may encounter higher production costs
as a result of acquiring significant quantities of aircraft before adequate
testing and demonstrations are complete. In addition, remaining testing
could identify problems that require costly modifications in order to
achieve satisfactory performance.

In a February 28, 2003 report to Representative John Tierney,5 we found
that F/A-22 production costs are likely to increase more than the latest
$5.4 billion cost growth recently estimated by the Air Force and the Office
of Secretary of Defense (OSD). First, the current OSD production estimate
does not include $1.3 billion included in the latest Air Force acquisition
plan. Second, schedule delays in developmental testing could further
postpone the start of the first F/A-22 multiyear contract, which has already
been delayed until fiscal year 2006. This could result in lower cost savings
from multiyear procurement. Last, we found several risk factors that may
increase future production costs, including the dependency of certain cost
reduction plans on Air Force investments that are not being made to
improve production processes, the availability of funding, and a reduction
in funding for support costs. In addition, DOD has not informed Congress
about the quantity of aircraft that can be procured within existing
production cost limits, which we believe could be fewer than the 276
currently planned. Further details on F/A-22 cost growth and the Air
Force’s attempt to offset it are provided in appendix I.




5
 U.S. General Accounting Office, Tactical Aircraft: DOD Needs to Better Inform Congress
about Implications of Continuing F/A-22 Cost Growth, GAO-03-280 (Washington, D.C.:
Feb. 28, 2003).



Page 14                                                    GAO-03-645T Best Practices
                       While DOD’s new acquisition policy is too late to influence the F/A-22
Real Change in         program, it is not too late for other major acquisition programs like the
Acquisition Outcomes   Missile Defense Agency’s suite of land, sea, air, and space defense
                       systems; the Army’s Future Combat Systems; and the Air Force and Navy’s
Requires Disciplined   Joint Strike Fighter. DOD’s revised acquisition policy represents tangible
Enforcement of         leadership action to getting better weapon system acquisition outcomes,
                       but unless the policies are implemented through decisions on individual
Acquisition Policy     programs, outcomes are not likely to change. Further, unless pressures are
                       alleviated in DOD to get new acquisition programs approved and funded
                       on the basis of requirements that must stand out, programs will continue
                       to be compromised from the outset with little to no chance of successful
                       outcomes. If new policies were implemented properly, through decisions
                       on individual programs, managers would face less pressure to promise
                       delivery of all the ultimate capabilities of a weapon system in one “big
                       bang.”

                       Both form and substance are essential to getting desired outcomes. At a
                       tactical level, we believe that the policies could be made more explicit in
                       several areas to facilitate such decisions. First, the regulations provide
                       little or no controls at key decision points of an acquisition program that
                       force a program manager to report progress against knowledge-based
                       metrics. Second, the new regulations, once approved, may be too general
                       and may no longer provide mandatory procedures. Third, the new
                       regulations may not provide adequate accountability because they may not
                       require knowledge-based deliverables containing evidence of knowledge
                       at key decision points.

                       At a strategic level, some cultural changes will be necessary to translate
                       policy into action. At the very top level, this means DOD leadership will
                       have to take control of the investment dollars and to say “no” in some
                       circumstances if programs are inappropriately deviating from sound
                       acquisition policy. In my opinion, programs should follow a knowledge-
                       based acquisition policy—one that embraces best practices—unless there
                       is a clear and compelling national security reason not to. Other cultural
                       changes instrumental to implementing change include:

                   •   Keeping key people in place long enough so that they can affect decisions
                       and be held accountable.
                   •   Providing program offices with the skilled people needed to craft
                       acquisition approaches that implement policy and to effectively oversee
                       the execution of programs by contractors.




                       Page 15                                           GAO-03-645T Best Practices
              •   Realigning responsibilities and funding between science and technology
                  organizations and acquisition organizations to enable the separation of
                  technology development from product development.
              •   Bringing discipline to the requirements-setting process by demanding a
                  match between requirements and resources.
              •   Requiring readiness and operating cost as key performance parameters
                  prior to beginning an acquisition.
              •   Designing and implementing test programs that deliver knowledge when
                  needed, including reliability testing early in design.

                  Ultimately, the success of the new acquisition policy will be seen in
                  individual program and resource decisions. Programs that are
                  implementing knowledge-based policies in their acquisition approaches
                  should be supported and resourced, presuming they remain critical to
                  national needs and affordable within current and projected resource
                  levels. Conversely, if programs that repeat the approaches of the past are
                  approved and funded, past policies—and their outcomes—will be
                  reinforced with a number of adverse implications.


                  DOD will continue to face challenges in modernizing its forces with new
Conclusions       demands on the federal dollar created by changing world conditions.
                  Consequently, it is incumbent upon DOD to find and adopt best product
                  development practices that can allow it to manage its weapon system
                  programs in the most efficient and effective way. Success over the long
                  term will depend not only on policies that embrace evolutionary,
                  knowledge-based acquisition practices but also on DOD leadership’s
                  sustaining its commitment to improving business practices and ensuring
                  that those adopted are followed and enforced.

                  DOD’s new acquisition policy embraces the best practice concepts of
                  knowledge-based, evolutionary acquisition and represents a good first step
                  toward achieving better outcomes from major acquisition programs. The
                  F/A-22 program followed a different path at its beginning, a big bang, high-
                  risk approach whose outcomes so far have been increased cost, quality
                  and reliability problems, growing procurement reductions, and delays in
                  getting the aircraft to the war fighter. Since this program is nearing the end
                  of development and already into production, it is too late to adopt a
                  knowledge approach, but it can limit further cost increases and adverse
                  actions by not ramping up production beyond current levels until
                  developmental and operational testing are completed and reliability goals
                  have been demonstrated. Regardless of the F/A-22’s current predicament,
                  the new policy can and should be used to manage all new acquisition


                  Page 16                                             GAO-03-645T Best Practices
programs and should be adapted to those existing programs that have not
progressed too far in development to benefit. At a minimum, the F/A-22
should serve as a lesson learned from which to effect a change in the
future DOD acquisition environment. The costs of doing otherwise are
simply too high for us to tolerate.


Mr. Chairman, this concludes my prepared statement. I would be happy to
respond to any questions that you or other members of the Subcommittee
may have.




Page 17                                         GAO-03-645T Best Practices
Appendix I: F/A-22 Production Cost Growth


                          Over the last 6 years, DOD has identified about $18 billion in estimated
                          production cost growth during the course of two DOD program reviews.
                          As a result, the estimated cost of the production program currently
                          exceeds the congressional cost limit. The Air Force has implemented cost
                          reduction plans designed to offset a significant amount of this estimated
                          cost growth. But the effectiveness of these cost reduction plans has varied.

                          During a 1997 review, the Air Force estimated cost growth of $13.1 billion.1
                          The major contributing factors to this cost growth were inflation,
                          increased estimates of labor costs and materials associated with the
                          airframe and engine, and engineering changes to the airframe and engine.
                          These factors made up about 75 percent of the cost growth identified in
                          1997.

                          In August 2001, DOD estimated an additional $5.4 billion in cost growth for
                          the production of the F/A-22, bringing total estimated production cost to
                          $43 billion. The major contributing factors to this cost growth were again
                          due to increased labor costs and airframe and engine costs. These factors
                          totaled almost 70 percent of the cost growth. According to program
                          officials, major contractors’ and suppliers’ inability to achieve the
                          expected reductions in labor costs throughout the building of the
                          development and early production aircraft has been the primary reason for
                          estimating this additional cost growth.

Mixed Success With Cost   The Air Force was able to implement cost reduction plans and offset cost
Reduction Plans           growth by nearly $2 billion in the first four production contracts awarded.
                          As shown in table 3, the total offsets for these contracts slightly exceeded
                          earlier projections by about $.5 million.




                          1
                              Based on a plan to procure 438 aircraft.



                          Page 18                                             GAO-03-645T Best Practices
Table 3: Comparison of Planned Versus Implemented Cost Reduction Offsets for
Awarded Production Contracts

    Dollars in millions
                                                 Planned     Implemented
    Production lot                                offsets         offsets      Difference
    Fiscal year 1999 (2 aircraft)                  $199.0          $200.5             $1.5
    Fiscal year 2000 (6 aircraft)                   329.3           336.4              7.1
    Fiscal year 2001 (10 aircraft)                  580.2           611.1             30.9
    Fiscal year 2002 (13 aircraft)                  827.2           788.2           (39.0)
    Total                                        $1,935.7        $1,936.2             $0.5
Source: Air Force.



Cost reduction plans exist but have not yet been implemented for
subsequent production lots planned for fiscal years 2003 through 2010
because contracts for these production lots have not yet been awarded. If
implemented successfully, the Air Force expects these cost reduction
plans to achieve billions of dollars in offsets to estimated cost growth and
to allow the production program to be completed within the current
production cost estimate of $43 billion.2 However, this amount exceeds the
production cost limit of $36.8 billion.

In addition, while the Air Force has been attempting to offset costs
through production improvement programs (PIPs), recent funding
cutbacks for PIPs may reduce their effectiveness. PIPs focus specifically
on improving production processes to realize savings by using an initial
government investment. The earlier the Air Force implements PIPs, the
greater the impact on the cost of production. Examples of PIPs previously
implemented by the Air Force include manufacturing process
improvements for avionics, improvements in fabrication and assembly
processes for the airframe, and redesign of several components to enable
lower production costs.

As shown in figure 3, the Air Force reduced the funding available for
investment in PIPs by $61 million for lot 1 and $26 million for lot 2 to cover




2
 The F/A-22 President’s budget for fiscal year 2004 would transfer $876 million in
production funding to help fund estimated cost increases in development. As a result, the
current production cost estimate is $42.2 billion.



Page 19                                                      GAO-03-645T Best Practices
cost growth in production lots 1 and 23. As a result, it is unlikely that PIPs
covering these two lots will be able to offset cost growth as planned.

Figure 3: Planned Versus Actual F/A-22 Production Improvement Program
Investment for Production Lots 1 (Fiscal Year 2001) and 2 (Fiscal Year 2002)




Figure 4 shows the remaining planned investment in PIPs through fiscal
year 2006 and the $3.7 billion in estimated cost growth that can potentially
be offset through fiscal year 2010 if the Air Force invests as planned in
these PIPs.




3
  Production lot 1 was awarded in fiscal year 2001 and production lot 2 was awarded in
fiscal year 2002.



Page 20                                                      GAO-03-645T Best Practices
Figure 4: Planned Offsets to Cost Growth From Investing in and Implementing PIPs




In the past, Congress has been concerned about the Air Force’s practice of
requesting fiscal year funding for these PIPs but then using part of that
funding for F/A-22 airframe cost increases. 4 Recently, Congress directed
the Air Force to submit a request if it plans to use PIP funds for an
alternate purpose.




4
    Report 107-298, Nov. 19, 2001.




Page 21                                               GAO-03-645T Best Practices
Related GAO Products


             Best Practices: Setting Requirements Differently Could Reduce Weapon
             Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February
             11, 2003.

             Best Practices: Capturing Design and Manufacturing Knowledge Early
             Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,
             2002.

             Defense Acquisitions: DOD Faces Challenges in Implementing Best
             Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

             Best Practices: Better Matching of Needs and Resources Will Lead to
             Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,
             2001.

             Best Practices: A More Constructive Test Approach Is Key to Better
             Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,
             2000.

             Defense Acquisitions: Employing Best Practices Can Shape Better
             Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April
             26, 2000.

             Best Practices: DOD Training Can Do More to Help Weapon System
             Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,
             D.C.: August 16, 1999.

             Best Practices: Better Management of Technology Development Can
             Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,
             D.C.: July 30, 1999.

             Best Practices: DOD Can Help Suppliers Contribute More to Weapon
             System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

             Best Practices: Successful Application to Weapon Acquisition Requires
             Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:
             February 24, 1998.

             Best Practices: Commercial Quality Assurance Practices Offer
             Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,
             1996.




             Page 22                                         GAO-03-645T Best Practices
Tactical Aircraft: Status of the F/A-22 Program. GAO-03-603T.
Washington, D.C.: April 2, 2003.

Tactical Aircraft: DOD Should Reconsider Decision to Increase F/A-22
Production Rates While Development Risks Continue. GAO-03-431.
Washington, D.C.: March 14, 2003.

Tactical Aircraft: DOD Needs to Better Inform Congress about
Implications of Continuing F/A-22 Cost Growth. GAO-03-280.
Washington, D.C.: February 28, 2003.

Tactical Aircraft: F-22 Delays Indicate Initial Production Rates Should
Be Lower to Reduce Risks. GAO-02-298. Washington, D.C.: March 5, 2002.

Tactical Aircraft: Continuing Difficulty Keeping F-22 Production Costs
within the Congressional Limitation. GAO-01-782. Washington, D.C.: July
16, 2001.

Tactical Aircraft: F-22 Development and Testing Delays Indicate Need
for Low-Rate Production. GAO-01-310. Washington, D.C.: March 15, 2001.

Defense Acquisitions: Recent F-22 Production Cost Estimates Exceeded
Congressional Limitation. GAO/NSIAD-00-178. Washington, D.C.: August
15, 2000.

Defense Acquisitions: Use of Cost Reduction Plans in Estimating F-22
Total Production Costs. GAO/T-NSIAD-00-200. Washington, D.C.: June 15,
2000.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal
Year 2001. GAO/OCG-00-8. Washington, D.C.: March 31, 2000.

F-22 Aircraft: Development Cost Goal Achievable If Major Problems Are
Avoided. GAO/NSIAD-00-68. Washington, D.C.: March 14, 2000.

Defense Acquisitions: Progress in Meeting F-22 Cost and Schedule Goals.
GAO/T-NSIAD-00-58. Washington, D.C.: December 7, 1999.

Fiscal Year 2000 Budget: DOD’s Procurement and RDT&E Programs.
GAO/NSIAD-99-233R. Washington D.C.: September 23, 1999.

Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal
Year 2000. GAO/OCG-99-26. Washington, D.C.: April 16, 1999.

Page 23                                          GAO-03-645T Best Practices
           Defense Acquisitions: Progress of the F-22 and F/A-18E/F Engineering
           and Manufacturing Development Programs. GAO/T-NSIAD-99-113.
           Washington, D.C.: March 17, 1999.

           F-22 Aircraft: Issues in Achieving Engineering and Manufacturing
           Development Goals. GAO/NSIAD-99-55. Washington, D.C.: March 15, 1999.

           F-22 Aircraft: Progress of the Engineering and Manufacturing
           Development Program. GAO/T-NSIAD-98-137. Washington D.C.: March 25,
           1998.

           F-22 Aircraft: Progress in Achieving Engineering and Manufacturing
           Development Goals. GAO/NSIAD-98-67. Washington, D.C.: March 10, 1998.

           Tactical Aircraft: Restructuring of the Air Force F-22 Fighter Program.
           GAO/NSIAD-97-156. Washington, D.C.: June 4, 1997.

           Defense Aircraft Investments: Major Program Commitments Based on
           Optimistic Budget Projections. GAO/T-NSIAD-97-103. Washington, D.C.:
           March 5, 1997.

           F-22 Restructuring. GAO/NSIAD-97-100BR. Washington, D.C.: February
           28, 1997.

           Tactical Aircraft: Concurrency in Development and Production of F-22
           Aircraft Should Be Reduced. GAO/NSIAD-95-59. Washington, D.C.: April
           19, 1995.

           Tactical Aircraft: F-15 Replacement Issues. GAO/T-NSIAD-94-176.
           Washington, D.C.: May 5, 1994.

           Tactical Aircraft: F-15 Replacement Is Premature as Currently Planned.
           GAO/NSIAD-94-118. Washington, D.C.: March 25, 1994.




(120231)
           Page 24                                          GAO-03-645T Best Practices
This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.