oversight

Defense Acquisitions: Assessments of Major Weapon Programs

Published by the Government Accountability Office on 2003-05-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to Congressional Committees




May 2003
             DEFENSE
             ACQUISITIONS
             Assessments of Major
             Weapon Programs




GAO-03-476
             a
                                                May 2003


                                                DEFENSE ACQUISITIONS
                                                Assessments of Major Weapon Programs
Highlights of GAO-03-476, a report to
Congressional Committees




The weapons the Department of                   GAO assessed 26 defense programs ranging from the Marine Corps’
Defense (DOD) develops have no                  Advanced Amphibious Assault Vehicle to the Missile Defense Agency’s
rival in superiority. How they are              Theater High Altitude Area Defense system. GAO’s assessments are
developed can be improved,                      anchored in a knowledge-based approach to product development that
without sacrificing the superiority             reflects best practices of successful programs. This approach centers on
of the outcome. GAO’s reviews
over the past 20 years have found               attaining high levels of knowledge in three elements of a new product or
consistent problems with weapon                 weapon—technology, design, and production. If a program is not attaining
investments—cost increases,                     this level of knowledge, it incurs increased risk of technical problems,
schedule delays and performance                 accompanied by cost and schedule growth (see figure). If a program is
shortfalls—along with underlying                falling short in one element, like technology maturity, it is harder to attain
causes, such as pressure on                     knowledge in succeeding elements.
managers to promise more than
they can deliver. The best
practices of successful product
developments offer a knowledge-
based approach DOD can use to
improve the way it develops new
weapons.

This report is new for GAO, and
draws on its work in best practices
for product development. GAO’s
goal for this report is to provide
congressional and DOD decision
makers with an independent,
knowledge-based assessment of
defense programs that identifies
potential risks, and offers an
opportunity for action when a
program’s projected attainment of
knowledge diverges from the best
practice. It can also highlight those
programs that employ practices                  All of the programs GAO assessed proceeded with less knowledge at critical
worthy of emulation by other                    junctures than suggested by best practices, although several came close to
programs. GAO plans to update and               meeting best practice standards. GAO also found that programs generally
issue this report annually to the               did not track statistical process control data, a key indicator for production
congressional defense committees.               maturity. Program stakeholders can use these assessments to recognize the
                                                gaps in knowledge early and to take advantage of opportunities for
                                                constructive intervention—such as adjustments to schedule, trade-offs in
                                                requirements, and additional funding.
GAO makes no recommendations.
Program office comments are
included in the assessments of                  GAO has summarized the results of its assessments in an easy to read two-
each individual program.                        page format. Each two-page assessment contains a profile of the product
www.gao.gov/cgi-bin/getrpt?GAO-03-476.
                                                that includes a description; a timeline of development; a baseline
                                                comparison of cost, schedule, and quantity changes to the program; and a
To view the full report, including the scope    graphical and narrative depiction of how the product development
and methodology, click on the link above.
For more information, contact Paul Francis at
                                                knowledge of an individual program compared to best practices. Each
(202) 512-4841 or francisp@gao.gov.             program office submitted comments and they are included with each
                                                individual assessment as appropriate.
Contents



Foreword                                                                                             iii


Letter                                                                                                1
                           A Knowledge-Based Approach Can Lead to Better Acquisition
                             Outcomes                                                                 2
                           Observations                                                               5
                           Agency Comments                                                            7
                           Scope of Our Review                                                        7


Appendixes
             Appendix I:   Assessments of Individual Programs                                         9
                           Introduction                                                               9
                           Advanced Amphibious Assault Vehicle (AAAV)                                15
                           Airborne Laser (ABL)                                                      17
                           Advanced Extremely High Frequency (AEHF) Communications
                           Satellite                                                                 19
                           AN/APG-79 Active Electronically Scanned Array (AESA) Radar                21
                           AIM-9X Short-Range Air-to-Air Missile                                     23
                           Advanced Threat Infrared Countermeasures/Common Missile Warning
                           System (ATIRCM/CMWS)                                                      25
                           Advanced Wideband Satellite (AWS)                                         27
                           Cooperative Engagement Capability (CEC)                                   29
                           CH-47F Improved Cargo Helicopter                                          31
                           RAH-66 Comanche                                                           33
                           EX-171 Extended Range Guided Munition (ERGM)                              35
                           Excalibur Artillery Round                                                 37
                           F/A-18E/F Super Hornet                                                    39
                           F/A-22 Raptor                                                             41
                           Joint Air-to-Surface Standoff Missile (JASSM)                             43
                           Joint Common Missile                                                      45
                           Joint Primary Aircraft Training System (JPATS)                            47
                           F-35 Joint Strike Fighter (JSF)                                           49
                           Joint Standoff Weapon (JSOW)                                              51
                           National Polar-orbiting Operational Environmental Satellite System
                           (NPOESS)                                                                  53



                           Page i                                GAO-03-476 Acquisition Trends and Risks
                          Contents




                          Patriot Advanced Capability 3 (PAC-3) Program                                              55
                          Space Based Infrared System (SBIRS) High                                                   57
                          Theater High Altitude Area Defense (THAAD)                                                 59
                          Tactical Tomahawk Missile                                                                  61
                          V-22 Osprey                                                                                63
                          Wideband Gapfiller Satellite (WGS) Communications System                                   65
          Appendix II:    Methodology                                                                                67
                          System Profile Assessment                                                                  67
                          Product Knowledge Assessment                                                               68
          Appendix III:   GAO Contact and Acknowledgments                                                            71


Figures                   Figure 1: Knowledge Build at Key Points in Product Development
                                    Reduces the Risk of Unknowns                                                      4
                          Figure 2: Graphic Depiction of Best Practices for Technology,
                                    Design, and Production Knowledge                                                 10
                          Figure 3: Depiction of a Notional Weapon System Program’s
                                    Knowledge as Compared with Best Practices                                        12




                          Abbreviations

                          DOD          Department of Defense
                          GAO          General Accounting Office
                          USAF         United States Air Force
                          USN          United States Navy
                          USMC         United States Marine Corps

                           This is a work of the U.S. Government and is not subject to copyright protection in the
                           United States. It may be reproduced and distributed in its entirety without further
                           permission from GAO. It may contain copyrighted graphics, images or other materials.
                           Permission from the copyright holder may be necessary should you wish to reproduce
                           copyrighted materials separately from GAO’s product.




                          Page ii                                           GAO-03-476 Acquisition Trends and Risks
A
United States General Accounting Office
Washington, D.C. 20548
                                                                                           Comptroller General
                                                                                           of the United States




                                    May 15, 2003                                                                    Foreword




                                    Congressional Committees

                                    Recent military operations in Iraq have soundly demonstrated the
                                    superiority of United States military capabilities. The Department of
                                    Defense (DOD) develops weaponry that is unmatched in levels of
                                    technological sophistication and lethality. Despite their superiority, weapon
                                    systems routinely take much longer to field, cost more to buy, and require
                                    more support than investment plans provide for. In a constrained funding
                                    environment, unforeseen cost growth in weapon systems forecloses other
                                    investment choices for the government, both within and outside of DOD.
                                    DOD’s investment in major weapon systems is expected to grow
                                    considerably in the future as DOD works to keep legacy systems while
                                    investing in future capabilities such as unmanned aircraft, satellite
                                    networks, and information communication systems. For example, the
                                    investment in weapons from fiscal years 2003 through 2009 will exceed
                                    $1 trillion. Such an investment clearly requires DOD to be as efficient and
                                    effective as possible in the development and acquisition of weapon
                                    systems.

                                    In the last several years, we have undertaken a body of work that examines
                                    weapon acquisition issues from a different, more cross-cutting
                                    perspective—one that draws lessons learned from best product
                                    development practices to see if they apply to weapon system development.
                                    We found that programs managed with a knowledge-based approach—
                                    where product knowledge is demonstrated at critical points in a
                                    development cycle—place themselves on a low-risk path to production.
                                    These programs are more likely to be executed within cost and schedule
                                    estimates. We believe that by employing this approach, DOD can still field
                                    superior weapons without attendant cost and schedule growth.

                                    This report is a new product for GAO. It provides decision makers with a
                                    snapshot of program performance and risk and is focused on each system’s
                                    developmental progress vis a vis best practices. Each assessment is
                                    summarized in an easy to read, visually descriptive 2-page format that
                                    provides a fact-based analysis of each program’s cost, schedule, and
                                    development status. We plan to issue this report annually in early spring,
                                    and we intend to increase the number of systems reviewed each year. We
                                    have briefed numerous committee staff on the product and received
                                    positive feedback regarding the report’s utility and breath of coverage.




                                    Page iii                                  GAO-03-476 Acquisition Trends and Risks
The continuing war on terrorism, regional instability, the challenge of
transforming the military, as well as the federal government’s short-and
long-term budget pressures have created a challenging environment for
DOD. It faces a number of difficult missions that will put its strategies and
resources under enormous strain. Consequently, it is important that
weapon systems be acquired using a knowledge-based approach to ensure
that their development is within cost, schedule, and performance
parameters. We believe that this report can provide useful insights on key
risks in development, allow decision makers to take corrective actions, and
thereby place programs in a better position to succeed.




David M. Walker
Comptroller General
of the United States




Page iv                                   GAO-03-476 Acquisition Trends and Risks
A
United States General Accounting Office
Washington, D.C. 20548



                                    May 15, 2003                                                                     Leter




                                    Congressional Committees

                                    The Department of Defense (DOD) is on the threshold of several major
                                    investments in programs that are likely to dominate budget and doctrinal
                                    debates well into the next decade. These programs include, among others,
                                    the Missile Defense Agency’s suite of land, sea, air, and space defense
                                    systems; the Army’s Future Combat System; and the Air Force and Navy’s
                                    Joint Strike Fighter. In fiscal year 2003, the Congress appropriated
                                    $127 billion to DOD for the research, development, and procurement of
                                    weapon systems. Funding for weapon systems is projected to continue
                                    growing to $182 billion in fiscal year 2009—an increase of over 43 percent.
                                    In total, the investment in weapons from fiscal years 2003 through 2009 will
                                    exceed $1 trillion. Thus, it is essential that sound foundations for these and
                                    other weapon system investments be laid now so that the resulting
                                    programs can be executed within estimates of available resources.

                                    The challenge of putting new programs on a better footing than their
                                    predecessors is a daunting one. Clearly, the acquisition process produces
                                    superior weapons. But it does so at a high price. Weapon systems routinely
                                    take much longer to field, cost more to buy, and require more support than
                                    investment plans provide for. These consequences reduce the buying
                                    power of the defense dollar, delay capabilities for the warfighter, and force
                                    unplanned—and possibly unnecessary—trade-offs among programs.

                                    DOD has undertaken a number of acquisition reforms over the preceding
                                    two decades in response to those problems, but while there have been
                                    individual successes, these reforms have not yet yielded consistent
                                    improvements in program outcomes. More recently, DOD leadership has
                                    embraced an evolutionary acquisition approach, coupled with time-phased
                                    requirements. This approach supports developing weapons in smaller,
                                    more predictable iterations of increasing capabilities, rather than the past
                                    approach of attempting to achieve a weapon’s maximum capability in one
                                    design leap. DOD is also striving to give programs, such as missile defense,
                                    more flexibility to make trade-offs between cost, schedule, and
                                    performance that can lead to better investment decisions. It is also
                                    currently looking at how to revise its planning, programming, and
                                    budgeting process that has been in place for over 40 years.

                                    Key to any effort to improve weapon system outcomes is using the lessons
                                    that can be learned from the best practices of successful commercial and



                                    Page 1                                     GAO-03-476 Acquisition Trends and Risks
                       defense product development programs. We have found that these
                       practices can be collectively described as a knowledge-based approach
                       whose success depends on the timely attainment and use of a product’s
                       technology, design, and production maturity. In this report, we compare the
                       knowledge gained on 26 DOD weapon system programs with best
                       practices. Our objective is to provide decision makers a means to quickly
                       gauge the progress and potential risks—based on demonstrated
                       knowledge—of the individual weapon system programs.



A Knowledge-Based      All product development efforts, whether for a car, a plane, a missile, or a
                       satellite, go through a process of building knowledge. Ultimately, this
Approach Can Lead to   process brings together and integrates all of the technologies, components,
Better Acquisition     and subsystems needed for the product to work and to be reliably
                       manufactured. The product development process can be characterized as
Outcomes               the reduction of risk and the resolution of unknowns through the
                       attainment of knowledge.

                       About 7 years ago, at the request of the Senate Committee on Armed
                       Services, we began an extensive body of work identifying best practices in
                       product development, both in DOD and in the commercial sector. Of
                       particular interest were cases in which increasingly sophisticated products
                       were being developed in significantly less time and at lower cost than their
                       predecessors. A major reason for these successes was the use of a product
                       development process that was anchored in knowledge. Product developers
                       employed specific practices to ensure that a high level of knowledge
                       regarding critical facets of the product was achieved at key junctures in
                       development. We have characterized these junctures as three knowledge
                       points. We have also identified key indicators that can be used to assess the
                       attainment of each knowledge point. When tied to major events on a
                       program’s schedule, they can disclose whether gaps or shortfalls exist in
                       demonstrated knowledge, which can presage future cost, schedule, and
                       performance problems. These knowledge points and associated indicators
                       are defined as follows.

                       • Knowledge point 1: Resources and needs are matched. This level of
                         knowledge is attained when a match is made between a customer’s
                         needs and the developer’s technical, financial, and other resources.
                         Technology maturity is a particularly important indicator of resource
                         availability. A best practice is to achieve a high level of technology
                         maturity at the start of product development. This means that the




                       Page 2                                    GAO-03-476 Acquisition Trends and Risks
   technologies needed to meet essential product requirements have been
   demonstrated to work in their intended environment.

• Knowledge point 2: The product design is stable. This level of
  knowledge is attained when the product’s design demonstrates its
  ability to meet the customer’s requirements. A best practice is to achieve
  design stability at the system-level critical design review, usually held
  midway through development. Completion of engineering drawings at
  the system design review provides tangible evidence that the design is
  stable.

• Knowledge point 3: Production processes are mature. This level of
  knowledge is attained when it is demonstrated that the product can be
  manufactured within cost, schedule and quality targets. A best practice
  is to achieve production maturity at the start of production. This means
  that all key manufacturing processes produce output within statistically
  acceptable limits for quality.

As illustrated in figure 1, the process is building block in nature as the
attainment of each successive knowledge point builds on the proceeding
one. While the knowledge itself builds continuously without clear lines of
demarcation, the attainment of knowledge points is sequential. In other
words, production maturity cannot be attained if the design is not mature,
and design maturity cannot be attained if the key technologies are not
mature.




Page 3                                    GAO-03-476 Acquisition Trends and Risks
Figure 1: Knowledge Build at Key Points in Product Development Reduces the Risk of Unknowns




                                        For the most part, all three knowledge points are eventually attained on a
                                        completed product. The difference between highly successful product
                                        developments—those that deliver superior products within cost and
                                        schedule projections—and problematic product developments is how this
                                        knowledge is built and how early in the development cycle each knowledge
                                        point is attained. When knowledge is built more slowly than these points
                                        suggest, less knowledge is on hand at key decisions or events, such as the
                                        decisions to start a development program, hold the critical design review,
                                        and start production. This invites greater cost, schedule, and performance
                                        risks because (1) problems are more likely to be discovered late in the
                                        process and will therefore be more difficult and costly to correct and (2) a
                                        variety of pressures encourage program managers to underestimate the
                                        difficulties.

                                        It is important to note that successful product developers treat technology
                                        development as a different and separate effort that precedes product
                                        development. This treatment of technology development is key to reaching
                                        the first knowledge point at the start of product development, as it is a
                                        prerequisite for capturing design and production knowledge early in
                                        product development. This approach to attaining knowledge puts program
                                        managers—and programs—in a better position to succeed.


                                        Page 4                                      GAO-03-476 Acquisition Trends and Risks
Observations   When programs proceed with less knowledge than suggested by best
               practices, cost, schedule, and performance problems often result. To
               varying degrees, all the programs we assessed proceeded with lower levels
               of knowledge at critical junctures and thus attained key elements of
               product knowledge later in development. In some programs, the
               consequences of proceeding with early knowledge deficits have already
               been felt. For example:

               • The F-22 Fighter began product development with key technologies
                 immature—deferring knowledge point 1—and subsequently had only a
                 quarter of the desired amount of engineering drawings completed at the
                 critical design review—deferring knowledge point 2. The program has
                 experienced substantial cost increases and schedule delays in the latter
                 stages of development.

               • The Patriot Advanced Capability missile also reached knowledge points
                 1 and 2 later than best practices. The seeker technology did not
                 demonstrate maturity until close to the production decision and the
                 design remains unstable. Each seeker still needs to be reworked about
                 3 times on average before it passes quality inspections. The cost of the
                 seeker has increased by 76 percent and contributed to a 2-year delay in
                 the program’s schedule.

               • The Extended Range Guided Munition program began with only one of
                 its 20 critical technologies mature—deferring knowledge point 1. While
                 progress has been made, program officials do not expect to achieve
                 maturity on all technologies until after the design review. The lack of
                 mature technologies contributed to subsequent test failures, cost
                 increases, and schedule delays.

               If programs attain more knowledge as suggested by best practices, they are
               in a better position to succeed in meeting cost, schedule, and performance
               expectations. We found some programs that did attain key product
               knowledge earlier than most. For example:

               • The National Polar-orbiting Operational Environmental Satellite System
                 program ensured that its pacing technologies were demonstrated before
                 committing to product development. The program plans to demonstrate
                 three critical sensors on a demonstrator satellite prior to their inclusion
                 on the new satellite.




               Page 5                                    GAO-03-476 Acquisition Trends and Risks
• The Theater High Altitude Area Defense System made significant strides
  in product development, following a problematic preliminary
  development phase. In 2000, we reported that the program’s delayed
  demonstration of technologies and components and reliance on full-
  system testing to discover problems, was a very costly method to
  mature the system’s design and nearly caused the cancellation of the
  program. The program has since structured a product development
  phase that places a much greater emphasis on early demonstration of
  components, a testing program that incorporates sufficient time
  between tests for learning, and a plan to achieve design stability by
  releasing 90 percent of engineering drawings by the time of the critical
  design review—knowledge point 2.

In general, we found that the greatest absence of knowledge was in the
area of production. Almost no programs collected statistical process
control data, the indicator for production maturity. Unlike technology
readiness levels, which can be applied at any time, and engineering drawing
release data, which is captured on all programs, few programs collected
statistical process control data. While the absence of this data does not
necessarily mean that production processes were immature, attained
knowledge could not be assessed against an objective standard. Other
indicators of production maturity, such as scrap and rework rates, can
indicate positive trends, but are not prospective—that is, they are not
useful in guiding preparations for production. To some extent, statistical
process control data is not being collected because DOD has been
delegating more responsibility to prime contractors and reducing the
amount of data requested. The lack of such data may put program offices in
a disadvantaged position to gain insights about a contractor’s production
progress. We have recently issued a report that recommends that DOD
collect statistical process control data on its weapon system programs and
DOD has agreed with this recommendation.1

We conducted our review from September 2002 through May 2003 in
accordance with generally accepted government auditing standards.




1
 U.S. General Accounting Office, Best Practices: Capturing Design and Manufacturing
Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 15,
2002).




Page 6                                         GAO-03-476 Acquisition Trends and Risks
Agency Comments       DOD did not provide general comments on a draft of this report, but did
                      provide technical comments on individual assessments. These comments,
                      along with program office comments, are included with each individual
                      assessment as appropriate.



Scope of Our Review   We selected programs for the assessments based on several factors,
                      including (1) high dollar value, (2) stage in acquisition, and (3)
                      congressional interest. The majority of the 26 programs covered in this
                      report are considered major defense acquisition programs by DOD. A
                      program is defined as major if its estimated research and development
                      costs exceed $365 million or its procurement exceeds $2.19 billion in fiscal
                      year 2000 constant dollars.

                      We plan to include more programs in subsequent years, with a greater
                      focus on programs early enough in development that the assessments can
                      be used to improve the program’s prospects for success, and issue this
                      report annually to the congressional defense committees. The individual
                      assessment of each program can be found in appendix I. Appendix II
                      contains detailed information on our methodology.


                      We are sending copies of this report to interested congressional
                      committees; the Secretary of Defense; the Secretaries of the Army, Navy,
                      and Air Force; and the Director, Office of Management and Budget. We will
                      also make copies available to others upon request. In addition, the report
                      will be available at no charge on the GAO Web site at http://www.gao.gov. If
                      you have any questions on this report, please contact me at (202) 512-4841
                      or Paul Francis at (202) 512-2811. Major contributors to this report are
                      listed in appendix III.




                      Jack L. Brock,
                      Managing Director
                      Acquisition and Sourcing Management



                      Page 7                                    GAO-03-476 Acquisition Trends and Risks
List of Congressional Committees

The Honorable John W. Warner
Chairman
The Honorable Carl Levin
Ranking Member
Committee on Armed Services
United States Senate

The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Duncan Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives

The Honorable Jerry Lewis
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 8                             GAO-03-476 Acquisition Trends and Risks
Appendix I

Assessments of Individual Programs                                                             AA
                                                                                                ppp
                                                                                                  ep
                                                                                                   ned
                                                                                                     n
                                                                                                     x
                                                                                                     id
                                                                                                      e
                                                                                                      x
                                                                                                      Iis




Introduction   For the 26 programs, each assessment provides the historical and current
               program status and offers the opportunity to take early corrective action
               when a program’s projected attainment of knowledge diverges significantly
               from the best practices. The assessments also identify programs that are
               employing practices worthy of emulation by other programs. If a program
               is attaining the desired levels of knowledge, it has less risk—but not zero
               risk—of future problems. Likewise, if a program shows a gap between
               demonstrated knowledge and best practices, it indicates an increased
               risk—not a guarantee—of future problems. The real value of the
               assessments is recognizing gaps early, which provides opportunities for
               constructive intervention—such as adjustments to schedule, trade-offs in
               requirements, and additional funding—before cost and schedule
               consequences mount.

               Our assessment of each program is summarized in two components—(1) a
               system profile and (2) a product knowledge assessment.

               The system profile presents a general description of the product in
               development; a picture of the product or a key element of the product; a
               schedule timeline identifying key dates in the program; a table identifying
               the prime contractor; the program office location, and the fiscal year 2004
               requested funding if available; and a table summarizing the cost, schedule
               and quantity changes to the program.

               The rest of the assessment analyzes the extent to which product knowledge
               at the three key knowledge points has been attained. We depict the extent
               of knowledge in a stacked bar graph and provide a narrative summary at
               the bottom of the first page. The second page is devoted to a narrative
               assessment of technology, design and production maturity, as well as other
               program issues identified and comments from the program office.

               The product knowledge figure is based on the three knowledge points and
               the key indicators for the attainment of knowledge. A “best practice” line
               is drawn based on the ideal attainment of the three types of knowledge at
               the three knowledge points (see fig. 2).




               Page 9                                    GAO-03-476 Acquisition Trends and Risks
Appendix I
Assessments of Individual Programs




Figure 2: Graphic Depiction of Best Practices for Technology, Design, and
Production Knowledge




The first major point on the best practice line represents two facts: a
commitment to a new product development has been made and the key
technologies needed for the new product are mature.

When all critical technologies have reached a technology readiness level 7,
technology maturity—and thus knowledge point 1—has been attained. In
our assessment, the technologies that have reached technology readiness
level 7, a prototype demonstrated in an operational environment, are
considered mature and those that reach technology readiness level 6, a
prototype demonstrated in a relevant environment, are assessed as
attaining 50 percent of the desired level of knowledge. Satellite
technologies that achieved technology readiness level 6 were assessed as
fully mature due to the difficulty of demonstrating maturity in an
operational environment—space. (Technology readiness levels are more
fully explained in appendix II.) The second major point on the best practice
line captures technology maturity plus design maturity—knowledge point
2. A design is considered mature when 90 percent of the engineering
drawings have been released or deemed releasable to manufacturing. In the



Page 10                                      GAO-03-476 Acquisition Trends and Risks
Appendix I
Assessments of Individual Programs




successful programs we have studied, design maturity is attained about
halfway through the product development phase. The third major point on
the best practice line captures the sum of technology maturity, design
maturity, and production maturity. Production is considered mature when
all key production processes are in statistical control. Ideally, this occurs
before the first products for delivery to the customer are manufactured. As
can be seen, knowledge about the technology, design, and production of a
new product builds over time. While the knowledge itself builds
continuously without clear lines of demarcation, the attainment of
knowledge points is sequential. In other words, production maturity cannot
be attained if the design is not mature, and design maturity cannot be
attained if the key technologies are not mature.

Data for a given weapon system program is then plotted against the best
practices line. In the assessments that follow, a brown bar indicates the
technology knowledge attained by a weapon system program. The actual
technology readiness levels attained for a program’s key technologies are
measured at the start of development—normally milestone II or milestone
B in the Department of Defense’s (DOD) acquisition process. The closer a
program’s attained knowledge is to the best practice line, the more likely
the weapon will be delivered within its estimated cost and schedule. A
knowledge deficit at this point—indicated by a gap between the technology
knowledge attained by the weapon system and the best practices line—
means the program proceeded with immature technologies and may face a
greater likelihood of cost and schedule increases as technology risks are
discovered and resolved. A green bar indicates the design knowledge
attained by a weapon system program. This is calculated by measuring the
percent of engineering drawings released to manufacturing. The green bar
is stacked on top of the brown bar to indicate whether any cumulative
gap—considering both technology and design—exists at the halfway point
of product development. A blue bar indicates the production knowledge
attained by a weapon system program. This is calculated by measuring the
percentage of key production processes in statistical control. The blue bar
is stacked on top of the brown and green bars to indicate whether any
cumulative technology, design, and production gaps exist at the time
production begins. In some cases, we obtained projections from the
program office of future knowledge attainment. These projections are
depicted as dashed bars.

Figure 3 depicts an example of an assessment for a notional weapon
system.




Page 11                                   GAO-03-476 Acquisition Trends and Risks
Appendix I
Assessments of Individual Programs




Figure 3: Depiction of a Notional Weapon System Program’s Knowledge as
Compared with Best Practices




An interpretation of this notional example would be that the product
development began with key technologies immature, thereby missing
knowledge point 1. Knowledge point 2 was not attained at the design
review as some technologies were still not mature and only a small
percentage of engineering drawings had been released. Projections for the
production decision show that the program is expected to achieve a greater
level of maturity, but will still fall short. It is likely that this program would
have had significant cost and schedule increases.

We also found three situations in which programs were unable to provide
key knowledge indicators. We used three types of labels in the knowledge
figures to depict those situations. Programs with these labels are
distinguished from those that have elected not to collect data that can be
used to assess progress against best practices. First, a few programs are
planning to collect the relevant knowledge indicator, but they have not yet
begun collecting it. In these situations, we annotate the graph with the
phrase “Data unavailable.” Second, a few programs have not followed the
traditional acquisition model. For example, one program combined the



Page 12                                      GAO-03-476 Acquisition Trends and Risks
Appendix I
Assessments of Individual Programs




development start decision with the production decision. Another program
used commercial off-the-shelf components, which negated the need to
monitor production processes. In these situations, we annotate the graph
with the phrase “Not applicable.” Finally, some programs were unable to
provide or reconstruct the relevant knowledge indicator because the event
happened too many years ago. In these situations, we annotate the graph
with the phrase “Not assessed.”

Our assessments of the 26 systems follow.




Page 13                                 GAO-03-476 Acquisition Trends and Risks
Appendix I
Assessments of Individual Programs




Page 14                              GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                        Common Name: AAAV

                           Advanced Amphibious Assault Vehicle (AAAV)


 The Marine Corps’ AAAV is designed to transport
 troops from ships to shore at higher speeds and
 from farther distances than the existing AAV-7. It is
 designed to be more mobile, lethal, reliable, and
 effective in all weather conditions. AAAV will have
 two variants—a troop carrier for 17 Marines and a
 command vehicle to manage combat operations in
 the field.




  Prime contractor: General Dynamics
  Program office: Woodbridge, Va.
  FY 2004 funding request:                FY 2003 dollars in millions               Approved            Latest   Percent
   R&D $240.7 million                                                                   12/00            12/01   change
   Procurement $97.9 million                 Research & development cost             $1,395.7         $1,616.4      15.8
   Quantity 0 vehicles                       Procurement cost                        $6,256.7         $6,741.7       7.8
                                             Total program cost                      $7,732.7         $8,440.9       9.2
                                             Program unit cost                        $7.544            $8.235       9.2
                                             Total quantities                           1,025            1,025       0.0
                                             Acquisition cycle time (months)              138              150       8.7




                                                                    AAAV demonstrated most technology and design
                                                                    knowledge at critical junctures in the program. At
                                                                    the start of the program, all but one of the critical
                                                                    technologies were mature. The design was close to
                                                                    meeting best practice standards at the design
                                                                    review, signifying the design was stable. Early
                                                                    development of fully functional prototypes and
                                                                    other design practices facilitated design stability.
                                                                    However, late maturation of the remaining
                                                                    technology may lead to some redesign. Also, the
                                                                    demonstration of production maturity remains a
                                                                    concern because the program is currently uncertain
                                                                    about requiring the contractor to use statistical
                                                                    process controls to achieve quality objectives. The
                                                                    AAAV production decision is not scheduled until
                                                                    September 2005. Remaining efforts include
                                                                    developmental, operational, live fire, and reliability
                                                                    testing.


GAO-03-476 Acquisition Trends and Risks                                                                              Page 15
Common Name: AAAV                                                                                         Appendix I

AAAV Program                                                Other
                                                            The Marine Corps has recently restructured the AAAV
Technology Maturity
                                                            program to add 12 additional months of testing before
Four of the five critical technologies had                  the September 2005 production decision. This change
demonstrated an acceptable level of maturity at the         more than doubles the number of vehicle test months
start of product development. The remaining                 previously planned. The change also moves the initial
technology, moving map navigation, is not expected to       operational capability date from September 2007 to
achieve maturity until the spring of 2003. Program          September 2008. The program estimates a $480 million
officials stated that maturing this technology is           increase in acquisition costs—$101 million for added
contingent on developing and testing system                 testing, $75 million for development, and $304 million
hardware. As a backup, program officials said they          for recurring production.
could carry out the AAAV mission using existing
technology, but it would not provide full vehicle-to-       Program Office Comments
vehicle situational awareness.                              AAAV program officials concurred with our
                                                            assessment.
Design Maturity
The AAAV design is essentially complete. However,
late maturation of the new mapping system may lead
to some redesign, if testing identifies any problems.
At the critical design review, AAAV had completed
77 percent of the drawings—not up to the best
practice standard of 90 percent, but higher than many
DOD programs. Early engineering prototypes—fully
integrated and functional—allowed the program to
demonstrate that the design worked as required.
These early prototypes have completed over 4,000
hours of testing that resulted in design improvements
for subsequent prototypes.
To complete development, program plans call for
building and testing nine development prototypes and
one live fire test vehicle. These prototypes will be
production representative vehicles for developmental,
operational, live fire and reliability testing. The first
prototype is scheduled to be available by May 2003.

Production Maturity
Program officials are developing a production
readiness plan to ensure vehicles will meet cost,
schedule, and quality objectives. At this time, they are
uncertain whether this plan will require the contractor
to use statistical process controls, the best practice
standard. As the prime contractor currently produces
the nine developmental prototype vehicles, it is not
tracking statistical process control data. Instead, it is
using postproduction inspections, considered less
efficient and effective than statistical process controls
to achieve quality.




Page 16                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                                                 Common Name: ABL

                                               Airborne Laser (ABL)

 The Missile Defense Agency’s ABL is designed to
 destroy enemy ballistic missiles almost immediately
 after their launch. The system, carried aboard a
 highly modified Boeing 747 aircraft, uses a high-
 energy chemical laser to rupture the skin of enemy
 missiles; a beam control/fire control subsystem to
 guide the laser beam through the aircraft, focus the
 beam on the target, and maintain the beam’s quality
 as it travels through the atmosphere; and a battle
 management subsystem to plan and execute the
 engagement. We assessed all components.




  Prime contractor: Boeing
  Program office: Albuquerque, NM
  FY 2004 funding request:                FY 2003 dollars in millions                              Approved                     Latest        Percent
   R&D $610.0 million                                                                                   1/97                     12/02        change
   Procurement $0 million                    Research & development cost                            $2,400.5                  $4,415.7             84
   Quantity 0 systems                        Procurement cost                                       $3,170.6                       NA             NA
                                             Total program cost                                     $5,571.1                       NA             NA
                                             Program unit cost                                       $795.9                        NA             NA
                                             Total quantities                                              7                       NA             NA
                                             Acquisition cycle time (months)                             118                       NA             NA
                                          Note: Latest costs only through FY 2007. Procurement funding, quantities, and the initial capability date have
                                          yet to be determined. NA = not applicable.




                                                                           Only one of ABL’s critical subsystems has
                                                                           demonstrated acceptable levels of maturity. The
                                                                           Missile Defense Agency is developing an initial ABL
                                                                           system to demonstrate technology critical to the
                                                                           system’s design and plans to begin development of a
                                                                           second improved demonstration aircraft in 2003.
                                                                           Either of these aircraft, or later improved
                                                                           configurations, could be given to the Air Force for
                                                                           operational testing and production if system-level
                                                                           tests show that any one of them is capable of
                                                                           destroying a threat missile at an operational range.
                                                                           Although the agency’s development strategy
                                                                           incorporates some knowledge-based practices, it is
                                                                           difficult to see how the discipline of a knowledge-
                                                                           based approach can be achieved when uncertainty
                                                                           exists about whether the effort is a technology
                                                                           development or a product development.



GAO-03-476 Acquisition Trends and Risks                                                                                                             Page 17
Common Name: ABL                                                                                         Appendix I

ABL Program                                               The Missile Defense Agency has made changes that
                                                          are expected to improve its ability to evolve ABL’s
Technology Maturity                                       critical technologies, including adopting a flexible
                                                          requirements setting process, providing additional
Only one of ABL’s five critical subsystems, the
                                                          time and facilities to develop and test these
aircraft itself, represents mature technology. A second   technologies, and attaining the knowledge to match
subsystem, which directs the laser energy through the
                                                          the warfighters’ needs with demonstrated technology.
aircraft, consists of several technologies that have
                                                          On the other hand, it is not clear whether the start of a
been tested in a simulated environment. However,          block represents a technology development or a
three other subsystems consist of low-fidelity
                                                          product development. This uncertainty may hamper
prototype technologies that have only been tested in a
                                                          the application of knowledge standards and forfeit the
laboratory environment. They include the laser, the       discipline necessary to ensure successful product
battle management subsystem, and the ground
                                                          development.
support subsystem.
Problems associated with maturing technology have         Program Office Comments
consistently been a source of cost and schedule           In commenting on a draft of this assessment, program
growth throughout the life of the program. DOD            officials reemphasized their commitment to spiral
analysts attribute this growth to the increased           development and capabilities-based acquisition. They
complexity of designing laser subsystems, substantial     plan to use this strategy to improve the critical aspects
increases in engineering analysis and design, and         of the system by allowing the pace of technological
greater than anticipated aircraft engineering             development to dictate the introduction of improved
complexity.                                               capabilities into the system. They believe this strategy
The program is managed under the Missile Defense          is not inconsistent with knowledge-based acquisition.
Agency’s new capabilities-based acquisition strategy.     They also mentioned that laser power depends not
This approach develops an operational system              only on the number of laser modules but also on
through a series of block upgrades. The agency plans      module efficiency, optics, and pointing precision. They
to use the first two blocks, block 2004 and block 2008,   admit that the laser subsystem should be operated in
to demonstrate critical technologies, but if tests show   flight before any production decision is made.
either configuration has any battlefield utility, that    Program officials are conducting emergency
configuration could be deployed in the event of an        operational capability planning to support a possible
emergency.                                                emergency ABL deployment. This decision will be
The 2004 configuration will have a 6-module laser,        based on the potential threat and an assessment of the
rather than the 14 modules planned for the production     capabilities ABL may provide.
system. The optical components can withstand the          The program office indicated that all but one of the
heat produced by a 6-module laser, but the agency         battle management components have been tested in
would have to redesign optical components for the         an operational environment. This component is the
system to withstand the heat associated with an           active ranger system, which provides crucial angle
increase in laser power. In addition, the 2004            measurements and range data for engaging ballistic
configuration is far too heavy to allow the addition of   missiles.
laser modules that will likely be needed in an
operational ABL system.
To accommodate more modules, a weight reduction
program has begun that includes redesigning many
components and the increased use of composite
materials. The program is considering whether to use
a different aircraft configuration that would allow the
system’s weight to be moved forward to relieve stress
on the airframe. However, its use would require
additional design changes.



Page 18                                                                        GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                       Common Name: AEHF

        Advanced Extremely High Frequency (AEHF) Communications Satellite

 The Air Force’s AEHF is a satellite system intended
 to replace the existing Milstar system with
 improved, survivable, jam-resistant, worldwide,
 secure communication capabilities at lower launch
 costs. First launch of an AEHF satellite is expected
 in 2006. The system also includes a mission control
 segment with service-specific terminals to process
 satellite information. DOD is negotiating
 international partner participation in the program.
 We assessed the satellites and mission control
 segments.




  Prime contractor: Lockheed Martin
  Program office: El Segundo, Calif.
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $778.1 million                                                                  10/01            12/02   change
   Procurement $0 million                    Research & development cost            $4,339.3         $4,608.4        6.2
   Quantity 0 satellites                     Procurement cost                       $1,286.2          $509.1      -60.4
                                             Total program cost                     $5,625.5         $5,117.5       -9.0
                                             Program unit cost                      $1,125.1         $1,705.8      51.6
                                             Total quantities                              5                3     -40.0
                                             Acquisition cycle time (months)             111              118        6.3




                                                                    The AEHF satellite program demonstrated most
                                                                    technology knowledge at development start. Eleven
                                                                    of the 12 critical technologies were mature,
                                                                    according to best practice standards. The remaining
                                                                    technology is not projected to be mature prior to
                                                                    the critical design review, nor does it have a backup
                                                                    technology. However, some elements of this
                                                                    technology are mature. The program expects to
                                                                    complete 90 percent of its drawings by the critical
                                                                    design review. The manufacture of the
                                                                    communications and transmission security
                                                                    subsystem is a major challenge facing the program
                                                                    as upgrades are being added into the new
                                                                    cryptological equipment. If production of this
                                                                    subsystem slips, first launch could slip
                                                                    correspondingly as no backup exists.




GAO-03-476 Acquisition Trends and Risks                                                                             Page 19
Common Name: AEHF                                                                                      Appendix I

AEHF Program                                             security subsystem’s functions as it is integrated into
                                                         the AEHF satellite's communications subsystem.
Technology Maturity                                      However, continued complications in fabrication
                                                         could potentially place the entire program at cost,
Eleven of the 12 critical technologies have reached
                                                         schedule, and performance risk.
maturity according to best practice standards. The
program does not project achieving maturity on the
                                                         Other Program Issues
remaining technology—the phased array antenna—by
the design review in June of 2004, nor does it have a    In December 2002, the Deputy Secretary of Defense
backup capability. However, some elements of this        decided to change the acquisition strategy of AEHF
technology have been demonstrated in an operational      from a five-satellite program to a three-satellite
environment.                                             program. Under the revised strategy, full capability
                                                         may no longer be satisfied by an AEHF-only
Design Maturity                                          constellation.
The program has completed 150 or more of the 6,000
                                                         Program Office Comments
total drawings for release to manufacturing. Program
officials project completing 90 percent of drawings by   In commenting on a draft of this assessment, program
the system critical design review in June 2004. The      officials stated that the program is executing very well
program has completed key segment level preliminary      since contract definitization in August 2002, with cost
design reviews and is expected to complete all design    and schedule variance at less than 1 percent.
reviews by the second quarter of fiscal year 2004.       Currently, at approximately 33 percent complete
Program officials consider the design and                toward first launch, the total program is on track and
development of the satellite subcomponents low risk      estimated to finish on time and on budget. The system
because those components have been used on other         preliminary design review has been completed.
space systems. However, the integration of these         Critical design reviews are on track for completion by
subcomponents into a subsystem, such as the phased-      Spring 2004. Funding cuts have, in the past, caused
array antenna, has yet to be successfully                schedule slips and cost increases. Given the focus on
demonstrated at the AEHF satellite frequencies.          the critical design review, the impacts of changing
                                                         requirement will have increasing deleterious effects.
Program officials assessed the software development
                                                         The program remains focused on addressing critical
for the mission control system as moderate risk and
                                                         risks that threaten cost, schedule, and performance.
have developed a risk mitigation strategy. This
                                                         New system security requirements recently received
strategy includes consulting with the National
                                                         from the National Security Agency for the space,
Software Engineering Institute and the Aerospace
                                                         mission control, and terminal segments are being
Corporation and conducting a software development
                                                         evaluated. After aggressive risk management, the most
capability evaluation. Also, the program office has
                                                         likely impacts include additional testing, verification,
incorporated spiral development and the use of
                                                         and program documentation. The program has also
software emulators so users and developers can see
                                                         begun developing engineering models for all of the
how the software will look and work. Until these
                                                         critical subsystems. These efforts are on track and
actions are completed, software may be at risk for
                                                         proceeding well.
unplanned cost and schedule growth.

Production Maturity
Any future problems with the fabrication of the
communications and transmission security
microprocessor, a component designed to limit access
to satellite transmissions to authorized users, could
delay the production schedule and the launch of the
first satellite planned for December 2006. Program
officials have started a number of risk reduction
efforts, including a chip emulator whose purpose is to
simulate the communications and transmission


Page 20                                                                      GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                           Common Name: AESA

               AN/APG-79 Active Electronically Scanned Array (AESA) Radar

 The Navy’s AESA radar is one of the top upgrades
 for the F/A-18E/F aircraft. It is to be the aircraft’s
 primary search/track and weapon control radar and
 is designed to correct deficiencies in the current
 radar. According to the Navy, the AESA radar is key
 to maintaining the Navy’s air-to-air fighting
 advantage and will improve the effectiveness of the
 air-to-ground weapons. When completed, the radar
 will be inserted in new production aircraft and
 retrofitted into existing aircraft.




  Prime contractor: Raytheon
  Program office: Patuxent River, Md.
  FY 2003 funding request:                FY 2003 dollars in millions                  Approved           Latest    Percent
   R&D $107.1 million                                                                       6/01           12/01    change
   Procurement $24.06 million                Research & development cost                 $518.9           $494.2        -4.7
   Quantity 0 radars                         Procurement cost                                NA              NA          NA
                                             Total program cost                              NA              NA          NA
                                             Program unit cost                               NA              NA          NA
                                             Total quantities                                NA              NA          NA
                                             Acquisition cycle time (months)                  69              68        -1.4
                                          Note: NA = not applicable.




                                                                       The AESA radar’s demonstrated knowledge is
                                                                       difficult to characterize. The fact that almost all of
                                                                       the engineering drawings have been completed
                                                                       suggests design stability. However, until the
                                                                       technologies are demonstrated, the potential for
                                                                       design changes remains. The AESA radar is also
                                                                       dependent on other programs that could pose
                                                                       significant risk to the radar’s cost, schedule, and
                                                                       technical performance. The technology and design
                                                                       risks are significant given that the AESA radar is
                                                                       only a few months from a production decision. The
                                                                       Navy is currently reassessing the radar’s technology
                                                                       maturity. Although many of the F/A-18E/F aircraft
                                                                       will be retrofitted with the AESA radar, full funding
                                                                       for the retrofitting has not been budgeted. If the
                                                                       radar is not ready for production as scheduled,
                                                                       more aircraft will have to be retrofitted.



GAO-03-476 Acquisition Trends and Risks                                                                                 Page 21
Common Name: AESA                                                                                          Appendix I

AESA Radar Program                                           planned total buy of 548 F/A-18E/F aircraft will not
                                                             receive the radar as they are being produced. Plans are
Technology Maturity                                          to retrofit the radar onto 136 aircraft at a projected
                                                             cost of $3.14 million each. This cost does not include
The AESA program’s four critical technologies were
                                                             the cost of new APG-73 mechanical scanned radars
not mature at the start of development in February           that will be installed in the aircraft until AESA radars
2001, and they were not mature as of May 2002. The
                                                             are available for retrofit. If delays occur in the AESA
Navy is currently reassessing the maturity of these
                                                             radar deliveries, retrofit costs will increase.
technologies. At the time of its last assessment, two of
the technologies had been tested using simulation and        The AESA radar is projected to weigh about
two had been tested in the laboratory. Program               270 pounds more than the current radar and will
officials indicated that they have several options for       require a more capable cooling system than the one
dealing with immature technologies, including                currently on the aircraft. The Navy expects some
utilizing backup technologies. Initial flight tests of the   minor degradation in aircraft performance, such as
radar in an aircraft are scheduled for June 2003—            slightly decreased range, as a result of the increased
concurrent with the production decision. All four            weight and new cooling system.
technologies are not expected to be mature until
                                                             The AESA program is linked to a number of other
late 2004.
                                                             corporate and Navy programs. For example, the radar
                                                             will use a 32-port fiber channel fabric module
Design Maturity
                                                             developed by Boeing as a commercial venture.
At the design review, 67 percent of the currently            Technical difficulties with the module have caused
projected total drawings were completed. In the              schedule delays and may impact cost and performance
period between June 2002 and December 2002, the              of the radar. Also, Raytheon is developing some
number of total expected drawings increased by 21            hardware and software for the radar with company
percent. Program officials stated that the increase was      funds or in coordination with other programs.
due to new or modified drawings for systems                  Disruptions in these efforts could adversely impact the
supporting the radar such as the radome, shield, and         AESA program.
aircraft airframe. Program officials indicated that they
currently have 98 percent of the drawings complete;          Program Office Comments
however, the technology maturation process may lead
                                                             The AESA program did not provide a general
to more design changes.
                                                             statement in response to our review but did provide
Production Maturity                                          technical comments that were incorporated where
                                                             appropriate.
We could not assess the AESA program’s production
maturity against best practices, as statistical control
data was not available.

Other Program Issues
Program officials estimate that the first low-rate
production unit will exceed its cost target by 27
percent. Subcontractor development cost was
considered to be the biggest contributor to this
increase. The effects of the cost increase may be
minimized in low-rate production lots 1-3 because of
firm fixed price contract options. Program officials
stated that cost reduction initiatives were underway to
reduce the cost overruns by half by full-rate
production.
Delivery of the first production AESA radars for
insertion into F/A-18E/F aircraft on the production line
is scheduled for fiscal year 2005. As a result, 254 of the

Page 22                                                                          GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                   Common Name: AIM-9X

                                  AIM-9X Short-Range Air-to-Air Missile

 The AIM-9X is a follow-on version of the existing
 AIM-9M short-range missile for Air Force and Navy
 fighters. The AIM-9X is designed to be a highly
 maneuverable, launch-and-leave missile; capable of
 engaging targets using passive infrared guidance to
 provide full day/night operations and improved
 resistance to countermeasures and expanded target
 acquisition. The full capabilities of the AIM-9X will
 not be achieved without completing development of
 the helmet mounted cueing system—a separate
 development program that we did not assess.




  Prime contractor: Raytheon Missile System Company
  Program office: Patuxent River, Md.
  FY 2004 funding request:                FY 2003 dollars in millions          Approved         Latest      Percent
   R&D $2.7 million                                                                 1/97         12/01      change
   Procurement $104.9 million               Research & development cost           $577.0        $594.6           3.0
   Quantity 531 missiles                    Procurement cost                    $2,116.1      $2,055.1          -2.9
                                            Total program cost                  $2,693.0      $2,649.6          -1.6
                                            Program unit cost                     $0.268        $0.261          -2.5
                                            Total quantities                      10,049        10,142           0.9
                                            Acquisition cycle time (months)           92           105         14.1




                                                                 The AIM-9X program entered production in
                                                                 September 2000 without assuring that the
                                                                 manufacturing processes were mature. However,
                                                                 because the missile is a follow-on to the AIM-9M
                                                                 missile, program officials believe that they have
                                                                 significant production knowledge. The program
                                                                 ensured, prior to entering low-rate initial
                                                                 production, that the missile design was stable. The
                                                                 program did attain knowledge early in development
                                                                 by using proven technologies from predecessor
                                                                 systems and other programs, as well as testing
                                                                 numerous prototype versions of the missile. As a
                                                                 result, the program released the majority of its
                                                                 engineering drawings at the design review.




GAO-03-476 Acquisition Trends and Risks                                                                         Page 23
Common Name: AIM-9X                                                                                         Appendix I

AIM-9X Program                                               rework data. The AIM-9X acquisition cost and
                                                             schedule history shows the program has been able to
Technology Maturity                                          meet its goals.
All of the AIM-9X critical technologies are mature
                                                             Program Office Comments
because they have been demonstrated in
developmental tests using actual hardware in realistic       In commenting on a draft of this assessment, program
conditions. Specifically, the program used prototypes        officials acknowledged they did not contractually
to test new technologies and existing missile                require collection of statistical process control data on
components that are being employed in a new                  critical manufacturing processes. Program officials
operational environment.                                     stated their strategy for demonstrating manufacturing
                                                             process maturity includes building, testing, and
Design Maturity                                              evaluating production representative missiles;
                                                             conducting multiple readiness reviews; utilizing low-
The design of the AIM-9X is complete, and 100 percent
                                                             rate initial production to test production processes;
of the drawings have been released to manufacturing.
                                                             and maturing production processes before full-rate
The AIM-9X program built and tested 43 prototypes of         production.
various configurations during development to help
mature the missile's design. Hardware and software
performance was assessed at subsystem and system
levels, and design changes were incorporated into the
prototypes until a mature and stable missile
configuration was demonstrated. The AIM-9X program
held design reviews for the 11 subsystems between
October 1997 and March 1998. The early design
reviews, prototypes, and early testing allowed the
program to achieve a stable design at the system
design review in March 1998. At that time, the
contractor had released 94 percent of its engineering
drawings to manufacturing.

Production Maturity
The AIM-9X program does not contractually require
collection of statistical process control data on critical
manufacturing process, but it has undertaken an
acquisition strategy to incentivize the contractor to
reach cost and quality goals. However, the contractor
and program officials believe that they have significant
knowledge about producing the missile. The AIM-9X is
a variant of the AIM-9M missile and uses components
produced for other weapon systems, providing the
program with significant production knowledge. In
addition, to improve the production capabilities, the
contractor built developmental units on production
equipment. Program officials believe this practice has
allowed them to mature the manufacturing processes.
According to program officials, most of the critical
processes on the AIM-9X are at the subcontractor level
and a process exists to attain cost and quality goals.
This is accomplished primarily by postproduction
inspections to track production yield, scrap, and




Page 24                                                                           GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                               Common Name: ATIRCM/CMWS

  Advanced Threat Infrared Countermeasures/Common Missile Warning System

 The Army’s and Special Operations’ ATIRCM/CMWS
 is a component of the Suite of Integrated Infrared
 Countermeasures planned to defend U.S. aircraft
 from advanced infrared-guided missiles. The system
 will be employed on Army and Special Operations
 aircraft. ATIRCM/CMWS includes an active infrared
 jammer, a missile warning system, and a
 countermeasure dispenser capable of loading and
 employing expendables, such as flares, chaff, and
 smoke.




  Prime contractor: BAE Systems
  Program office: Huntsville, Ala.
  FY 2004 funding request:                FY 2003 dollars in millions                 Approved             Latest    Percent
   R&D $7.2 million                                                                        3/96             12/01    change
   Procurement $75.7 million                 Research & development cost                 $568.0            $540.1        -4.9
   Quantity 2 units                          Procurement cost                          $2,338.9          $1,971.9      -15.7
                                             Total program cost                        $2,906.9          $2,512.0      -13.6
                                             Program unit cost                           $0.940            $2.330      148.0
                                             Total quantities                             3,094             1,078      -65.2
                                             Acquisition cycle time (months)          Classified        Classified        NA
                                          Note: NA = not available.




                                                                      The ATIRCM/CMWS is scheduled to enter
                                                                      production in May 2003 with no assurance that
                                                                      production processes are in control. The CMWS
                                                                      portion of the ATIRCM/CMWS program entered
                                                                      limited production in February 2002 to meet an
                                                                      urgent need. Full-rate production for ATIRCM/
                                                                      CMWS was delayed because of reliability problems,
                                                                      which may indicate that production processes were
                                                                      not in control. These problems are, at least in part, a
                                                                      consequence of design proceeding with known
                                                                      shortfalls in knowledge: key technologies were
                                                                      demonstrated late in development and only a small
                                                                      number of design drawings were completed by
                                                                      design review. Resolving these knowledge shortfalls
                                                                      has led to cost and schedule increases. While the
                                                                      key technologies appear mature, reliability and
                                                                      producibility issues could necessitate design
                                                                      changes.


GAO-03-476 Acquisition Trends and Risks                                                                                  Page 25
Common Name: ATIRCM/CMWS                                                                                Appendix I

ATIRCM/CMWS Program                                      The program delayed the production decision for the
                                                         combined system an additional year to the currently
Technology Maturity                                      scheduled May 2003 date primarily due to reliability
                                                         issues. Reliability testing was halted because of
The five critical technologies for the system are
                                                         numerous failures with the ATIRCM subsystem.
mature, but they did not mature until after the system   Reliability failure can be an indicator of producibility
design review. Most of the early technology
                                                         and process control problems. The program plans to
development effort was focused on the application to
                                                         build and develop six additional subsystems during
rotary wing aircraft. However, when product              2002 and 2003. The full-rate production decision for
development began in 1995, the requirements were
                                                         the complete system is now scheduled for 2005.
expanded by Office of the Secretary of Defense
direction to include Navy and Air Force fixed wing       Other Program Issues
aircraft. According to program officials, they did not
fully anticipate the additional technology needed to     The Army procured an initial 32 systems in fiscal year
meet these much more demanding requirements. This        2002 that only included the CMWS. The Army plans to
change caused problems that largely contributed to       procure a total of 99 ATIRCM/CMWS systems to outfit
cost increases of more than 150 percent to the           special operations aircraft between fiscal year 2002
development contract. The Navy and the Air Force         and 2009.
subsequently dropped out of the program, rendering
the extra effort needless.                               Program Office Comments
                                                         In commenting on a draft of this assessment, program
Design Maturity                                          officials stated that the Army eliminated the program's
The basic design of the system is complete, with         funding for fiscal years 2002 and 2003. In fiscal year
100 percent of the drawings released to                  2003, the Special Operations Command funded the
manufacturing. However, reliability and producibility    urgent procurement of 32 CMWSs. Subsequently, the
issues could require design changes. The design was      Army reinstated the program for fiscal years 2004–
particularly immature at the critical design review,     2009. The program office stated that the loss of
with only 22 percent of the drawings complete. A         funding in fiscal year 2003 slowed the program
major cause was that the technology requirements         markedly. The program's acquisition strategy remains
were not well understood until the system design         to equip Special Operations forces before equipping
review, leading to the discovery that a major redesign   the remainder of the Army.
was needed to meet requirements. It was not until
                                                         The system was modified in 2002 to address ATIRCM
2 years after the design review that 90 percent of the
                                                         reliability, producibility, and built-in-test issues. Six
drawings were released and the design was
                                                         ATIRCM systems are being manufactured and tested
considered stable. According to program officials, the
                                                         to demonstrate and verify the enhancements. ATIRCM
immature design caused inefficient manufacturing,
                                                         is scheduled to begin low-rate initial production in
rework, and testing and contributed to the 3-year
                                                         May 2003, and CMWS is scheduled to begin low-rate
schedule delay.
                                                         initial production in January 2004. The program office
                                                         stated that low-rate production is required to maintain
Production Maturity
                                                         a production base. The system's operational testing is
The ATIRCM/CMWS program does not collect                 planned for March 2005. According to the program
statistical control data on its critical manufacturing   office, the prime contractor indicated that statistical
processes. Program officials have identified the         process control is not within its corporate philosophy,
absence of statistical process control data as a         particularly for a program with such low production
weakness and believe it should be instituted. Despite    rates and quantities.
this shortfall in knowledge, the Army entered limited
CMWS subsystem production in February 2002 to
meet an urgent need of the U.S. Special Operations
Command.




Page 26                                                                       GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                                           Common Name: AWS

                                   Advanced Wideband Satellite (AWS)

 The Advanced Wideband Satellite system is
 designed to provide improved, survivable, jam-
 resistant, worldwide, secure and general purpose
 communications to support the National
 Aeronautics and Space Administration, DOD and
 the intelligence community. It will replace the
 current Milstar satellite system and supplement the
 AEHF satellite system, reviewed elsewhere in this
 report. It will be the cornerstone of a DOD
 architecture that includes the multiple satellite
 systems.




  Prime contractor: In competition
  Program office: El Segundo, Calif.
  FY 2004 funding request:                FY 2003 dollars in millions        President’s budget                          Latest         Percent
   R&D $439.3 million                                                                     02/03                           00/00         change
   Procurement $0.0 million                  Research & development cost               $5,814.5                            $0.0             0.0
   Quantity 0 satellites                     Procurement cost                          $2,343.9                            $0.0             0.0
                                             Total program cost                        $8,158.4                            $0.0             0.0
                                             Program unit cost                         $2,039.6                            $0.0             0.0
                                             Total quantities within budget                   4                               0               0
                                             Acquisition cycle time (months)                 75                               0               0
                                          Note: Program costs and schedule have yet to be formally approved by DOD. Costs only through FY 2009.




                                                                       The AWS/TSAT program is scheduled to enter
                                                                       product development with only one of its five
                                                                       critical technologies mature, according to best
                                                                       practices. The initial product development period
                                                                       will likely require concurrent technology and
                                                                       product development activities to maintain
                                                                       schedule. Although the new draft space acquisition
                                                                       guidance allows this approach, it is contrary to the
                                                                       best practice of separating technology development
                                                                       from product development.




GAO-03-476 Acquisition Trends and Risks                                                                                                       Page 27
Common Name: AWS                                                                      Appendix I

AWS/TSAT Program

Technology Maturity
Of the five AWS/TSAT key space segment
technologies, one is mature while the other four are
scheduled to reach maturity by January 2006, more
than 2 years after development start. Three of the four
immature technologies have a backup technology
available in case of development difficulties. However,
use of these technologies would degrade system
overall performance. The Single Access Laser
Communications technology has no backup and,
according to program officials, any delay in maturing
this technology would result in a slip in the expected
launch date.

Other Program Issues
The program plans a development cycle that is,
according to DOD documentation, aggressive. The
satellite development cycle is planned to be
75 months: 27 months for technology development;
15 months for product development; and 33 months
for satellite build, test and launch. This period of time
is substantially shorter than the development cycle for
the AEHF satellite (118 months vs. 75 months), though
the AWS/TSAT system is expected to provide a
transformational leap in satellite communications
capability.
The program is managed under the new National
Security Space Acquisition process, which makes no
clear distinction between the end of technology
development and the start of product development.
Therefore, the AWS/TSAT acquisition strategy may
allow the system's technology development and
product development to be conducted concurrently
prior to the production decision. DOD’s acquisition
system policy states that one of the entrance criteria
for the system development and demonstration phase
is technology maturity. The AWS/TSAT acquisition
strategy does not ensure that technology maturity will
be achieved prior to the start of product development
consistent with best practices.

Program Office Comments
In commenting on a draft of this assessment, program
officials stated that the National Security Space
Acquisition Policy was developed to streamline the
decision-making framework and to tailor it for space
systems, in order to more efficiently field systems that
incorporate rapidly changing technology advances.


Page 28                                                     GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                    Common Name: CEC

                              Cooperative Engagement Capability (CEC)

 The Navy’s CEC is designed to connect radar
 systems to enhance detection and engagement of
 air targets. Ships and planes equipped with their
 version of CEC hardware and software will share
 real-time data to create composite radar tracks,
 essentially allowing the battle group to see the same
 radar picture. A CEC-equipped ship will then be
 able to detect and launch missiles against targets its
 radar cannot see. We assessed block 1 of the CEC.
 The Navy is developing a more advanced
 block 2 CEC.




  Prime contractor: Raytheon Systems Corporation
  Program office: Washington, D.C.
  FY 2004 funding request               FY 2003 dollars in millions             Approved          Latest   Percent
   R&D $72.5 million                                                                 5/95           4/02   change
   Procurement $128.6 million              Research & development cost           $1,154.0       $2,052.8      77.9
   Quantity 21 systems                     Procurement cost                      $1,292.5       $2,127.6      64.6
                                           Total program cost                    $2,493.9       $4,180.4      67.6
                                           Program unit cost                        $13.6          $15.4      12.8
                                           Total quantities                           183            272      48.6
                                           Acquisition cycle time (months)             16             16       0.0




                                                                The technologies and design of the CEC program
                                                                block 1 are fully mature. The program’s production
                                                                maturity could not be assessed. The program lacks
                                                                the necessary data primarily because the
                                                                government does not collect it on the commercially
                                                                available portions. However, program and
                                                                contractor officials consider the processes to be
                                                                capable of producing a quality product on time and
                                                                on cost. Block 1 of the CEC program was approved
                                                                in April 2002 for full-rate production for the
                                                                shipboard version and continued low-rate initial
                                                                production for the airborne version.




GAO-03-476 Acquisition Trends and Risks                                                                        Page 29
Common Name: CEC                                                                                         Appendix I

CEC Program                                                Some solutions for interoperability and integration
                                                           issues will also be assessed in follow-on testing.
Technology Maturity                                        However, many of these issues are expected to be
                                                           resolved through the introduction of block 2. The plan
In January 2002, the Office of Naval Research assessed
                                                           was approved in April 2002. Block 2 is expected to
CEC’s six critical technologies. Five of the               provide cost, performance, and functional
technologies assessed as mature were incorporated
                                                           improvements over the current system, though its
into the shipboard version when it successfully
                                                           details are yet to be defined. Among the anticipated
completed the operational evaluation in May 2001. The      characteristics of block 2 is interoperability with
sixth technology, a data processor, was not assessed
                                                           legacy combat systems.
as part of the operational evaluation but was
determined to be mature.                                   Program Office Comments
Design Maturity                                            In commenting on a draft of this assessment, program
                                                           officials stated that a production readiness review
CEC’s basic design appears complete, as all of the
                                                           conducted in October 2001 found CEC production to
drawings needed to build the shipboard version have        be mature. They evaluated all areas of production,
been released to manufacturing.
                                                           including quality, configuration management,
CEC program officials noted that new drawings              processes and procedures, drawings, and testing. They
continue to be released. They explained that as            stated that the contractor is delivering systems on
commercially available technologies, which comprise        schedule and within cost. To date, 29 systems over 5
approximately 60 percent of CEC’s hardware, become         years have been successfully delivered, installed,
more advanced, portions of the system will need to be      tested, and many have been deployed. Following
and redesigned to incorporate those advances.              operational testing and evaluation, the Navy found
                                                           CEC to be operationally suitable and effective and the
Production Maturity                                        DOD Director for Operational Test and Evaluation
                                                           found CEC demonstrated the highest reliability of any
We could not assess the CEC program’s production           system tested so far of comparable complexity.
maturity against the best practice as data were not        According to program officials, CEC’s use of
available. According to program officials, the             commercial off-the-shelf components enables the
noncommercially available portions of CEC do not           program to select mature cost-effective components
involve any critical manufacturing processes. Officials    from industry, instead of manufacturing them in-
indicated that they do not have insight into               house. In recognition of the above, DOD approved the
manufacturing processes for the commercially               program for full-rate production in April 2002.
available portions, including whether these processes
are critical and whether the contractor has them under
statistical control.
The program officials and the contractor are confident
that a quality product can be delivered on time and
within cost based on the contractor’s adherence to
industry standards and past performance on the low-
rate initial production contracts for the shipboard
version.

Other Program Issues
Battle group-level interoperability, integration, and
built-in-test false alarm rates were identified as areas
needing improvement following the operational
evaluation. Program officials expect a solution for the
alarm rates to be in place for a follow-on operational
test and evaluation planned for 2004.



Page 30                                                                        GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                      Common Name: CH-47F

                                     CH-47F Improved Cargo Helicopter

 The Army’s CH-47F heavy lift helicopter is intended
 to provide transportation for tactical vehicles,
 artillery, engineer equipment, personnel, and
 logistical support equipment. It is expected to
 operate in both day and night. The purpose of the
 CH-47F program is to improve the performance and
 extend the useful life of the CH-47. This effort
 includes installing a digitized cockpit, rebuilding the
 airframe, and reducing aircraft vibration through
 airframe stiffening.




  Prime contractor: Boeing
  Program office: Huntsville, Ala.
  FY 2004 funding request:                FY 2003 dollars in millions               Approved            Latest   Percent
   R&D $0 million                                                                        5/98            10/02   change
   Procurement $516.0 million                Research & development cost              $148.0            $169.6      14.6
   Quantity 16 aircraft                      Procurement cost                        $2,613.9         $5,927.7     126.8
                                             Total program cost                      $2,761.9         $6,097.3     120.7
                                             Program unit cost                        $9.145           $17.986      96.7
                                             Total quantities                             302              339      12.3
                                             Acquisition cycle time (months)               82               99      20.7




                                                                    The CH-47F helicopter began low-rate production in
                                                                    December 2002, although key production processes
                                                                    were not in control. Program officials believe that
                                                                    CH-47F production is low risk because no new
                                                                    technology is being inserted into the aircraft, two
                                                                    prototypes have been produced, and the production
                                                                    process has been demonstrated during the
                                                                    development phase. The CH-47F technologies and
                                                                    design appear mature, although a low percentage of
                                                                    engineering drawings were released at the design
                                                                    review. Production unit costs have more than
                                                                    doubled due to contractor rate increases, increases
                                                                    in system capabilities, and initial underestimation of
                                                                    program cost.




GAO-03-476 Acquisition Trends and Risks                                                                              Page 31
Common Name: CH-47F                                                                                       Appendix I

CH-47F Program Technology Maturity                          and (4) initial underestimation of program costs.
                                                            According to the program manager, the Army has fully
Although we did not assess technology maturity in           funded the program’s cost growth of about $2.5 billion
detail, the CH-47F is a modification of the existing        (then-year dollars). This increase in program cost
CH-47D helicopter. Program officials believe that all       necessitated rebaselining the CH-47F program. The
critical technologies are mature and have been              Army approved the CH-47F acquisition program
demonstrated prior to integration into the CH-47F           baseline.
development program.
                                                            Program Office Comments
Design Maturity                                             The CH-47F program office generally concurred with
The CH-47F design is complete, with 100 percent of          this assessment.
the drawings released to manufacturing. However, at
the design review only 37 percent of the system’s
engineering drawings were complete. Since that time,
the number of drawings completed has increased
substantially. The majority of the new drawings were
instituted to correct wire routing and installation on
the aircraft, changes program officials believed could
not be determined until after the first prototype was
developed.

Production Maturity
CH-47F production maturity could not be determined
because the program does not use statistical process
control to ensure that production processes are stable.
Program officials believe the production is low risk
because two prototypes have been produced and the
production processes have been demonstrated during
the development phase. The Army plans to conduct
operational testing in fiscal year 2004 to demonstrate
its readiness to proceed into full-rate production. Prior
to that decision, the Army plans to complete a risk
assessment for the CH-47F to eliminate any
production risk that remains.

Other Program Issues
Both the total cost and the program unit cost for the
CH-47F production program have more than doubled.
This growth triggered a Nunn-McCurdy breach (see
10 U.S.C. 2433) in December 2001, requiring a review
by the Secretary of Defense and a report to Congress.
As a result, the Secretary of Defense has certified to
Congress that the CH-47F is essential for national
security, there are no alternatives, the new cost
estimates are reasonable, and the management
structure is in place to continue to keep costs under
control. According to the program office, the cost
increases were due to (1) prime contractor labor rate
increases and material cost growth, (2) additional
system capabilities required by the Army,
(3) recapitalization of 36 Special Operations aircraft,

Page 32                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                 Common Name: Comanche

                                               RAH-66 Comanche
The Army’s Comanche is a multi-mission helicopter
intended to perform tactical armed reconnaissance. It
is designed to operate in adverse weather across a
wide spectrum of threat environments and provide
improved speed, agility, reliability, maintainability, and
low observability over existing helicopters. It is also
expected to lower operating costs through the use of
integrated diagnostics, a composite airframe, and a
bearingless rotor system. It will replace the AH-1,
OH-6, and OH-58 helicopters.




  Prime contractor: Boeing-Sikorsky
  Program office: Huntsville, Ala.
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $1.1 billion                                                                     7/00            10/02   change
   Procurement $0 million                    Research & development cost            $8,886.4        $12,556.3      41.3
   Quantity 0 aircraft                       Procurement cost                      $30,550.9        $21,939.5     -28.2
                                             Total program cost                    $39,824.0        $34,545.0     -13.2
                                             Program unit cost                       $32.831          $53.146      61.9
                                             Total quantities                          1,213              650     -46.4
                                             Acquisition cycle time (months)             222              250      14.9




                                                                    Most of the Comanche’s critical technologies have
                                                                    demonstrated acceptable levels of maturity, and the
                                                                    program appears very close to meeting the best
                                                                    practice standard for a stable design. This level of
                                                                    maturity follows many years of difficult
                                                                    development. Since the program’s first cost
                                                                    estimate was originally approved in 1985, the
                                                                    research and development cost has almost
                                                                    quadrupled and the time to obtain an initial
                                                                    capability has increased from 9 years to over
                                                                    21 years. The program has recently undergone
                                                                    another major restructuring to incorporate an
                                                                    evolutionary acquisition approach and reduce
                                                                    concurrency and lower overall risk. This
                                                                    restructuring shows promise of being a knowledge-
                                                                    based program that matches program resources
                                                                    with user requirements.



GAO-03-476 Acquisition Trends and Risks                                                                             Page 33
Common Name: Comanche                                                                                    Appendix I

RAH-66 Comanche                                             program requirements with force requirements and
                                                            program risks. Weight issues were addressed through
Technology Maturity                                         increased engine performance. Initial operational
                                                            capability was moved from December 2008 to
Seven of the Comanche’s eight critical technologies
                                                            September 2009 to reduce risk and significantly
are considered mature. Only one critical technology,        increase the amount of testing conducted.
the radar cross-section needed for low observability,
requires additional development. The Army expects
                                                            Program Office Comments
that this technology will reach maturity in fiscal
year 2005, a year before the production decision.           In commenting on a draft of this assessment, program
                                                            officials generally concurred with our assessment.
Design Maturity                                             They added that in October 2002, the Office of the
                                                            Secretary of Defense approved the Comanche
The Comanche program has released 73 percent of the         program as an evolutionary acquisition approach. The
engineering drawings to manufacturing. The program
                                                            Comanche quantity was reduced from 1213 to 650
has improved its ability to reach design maturity by
                                                            based on emerging results of the Comanche’s role in
rescheduling the design review from July 2002 to April      the Objective Force. This reduction in quantities,
2003. The program estimates that it will complete
                                                            combined with the research and development cost
90 percent of the drawings by the design review under
                                                            growth, resulted in a program acquisition unit cost
the proposed plan, instead of the former 59 percent         increase of approximately 62 percent. Excluding
under the previous program.
                                                            impacts of the quantity reduction, the average
Critical technologies have not yet been integrated and      procurement unit cost increased 18 percent and the
demonstrated on the Comanche airframe. Prior to the         program acquisition unit cost increased 23 percent.
proposed program restructure, integration of critical
technologies was considered high risk, even though
most of the technologies had reached maturity on
other platforms. Program officials believe that the
restructured program reduces integration risks and
that the longer development schedule will allow for
reduced concurrent development and additional
integration time and facilities, thereby reducing
critical risks.
The longer schedule also provides additional time for
near-term development testing, use of a production
representative aircraft for initial operational testing,
and full qualification testing. Additionally, the phasing
of development and operational tests was revised and
expanded to reduce overall program risk.

Other Program Issues
Continuing cost and schedule issues have led to the
most recent restructuring of the program. In
October 2002, the Office of the Secretary of Defense
approved the Comanche program to continue under
an evolutionary acquisition approach. However,
because of uncertainties with future funding and
capabilities, quantities were reduced from 1213 to 650
aircraft. This reduction in quantities, combined with
the research and development cost growth, resulted in
a unit cost increase of approximately 62 percent.
Program officials stated that the restructuring added a
more robust internal review process and balanced

Page 34                                                                        GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                       Common Name: ERGM

                        EX-171 Extended Range Guided Munition (ERGM)

 The Navy’s ERGM is a rocket-assisted projectile
 that is fired from a gun aboard ships. It can be
 guided to land targets at ranges of between about 10
 and 50 nautical miles to provide fire support for
 ground troops. ERGM is expected to offer increased
 range and accuracy compared to the Navy’s current
 gun range of 13 nautical miles. ERGM requires
 modifications to existing 5-inch guns, a new
 munitions-handling system (magazine), and a new
 fire control system. We assessed the projectile only.




  Prime contractor: Raytheon
  Program office: Washington, D.C.
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $28.6 million                                                                    4/97            12/02   change
   Procurement $3.8 million                  Research & development cost              $77.6            $326.1     320.5
   Quantity 0 rounds                         Procurement cost                        $308.4            $159.4     -48.3
                                             Total program cost                      $386.0            $485.5      25.8
                                             Program unit cost                       $0.045            $0.150     233.8
                                             Total quantities                         8,570              3230     -62.3
                                             Acquisition cycle time (months)              50              121     146.9




                                                                    The ERGM program began product development
                                                                    with very few of its critical technologies mature
                                                                    according to the best practices standards. While
                                                                    significant progress has been made in the past 7
                                                                    years, program officials do not expect to achieve
                                                                    maturity on all critical technologies until after the
                                                                    design review. No production representative
                                                                    engineering drawings had been released at the time
                                                                    of our assessment, and none are projected by the
                                                                    system design review. The program office currently
                                                                    expects to release these 1 year later. In June 2002,
                                                                    the program conducted a successful test of a guided
                                                                    tactical round under realistic launch conditions.
                                                                    This test did not evaluate the performance of a new
                                                                    warhead design.




GAO-03-476 Acquisition Trends and Risks                                                                             Page 35
Common Name: ERGM                                                                                          Appendix I

ERGM Program                                                 fiscal years 2002 and 2003, stretched out program
                                                             milestones and will delay deployment of ERGM until
Technology Maturity                                          2006.
Fourteen of ERGM’s 20 critical technologies have
                                                             Other Program Issues
demonstrated technological maturity. The remaining 6
technologies are approaching maturity, and program           Future program costs are not accurately reflected in
officials expect that all 20 critical technologies will be   the latest program cost estimate and the fiscal year
demonstrated in an operational environment by the            2004/2005 budget request. The cost estimate is based
end of 2003, approximately 7 months after the design         on a much lower production quantity than is contained
review. Three of the technologies yet to reach maturity      in either the approved or the current draft revision of
are part of the new unitary warhead design, and a            the ERGM acquisition program baseline. The budget
fourth is related to this change. Program officials          request does not fully fund the 80 operational test
recently identified the unitary warhead’s safe/arm           rounds currently required.
device and fuze as a critical technology, after a Navy
                                                             Two testing issues could affect the program. The
safety review concluded that it needed to be
                                                             Director of Operational Test and Evaluation has raised
redesigned to meet applicable DOD standards.
                                                             a concern about test range restrictions that could limit
The ERGM program began development with only one             realistic operational testing. Finally, the project
of its critical technologies mature. Having only one         manager stated that the availability of a fully capable
critical technology mature at the start of product           ship to support development testing could be an issue
development has caused cost and schedule problems.           due to funding shortfalls for magazine modifications
For example, when the program began, none of the             on these ships.
components of the rocket motor had been integrated
                                                             Some of the cost increases and schedule slippages to
into an ERGM representative design. Subsequent
                                                             date may be attributed to the fact that the contractor
problems with the performance reliability of the motor
                                                             relocated the program in 1998, resulting in a loss of
resulted in cost growth of more than $13 million.
                                                             trained personnel and development inertia.
Design Maturity
                                                             Program Office Comments
None of ERGM’s approximately 127 production
                                                             In commenting on a draft of this assessment, program
representative engineering drawings have been
                                                             officials stated that although production
released to manufacturing. The program office plans
                                                             representative drawings will not be available at the
for all of these drawings to be released in June 2004,
                                                             design review, the entire ERGM design would be
about one year after the design review. In the
                                                             under configuration control. Design maturity at that
meantime, the design review will be used to validate
                                                             time will be sufficient to produce all-up rounds for
the design of the development test rounds. The June
                                                             land and ship-based development testing. Based on
2004 drawing release, which will reflect knowledge
                                                             data obtained from flight and qualification tests in
gained from 8 of 18 flight tests and some qualification
                                                             fiscal year 2004, minor revisions to the ERGM
tests, will be used to build the 80 production
                                                             technical data package may be made. Production
representative operational test rounds. Program
                                                             representative drawings will be finalized by June 2004.
officials pointed out that progress has been made in
                                                             Program officials stated that they are highly
maturing the design. For example, the main elements
                                                             encouraged by the significant progress in ERGM
of the design were validated during the guided gunfire
                                                             development activities over the last 18 months. They
test in June 2002.
                                                             further stated that they have a high degree of
In January 2002, in order to meet lethality and safety       confidence that ERGM will meet all performance
requirements, the Navy decided to make a significant         requirements, while meeting the production cost goals
change to the warhead design, moving from a                  specified in the acquisition program baseline.
multiple-submunition design to a single explosive—or
unitary—warhead. This decision, coupled with the
decision to stay within planned funding levels for




Page 36                                                                          GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                   Common Name: Excalibur

                                              Excalibur Artillery Round

 The Army’s Excalibur is a family of extended range,
 precision, 155-mm artillery projectiles. It is
 designed to increase soldier survivability by
 allowing the Future Combat Systems’ nonline of
 sight cannon to fire from farther away and defeat
 threats more quickly, while reducing logistic
 support. It also is intended to be more effective
 when fired at urban targets, through a combination
 of altered trajectory and global positioning system
 accuracy.




  Prime contractor: Raytheon Missile Systems
  Program office: Picatinny Arsenal, NJ.
  FY 2004 request:                        FY 2003 dollars in millions            Approved            Latest   Percent
   R&D $134 million                                                                   4/97             2/03   change
   Procurement $0 million                   Research & development cost             $59.8            $662.8   1,008.7
   Quantity 0 projectiles                   Procurement cost                       $676.2          $4,135.9     511.6
                                            Total program cost                     $736.0          $4,798.7     552.0
                                            Program unit cost                      $0.004            $0.062   1,578.8
                                            Total quantities                      200,000            77,677     -61.2
                                            Acquisition cycle time (months)           160               136     -15.0




                                                                  The Excalibur program’s three critical technologies
                                                                  are not fully mature, even though product
                                                                  development began over 5 years ago. The
                                                                  technologies appear to be approaching maturity,
                                                                  and program officials project demonstrating
                                                                  technology and design maturity before the design
                                                                  review in 2005. Currently, 13 percent of the
                                                                  drawings are at the level that could be released to
                                                                  manufacturing. Program officials expect to have a
                                                                  stable design by the design review. The program has
                                                                  undergone a major restructuring effort. It has
                                                                  encountered a number of challenges since
                                                                  development began, including a substantial
                                                                  decrease in planned quantities, a relocation of the
                                                                  contractor’s plant, limited early funding, technical
                                                                  problems, changes in program direction, and a
                                                                  merger with another program.



GAO-03-476 Acquisition Trends and Risks                                                                           Page 37
Common Name: Excalibur                                                                                  Appendix I

Excalibur Program                                         Program Office Comments
                                                          In commenting on a draft of this assessment, program
Technology Maturity
                                                          officials generally agreed with the information in this
None of the Excalibur’s three critical technologies are   report. However, they provided the following
fully mature according to best practice standards.        clarifying comments.
According to program officials, all three have been
                                                          Concerning the Excalibur design maturity, program
demonstrated in a relevant environment and are
                                                          officials stated that approximately 600 drawings are
expected to reach maturity before the design review in
                                                          anticipated at the subsystem level. But because the
March 2005. The Excalibur’s design and requisite
                                                          program is still in research and development, no
technologies have changed since product
                                                          drawings have been officially released to
development was started. The three critical
                                                          manufacturing. The program is fabricating hardware
technologies for the current design are the guidance
                                                          in a research and development environment.
control system, the airframe, and the warhead. The
warhead was not considered a critical technology in
1997 because the Excalibur design called for a
warhead that was under production for other
munitions. Based on Army direction, the program has
undertaken development of a different warhead that is
currently undergoing testing.

Design Maturity
About 13 percent of the Excalibur’s engineering
drawings are at a level that could be released to
manufacturing. The program office plans to have all of
its drawings complete and released to manufacturing
by the design review in March 2005. However,
program officials could not estimate the total number
of drawings expected.

Other Program Issues
The program has gone through many changes since
the beginning of product development in May 1997. It
was almost immediately restructured due to limited
funding, and it was restructured again in 2001. In
response to congressional direction, the program was
restructured to merge with the joint Swedish/U.S.
program known as Trajectory Correctable Munitions.
The merger should help the program deal with design
challenges, including issues related to its folding fin
design. Also, in May 2002, the Office of the Secretary
of Defense directed the program to develop the
Excalibur for the Future Combat Systems nonline of
sight cannon and to field it in fiscal year 2008.
Although program officials have not yet released the
new cost and schedule estimates, the net effect of
these changes has been to increase the program’s
schedule and to substantially decrease planned
procurement quantities. As a result, the program’s
overall costs and unit costs have dramatically
increased.


Page 38                                                                       GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                     Common Name: F/A-18E/F

                                              F/A-18E/F Super Hornet



 The Navy’s F/A-18E/F is a multi-mission tactical
 aircraft designed to meet fighter escort,
 interdiction, fleet air defense, and close air support
 mission requirements. The program was approved
 as a major modification to earlier F/A-18 aircraft in
 1992. It is intended to complement and replace the
 Navy's F/A-18C/D and F-14 aircraft.




  Prime contractor: Boeing
  Program office: Patuxent River, Md.
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $179.0 million                                                                   6/92            12/01   change
   Procurement $3.0 billion                  Research & development cost            $6,174.4         $6,181.6       0.1
   Quantity 42 aircraft                      Procurement cost                      $61,215.3        $41,368.2     -32.4
                                             Total program cost                    $67,389.7        $47,549.8     -29.4
                                             Program unit cost                       $67.390          $86.770      28.8
                                             Total quantities                          1,000              548     -45.2
                                             Acquisition cycle time (months)             102              112       9.8




                                                                    The F/A-18E/F went into full-rate production in
                                                                    June 2000. Although the program proceeded
                                                                    without obtaining full product knowledge at key
                                                                    decision points, it embraced the concepts of
                                                                    attaining design and manufacturing knowledge
                                                                    early in development. The program released just
                                                                    over half of its engineering drawings by its design
                                                                    review. When low-rate production began, nearly all
                                                                    of the drawings were released and about 75 percent
                                                                    of the manufacturing processes were in control. The
                                                                    Navy reduced some program risk because aviation
                                                                    electronics from an earlier version of the
                                                                    F/A-18 were incorporated into the baseline
                                                                    F/A-18E/F. Furthermore, focus was placed on
                                                                    commonality between the F/A-18 C/D and the
                                                                    F/A-18 E/F, which further reduced risk.




GAO-03-476 Acquisition Trends and Risks                                                                             Page 39
Common Name: F/A-18E/F                                                                                  Appendix I

F/A-18E/F Program                                         The level of effort and timing to incorporate some of
                                                          the sensors—the AESA radar and the Advanced
Technology Maturity                                       Targeting sensor—may prove to be a challenge. We
                                                          have assessed the AESA radar elsewhere in this
We did not assess the technology maturity of the
                                                          report.
F/A-18E/F program because it is already in full-rate
production. Nevertheless, we did not identify any
                                                          Program Office Comments
technical challenges during the development of this
aircraft in our previous reviews.                         In commenting on a draft of this assessment, program
                                                          officials stated initial schedule delays were due to a
Design Maturity                                           procurement reduction of 10 aircraft in a 1998
                                                          Program Objective Memorandum. Since that time, the
The F/A-18E/F design appears complete. The program
                                                          contractor has consistently delivered aircraft ahead of
has released 100 percent of the design drawings to        schedule. Program officials also noted that the aircraft
manufacturing. At the time of the critical design
                                                          are demonstrating two to three times the quality of the
review in July 1994, 56 percent of the engineering
                                                          F/A-18C/D and have provided measurable
drawings were completed and released to                   improvements to squadron readiness. In addition, all
manufacturing for aircraft structure and systems.
                                                          F/A-18E/F preplanned upgrades continue to track to
According to program officials, they decided to
                                                          their program schedules. The Joint Mounted Helmet
proceed despite the low level of completed drawings       Cueing System has completed operational evaluation,
because the knowledge gathered from earlier
                                                          and the system has been incorporated into lot 24 of the
F/A-18C/D models gave them confidence that the
                                                          aircraft (deliveries of which began in September
design was stable. By the time of the low-rate initial
                                                          2001). Deliveries of the Advanced Targeting Forward
production decision, 99 percent of the drawings had       Looking Infrared Sensor production units began in
been released.
                                                          April 2002, and the units were deployed in January
                                                          2003. Finally, program officials stated that the AESA
Production Maturity                                       radar program continues to execute as planned, and
According to program officials, they currently have       the program has received the first engineering and
100 percent of their critical manufacturing processes     manufacturing development unit.
under control, according to the best practice standard.
Therefore, they are no longer tracking processes using
statistical process control. However, defects are still
monitored through inspections, failures, and age
exploration testing, and during maintenance. If
production problems are identified, the program
would resume statistical process control analysis
where necessary.
Program officials estimate that about 75 percent of
key manufacturing processes were in control at the
low-rate production decision in March 1997. Program
officials stated that they concentrated on maturing
their manufacturing processes before starting
production. As a result of these efforts, labor
efficiency rates have steadily improved.

Other Program Issues
The F/A-18E/F will not reach its full potential until
after the incorporation of several preplanned
upgrades—the Active Electronically Scanned Array
(AESA) radar, the Joint Mounted Helmet Cueing
System coupled with the AIM-9X missile, and the
Advanced Targeting Forward Looking Infrared sensor.

Page 40                                                                       GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                       Common Name: F/A-22

                                                      F/A-22 Raptor

 The Air Force’s F/A-22, originally planned to be an
 air superiority fighter, will also have air-to-ground
 attack capability. It is being designed with advanced
 features, such as stealth characteristics, to make it
 less detectable to adversaries and capable of high
 speeds for long ranges. It also has integrated
 aviation electronics (avionics) designed to greatly
 improve pilots' awareness of the situation
 surrounding them. It is designed to replace the Air
 Force’s F-15 aircraft.




  Prime contractor: Lockheed Martin
  Program office: Dayton, Ohio
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $936.5 million                                                                   2/92            12/02   change
   Procurement $4.2 billion                  Research & development cost           $20,938.2        $30,836.1      47.3
   Quantity 22 aircraft                      Procurement cost                      $54,272.2        $39,049.2     -28.0
                                             Total program cost                    $75,461.2        $70,469.4       -6.6
                                             Program unit cost                       $116.5           $253.5      117.7
                                             Total quantities                           648               278     -57.1
                                             Acquisition cycle time (months)            203               230      13.3




                                                                    Because the F/A-22 Program Office stopped
                                                                    collecting process control data in 2000, the program
                                                                    began production in 2001 with no proof that
                                                                    processes were in control, as defined by best
                                                                    practice standards. Technology appears mature and
                                                                    the design appears stable; however, problems with
                                                                    the vertical tail and the avionics have been
                                                                    discovered recently, which require design
                                                                    modifications. Delays in capturing technology,
                                                                    design, and production knowledge and these latest
                                                                    problems contributed to cost increases and
                                                                    schedule delays. The potential exists for further
                                                                    cost increases and schedule delays as a significant
                                                                    amount of the test program remains, including
                                                                    operational tests. Also, the latest production cost
                                                                    estimate is likely to increase because of several
                                                                    factors, and the estimate assumes over $25 billion in
                                                                    offsets from cost reduction plans.



GAO-03-476 Acquisition Trends and Risks                                                                             Page 41
Common Name: F/A-22                                                                                       Appendix I

F/A-22 Program                                             practice standards. In September 2001, the Air Force
                                                           awarded a contract for 10 aircraft to begin F/A-22
Technology Maturity                                        production.
Although we did not assess the F/A-22 key
                                                           Other Program Issues
technologies using technology readiness levels, the
three critical technologies (supercruise, stealth, and     In September 2001, the Air Force acknowledged an
integrated avionics) appear mature. Two of these           estimated production cost increase of $5.4 billion
technologies, integrated avionics and stealth, were        (then-year dollars) over the congressional cost limit.
late to mature. It was not until September 2000, or        We believe conditions exist that makes it likely
over 9 years into product development, that the            production costs will increase even further. In
integrated avionics reached maturity. During               addition, the Air Force is counting on over $25 billion
development, the integrated avionics was a source of       in cost reduction plans to offset estimated cost growth
schedule delays and cost growth. Since 1997, avionics      and enable the program to meet the production cost
software development and flight-testing have been          estimate. If these cost reduction initiatives are not
delayed, and the cost of avionics development has          achieved as planned, production costs could increase.
increased by over $980 million. Moreover, the Air          Further, the contractor has yet to demonstrate it can
Force did not complete an evaluation of stealth            efficiently build the development aircraft, and
technology on a full-scale version of the aircraft until   estimates of the cost to build the production aircraft
several years into product development.                    continue to increase.
                                                           In December 2002, DOD estimated development costs
Design Maturity
                                                           would increase by $876 million and that the funding
The basic design of the F/A-22 is essentially complete,    necessary to cover this cost increase would be
as engineering drawings are complete. However,             transferred from production funding. Avionics
design changes have been necessary as a result of          problems discovered in flight-testing are the primary
flight tests and structural tests. For example, problems   contributor to a six-month extension to the
with excessive movement of the vertical tails and          development program.
avionics failures in flight tests were discovered, and
they will require costly design modifications. The Air     Program Office Comments
Force still has to complete a significant amount of
development testing and operational testing. Until         In commenting on a draft of this assessment, program
initial operational testing is completed as planned in     officials stated that the report implies that had the F/A-
June 2004, the possibility of additional design changes    22 deferred product development until engineering
remains.                                                   and testing were accomplished, at a level providing
                                                           higher product knowledge, substantial cost increases
Design knowledge for the F/A-22 was built slowly.          and schedule delays would have been prevented. The
Only 26 percent of the total drawings were released at     issues cited as examples do not pose a substantial risk
the 1995 design review. The program released               to either cost or schedule and have either been fixed
90 percent of the drawings over 3 years later, after the   through minor design change or are anticipated to be
first two development aircraft had been delivered.         resolved without major impact to continued testing
Late drawing release contributed to parts shortages        and production. A program of this nature is expected
and work performed out of sequence during assembly,        to have both design and technological maturities to
which drove up costs and contributed to delaying           overcome and there will be some element of risk
flight tests by 83 months.                                 throughout its development and into the production
                                                           process.
Production Maturity
The program office stopped collecting process control
information in November 2000. The contractor
estimated that nearly half of the key processes had
reached a marginal level of control, but not up to best




Page 42                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                      Common Name: JASSM

                            Joint Air-to-Surface Standoff Missile (JASSM)


 JASSM is a joint Air Force and Navy program
 designed to attack surface targets outside of the
 range of area defenses. JASSM will be delivered by
 a variety of aircraft, including the F-16 C/D, the
 B-52H, the F/A-18E/F, the B-2, and the B-1B. The
 system includes the missile, software, and software
 interfaces with the host aircraft and mission
 planning system.




  Prime contractor: Lockheed Martin
  Program office: Eglin, Fla.
  FY 2004 funding request                 FY 2003 dollars in millions               Approved           Latest   Percent
   R&D $56.3 million                                                                    11/98           12/02   change
   Procurement $102.5 million                Research & development cost               $863.6        $1,209.5      40.1
   Quantity 250 missiles                     Procurement cost                        $1,078.7        $2,568.4     138.1
                                             Total program cost                      $1,962.9        $3,777.8      92.5
                                             Program unit cost                         $0.795          $0.852       7.2
                                             Total quantities                           2,469           4,434      79.6
                                             Acquisition cycle time (months)               75              87      16.0




                                                                    The JASSM program entered production in
                                                                    December 2001 without ensuring that production
                                                                    processes were in control, according to best
                                                                    practice standards. However, program officials
                                                                    indicated that they have demonstrated the
                                                                    production processes and that they sample
                                                                    statistical data at the subsystem level. The program
                                                                    ensured that the technology was mature and that
                                                                    the design was stable at critical points in
                                                                    development, closely tracking best practice
                                                                    standards. Redesign remains one area of concern
                                                                    because recent test failures have led to the delay of
                                                                    operational tests. The program has identified fixes
                                                                    to the problems, and a retrofit plan is in progress.
                                                                    The contractor’s ability to attain a higher
                                                                    production rate is another area of concern.




GAO-03-476 Acquisition Trends and Risks                                                                             Page 43
Common Name: JASSM                                                                                       Appendix I

JASSM                                                      capacity and expanding facilities to support full-rate
                                                           production plus anticipated foreign military sales.
Technology Maturity                                        Program officials believe that none of the
                                                           manufacturing processes that affect critical system
The JASSM program used existing technologies and so
                                                           characteristics are problematic, although there are
its level of technology maturity is high. Although none    key production processes that have cost implications,
of the subsystems involve new technologies, three
                                                           such as the bonding for the low observable materials
critical technologies are new applications of existing
                                                           and the painting/coating application.
technologies. These three technologies are the global
positioning system anti-spoofing receiver module, the
                                                           Program Office Comments
low observable technology, and the composite
materials. The program office reports these                In commenting on a draft of this assessment, program
technologies to be mature.                                 officials stated that JASSM has established a new
                                                           benchmark for missile development by ensuring
Design Maturity                                            weapon system design maturity and production
                                                           capability were demonstrated during development
The contractor has released 100 percent of the             prior to entering low-rate initial production. JASSM’s
drawings to manufacturing. The two remaining
                                                           acquisition strategy incorporated existing technology
concerns are the software for the missile and the
                                                           to reduce program risk and speed up delivery of the
status of integration with aircraft, although program      weapon to the warfighter. The officials further stated
officials believe the risks are low.
                                                           that JASSM’s development cycle is 33 percent faster
Recent failures in development and operational tests       than comparable weapon systems, with production
have led to the delay of the remaining JASSM               unit prices 50 percent less than weapon systems with
operational tests. During an operational test on           less capability. The contractor was contracted to
October 10, 2002, the missile flew its planned route       produce 82 all-up production prove-out test rounds
and penetrated the target, but it failed to detonate.      during development on the production line prior to
According to program officials, this failure occurred      low-rate initial production missile delivery. Program
because the requested test methodology was                 officials noted that establishment of production
experimental and exceeded original design                  representative hardware during development was key
requirements for the fuze. On October 24, 2002, during     to the contractor’s ability to prove out all production
the last of 11 developmental tests, the missile went out   processes. The contractor has a capitalization plan to
of control and crashed at the test site. According to      meet full-rate production quantities.
program officials, this failure was due to a failed
actuator. Program officials believe they have identified
the problems in both cases and have a retrofit plan.
Retrofits will be tested in spring 2003. However, if
additional problems occur, they will have to be
corrected while JASSM is in production, which may
require additional retrofitting of missiles already
produced.

Production Maturity
Program officials do not collect production process
control data at the system level. However, they stated
that all production processes had been demonstrated
and that statistical data is collected at the subsystem
level and is sampled as required. Program officials
indicated that the contractor will produce at the rates
required for the first production lot and 76 missiles
will be delivered. A contract for the second lot, 100
missiles, has been signed. Production concerns
remaining include achieving full-rate production


Page 44                                                                        GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                 Common Name: Joint Common Missile

                                             Joint Common Missile

 The Joint Common Missile is an air-launched and
 potentially a ground-launched missile designed to
 target tanks; light armored vehicles; missile
 launchers; command, control, and communications
 vehicles; bunkers; and buildings. It is designed to
 provide line of sight and beyond line-of-sight
 capabilities. It can be employed in a fire-and-forget
 mode—providing maximum survivability—or a
 precision attack mode, providing the greatest
 accuracy. The Joint Common Missile will be a joint
 Army and Navy program with USMC participation.




  Prime contractor: In competition
  Program office: Huntsville, Ala.
  FY 2004 funding request:                FY 2003 dollars in millions                                                         Latest        Percent
   R&D $183.8 million                                                                             Approved                     11/02        change
   Procurement $0 million                    Research & development cost                                NA                  $563.92               0
   Quantity 0 missiles                       Procurement cost                                           NA                  $1,597.3              0
                                             Total program cost                                         NA                  $2,161.2              0
                                             Program unit cost                                          NA                      TBD               0
                                             Total quantities                                           NA                     8,425              0
                                             Acquisition cycle time (months)                            NA                        60              0
                                          Note: Funding from FY 2004 President’s Budget. Total Army and Navy Acquisition Objective is 77,400. Official
                                          cost position to be finalized by 6/2003. NA = not applicable.




                                                                        The Joint Common Missile is scheduled to enter
                                                                        product development before any of its critical
                                                                        technologies are fully mature, according to best
                                                                        practices. Furthermore, program officials currently
                                                                        project that the critical technologies will not reach
                                                                        maturity until a year after the design review. The
                                                                        Army will initially focus development on an air-
                                                                        launched version.




GAO-03-476 Acquisition Trends and Risks                                                                                                           Page 45
Common Name: Joint Common Missile                                                                       Appendix I

Joint Common Missile System                               risk. Prototype testing of a multi-mode seeker (tower
                                                          and captive flight), a multi-purpose warhead (heavy
Technology Maturity                                       armor and building structures), and a rocket motor
                                                          (high maximum to minimum thrust profiles over
None of the Joint Common Missile’s three critical
                                                          operational temperatures) is currently being
technologies have demonstrated full maturity. These       conducted with results to be available in sufficient
critical technologies include a multi-mode seeker for
                                                          time to support the milestone decision to begin the
increased countermeasure resistance, a boost-sustain
                                                          development phase.
propulsion for increased standoff range, and a multi-
purpose warhead for increased lethality capability.
Program officials noted that many of the components
of these technologies are currently in production on
other missile systems, but that they have not been
fully integrated. While backup technologies exist for
each of the critical technologies, substituting any of
them would result in degraded performance or
increased costs.

Design Maturity
Program officials project that full integration of the
subsystems into the Joint Common Missile will be
mature one year after the system design review, which
is scheduled for July 2004.

Other Program Issues
The current cost estimates are from the fiscal year
2004 President’s budget. This cost estimate will be
updated at the conclusion of the Army's formal
estimating process. The formal estimating process
began in January 2003 for presentation at the
milestone decision review in September 2003.
According to program officials the Army's acquisition
objective is 54,400 missiles and the Navy's acquisition
objective is 23,000. Program officials also indicated
that the modular design will reduce life-cycle costs,
including demilitarization, and will enable continuous
technology insertion to ensure improvements against
advancing threats.

Program Office Comments
In commenting on a draft of this assessment, program
officials stated that they plan to demonstrate the
technological maturity required by DOD acquisition
system policy before beginning the development
phase in September 2003. Program officials further
stated that the technological maturity projected
represents a major achievement in the technology's
demonstrated readiness in a relevant environment and
provides the critical technologies the maturity
necessary to accomplish system integration of
demonstrated subsystems, thereby reducing program


Page 46                                                                       GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                    Common Name: JPATS

                           Joint Primary Aircraft Training System (JPATS)

 JPATS is a joint acquisition by the Air Force and the
 Navy to replace the aging primary trainer aircraft
 fleet. JPATS is a variant of the Beech Pilatus PC-9
 commercial aircraft, but it has been modified
 significantly to incorporate military unique
 requirements. The JPATS program includes the
 aircraft; the ground-based training system
 (simulators, course materials), and an integrated
 training management system. We assessed the
 aircraft.




  Prime contractor: Raytheon
  Program office: Wright Patterson AFB, Ohio
  FY 2004 funding request:                FY 2003 dollars in millions             Approved           Latest   Percent
   R&D $0 million                                                                      8/95           12/01   change
   Procurement $283.0 million                Research & development cost            $349.1           $294.2     -15.7
   Quantity 52 aircraft                      Procurement cost                      $2,720.3        $4,316.8      58.7
                                             Total program cost                    $3,138.6        $4,674.8      49.0
                                             Program unit cost                      $4.408           $5.970      35.4
                                             Total quantities                           712             783      10.0
                                             Acquisition cycle time (months)             97             113      16.5




                                                                   The JPATS aircraft entered full production in
                                                                   December 2001 without ensuring that the
                                                                   manufacturing processes were mature. The aircraft
                                                                   entered limited production in 1995 before achieving
                                                                   design stability. DOD considered the aircraft a
                                                                   mature commercial product that did not require
                                                                   extensive product development. However, program
                                                                   officials underestimated the number of design
                                                                   changes needed to accommodate the military
                                                                   unique requirements. The design has subsequently
                                                                   changed about 70 percent from the commercial
                                                                   baseline. The JPATS initial operating capability
                                                                   occurred in 2002, 2 years later than originally
                                                                   planned.




GAO-03-476 Acquisition Trends and Risks                                                                           Page 47
Common Name: JPATS                                                                                          Appendix I

JPATS Program                                                Program Office Comments
                                                             In commenting on a draft of this assessment, program
Technology Maturity
                                                             officials disagreed with our analysis of production
Although we did not assess the JPATS aircraft key            maturity. They stated that statistical process control is
technologies, the aircraft is a derivative of a              not the only determinant of maturity. The production
commercial aircraft and the technologies appear              line was certified by the International Organization for
mature.                                                      Standardization in 1994 and by the Federal Aviation
                                                             Administration in 1999, and is currently producing
Design Maturity                                              aircraft according to these guidelines. Assembly labor
The basic design of the aircraft is currently complete.      hours per aircraft are on a 78 percent learning curve,
                                                             and they have decreased 65 percent since the first
However, the military unique design was only about
                                                             operational aircraft was delivered. The production line
5 percent complete shortly after the program was
approved to enter limited production in 1995. The            rate increased to five aircraft per month by the end of
                                                             2002, and remains there still, even as design changes
design has changed about 70 percent from its
                                                             are incorporated into the production line. After initial
commercial baseline. Testing has revealed tangible
examples of design immaturity. Several subsystems,           production difficulties, over the past year the
                                                             contractor has been delivering aircraft ahead of
including the engine, the UHF radio, and the
                                                             schedule while incorporating engineering changes to
environmental control system, have required
extensive modification or redesign. These and other          increase the suitability of the system. Program
                                                             officials also stated that the cycle time should be
problems have delayed both aircraft testing and the
                                                             reduced by 6 months because the JPATS program was
production decision.
                                                             unable to award a contract or proceed with contract
In November 2001, operational testers concluded that         performance pending the disposition of several bid
JPATS was operationally effective but not                    protests.
operationally suitable. They cited concerns about the
aircraft's reliability, availability, and maintainability.   GAO Comments
They also reported that the full JPATS had not yet
been tested due to uncorrected deficiencies in the           Our prior work has shown that leading commercial
aircraft and the immaturity of the software-intensive        firms rely on statistical control data as the best
training information management system. The                  indicator of production readiness. Despite its
contractor is incorporating changes to the aircraft as a     commercial origins, the JPATS program entered
result of operational test issues. Operational testers       limited and full production without this information.
expressed concern that some changes may adversely            Subsequent testing has uncovered numerous problems
impact other critical subsystems. Despite these issues,      that require modification and retrofit. Although the
the Air Force proceeded into full-rate production the        aircraft has been production certified by the Federal
following month.                                             Aviation Administration, its regulations merely require
                                                             the contractor to maintain a generic quality control
Production Maturity                                          system and do not provide assurance that the
                                                             components can be built within cost and on schedule.
Production maturity remains at issue because                 We used DOD official documents to determine
information about the contractor's manufacturing             acquisition cycle time.
process controls is not available. The Air Force did not
require this information because the aircraft was
considered a commercial derivative.
Other factors could affect production maturity. In
2002, two key modifications—the environmental
control system and the UHF radio—began to be
incorporated on the aircraft. The program office has
also identified additional retrofit requirements and is
evaluating a replacement for the collision warning
system. The rework associated with these changes
may affect aircraft production efficiencies.

Page 48                                                                           GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                           Common Name: JSF

                                          F-35 Joint Strike Fighter (JSF)

 The JSF program goals are to develop and field a
 family of stealthy, strike fighter aircraft for the
 Navy, Air Force, Marine Corps, and U.S. allies, with
 maximum commonality to minimize life-cycle costs.
 The carrier suitable version will complement the
 Navy F/A-18 E/F. The Air Force version will
 primarily be an air-to-ground replacement for the
 F-16 and the A-10 and complement the F/A-22. The
 short take-off and vertical landing version will
 replace the Marine Corps F/A-18 and AV-8B.
 Significant foreign military purchases are expected.




  Prime contractor: Lockheed Martin
  Program office: Arlington, Va.
  FY 2004 funding request:                  FY 2003 dollars in millions              Approved           Latest   Percent
   R&D $4.4 billion                                                                      10/01           12/01   change
   Procurement $0 million                      Research & development cost           $32,788.6       $32,880.8       0.3
   Quantity 0 aircraft                         Procurement cost                     $145,733.8      $147,604.7       1.3
                                               Total program cost                   $180,047.0      $180,485.5       0.2
                                               Program unit cost                         $62.8           $63.0       0.2
                                               Total quantities (U.S. only)              2,866           2,866       0.0
                                               Acquisition cycle time (months)             185             185       0.0




                                                                      The JSF program entered the development phase
                                                                      without demonstrating that its eight critical
                                                                      technologies had reached maturity according to
                                                                      best practice standards. Two technologies,
                                                                      propulsion and critical fabrication techniques, were
                                                                      very close to maturity. DOD conducted an
                                                                      independent review in 2001 and concluded that the
                                                                      technology maturity was sufficient to proceed into
                                                                      product development. The JSF program no longer
                                                                      focuses on the previous 8 technology areas, instead
                                                                      it uses a different method of integration and risk
                                                                      management that currently tracks 23 program level
                                                                      risks. We were unable to assess the new risk areas,
                                                                      but program data indicates that the majority are
                                                                      moderate risk. The program expects to have 80 to
                                                                      90 percent of its critical build-to-packages
                                                                      completed by the final design review in 2005.



GAO-03-476 Acquisition Trends and Risks                                                                              Page 49
Common Name: JSF                                                                                          Appendix I

JSF Program                                                JSF preliminary design review in late March 2003. We
                                                           were unable to review the results of those meetings
Technology Maturity                                        prior to the release of this report, but program office
                                                           data indicates the discovery of higher risk levels for
During its concept development phase, the Joint
                                                           the propulsion system and overall aircraft weight.
Strike Fighter had eight critical technologies: short
take-off vertical landing/integrated flight propulsion
                                                           Other Program Issues
control, prognostic and health management,
integrated support systems, subsystems technology,         Due to the highly complex nature of the JSF design,
integrated core processor, radar, mission systems          the Director, Operational Test and Evaluation, expects
integration, and manufacturing. We reported in             numerous test challenges for the program. These
May 2000 and again in October 2001 that low levels of      challenges include the integration of highly advanced
maturity in these technologies could increase the          sensors with the avionics systems, vertical thrust
likelihood of program cost and schedule growth.            capability for the Marine Corps version, and
                                                           performance and maintenance requirements of the
The program experienced cost growth and schedule           low observable capabilities. The program has received
concerns during the concept demonstration phase,           authority for its low-rate production quantity to reach
prior to starting product development in October 2001.     15 percent—427 aircraft—of the total production run.
This included manufacturing delays for hardware used
on the propulsion system for the Marine Corps              Program Office Comments
version. To reduce cost and schedule delays, the
program eliminated planned risk-reduction efforts and      In commenting on a draft of this assessment, program
delayed other technology demonstrations until after        officials stated that, prior to the start of the
product development began.                                 development phase, JSF’s key technologies had gone
                                                           through an extensive series of tests and
An independent review performed by DOD in 2001,            demonstrations, culminating in four experimental
using a different method than technology readiness         aircraft proving flight capabilities for each service
levels, concluded that the overall technology maturity     variant in over 200 hours of flight. An independent
of the JSF program was sufficient to enter into            DOD review concluded that JSF had demonstrated
product development. Today, the program no longer          sufficient technical maturity for low risk entry into the
monitors the eight specific technologies from the          development phase. For this phase, the program
previous phase. Instead, the program is using              officials stated that JSF has adapted the contractor's
Lockheed Martin’s Key System Development                   risk mitigation approach. Risk mitigation assessments
Integration approach to monitor overall technology         in February 2003 indicated that most program level
development and design integration. Further, the           risks were rated moderate using the contractor’s
program tracks 23 program level risk areas and has         approach. Cost and schedule planning for the
assessed 19 as moderate and 2 as high. Five of eight       development phase has evolved as the services
critical technologies from the concept development         iterated system operational requirements with life
phase are contained within elements of these program       cycle cost. The JSF air system preliminary design
level risks. We have not evaluated the current JSF         review is scheduled in March 2003, and the first of
technique for assessing risks.                             three critical design reviews is to occur in April 2004.
                                                           Finally, program officials stated that the program is
Design Maturity                                            being executed in accordance with its cost, schedule,
The program has committed time and funding to the          and technical baselines.
system development and demonstration phase that
should improve its chances for success. Specifically,
the new program structure will now include additional
test aircraft, increased software on the aircraft, and a
greater number of flight test hours. Program
documents indicate that the 1996 estimated cost and
schedule for JSF’s development phase have increased
by 56 percent and 40 percent, respectively, due to
changes in program scope. Meetings were held for the


Page 50                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                   Common Name: JSOW

                                          Joint Standoff Weapon (JSOW)

 JSOW is a joint Air Force and Navy guided bomb to
 attack targets from outside of the range of most
 enemy air defenses. There are three JSOW variants
 that use a common air vehicle. Two variants (JSOW
 A and B) carry submunitions to attack soft targets
 or armored vehicles. The unitary variant (JSOW C)
 uses a seeker, autonomous targeting acquisition
 software, and a single warhead to attack targets. We
 assessed the unitary variant and the common air
 vehicle.




  Prime contractor: Raytheon Systems Company
  Program office: Patuxent River, MD
  FY 2004 funding request:              FY 2003 dollars in millions             Approved           Latest   Percent
   R&D $0.8 million                                                                  6/92           12/01   change
   Procurement $65.89 million             Research & development cost             $325.2           $311.5       -4.2
   Quantity 175 missiles                  Procurement cost                       $3,871.4         $803.8      -79.2
                                          Total program cost                     $4,196.6        $1,115.3     -73.4
                                          Program unit cost                       $0.538           $0.372     -30.9
                                          Total quantities                          7,800           3,000     -61.5
                                          Acquisition cycle time (months)              89             112      25.8




                                                               The JSOW program is scheduled to begin low-rate
                                                               production in March 2003 without knowing that
                                                               production processes are in control, according to
                                                               best practice standards. The program instead relies
                                                               on an after-production process of inspection to
                                                               discover defects. Immature technology at the start of
                                                               development at least partially delayed design
                                                               maturity, and developmental testing of the seeker is
                                                               not complete.




GAO-03-476 Acquisition Trends and Risks                                                                         Page 51
Common Name: JSOW                                                                                         Appendix I

JSOW Program                                               Program Office Comments
                                                           In commenting on a draft of this assessment, program
Technology Maturity
                                                           officials stated that the contractor has completed
The JSOW unitary’s technology appears mature. The          17 consecutive months of on-schedule deliveries,
program office identified the imaging infrared seeker      increasing the inventory to over 850 combat ready
with the autonomous acquisition software as the only       assets. In addition, program officials noted that the Air
critical technology for the system. The seeker was not     Force has upgraded its JSOW inventory to mission
mature at the start of development, but it did             ready as a result of a successful resolution of
demonstrate maturity in October 2001—over three-           remaining manufacturing, navigation, and vibration
fourths through development—when it was flown              tolerance issues. The JSOW unitary continues
aboard an aircraft in a captive flight test. Program       development and its performance is being monitored
officials stated that in three free-flight tests, the      by the program office.
seeker's performance substantially exceeded
requirements.

Design Maturity
The JSOW unitary variant’s basic design appears
complete. At the system design review in May 2002,
the program office had completed 99 percent of the
drawings. The Navy included nine developmental tests
in its development program—three sled tests with the
warhead, three free flights with the seeker, and three
combined warhead/seeker tests. The Navy has
completed two of the warhead sled tests and the
seeker free-flight tests.

Production Maturity
JSOW production maturity could not be determined
because the contractor does not use statistical process
controls to ensure that production processes are
stable and units are produced with few, if any, defects.
Rather, the contractor uses a process of post-
production inspection to control production quality.
The contractor collects this postproduction data on a
factorywide basis that includes JSOW production but
is not specific to it.
According to the program office, the contractor
delivered end items in the past that included
manufacturing defects. The program office attributes
these defects at least partially to suppliers and to
reorganization and relocation of the prime contractor
to Tucson, Arizona. To mitigate the risk of further
manufacturing problems, the Navy has instituted a
series of reviews of major suppliers. The Navy will
conduct an additional production readiness review
after the low-rate production is approved. Program
officials report that the contractor is meeting its
revised production schedule and that the scrap and
rework rates remain low.



Page 52                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                  Common Name: NPOESS

   National Polar-orbiting Operational Environmental Satellite System (NPOESS)

 The NPOESS is a joint National Oceanic and
 Atmospheric Administration (NOAA), DOD, and
 National Aeronautics and Space Administration
 satellite program to monitor the weather and
 environment. Current NOAA and DOD satellites will
 be merged into a single national system (NPOESS),
 with projected savings of at least $1.3 billion. The
 program consists of five segments: space;
 command, control, and communications; interface
 data processing; launch; and field terminals.




  Prime contractor: Northrop Grumman Space Technology
  Program office: Silver Spring, Md.
  FY 2004 funding request:              FY 2003 dollars in millions             Approved            Latest   Percent
   R&D $544.4 million                                                                8/02             9/02   change
   Procurement $0 million                 Research & development cost            $4,029.2         $4,431.6      10.0
   Quantity 0 satellites                  Procurement cost                       $1,155.6         $1,264.5       9.4
                                          Total program cost                     $5,628.2         $6,183.4       9.9
                                          Program unit cost                       $938.0          $1,030.6       9.9
                                          Total quantities                              6                6       0.0
                                          Acquisition cycle time (months)             172              174       1.2




                                                                 The NPOESS program entered product
                                                                 development in August 2002 with most of its
                                                                 technologies mature. The program also completed a
                                                                 significant portion of the engineering drawings well
                                                                 in advance of the design review; however, the total
                                                                 number has yet to be determined. Over 5 years ago,
                                                                 program officials considered the program to have
                                                                 several high-risk areas. Since then, officials have
                                                                 implemented several efforts, which are expected to
                                                                 reduce all program areas to low risk by the first
                                                                 NPOESS launch, currently scheduled for the 2008-
                                                                 2009 time frame. Perhaps the most significant step
                                                                 taken to reduce risk was to put the pacing space
                                                                 sensor technologies into full development in
                                                                 advance of the satellite system itself.




GAO-03-476 Acquisition Trends and Risks                                                                          Page 53
Common Name: NPOESS                                                                                        Appendix I

NPOESS Program                                               radiometer suite, the cross-tracked infrared sounder,
                                                             the advanced technology microwave sounder and the
Technology Maturity                                          ozone mapper/profiler suite. This satellite will provide
                                                             the program office and the data processing centers
The NPOESS spacecraft and the sensors under
                                                             with an early opportunity to work with the sensors,
development consist of 14 key technologies; twelve           ground control, and data processing systems, thus
were mature at the start of development in August
                                                             allowing lessons learned to be incorporated into the
2002.
                                                             NPOESS satellites.
In 1997, the program office determined that the space
segment had high cost and technical risks and that the       Program Office Comments
interface data processing segment and overall system         The NPOESS integrated program office concurred
integration effort had high cost, schedule, and              with this assessment.
technical risks.
To reduce the risk to the data processing segment, two
contractors selected for program definition and risk
reduction each conducted four ground-based
demonstrations of the data processing hardware and
software components. Therefore, the program office
expects the data processing segment to be relatively
mature before product development.
Program officials indicated that they achieved
maturity by concentrating on the early development of
key individual sensors. The acquisition strategy
focused on maturing key sensor technologies using
individual development contracts structured to
demonstrate the maturity of each sensor through a
component-level design review prior to the system-
level design review. The two technologies that are not
mature are needed for two key sensors—the cross-
track infrared sounder and the conical microwave
imager/sounder. However, program officials project
that those two technologies will be mature by the
system design review in 2005.

Design Maturity
Although the total number of engineering drawings
has yet to be determined, program officials indicated
that at least 52 percent of the 6,829 currently identified
drawings were completed and released to
manufacturing by the end of January 2003. Program
officials further project that all of the currently
identifiable drawings will be complete by the system
design review in 2005.
The program is taking advantage of a unique
opportunity to demonstrate design maturity. The
NPOESS Preparatory Project, a planned
demonstration satellite, is to be launched in 2006,
about 2 to 3 years before the first NPOESS satellite
launch. The demonstration satellite is scheduled to
carry four critical sensors—the visible/infrared imager


Page 54                                                                          GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                      Common Name: PAC-3

                           Patriot Advanced Capability 3 (PAC-3) Program

 The Army’s Patriot system is a long-range, high-
 medium altitude air and missile defense system.
 PAC-3 is designed to enhance the Patriot’s ability to
 detect and identify missiles and other targets,
 increase system computer capabilities and the
 number of missiles in each launcher, improve
 communications, and incorporate a new hit-to-kill
 missile. The PAC-3 system has two primary
 components, the fire unit and the missile. We
 assessed both components.




  Prime contractor: Raytheon (prime)
   Lockheed Martin (missile segment)
  Program office: Huntsville, Ala.        FY 2003 dollars in millions              Approved           Latest   Percent
  FY 2004 funding request:                                                              2/95           12/02   change
   R&D $174.5 million                        Research & development cost            $2,760.7        $4,476.2      62.1
   Procurement $561.6 million                Procurement cost                       $3,721.9        $7904.9      112.4
   Quantity 108 missiles                     Total program cost                     $6,482.6       $12,381.2      91.0
                                             Program unit cost                       $5.170          $10.326      99.8
                                             Total quantities                          1,254           1,199       -4.4
                                             Acquisition cycle time                       66             136     106.1




                                                                    The PAC-3 program currently has only about one-
                                                                    fourth of its critical production processes under
                                                                    statistical control using best practice standards.
                                                                    Continuing problems with producing and testing the
                                                                    missiles are partially explained by the absence of
                                                                    process control and partially a consequence of
                                                                    maturing PAC-3’s design late in development.
                                                                    Technical and design challenges disrupted the early
                                                                    part of product development, causing cost and
                                                                    schedule increases and delays in attaining
                                                                    production knowledge. PAC-3’s basic design is now
                                                                    complete and the technology appears mature.
                                                                    However, the contractor must increase production
                                                                    earlier than planned because DOD decided to
                                                                    accelerate deliveries. This decision may present
                                                                    new production challenges because the contractor
                                                                    must find and train additional personnel.



GAO-03-476 Acquisition Trends and Risks                                                                            Page 55
Common Name: PAC-3                                                                                        Appendix I

Patriot PAC-3 Program                                      performance was adversely affected by PAC-3 missile
                                                           reliability and launch failures. According to program
Technology Maturity                                        officials, there were several anomalies caused by
                                                           manufacturing practices, software, and test hardware.
Although we did not assess the PAC-3 technologies
                                                           However, they believe there are no systemic issues
using technology readiness levels, the system's critical   and the anomalies have been corrected. A flight test to
technologies appear mature. However, a key
                                                           validate these corrections is scheduled for the spring
technology, the Ka band seeker, was particularly late
                                                           of 2003.
to mature. The seeker did not mature until 1999, close
to the low-rate production decision. Problems              The program has adopted an evolutionary acquisition
experienced during development increased the               approach, with production decisions every 2 to 3
seeker's cost by 76 percent and delayed the contractor     years. In October 2002, DOD decided to buy 208
in attaining design and production knowledge.              missiles covering the next 2 years. DOD plans to
                                                           accelerate the production rate immediately by adding
Design Maturity                                            a second manufacturing shift and test equipment.
                                                           Because production was not expected to be
PAC-3’s basic design is complete, with 100 percent of
                                                           accelerated to this level this early in production, the
the drawings released to manufacturing. Only
                                                           contractor must expeditiously find and train qualified
21 percent of the drawings were complete when the
                                                           personnel. The accelerated plan requires additional
program held its design review, which led to a number
                                                           funding of $239 million for fiscal years 2003 and 2004.
of problems. For example, the contractor attributed a
$101 million cost increase to first-time manufacturing
                                                           Program Office Comments
problems, such as some subsystems not fitting
together properly and some not passing ground or           In commenting on a draft of this assessment, program
environmental tests. These problems were a major           officials stated that they believe production processes
contributor to a 2-year schedule delay. To reduce          are in control. Program officials stated that they have
missile costs, the contractor has identified several       meticulously and methodically examined every critical
major design changes, which will be incorporated into      process from a labor and inspection standpoint to help
the design in 2004.                                        ensure a consistent and quality product. Despite the
                                                           less than fully successful operational tests, they also
Production Maturity                                        believe that they have the most successful
                                                           development flight test program in the history of
The program has 23 percent of the key manufacturing
                                                           missile development. They provided technical
processes used to assemble the missile and the seeker
                                                           comments, which were incorporated as appropriate.
under control. Production maturity has deteriorated
from the 35 percent that was in control at the October
1999 low-rate production decision. A switch in the
manufacturing facilities may have played a role.
According to program officials, the program entered
production before process control was emphasized to
the contractor. The contractor is still having
difficulties building the missile. For example, each
seeker still needs to be reworked about three times on
average before it passes quality inspections. Program
officials have added quality tests of components,
which have improved the situation, but the contractor
has not yet demonstrated that these tests will
eliminate the need for seeker rework in the future.

Other Program Issues
The Army conducted four operational tests in 2002;
none were completely successful. The PAC-3 system
defeated half of the targets in flight-testing. System


Page 56                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                   Common Name: SBIRS High

                             Space Based Infrared System (SBIRS) High


 SBIRS High will consist of a constellation of four
 satellites plus one spare, two sensors on a non-
 SBIRS satellite, and associated ground stations.
 SBIRS High is to provide missile warning and
 missile defense information and will be used to
 support the technical intelligence and battlespace
 characterization missions. The first launch of SBIRS
 High is scheduled for fiscal year 2007.




  Prime contractor: Lockheed Martin
  Program office: El Segundo, Cal.
  FY 2004 funding request:                FY 2003 dollars in millions                 Approved            Latest   Percent
   R&D: $617.2 million                                                                     3/98             6/02   change
   Procurement: $95.4 million                Research & development cost               $3,378.4         $6,077.3      79.9
   Quantity: 1 satellite                     Procurement cost                           $558.1          $1,417.4     154.0
                                             Total program cost                        $4,127.0         $8,241.2      99.7
                                             Program unit cost                          $825.4          $1,648.2      99.7
                                             Total quantities                                 5                5       0.0
                                             Acquisition cycle time (months)                NA               NA        NA
                                          Note: NA = not available.




                                                                      The SBIRS High program’s critical technologies
                                                                      have demonstrated acceptable levels of maturity.
                                                                      This level of maturity follows many years of difficult
                                                                      development. The level of design stability is
                                                                      unknown since the contractor was unable to
                                                                      provide information on the total number of
                                                                      releasable drawings at specific milestones.
                                                                      Similarly, production maturity could not be
                                                                      determined because the contractor does not collect
                                                                      statistical control data. The SBIRS High program is
                                                                      building the first two satellites using research and
                                                                      development funding with a first launch expected in
                                                                      fiscal year 2007. The program also recently
                                                                      underwent a major restructuring to reduce program
                                                                      risk.




GAO-03-476 Acquisition Trends and Risks                                                                                Page 57
Common Name: SBIRS High                                                                                  Appendix I

SBIRS High Program                                         10 U.S.C. 2433) occurred on December 31, 2001,
                                                           requiring a review by the Secretary of Defense and a
Technology Maturity                                        report to Congress. As a result, DOD certified to
                                                           Congress in May 2002 that the SBIRS High program is
The SBIRS High program’s three critical
                                                           essential for national security, there are no
technologies—the infrared sensor, thermal                  alternatives that provide equal or greater capability at
management, and the on-board processor—are now
                                                           less cost, cost estimates are reasonable, and the
mature. Program officials indicated that the hardware
                                                           management structure is in place to continue to keep
was built and tested in a thermal vacuum chamber           costs under control.
under expected flight conditions. When the program
began product development in 1996, none of its
                                                           Program Office Comments
critical technologies were mature, according to best
practice standards.                                        Program officials generally concurred with our
                                                           assessment and provided technical comments, which
Design Maturity                                            we have incorporated where appropriate. Program
                                                           officials added that the fiscal year 2004 budget fully
Program officials do not know how many total               funds their restructured program and directs the
drawings are expected for SBIRS High, and thus do
                                                           satellite procurement to begin in fiscal year 2006.
not track the number of releasable drawings. As a
result, we could not assess design stability relative to
best practices. Program officials did state that the
current number of releasable drawings is 2,342, about
twice the number at the time of the design review. This
means that at most, no more than half of the drawings
could have been releasable at the design review.
Design stability has been an issue for this program.
During development, the satellite was redesigned to
maintain key performance parameters. Redesign
efforts resulted in a 6-month slip to the spacecraft and
increased the requirement for ground processing.
On the other hand, the two sensors that will be aboard
non-SBIRS satellites are considered stable with
subsystem qualification nearing completion, and
integration and delivery of the flight payload are
expected within the year. The first of these sensors is
scheduled for delivery in May 2003—three months
behind schedule. This delay is attributed to problems
with radio waves emitted by the sensor's electronics
that interfere with the host satellite. Despite these
integration difficulties, data shows that the sensors
will perform much better than expected.

Production Maturity
We could not assess the SBIRS High production
maturity relative to best practice standards because
the contractor does not use statistical process control
to ensure that production processes are stable.

Other Program Issues
The total unit cost of the SBIRS High program rose
more than 25 percent in 1 year. The notification to
Congress of the Nunn-McCurdy breach (see

Page 58                                                                        GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                                      Common Name: THAAD

                            Theater High Altitude Area Defense (THAAD)

 THAAD is an element of the terminal defense
 segment of the Ballistic Missile Defense System. Its
 mission is to defend against short and medium
 range ballistic missiles. THAAD’s ability to intercept
 inside and outside the atmosphere makes effective
 countermeasures more difficult and allows multiple
 intercept opportunities. The system includes
 missiles, launchers, radars, command and control/
 battle management (C2/BM), and THAAD support
 equipment.




  Prime contractor: Lockheed Martin
  Project office: Huntsville, Ala.
  FY 2004 funding request:                FY 2003 dollars in millions                          Approved                   Latest        Percent
   R&D $730.6 million                                                                               1/92                    2/03        change
   Procurement $0 million                    Research & development cost                        $4,382.7               $10,548.0          138.0
   Quantity 0 missiles                       Procurement cost                                         NA                     NA             NA
                                             Total program cost                                      NA                      NA             NA
                                             Program unit cost                                       NA                      NA             NA
                                             Total quantities (U.S. only)                            NA                      NA             NA
                                             Acquisition cycle time (months)                         114                     NA             NA
                                          Note: Procurement schedule, funding, and quantities have yet to be determined. The THAAD schedule no
                                          longer includes production milestones. NA = not applicable.




                                                                        Most of THAAD’s critical technologies have
                                                                        demonstrated acceptable levels of maturity and the
                                                                        program appears close to meeting the best practice
                                                                        standard for a stable design. The program’s
                                                                        launcher and radar have essentially attained
                                                                        technological maturity, but the missile and the
                                                                        command and control/battle management
                                                                        components are somewhat less mature. This level
                                                                        of maturity follows many years of difficult
                                                                        development. It appears that the THAAD program
                                                                        has mostly recovered from initial problems driven
                                                                        by an early fielding requirement and poor quality
                                                                        control. The current THAAD acquisition strategy
                                                                        shows a much greater emphasis on attaining
                                                                        knowledge. The program expects to reach
                                                                        technological maturity and design stability by
                                                                        February 2004.



GAO-03-476 Acquisition Trends and Risks                                                                                                      Page 59
Common Name: THAAD                                                                                            Appendix I

THAAD Program                                                  missile in the more stressing flight environment inside
                                                               the atmosphere, and (3) prepares the system for initial
Technology Maturity                                            operational test and evaluation.
THAAD program officials assessed 47 technologies in
                                                               Program Office Comments
four major elements—command and control/battle
management; missile interceptor; launcher; and radar.          In commenting on a draft of this assessment, program
Of the four elements, the radar is currently the most          officials stated that, to ensure the highest probability
mature, followed by the launcher, command and                  of success in flight-testing, a substantial amount of
control/battle management, and the missile. The                ground testing is being conducted in the next year and
program has made progress on technology maturity               a half. This testing includes exhaustive engineering
since it began development despite early failures in           and qualification level testing on all flight components.
intercept attempts. Early flight-test failures were            Program officials further stated that the extensive
caused by a combination of the compressed test                 design, fabrication, and test preparation activity has
schedule and quality control problems. The program             been very successful to date, and the program remains
was restructured twice, before the first successful            healthy with a slightly ahead-of-schedule and under-
intercept occurred in 1999. The research and                   cost status.
development cost grew from $4.4 to $10.5 billion prior
to the program's transfer to the Missile Defense
Agency, partially as a result of these problems.
The current program strategy appears geared to
obtaining the necessary knowledge by providing more
time for maturing the technology before flight tests
and placing greater emphasis on risk reduction efforts.
This strategy includes utilizing technology readiness
levels to assess technological maturity.

Design Maturity
The program has released about 82 percent of total
drawings. Program officials expect to release about
91 percent of the drawings by the system-level design
review in February 2004. The Missile Defense Agency
is redesigning the missile to be more reliable and
testable, with significantly fewer parts than the
previous version. The first flight test of the redesigned
missile is not scheduled to occur until at least
6 months after the system design review. Depending
on the outcome, flight tests could require more design
changes and delay achieving design stability.

Other Program Issues
THAAD was recently transferred from the Army to the
Missile Defense Agency, which has restructured and
modified the contract to a block upgrade approach.
Therefore, limited information is currently available
on the total projected costs of this program.
In response to the prior program setbacks, the THAAD
project office is accelerating some risk reduction
activities, and it has planned a series of flight tests that
(1) tests the missile in a less stressing intercept
environment outside the atmosphere, (2) tests the

Page 60                                                                             GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                     Common Name: Tactical Tomahawk

                                          Tactical Tomahawk Missile

 The Navy’s Tactical Tomahawk (block IV) is a major
 upgrade to the Tomahawk Land Attack Missile
 (block III). The Tactical Tomahawk missile will
 provide ships and submarines with enhanced
 capability to attack targets on land. New features
 include improved antijamming global positioning
 system, in-flight retargeting, and ability to transmit
 battle damage imagery. The system includes the
 missile, the weapon control system, and the mission
 planning system. We assessed only the missile.




  Prime contractor: Raytheon Systems Company
  Program office: Patuxent River, Md.
  FY 2004 funding request:              FY 2003 dollars in millions             Approved            Latest   Percent
   R&D $71.4 million                                                                 9/99            12/01   change
   Procurement $277.6 million             Research & development cost             $559.5            $584.1        4.4
   Quantity 267 missiles                  Procurement cost                       $1,236.4         $1,546.0      25.0
                                          Total program cost                     $1,795.9         $2,130.0      18.6
                                          Program unit cost                       $1.316            $1.235       -6.1
                                          Total quantities                          1,365            1,725      26.4
                                          Acquisition cycle time (months)              58               69      19.0




                                                                 The Tactical Tomahawk missile entered low-rate
                                                                 production without ensuring that production
                                                                 processes were in control. Program officials
                                                                 indicated that they plan to collect production
                                                                 process control data over the next year, prior to
                                                                 award of the full-rate production contract in fiscal
                                                                 year 2004. At that time, program officials expect
                                                                 over 80 percent of the low-rate production missiles
                                                                 to be in various stages of assembly. The technology
                                                                 and design have reached acceptable levels of
                                                                 maturity. While engineering drawings have
                                                                 improved to 96 percent, the program only had about
                                                                 half of its drawings released at the design review.
                                                                 Program plans call for a full-rate production
                                                                 decision in May 2004.




GAO-03-476 Acquisition Trends and Risks                                                                          Page 61
Common Name: Tactical Tomahawk                                                                             Appendix I

Tactical Tomahawk Program                                   awarded in mid- January 2003 for 167 units. Program
                                                            officials stated that total quantities have increased to
Technology Maturity                                         2,396.
We did not assess the technology readiness levels of
                                                            Program Office Comments
the key technologies for the Tactical Tomahawk
missile. At the time of our review, critical technologies   In commenting on a draft of this assessment, program
were mature. According to the program office, the           officials stated that two development test flights,
critical technologies for the key subsystems—antijam        conducted prior to low-rate production awards,
global positioning system, digital scene matching area      demonstrated that the Tactical Tomahawk missile
correlator, and cruise engine—were modified                 design met or exceeded technical and key
derivatives from other programs or upgrades to              performance parameters. They also noted that, due to
existing Tomahawk subsystems and consequently               the stability of the design and successful completion
already mature. To date, subsystem and the majority         of all component and flight qualification testing, the
of missile-level qualification testing has been             Navy’s operational test agency issued a favorable
completed successfully.                                     operational assessment, stating that the Tactical
                                                            Tomahawk missile is potentially suitable and
Design Maturity                                             potentially operationally effective.
The basic design of the Tactical Tomahawk missile is
essentially complete. The critical design review
occurred in June 2000. At that time, approximately
47 percent of the drawings had been released to
manufacturing. In October 2002, at the first low-rate
initial production award, 723 of 750 total drawings, or
about 96 percent, had been released.

Production Maturity
Officials plan to collect statistical control data at the
start of the manufacturing process but do not expect
to have meaningful statistical data until sometime in
2004. Manufacture of the Tactical Tomahawk missile is
scheduled to begin at the subcontractor's facility in
2003 and missile assembly in 2004. Although two low-
rate production contracts have been awarded,
program officials stated that data regarding
manufacturing process controls currently is very
limited. Program officials told us that it is too soon to
know what percentage of critical manufacturing
processes will be under statistical control when the
full-rate production contract is awarded in mid-2004,
but that they plan to start collecting production
process control data over the next year.

Other Program Issues
The Tactical Tomahawk missile successfully
completed its first developmental flight test in
August 2002, and the first low-rate production
contract for 25 units was awarded in October 2002. A
second and final low-rate production contract was




Page 62                                                                          GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                         Common Name: V-22

                                                     V-22 Osprey

 The V-22 Osprey is a tilt-rotor, vertical takeoff and
 landing aircraft designed to meet the amphibious/
 vertical assault needs of the Marine Corps, long-
 range missions of Special Operations forces, and
 combat search and rescue needs of the Navy. The
 V-22 will replace the CH-46E and the CH-53A/D in
 the Marine Corps; the H-53 and H-60 will augment
 the C-130 in the Air Force and the Special
 Operations Command; and supplement the H-60 in
 the Navy. We assessed the block A version.




  Prime contractor: Bell-Boeing
  Program office: Patuxent River, Md.
  FY 2004 funding request:                FY 2003 dollars in millions              Approved            Latest   Percent
   R&D $543.9 million                                                                   2/87            12/01   change
   Procurement $1.11 billion                 Research & development cost            $3,568.5        $10,253.5     187.3
   Quantity 11 aircraft                      Procurement cost                      $29,499.2        $32,312.7       9.5
                                             Total program cost                    $33,264.7        $42,617.5      28.1
                                             Program unit cost                      $36.434           $93.051     155.4
                                             Total quantities                           913               458     -49.8
                                             Acquisition cycle time (months)             117              261     123.1




                                                                    The V-22 program plans to enter full-rate production
                                                                    without ensuring that the manufacturing processes
                                                                    are mature. Redesign of the aircraft’s hydraulic and
                                                                    electric system, and software changes have been
                                                                    made to address safety, reliability, maintainability,
                                                                    and logistics supportability. These design changes
                                                                    and others are undergoing developmental testing to
                                                                    ready the aircraft for an operational test and
                                                                    evaluation test period in late 2004 through early
                                                                    2005 to determine if the V-22 is operationally
                                                                    suitable and effective. The design changes,
                                                                    however, have not been incorporated into the low-
                                                                    rate production aircraft currently being produced.
                                                                    The value of contract modifications needed to
                                                                    address the cost of these design changes is not yet
                                                                    known. Also, parts shortages and quality issues are
                                                                    currently effecting low-rate production costs. Some
                                                                    key performance requirements have been
                                                                    eliminated.


GAO-03-476 Acquisition Trends and Risks                                                                             Page 63
Common Name: V-22                                                                                         Appendix I

V-22 Program                                               and incorporated into production aircraft. Delivery of
                                                           block A aircraft is expected to start in the fourth
Technology Maturity                                        quarter of fiscal year 2003. However, the cost of
                                                           contract modifications needed to reconfigure already
Although we did not specifically assess the V-22’s
                                                           produced aircraft and aircraft still on the assembly
technology maturity, the program office believes key       line to the bock A configuration has not been
technologies to be mature. An operational test report,
                                                           negotiated.
dated November 2000, determined that the V-22 was
not operationally suitable because of poor reliability,    Program Office Comments
maintainability, availability, human factors, and
interoperability problems. Immature technology, in         In commenting on draft of this assessment program
part, contributed to this assessment.                      officials stated that they have restructured the
                                                           program to gather more technical knowledge through
Design Maturity                                            a more rigorous “event-driven” flight test program.
                                                           Program officials strongly disagreed that the program
As a result of a crash in December 2000, the V-22 has
                                                           plans to enter full-rate production without ensuring
undergone several design changes. Specifically, the
aircraft’s hydraulic and electrical lines were             that manufacturing process are mature. V-22s are
                                                           currently being manufactured at a minimum
redesigned to improve safety, reliability,
                                                           sustaining rate (11 aircraft per year). A May 20th
maintainability, and logistics supportability. The V-22
flight control system software was also redesigned.        defense acquisition board review is scheduled to
                                                           consider increasing this rate. Manufacturing processes
The program office estimates that redesign of the V-22
                                                           and tooling are in place and being continually analyzed
resulted in 1,755 additional drawings, increasing the
                                                           and improved. Both companies utilize statistical
total number of drawings to 7,490. To date, all of these
drawings are complete.                                     process control techniques and numerous metrics to
                                                           assess program performance. They do not use the
The success of these design changes will be                process capability index, the only metric that GAO
determined as the aircraft undergoes additional            uses as a basis for their assessment. Program officials
developmental testing through 2005. Testing will           are also undertaking an affordability review to reduce
address many issues, including high rate of descent,       the aircraft unit cost to $58 million by 2010. High unit
handling qualities, austere environment operations,        costs are driven by the current low production
and ship operations. The operational assessment of         quantities and will remain the norm until production
these characteristics will not occur until late 2004 or    quantities increase.
early 2005. Recent decisions to defer some V-22
operational requirements previously considered             GAO Comments
critical until later blocks will void the need for some
design changes in the block A.                             Our prior work has shown that leading commercial
                                                           firms rely on statistical control data, specifically, the
Production Maturity                                        process capability index, as the best indicator of
                                                           production readiness. The V-22 program entered low-
Neither V-22 contractor collects statistical process       rate production without this information and has
control data on its critical manufacturing processes. A    experienced production quality problems.
recent program management assessment rated V-22
production as cautionary. Part shortages and quality
problems caused inefficiencies in shop and assembly
operations, as well as scrap, rework, repair, and
schedule delays.

Other Program Issues
Low-rate production of the V-22 continues. V-22s are
being fabricated and partially assembled, but not
delivered until the first set of upgrades—referred to as
block A—needed to bring the V-22 to a safe
operational and suitable configuration are approved

Page 64                                                                         GAO-03-476 Acquisition Trends and Risks
Appendix I                                                                                        Common Name: WGS

                Wideband Gapfiller Satellite (WGS) Communications System


 The Wideband Gapfiller Satellite system is a joint
 Air Force and Army program intended to provide
 communications to the U.S. warfighters, allies, and
 Coalition Partners during all levels of conflict short
 of nuclear war. It is the next generation wideband
 component in the DOD’s future Military Satellite
 Communications architecture.




  Prime contractor: Boeing Satellite Systems (BSS)
  Program office: El Segundo, Calif.
  FY 2004 funding request:                 FY 2003 dollars in millions              Estimate          Latest   Percent
   R&D $36.7 million                                                                   12/01           12/02   change
   Procurement $34.6 million                  Research & development cost             $181.2          $244.8      35.1
   Quantity 0 satellites                      Procurement cost                        $831.5        $1,389.8      67.2
                                              Total program cost                    $1,012.7        $1,634.6      61.4
                                              Program unit cost                       $337.6          $326.9       -3.2
                                              Total quantities                             3               5      66.7
                                              Acquisition cycle time (months)             50              55      10.0




                                                                   The WGS program’s critical technologies and design
                                                                   are mature, while its production processes are
                                                                   nearly mature. DOD plans to rely on commercial
                                                                   technologies that will not require extensive product
                                                                   development. However, two of these processes use
                                                                   statistical control rates that are below the level
                                                                   prescribed by best practice standards. The program
                                                                   recently added two satellites to better support
                                                                   intelligence, surveillance, and reconnaissance
                                                                   missions in the future.




GAO-03-476 Acquisition Trends and Risks                                                                            Page 65
Common Name: WGS                                                                     Appendix I

WGS Program
The WGS program’s two critical technologies—the
digital channelizer and the phased array antenna—are
mature. Most of these technologies are commercial
derivatives. For this reason, many of the satellite
technologies selected were already at high levels of
maturity. In fact, the program is leveraging
commercial technology and practices by modifying
commercial satellites to better support unique military
requirements.

Design Maturity
The WGS design is essentially complete, as the
program has released approximately 95 percent of the
expected drawings.

Production Maturity
The contractor has six of its eight key manufacturing
processes under control, according to the best
practice standards. Program officials indicated that
they are bringing the remaining processes under
statistical control.

Program Office Comments
In commenting on a draft of this assessment, program
officials stated that while critical technology areas
being applied to WGS are fairly mature, the
manufacturing of the systems using these technologies
is relatively new for the contractor. Risk of production
problems was to be reduced due to other commercial
satellite system developments and production ahead
of WGS in the development and production schedule.
However, due to the drastic loss of commercial
satellite orders, only one commercial satellite with
similar technologies as WGS is now leading WGS in
the manufacturing schedule. Recently identified
problems found on the “leader” program will impact
the WGS manufacturing schedule, and a first launch
schedule delay of 4 to 6 months can be expected due
to time needed to resolve the “leader” program
manufacturing problems. Satellites four and five have
been directed by DOD to be launched in fiscal year
2009 and fiscal year 2010, respectively. These dates are
outside the allowable dates of the WGS contract
option clauses and will require renegotiation to
finalize their cost. The cost is expected to increase to
compensate for loss of learning curve from over a 3-
year break in production, parts obsolescence, and
inflation.




Page 66                                                    GAO-03-476 Acquisition Trends and Risks
Appendix II

Methodology                                                                                      Appendx
                                                                                                       Ii




                 In conducting our work, we evaluated performance and risk data from each
                 of the programs included in this report. We summarized our assessments of
                 each individual program in two components—a system profile and a
                 product knowledge assessment. We did not validate or verify the data
                 provided by DOD. However, we took several steps to address data quality.
                 Specifically, we reviewed the data and performed various quality checks,
                 which revealed some discrepancies in the data. We discussed these
                 discrepancies with program officials and adjusted the data accordingly.



System Profile   In the past 3 years, DOD revised its policies governing weapon system
                 acquisitions and changed the terminology used for major acquisition
Assessment       events. In order to make DOD’s acquisition terminology more consistent
                 across the 26 program assessments, we standardized the terminology for
                 key program events. In the individual program assessments, program start
                 refers to the initiation of a program; DOD usually refers to program start as
                 milestone I or milestone A, which begins the concept and technology
                 development phase. Similarly, development start refers to the commitment
                 to product development that coincides with either milestone II or
                 milestone B, which begins DOD’s system development and demonstration
                 phase. The production decision generally refers to the decision to enter the
                 production and deployment phase, typically with low-rate initial
                 production. Initial capability refers to the initial operational capability,
                 sometimes also called first unit equipped or required asset availability.

                 The funding request information presented refers to the President’s fiscal
                 year 2004 budget request, except where noted. The program cost
                 comparisons are the latest estimates provided by the individual programs.
                 The quantities listed refer to total quantities, including both procurement
                 and development quantities.

                 To assess the cost, schedule, and quantity changes of each program, we
                 reviewed DOD’s selected acquisition reports or obtained data directly from
                 the program offices. In general, we compared the latest available selected
                 acquisition report information with a baseline for each program. For
                 systems that have started product development—those that are beyond
                 milestone II or B—we compared the latest available Selected Acquisition
                 Report to the development estimate from the first Selected Acquisition
                 Report issued after the program was approved to enter development. For
                 systems that have not yet started product development, we compared the
                 latest available data to the planning estimate issued after milestone I or A.
                 For systems not included in selected acquisition reports, we attempted to



                 Page 67                                   GAO-03-476 Acquisition Trends and Risks
                    Appendix II
                    Methodology




                    obtain comparable baseline and current data from the individual program
                    offices.

                    All cost information is presented in base year 2003 dollars, unless
                    otherwise noted, using Office of the Secretary of Defense approved
                    deflators to eliminate the effects of inflation. We have depicted only the
                    programs’ main elements of acquisition cost—research and development,
                    and procurement, however the total program costs displayed also include
                    military construction and acquisition operation and maintenance costs.
                    Because of rounding and these additional costs, in some situations the total
                    cost may not match the exact sum of the research and development and
                    procurement costs. The program unit costs are calculated by dividing the
                    total program cost by the total quantities planned. These costs are often
                    referred to as program acquisition unit costs.

                    The schedule assessment is based on acquisition cycle time, defined as the
                    number of months between the program start, usually milestone I or A, and
                    the achievement of initial operational capability or an equivalent fielding
                    date.

                    The intent of these comparisons is to provide an aggregate or overall
                    picture of a program’s history. These assessments represent the sum total
                    of the federal government’s actions on a program, not just those of the
                    program manager and the contractor. DOD does a number of detailed
                    analyses of changes that attempt to link specific changes with triggering
                    events or causes. Our analysis does not attempt to make such detailed
                    distinctions.



Product Knowledge   To assess the product development knowledge of each program at key
                    points in development, we submitted a data collection instrument to each
Assessment          program office. The results are graphically depicted in each two-page
                    assessment. The methodology used to generate each graph is discussed at
                    the beginning of appendix I. We also reviewed pertinent program
                    documentation, such as the operational requirements document, the
                    acquisition program baseline, test reports, and major program reviews.

                    To assess technology maturity, we asked program officials to apply a tool,
                    referred to as technology readiness levels, for our analysis. The National
                    Aeronautics and Space Administration originally developed technology
                    readiness levels, and the Army and Air Force Science and Technology
                    research organizations use them to determine when technologies are ready



                    Page 68                                   GAO-03-476 Acquisition Trends and Risks
Appendix II
Methodology




to be handed off from science and technology managers to product
developers. Technology readiness levels are measured on a scale of one to
nine, beginning with paper studies of a technology’s feasibility and
culminating with a technology fully integrated into a completed product.
Our best practices work has shown that a technology readiness level of 7—
demonstration of a technology in an operational environment—is the level
of technology maturity that constitutes a low risk for starting a product
development program.

In most cases, we did not validate the program offices’ selection of critical
technologies or the determination of the demonstrated level of maturity.
We sought to clarify the technology readiness levels in those cases where
information existed that raised concerns. If we were to conduct a detailed
review, we might adjust the critical technologies assessed, the readiness
level demonstrated or both. It was not always possible to reconstruct the
technological maturity of a weapon system at key decision points after the
passage of many years.

To assess design maturity, we asked program officials to provide the
percentage of engineering drawings completed or projected for completion
by the design review, the production decision, and as of our current
assessment. Completed engineering drawings were defined as the number
of drawings released or deemed releasable to manufacturing that can be
considered the “build to” drawings.

To assess production maturity, we asked program officials to identify the
number of critical manufacturing processes and, where available, to
quantify the extent of statistical control achieved for those processes. We
used a standard called the Process Capability Index, which is a process
performance measurement that quantifies how closely a process is running
to its specification limits.1 The index can be translated into an expected
product defect rate and we have previously found it to be a best practice.
We sought other data, such as scrap and rework trends in those cases
where quantifiable statistical control data was unavailable.




1
 Process Capability Index provides assurance that production processes are under
100 percent statistical control. A high index value equates to fewer defects per part based on
statistical process control data. The general rule of thumb used by the manufacturing
industry states that if the index value for a process is less than 1.33, then the process is not
capable of producing a part with acceptable consistency.




Page 69                                              GAO-03-476 Acquisition Trends and Risks
Appendix II
Methodology




Although the knowledge points provide excellent indicators of potential
risks, by themselves, they do not cover all elements of risk that a program
encounters during development, such as funding instability. Our detailed
reviews on individual systems normally provide for a fuller treatment of
risk elements.




Page 70                                   GAO-03-476 Acquisition Trends and Risks
Appendix III

GAO Contact and Acknowledgments                                                                   Appendx
                                                                                                        iI




GAO Contact       Paul Francis (202) 512-2811



Acknowledgments   David B. Best, James A. Elgas, and James L. Morrison made key
                  contributions to this report and were largely supported by GAO’s
                  Acquisition and Sourcing Management staff. The following staff were
                  responsible for individual programs.

                             Staff                                 System
                  Michael Hazard                Advanced Amphibious Assault Vehicle
                                                (AAAV)
                  Steven Martinez               Advanced Extremely High Frequency
                                                (AEHF) Communications Satellite
                  David Hubble                  Advanced Wideband
                                                Satellite/Transformational Communications
                                                Satellite (AWS/TSAT)
                  Gaines Hensley                AN/APG-79 Active Electronically Scanned
                                                Array (AESA) Radar
                  Marvin Bonner                 AIM-9X Short-Range Air-to-Air Missile
                  Thomas Gordon                 Airborne Laser (ABL)
                  Dana Solomon/Carrie           Advanced Threat Infrared
                  Wilson/Danny Owens            Countermeasures/Common Missile Warning
                                                System (ATIRCM/CMWS)
                  Johanna Ayers                 Cooperative Engagement Capability (CEC)
                  Leon Gill                     CH-47F Improved Cargo Helicopter
                  Wendy Smythe                  RAH-66 Comanche
                  Marti Dey/Ron Schwenn         EX-171 Extended Range Guided Munition
                                                (ERGM)
                  Larry Gaston                  Excalibur Artillery Round
                  Cheryl Andrew                 F/A-18E/F Super Hornet
                  Donald Springman              F-22 Raptor
                  Beverly Breen/Lynn            Joint Air-to-Surface Standoff Missile (JASSM)
                  Lavalle
                  Danny Owens                   Joint Common Missile
                  Rae Ann Sapp/Art Cobb         Joint Primary Aircraft Training System
                                                (JPATS)
                  Brian Mullins/Ron             F-35 Joint Strike Fighter (JSF)
                  Schwenn



                  Page 71                                   GAO-03-476 Acquisition Trends and Risks
           Appendix III
           GAO Contact and Acknowledgments




                       Staff                               System
           Carol Mebane                 Joint Standoff Weapon (JSOW)
           Bruce Thomas                 National Polar-orbiting Operational
                                        Environmental Satellite System (NPOESS)
           Matthew Lea                  Patriot Advanced Capability 3 Program
                                        (PAC-3)
           Tana Davis                   Tactical Tomahawk Missile
           William Lipscomb/Tana        Theater High Altitude Area Defense (THAAD)
           Davis
           Maricela Cherveny/Nancy      Space Based Infrared Satellite-High (SBIRS-
           Rothlisberger                High)
           Jerry Clark                  V-22 Osprey
           Art Gallegos/Tony            Wideband Gapfiller Satellite Communication
           Beckham                      System




(120185)   Page 72                                  GAO-03-476 Acquisition Trends and Risks
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Service Requested