oversight

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes

Published by the Government Accountability Office on 1999-07-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to the Chairman and Ranking
                   Minority Member, Subcommittee on
                   Readiness and Management Support,
                   Committee on Armed Services, U.S.
                   Senate
July 1999
                   BEST PRACTICES

                   Better Management of
                   Technology
                   Development Can
                   Improve Weapon
                   System Outcomes




GAO/NSIAD-99-162
United States General Accounting Office                                                  National Security and
Washington, D.C. 20548                                                            International Affairs Division



           B-280233                                                                                         Letter

           July 30, 1999

           The Honorable James Inhofe
           Chairman
           The Honorable Charles Robb
           Ranking Minority Member
           Subcommittee on Readiness and Management Support
           Committee on Armed Services
           United States Senate

           As you requested, this report assesses how best practices offer improvements to the way the
           Department of Defense (DOD) incorporates new technology into weapon system programs. It also
           assesses the factors that can make it difficult to mature technologies before they are included on
           weapon system programs and what can be done about them. We make recommendations to the
           Secretary of Defense on how advanced technologies can be better managed so they pose less risk
           when they are included in weapon system designs.

           We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the
           Honorable Louis Caldera, Secretary of the Army; the Honorable Richard Danzig, Secretary of the
           Navy; the Honorable F. Whitten Peters, Acting Secretary of the Air Force; the Honorable Jacob J. Lew,
           Director, Office of Management and Budget; and to interested congressional committees. We will also
           make copies available to others upon request.

           If you have any questions regarding this report, please call me at (202) 512-4841. Other key contacts
           are listed in appendix III.




           Katherine V. Schinasi
           Associate Director
           Defense Acquisitions Issues




                      Leter
Executive Summary



Purpose              The Department of Defense (DOD) plans to increase its investment in new
                     weapons to about $60 billion in fiscal year 2001—a 40-percent increase
                     over fiscal year 1997. DOD has high expectations from this investment: that
                     new weapons will be better and less expensive than their predecessors and
                     will be developed in half the time. With its traditional management
                     approach—which has produced superior weapons, but at much greater
                     cost and time than planned—DOD will not meet these expectations.
                     Leading commercial firms have changed their practices for developing
                     products and have achieved the kinds of results DOD seeks. Maturing new
                     technology before it is included in products is one of the main determinants
                     of these firms’ successes. This practice holds promise for DOD, for
                     immature technologies have been a main source of problems on weapon
                     systems. In response to a request from the Chairman and the Ranking
                     Minority Member, Subcommittee on Readiness and Management Support,
                     Senate Committee on Armed Services, GAO assessed (1) the impact of
                     technology maturity on product outcomes, (2) best practices for managing
                     new technologies and incorporating them into products, and (3) ways DOD
                     can adapt these practices to get better outcomes on weapon system
                     programs.



Background           GAO reviewed commercial and DOD experiences in incorporating
                     23 different technologies into new product and weapon system designs.
                     The technologies were drawn from (1) six commercial firms recognized for
                     their success in developing technically advanced products more quickly
                     than the products’ predecessors and (2) five DOD weapon system
                     programs that incorporated advanced technologies, including some that
                     did not encounter problems and some that did. GAO asked the managers of
                     these technologies to assess the maturity of the technologies at the point
                     they were included in product development by applying a tool, referred to
                     as technology readiness levels (TRLs). The National Aeronautics and Space
                     Administration and the Air Force Research Laboratory use TRLs to
                     determine the readiness of technologies to be incorporated into a weapon
                     or another type of system. Readiness levels are measured along a scale of
                     one to nine, starting with paper studies of the basic concept, proceeding
                     with laboratory demonstrations, and ending with a technology that has
                     proven itself on the intended product.

                     The Air Force Research Laboratory considers TRL 6 an acceptable risk for
                     a weapon system entering the program definition stage, the point at which
                     DOD launches its weapon programs, and TRL 7 an acceptable risk for the



             Leter   Page 2                                          GAO/NSIAD-99-162 Best Practices
                      Executive Summary




                      engineering and manufacturing development stage. This is an important
                      distinction because leading commercial firms launch a new product later
                      than DOD, after technology development is complete. They refer to this
                      point as the beginning of product development, the point at which they
                      commit to developing and manufacturing the product. Typically,
                      technology is still being developed when weapon system programs are
                      launched; the point at which a weapon system is far enough along to
                      compare to a commercial product development is likely to be at or after the
                      start of engineering and manufacturing development.



Results in Brief      The experiences of DOD and commercial technology development cases
                      GAO reviewed indicate that demonstrating a high level of maturity before
                      new technologies are incorporated into product development programs
                      puts those programs in a better position to succeed. The TRLs, as applied
                      to the 23 technologies, reconciled the different maturity levels with
                      subsequent product development experiences. They also revealed when
                      gaps occurred between a technology’s maturity and the intended product’s
                      requirements. For technologies that were successfully incorporated into a
                      product, the gap was recognized and closed before product development
                      began, improving the chances for successful cost and schedule outcomes.
                      The closing of the gap was a managed result. It is a rare program that can
                      proceed with a gap between product requirements and the maturity of key
                      technologies and still be delivered on time and within costs.

                      Two conditions were critical to closing the maturity gap. First, the right
                      environment for maturing technologies existed. Key to this environment
                      was making a science and technology organization, rather than the
                      program or product development manager, responsible for maturing
                      technologies to a high TRL. When a maturity gap persisted, managers were
                      given the flexibility to take the time to mature the technology or decrease
                      product requirements so that they could use another, already mature
                      technology. Second, both technology and product managers were
                      supported with the disciplined processes, readily available information,
                      readiness standards, and authority to ensure technology was ready for
                      products. This support enabled these managers to safeguard product
                      development from undue technology risks. On the other hand, immature
                      technologies were sometimes incorporated into products for reasons such
                      as inflexible performance requirements, increasing the likelihood of cost
                      overruns and delays in product development. Product managers had little
                      choice but to accept the technologies and hope that they would mature




              Leter   Page 3                                         GAO/NSIAD-99-162 Best Practices
                            Executive Summary




                            successfully. However, the pressures of product development made for an
                            environment less conducive to maturing technology.

                            For several reasons, DOD is likely to move technologies to product
                            development programs before they are mature. Science and technology
                            organizations, which traditionally operate within fixed budget levels, do
                            not necessarily have the funds to mature technology to the higher TRLs.
                            Programs are more able to command the large budgets necessary for
                            reaching these levels. The pressures exerted on new programs to offer
                            unique performance at low cost encourage acceptance of unproven
                            technologies. The technologies GAO reviewed indicate these conditions
                            can be overcome on individual cases. DOD has several initiatives
                            underway, such as advanced technology demonstrations, that could make
                            it more feasible for science and technology organizations to mature
                            technology before it is moved to product development programs. The
                            challenge will be whether the lessons learned from these cases and
                            initiatives offer an approach that has a DOD-wide application.

                            GAO makes recommendations to the Secretary of Defense on ways to
                            pursue advanced technologies while lessening their potential for causing
                            problems on weapon system programs.



Principal Findings

Maturity of Technology at   The 23 technologies GAO reviewed spanned a wide range of readiness
Program Start Is an         levels—from a low of TRL 2 to a high of TRL 9—when they were included
                            in product development programs. Programs with key technologies at
Important Determinant of
                            readiness levels 6 to 8 at the time of program launch met or were meeting
Success                     cost, schedule, and performance requirements. All of the commercial
                            technologies and a few of the DOD technologies fell into this category. For
                            example, Ford managed its voice-activated control technology to TRL 8—a
                            10-year effort—before introducing it on the 1999 Jaguar. Similarly, the
                            Defense Advanced Research Projects Agency matured a revolutionary
                            periscope technology to TRL 9 before it was included on the Virginia class
                            attack submarine. DOD programs that accepted technologies at a readiness
                            level of 5 or less experienced significant cost and schedule increases due,
                            in part, to problems with the technologies. DOD’s acceptance of
                            technologies at level 4 or lower was not unusual. For example, the key
                            technologies for the Army’s brilliant antiarmor submunition were at levels 2



                            Page 4                                          GAO/NSIAD-99-162 Best Practices
                              Executive Summary




                              and 3 when weapon system development began. At these levels, DOD had a
                              significant gap in technology maturity at the start of the program. The gap
                              was not closed until well into the development program, and problems with
                              the technologies were a main contributor to the program’s 88-percent cost
                              growth and 62-percent slip in schedule.


Controllable Conditions       Closing the technology development gap before beginning product
Affect How Well a             development was the result of good technology maturation practices and
                              sound methods for moving technologies to products. The more successful
Technology’s Inclusion on a
                              of the 23 technologies were managed by science and technology
Product Can Be Managed        organizations until they reached at least TRL 6 and more, often TRL 8 or
                              higher. This environment was an important condition for successfully
                              maturing technologies, as it allowed room for unexpected results such as
                              test “failures,” which are considered normal events in developing
                              technologies. To match technology maturity and product requirements,
                              managers also had the option of waiting until technologies matured or
                              changing product requirements so that an already mature technology could
                              be used. For example, Hughes deferred the development of the HS-702
                              satellite until critical solar cell technology had matured—a process that
                              took over 10 years. Also, Navy managers accepted an existing weapon
                              ejection system on the Virginia class attack submarine when technology
                              failed to mature as expected. In contrast, performance requirements for the
                              Comanche helicopter were inflexible; requirements mandated the inclusion
                              of advanced sensors and avionics technologies, despite their immaturity.
                              The Comanche program has experienced cost growth and schedule delays,
                              partly attributable to the inclusion of these technologies.

                              In the more successful cases, technology and product managers were given
                              the authority and tools to move technology only when it was at high
                              readiness levels. Disciplined processes provided managers credible
                              information on the status of technologies and high standards for assessing
                              readiness. Science and technology managers developed technologies to
                              standards acceptable to product managers who could reject those
                              technologies that fell short. For example, Ford’s science and technology
                              managers use agreed-upon standards for judging technology readiness, and
                              all new technologies follow the same maturation process. Ford’s product
                              managers are also empowered to say no when technologies are not deemed
                              mature. Recently, the Jaguar vehicle team rejected night vision technology
                              at TRL 8 because it did not meet cost objectives. DOD program managers
                              that had to accept immature technologies had less information available to
                              guide them. For example, key technologies for the brilliant antiarmor



                              Page 5                                         GAO/NSIAD-99-162 Best Practices
                          Executive Summary




                          submunition program bypassed Army science and technology
                          organizations, forcing the program manager to accept the technologies
                          with little information about their readiness. Often, the tools used to assess
                          the technologies’ status failed to identify high risks; the TRLs indicate that
                          risks on the problematic technologies were often high. Also, the greater
                          pressures to meet cost and schedule goals in product development
                          provided a less forgiving environment for fledgling technologies.


Impediments to Adopting   Leading commercial firms have put the organizations, tools, and other
Best Practices for        practices in place to foster technology development and improve the
                          outcomes of product developments as a matter of necessity. The large
Technology Inclusion in
                          investment required for a new product and the risks to that investment if
DOD Are Surmountable      the product does not meet customer needs reinforce these practices. The
                          DOD cases that followed a similar approach—the Advanced Amphibious
                          Assault Vehicle and the Virginia class attack submarine—have so far
                          avoided problems with key technologies. Yet these cases are not the norm
                          for DOD programs. DOD programs operate under conditions that make it
                          more difficult—and less rewarding—to separate technology from product
                          development and to allow technology to reach high maturity before being
                          included in an acquisition program.

                          It is easier for weapon system programs to fund technology development at
                          higher readiness levels because they attract much bigger budgets than
                          science and technology projects. DOD typically does not fund science and
                          technology organizations to take technology past the feasibility stage—
                          TRL 5. As a practical matter, it is often necessary to move immature
                          technology to a weapon system program to get needed funds and
                          management support. New programs are pressured to include immature
                          technologies that offer significant performance gains. These pressures
                          come from the user’s perception of the threat, technologists that see the
                          program as an opportunity to apply a new technology, and funding
                          competition that rewards weapon systems with unique features.

                          DOD and the services have several initiatives for improving the technology
                          development process and reducing weapon system cycle times. These
                          include defense technology objectives, advanced technology
                          demonstrations, advanced concept technology demonstrations, the Army’s
                          new scout/cavalry vehicle, and the Air Force’s Integrated High
                          Performance Turbine Engine Technology Program. These initiatives are
                          aimed at putting the science and technology organizations and funding in




                          Page 6                                           GAO/NSIAD-99-162 Best Practices
                  Executive Summary




                  place to bring technologies to higher readiness levels before they are
                  included in weapon system programs.



Recommendations   GAO recommends that the Secretary of Defense adopt a disciplined and
                  knowledge-based approach of assessing technology maturity, such as
                  TRLs, DOD-wide, and establish the point at which a match is achieved
                  between key technologies and weapon system requirements as the proper
                  point for committing to the development and production of a weapon
                  system. GAO also recommends that the Secretary (1) require that
                  technologies needed to meet a weapon’s requirements reach a high
                  readiness level (analogous to TRL 7) before making that commitment,
                  (2) extract lessons from successful technology inclusion cases for
                  application to future technology inclusion efforts, and (3) empower
                  program managers to refuse to accept key technologies with low levels of
                  maturity by making decisions on individual programs that reinforce a best
                  practice approach to technology maturation and inclusion. These
                  recommendations appear in full in chapter 5.



Agency Comments   DOD generally agreed with the report and its recommendations. A detailed
                  discussion of DOD’s comments appear in appendix I.




                  Page 7                                          GAO/NSIAD-99-162 Best Practices
Contents



Letter

Executive Summary                                                                                   2


Chapter 1                Separating Technology Development From Product Development
                           Is a Best Practice                                                      12
Introduction             Technology and Product Development Conducted at the Same Time
                           Within DOD                                                              15
                         Shorter Acquisition Cycle Times Are Needed                                17
                         Objectives, Scope, and Methodology                                        18


Chapter 2                Technology Maturity Can Be Measured and Its Consequences for
                           Products Can Be Forecast                                                22
Maturity of Technology   Technologies With High Readiness Levels at Launch Were Better
at Program Start Is an     Able to Meet Product Objectives                                         25
Important Determinant
of Success

Chapter 3                Providing the Right Environment Is Critical to the Successful
                           Maturation of Technology                                                35
Controllable             Good Technology Handoff Decisions Depend on the Tools and
Conditions Affect How      Authority Given to Managers                                             41
Well a Technology’s
Inclusion on a Product
Can Be Managed

Chapter 4                Several Factors Make It Difficult to Mature Technologies Before
                           They Are Included on Weapon Systems                                     50
Impediments to           Services Encouraged to Use Best Practices                                 54
Adopting Best            Two Unique DOD Projects May Provide Lessons on How to Enable
                           S&T Organizations to Manage Technology Further                          56
Practices for
Technology Inclusion
in DOD Are
Surmountable


                         Page 8                                        GAO/NSIAD-99-162 Best Practices
                       Contents




Chapter 5              Conclusions                                                             61
                       Recommendations                                                         63
Conclusions and        Agency Comments and Our Evaluation                                      65
Recommendations
Appendixes             Appendix I: Technology Readiness Levels and Their Definitions           68
                       Appendix II: Comments From the Department of Defense                    69
                       Appendix III: GAO Contacts and Staff Acknowledgments                    73


Related GAO Products                                                                           76




Tables                 Table 2.1: Cost and Schedule Experiences on Product
                         Developments                                                          27
                       Table 3.1: TRLs of Technologies Managed by S&T Organizations            36




                       Page 9                                      GAO/NSIAD-99-162 Best Practices
          Contents




Figures   Figure 1.1: Cycle for Providing Users a Product With Better
            Capabilities                                                            13
          Figure 1.2: DOD’s Weapon System Acquisition Cycle                         16
          Figure 1.3: Allocation of DOD’s Fiscal Year 1999 Research and
            Development Funds                                                       17
          Figure 2.1: Using TRLs to Match Technology With Product Launch
            Requirements                                                            24
          Figure 2.2: Readiness Levels of Technologies at the Time They Were
            Included in Product Designs                                             26
          Figure 2.3: Time Line for Ford’s Development of Voice Activated
            Controls Technology                                                     28
          Figure 2.4: Jaguar                                                        29
          Figure 2.5: Virginia Class Attack Submarine                               30
          Figure 2.6: Brilliant Anti-Armor Submunition                              32
          Figure 2.7: Comanche Helicopter                                           33
          Figure 3.1: Hughes Solar Cell Arrays                                      38
          Figure 3.2: Integrated Avionics for Comanche Helicopter                   40
          Figure 3.3: Process for Closing the Gap Between the Readiness of
            Adaptive Cruise Control Technology and Jaguar Requirements              43
          Figure 3.4: Process for Closing the Gap Between the Readiness of
            Propulsion Technologies and AAAV Requirements                           46
          Figure 3.5: AAAV                                                          47
          Figure 3.6: Assimilation of New Technology Into the BAT Program           48
          Figure 4.1: Airborne Laser                                                52
          Figure 4.2: Comparison of Traditional Technology Development
            Process With the Army’s Fast Track Approach                             57
          Figure 4.3: Future Scout and Cavalry System                               59




          Page 10                                       GAO/NSIAD-99-162 Best Practices
Abbreviations

AAAV      Advanced Amphibious Assault Vehicle
ABL       Airborne Laser
ACTD      Advanced Concept Technology Demonstration
ATD       Advanced Technology Demonstration
BAT       Brilliant Anti-Armor Subminition
DARPA     Defense Advanced Research Projects Agency
DEAL      Deliverables Agreement Log
DOD       Department of Defense
DTO       defense technology objective
FLIR      forward-looking infrared
NASA      National Aeronautics and Space Administration
S&T       science and technology
TRL       technology readiness level



Page 11                                     GAO/NSIAD-99-162 Best Practices
Chapter 1

Introduction                                                                                          Chapte1
                                                                                                            r




                        A central piece of the National Military Strategy is the military capability
                        represented by advanced weaponry. The Department of Defense (DOD)
                        plans to increase its annual investment in new weapons to about $60 billion
                        by fiscal year 2001—a 40-percent increase over fiscal year 1997. DOD has
                        high expectations from this investment: that new weapons will be better
                        and less expensive than their predecessors and will be developed in half
                        the time. These expectations frame a great challenge for managers of
                        programs. The traditional management approach—which has produced
                        superior weapons but at much greater cost and time than planned—will not
                        meet these expectations. Cycle times—the time to develop a new
                        weapon—can be so long that the technology a weapon is designed with
                        becomes obsolete before it can be produced. Costs of new weapons have
                        reached the point that significantly fewer can be bought than planned.
                        These are not new issues, but they have become more pressing as the pace
                        and sophistication of foreign and commercial technology have increased,
                        complicating a national security environment of unknown threats.

                        Leading commercial firms have changed the way they develop products
                        and have achieved the kinds of results DOD seeks, often yielding more
                        sophisticated products in half the time formerly needed. Industry experts
                        estimate that resolving technology problems before product development
                        begins results in 10 times the savings compared to correcting problems
                        afterward. In this sense, technology maturity breeds product success. The
                        practices leading firms use to mature and transition technology to products
                        hold promise for DOD, for immature technologies have been main sources
                        of problems on weapon systems. We have previously reported on the
                        different elements of knowledge firms insist on to get better products to
                        market faster. Of these, no element is more important than having
                        technology, advanced enough to meet requirements but also mature
                        enough to be predictably managed, available at the start of the product
                        development cycle. Maturing new technology before it is included on a
                        product is perhaps the most important determinant of the success of the
                        eventual product—or weapon system. It is the topic of this report.



Separating Technology   The cycle for placing better capabilities in the hands of users—both
                        military and commercial—can be described as consisting of technology
Development From
Product Development
Is a Best Practice


                        Page 12                                         GAO/NSIAD-99-162 Best Practices
                                           Chapter 1
                                           Introduction




                                           development, product development, and production. In a 1998 report,1 we
                                           characterized the knowledge needed on a new product as consisting of
                                           three knowledge points: when a match is made between a customer’s
                                           requirements and the available technology; when the product’s design is
                                           determined to be capable of meeting performance requirements; and when
                                           the product is determined to be producible within cost, schedule, and
                                           quality targets (see fig. 1.1). We found that this knowledge, when obtained
                                           at the right time and in the right sequence—technology, design, and
                                           manufacturing—was a best practice. This practice lowered product
                                           development risks, reduced cycle times and costs, and resulted in
                                           smoother production programs.


Figure 1.1: Cycle for Providing Users a Product With Better Capabilities




             Technology                                 Product
                                                                                                      Production
             Development                              Development


                                        Product
                                        launch


                          Knowledge                  Knowledge               Knowledge
                           Point 1                    Point 2                 Point 3

                                                                             Control of
                          Technology                   Design
                                                                            manufacturing
                           readiness                   maturity
                                                                              process




                                           Leading commercial firms recognize a distinct difference between
                                           technology development and product development; accordingly, they
                                           develop technology before introducing it into product development
                                           programs. They minimize risk, improve cost and schedule outcomes,
                                           reduce cycle time, and improve quality during product development by


                                           1
                                           Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD’s
                                           Environment (GAO/NSIAD-98-56, Feb. 24, 1998).




                                           Page 13                                                       GAO/NSIAD-99-162 Best Practices
Chapter 1
Introduction




gaining significant knowledge about a technology before launching the
product development. Scientists and technologists—different people than
those that manage product developments—manage the development of
technology until it is ready to be included in the design of a product.

Program launch is the point at which a firm defines a product’s
performance, cost, and schedule estimates and begins making a large
investment in human capital, facilities, and materials—an investment that
increases continuously as the product approaches the point of
manufacture. It includes a commitment to manufacture the product.
Therefore, program launch and the start of product development are
synonymous within commercial firms. Protecting this investment provides
a strong incentive for firms to minimize the potential for technology
development problems during the product phase and cause delays.
Confining delays in maturing technology to a time prior to launch—in an
environment where small teams of technologists work in laboratories and
are dedicated to perfecting the technology—is critical to saving time and
money. If delays occur during product development, when a large
engineering force is in place to design and manufacture the product, they
would be much more costly. In fact, industry experts estimate that
identifying and resolving a problem before product development can reap a
10-fold savings compared to correcting the problem after launch and that
correcting the same problem in the manufacturing stage would be even
more costly.

Leading commercial firms have found that managing technology
development separately from and before product development is a major
reason they have been able to reduce product cycle times. As a whole,
50 to 70 percent reductions in cycle times are not unrealistic achievements
by leading commercial firms. For instance, leaders in the automobile
industry have reduced cycle times from 7 years to 2 years, or by about
70 percent. The consumer electronics industry has recently reduced its
cycle time from 2 years to 6 months, and the commercial aircraft industry
has achieved reductions of 50 percent. Leading commercial firms have
found that reducing the product development cycle time brings products to
market faster, results in an increased market share, and helps to keep
products from becoming technologically obsolete.




Page 14                                        GAO/NSIAD-99-162 Best Practices
                        Chapter 1
                        Introduction




Technology and          DOD’s process for developing and manufacturing weapon systems is
                        described as a cycle consisting of phases. These phases are concept
Product Development     exploration, program definition and risk reduction, engineering and
Conducted at the Same   manufacturing development, and production and fielding. The basic
                        process of gathering knowledge about technology, design, and
Time Within DOD         manufacturing is followed, but in practice, the DOD cycle does not make a
                        clear distinction between technology development and product
                        development. The launch of a program in DOD usually takes place several
                        years before the beginning of product development does in leading
                        commercial firms.

                        In fact, a new weapon system program is normally launched at the start of
                        the program definition and risk reduction phase, which is often in the midst
                        of technology development, while most product development activities do
                        not begin until the engineering and manufacturing development phase.
                        Consequently, technology, design, and manufacturing knowledge is
                        attained concurrently--in the higher cost environment that characterizes
                        product development--throughout the weapon system phases. In our
                        February 1998 report, we noted that such technology development
                        problems are a major cause of cost increases and schedule delays on DOD
                        weapon system programs. The phases in DOD’s weapon system acquisition
                        cycle and the knowledge gathering process, as it is typically followed, are
                        shown in figure 1.2.




                        Page 15                                         GAO/NSIAD-99-162 Best Practices
                                        Chapter 1
                                        Introduction




Figure 1.2: DOD’s Weapon System Acquisition Cycle



                                        Program                  Engineering and
           Concept                                                                           Production
                                     definition and               manufacturing
          exploration                                                                        and fielding
                                     risk reduction               development



                            Program launch                   Begin product
                                                             development


                                                       Technology maturity

               Knowledge                                         Design maturity
               attainment
                                                                Manufacturing processes controlled



                                        DOD’s process also has organizational and budgetary implications.
                                        Activities accomplished in the first three phases of the acquisition cycle use
                                        research and development funds whereas production programs use
                                        procurement funds. Generally, DOD’s science and technology (S&T)
                                        community is responsible for basic research, applied research, and
                                        advanced technology development to produce generic, rather than
                                        weapon-specific, technologies. Its goal is to conduct research, develop
                                        technology, and farm these efforts for potential military application, such
                                        as a weapon system. The S&T community also uses research and
                                        development funds, but its work generally precedes the acquisition cycle.
                                        Weapon system program managers, who receive most of DOD’s research
                                        and development budget, apply generic technologies to specific weapon
                                        systems. However, they often become responsible for completing
                                        development of generic technologies as well. The allocation of DOD’s fiscal
                                        year 1999 research and development funds to these categories is shown in
                                        figure 1.3.




                                        Page 16                                          GAO/NSIAD-99-162 Best Practices
                                            Chapter 1
                                            Introduction




Figure 1.3: Allocation of DOD’s Fiscal Year 1999 Research and Development Funds

                                                           3%   Basic research - $1.1 billion
                                                                       9%   Applied research - $3.2 billion



                                                                                 9%   Advanced technology
                                                                                      development - $3.5 billion




                 Weapon specific      79%
        development - $29.6 billion


                                            Source: DOD


                                            S&T officials stated their role is to show that technology is feasible through
                                            laboratory experiments or demonstrations. It is often at this point that the
                                            technology’s military potential will be identified and the technology will be
                                            harvested for inclusion on a weapon system. Because the technology is still
                                            not mature, its development will be completed as part of the weapon
                                            system’s design and development, under the authority of the weapon
                                            system manager and apart from the S&T community.



Shorter Acquisition                         DOD’s weapon acquisition cycle times average between 10 to 15 years—far
                                            longer than the cycle time for commercial products. To an extent, DOD’s
Cycle Times Are                             cycle times are longer because they start earlier than commercial cycles
Needed                                      and often entail more complex products. Compounding the length of the
                                            weapon system development cycle is its unpredictability. Over the years,
                                            we have issued numerous reports highlighting cost overruns and schedule
                                            delays during the product development cycle, for which technology
                                            development problems were a major cause. These problems require
                                            additional technology development activities to take place at a time when



                                            Page 17                                                   GAO/NSIAD-99-162 Best Practices
                         Chapter 1
                         Introduction




                         the product should be undergoing design and manufacturing development.
                         As a result, the pace of technology advances outruns the time to develop a
                         weapon system and some of the more mature components designed into a
                         weapon system become obsolete before the weapon is manufactured. For
                         example, the F-22 will have almost 600 obsolete components by fiscal year
                         2000 while the aircraft is still in development.

                         The longer a weapon system’s development cycle, the more prone the
                         program is to management and funding changes. According to DOD, an
                         11-year development program historically encounters a 30-percent cost
                         growth over time. Based on historical averages, DOD calculates that the
                         typical program will have four different program managers, eight defense
                         acquisition executives, and seven Secretaries of Defense—all of who are
                         major influences and decisionmakers on the program. In addition, the
                         program will have gone through 11 annual budget cycles in which funding
                         changes could have occurred and affected the program’s content.

                         The Under Secretary of Defense for Acquisition and Technology has stated
                         that cycle time reduction is necessary to meet DOD’s goals of delivering
                         emerging technologies to warfighters in less time and at lower costs. The
                         Under Secretary has set a goal to reduce the average acquisition cycle time
                         for all program starts in fiscal year 1999 and beyond by 50 percent over
                         historical averages. Reductions in cycle times will (1) allow for earlier
                         fielding of increased capabilities, (2) reduce costs, (3) free up funds for
                         more programs, (4) reduce the potential for components becoming
                         obsolete, and (5) take more frequent advantage of technology advances
                         found in the commercial world. An emphasis on shorter cycle times may
                         also reduce the tendency to add technological advances that are unproven
                         and immature into weapon acquisition programs. To help achieve this goal,
                         DOD is working on several efforts such as Defense Acquisition Pilot
                         Programs, the Defense Reform Initiatives, and many acquisition reform
                         projects. The Under Secretary has also advocated adopting the practices of
                         leading commercial firms and taking a more evolutionary approach to
                         developing weapon systems, which would lessen the amount of technology
                         development initially attempted within a weapon system program.



Objectives, Scope, and   The Chairman and the Ranking Minority Member, Subcommittee on
                         Readiness and Management Support, Senate Committee on Armed
Methodology              Services, requested that we examine various aspects of the acquisition
                         process to determine whether the application of best practices can improve
                         program outcomes. To date, we have issued reports on advanced quality



                         Page 18                                        GAO/NSIAD-99-162 Best Practices
Chapter 1
Introduction




concepts, earned value management, management of a product from
development to production, and management of key suppliers (see related
GAO products). This report covers the inclusion of technology into weapon
system programs, and is, in a sense, a prequel to our report on product
development. Our overall objective was to determine whether best
practices offer methods to improve the way DOD matures new technology
so that it can be assimilated into weapon system programs with less
disruption. Specifically, we assessed (1) the impact of technology maturity
on product outcomes, (2) best practices for managing new technologies
and incorporating them into products, and (3) ways DOD can adapt best
practices to achieve better outcomes on weapon system programs.

Our methodology consisted of analyzing 23 commercial and DOD
technologies that had transitioned or attempted to transition into product
development programs. The technologies were drawn from six commercial
firms recognized for their success in developing technically advanced
products more quickly than their predecessors and five weapon system
programs that incorporated advanced technologies, including some that
did not encounter problems and some that did. We asked the managers of
these technologies to apply a tool, referred to as technology readiness
levels (TRLs), for our analysis. The managers used TRLs to judge the
maturity of the technologies at the time they had entered product
development or were included in programs. The National Aeronautics and
Space Administration (NASA) originally developed TRLs, and the Air Force
Research Laboratory uses them to determine when technologies are ready
to be handed off from S&T managers to product development managers.
We held discussions with the DOD and NASA users of TRLs to better
understand their applicability to our review. They stated that TRLs can be
used as general indicators of a technology’s readiness level and associated
risk of including the technology into a product development program, given
its TRL at that time. TRLs are more fully explained in chapter 2.

To understand the best practices the commercial sector used to include
technologies in product development programs, we conducted literature
searches and focused those searches as the review progressed. On the
basis of the searches, we identified a number of commercial firms with
innovative technology development processes for including new or
advanced technologies into new products. We used structured interview
questions sent in advance of our visits to gather uniform and consistent
information about each firm’s process and practice and the results
achieved. In addition, we examined four specific technology cases—Ford’s
night vision, adaptive cruise control, and voice activated controls and



Page 19                                        GAO/NSIAD-99-162 Best Practices
Chapter 1
Introduction




Hughes’ solar cell array—to better understand their processes and
practices. The commercial firms we visited were

• Ethicon-Endo Surgery (medical device manufacturer), Division of
  Johnson and Johnson, Cincinnati, Ohio;
• Ford Motor Company (automobile manufacturer), Dearborn, Michigan;
• Harris Semiconductor (semiconductor manufacturer), Melbourne,
  Florida;
• Hughes Space and Communications (satellite and spacecraft
  manufacturer), Los Angeles, California;
• 3M (commercial products manufacturer), St. Paul, Minnesota; and
• Motorola Corporate Research Headquarters (communications
  technology manufacturer), Schaumburg, Illinois, and Motorola Land
  Mobile Products Sector, Plantation, Florida.

We also attended and participated in conferences and workshops with
recognized leaders in the acquisition field to obtain information on how
organizations are improving their acquisition processes. Finally, we
interviewed officials from trade organizations concerning the application
of commercial practices to DOD operations.

To better understand DOD’s technology inclusion process, we selected 19
advanced technologies that had been included in 5 DOD weapon system
programs that were in various stages of the acquisition process. We
collected technical reports, acquisition management, and risk management
documentation about the technologies. In addition, we interviewed S&T
and acquisition program management officials about each technology’s
development history, costs, and current status. The technologies and
programs reviewed were

• acoustic sensor, infrared seeker, inertial measurement unit, tandem
  shaped charge warhead, and processor technologies from the Army’s
  Brilliant Anti-Armor Submunition (referred to as BAT) Program;
• rotor, engine, integrated avionics, forward looking infrared, and helmet
  mounted display technologies from the Army’s Comanche Helicopter
  Program;
• nonpenetrating periscope and weapon ejection system technologies
  from the Navy’s Virginia class attack submarine program;
• high speed planing craft, power dense diesel engine, lightweight
  composite armor, high power water jet, moving map and advanced
  navigation technologies from the Marines’ Advanced Amphibious
  Assault Vehicle (AAAV) Program; and



Page 20                                        GAO/NSIAD-99-162 Best Practices
Chapter 1
Introduction




• laser and beam control technologies from the Air Force’s Airborne Laser
  (ABL) Program.

To determine relevant DOD policy and initiatives, we obtained documents
and interviewed officials of the Office of the Secretary of Defense; the
Defense Advanced Research Projects Agency (DARPA); and Army, Navy
and Air Force Science and Technology organizations. We also had
discussions with former DOD officials and industry experts about DOD
acquisition policies and practices.

Even though we selected firms with product lines of varying complexity,
we did not concentrate only on firms whose products had the most in
common with weapon systems. Such an approach would have limited our
ability to include firms recognized as the best at including new, advanced
technologies into programs. In our analysis, we concentrated on the
criteria and knowledge used to support technology readiness decisions.
Although the approach from product to product may vary, the basic
processes and standards leading commercial firms applied to technology
inclusion decisions were consistent. We were limited, however, in our
ability to obtain and present some relevant data that commercial
companies considered proprietary in nature. This information included
funding amounts for investing in technology development, details on
technological innovations, and some specific data from recent technology
inclusion successes. Our report highlights the best commercial practices
for including technology into product development programs. As such, they
are not intended to describe all commercial industry practices or to suggest
that commercial firms do not have any flaws.

We conducted our review between March 1998 and June 1999 in
accordance with generally accepted government auditing standards.




Page 21                                         GAO/NSIAD-99-162 Best Practices
Chapter 2

Maturity of Technology at Program Start Is an
Important Determinant of Success                                                                        Chapte2
                                                                                                              r




                       The experiences of the DOD and commercial technology development
                       cases we reviewed indicate that demonstrating a high level of maturity
                       before allowing new technologies into product development programs puts
                       those programs in a better position to succeed. Simply put, the more
                       mature technology is at the start of the program, the more likely the
                       program will succeed in meeting its objectives. Technologies that were
                       included in a product development before they were mature later
                       contributed to cost increases and schedule delays in those products.

                       We found an analytical tool—TRLs—that can assess the maturity level of
                       technology as well as the risk that maturity poses if the technology is
                       included in a product development. The tool associates different TRLs with
                       different levels of demonstrated performance, ranging from paper studies
                       to proven performance on the intended product. The value of using the tool
                       is that it can presage the likely consequences of incorporating a technology
                       at a given level of maturity into a product development, enabling
                       decisionmakers to make informed choices. TRLs proved to be reliable
                       indicators of the relative maturity of the 23 technologies reviewed, both
                       commercial and military, and their eventual success after they were
                       included in product development programs.



Technology Maturity    Successful technologies progress from initial concept to proven
                       performance, whether they are developed in the laboratory or in the
Can Be Measured and    factory, by commercial industry or DOD. The Air Force Research
Its Consequences for   Laboratory has adapted and uses TRLs to measure the key steps in this
                       progression toward inclusion into weapon systems. TRLs are measured
Products Can Be        along a scale of one to nine, starting with paper studies of the basic concept
Forecast               and ending with a technology that has proven itself in actual usage on the
                       intended product. A detailed description of TRLs is provided in
                       appendix II, but the following hypothetical example about an airborne
                       communications radio can illustrate the readiness levels.

                       First, the idea for a new radio is conceived. The idea reaches TRL 3 when
                       analytical studies and some tests of the technology’s elements, such as a
                       circuit, back it up. When initial hand-built versions of all of the radio’s basic
                       elements are connected and tested together, the radio reaches TRL 5. This
                       is sometimes referred to as a “breadboard” article; although it may function
                       like a radio, it does not look like one because the individual parts are
                       attached to plywood and hand-wired together. When the technology is built
                       into a generic model, which is well beyond the breadboard tested in TRL 5,
                       and demonstrated in a laboratory environment, the radio reaches TRL 6.



                       Page 22                                            GAO/NSIAD-99-162 Best Practices
Chapter 2
Maturity of Technology at Program Start Is
an Important Determinant of Success




This model represents the last level of demonstration before the radio
becomes tailored for application to a specific aircraft. When the
components are assembled inside a case that resembles the final radio
design and are demonstrated aboard a surrogate for the intended aircraft,
the radio reaches TRL 7. TRL 8 is reached when the radio is put in its final
form, installed in the intended aircraft’s cockpit, and tested in conjunction
with the other aircraft equipment with which it must interface. TRL 9 is
achieved when the radio is successfully operated on the aircraft through
several test missions. Unexpected problems can arise at every level, and
effort must be expended to overcome them. This effort takes time and can
delay the progress to the next readiness level.

Once a technology’s readiness level has been established, the risks of
including that technology in a product development can be assessed.
Unlike S&T projects, for which the main objective is to develop knowledge,
a product development’s objective is to deliver products that meet strict
cost, schedule, and performance targets. We found that most leading
commercial firms, after they had translated their own methods of assessing
risk into TRLs, determined that a TRL 8 was required before they allowed a
new technology into a product development.1 DOD launches a program in
the program definition and risk reduction phase—much earlier than the
leading commercial firms do. According to the Air Force Research
Laboratory, a TRL 6 is required for a technology to be an acceptable risk for
a program in that phase. When weapon system development reaches the
engineering and manufacturing development phase, it more nearly
approximates the point at which a commercial product development
program would start. The Air Force Research Laboratory depicts a
technology at TRL 7 as an acceptable risk for this phase—technologies at
lower levels would be considered high risks.

The lower the level of technology readiness, the more ground must be
covered to bring the technology to the point at which it can meet the
intended product’s cost, schedule, and performance requirements with
little risk (see fig. 2.1).




1
 An exception to this is space systems technology. Space-based technologies are generally included on a
development program once they have been prototyped and ground tested—a TRL 6, the highest level
attainable short of space operation.




Page 23                                                          GAO/NSIAD-99-162 Best Practices
                                        Chapter 2
                                        Maturity of Technology at Program Start Is
                                        an Important Determinant of Success




Figure 2.1: Using TRLs to Match Technology With Product Launch Requirements

                                            High risk for                              Low risk for
                                           product launch                            product launch
     Product
     Requirements
                                                                                            8     9
                                        Risks or
                                       unknowns                              6       7




                                                  4             5



      TR L
                          1      2     3


                                        The gap between the maturity of the technology and the product’s
                                        requirements represents the risks or unknowns about the technology. As
                                        each succeeding level of readiness is demonstrated, unknowns are
                                        replaced by knowledge and the gap becomes smaller. Ideally, the gap is
                                        closed before a new technology is included in a new product’s design,
                                        although the Air Force Research Laboratory accepts the amount of risk at
                                        TRL 7 for a program entering engineering and manufacturing development.
                                        Technologies that reach TRL 7 or higher at the start of product
                                        development allow product managers to focus their attention on
                                        integrating the technologies and proving out the product design.
                                        Technologies that are included at lower maturity levels require more of the
                                        product managers’ attention and resources, as basic knowledge about
                                        those technologies must still be gained.

                                        Thus, a major purpose served by TRLs is to reveal the gap between a
                                        technology’s maturity and the maturity demanded for successful inclusion
                                        in the intended product. With TRLs as guides, the options available to
                                        decisionmakers can be framed. Given that a key determinant of achieving
                                        cost and schedule outcomes for a product development is the technology’s
                                        maturity at product launch, decisionmakers can either (1) delay product



                                        Page 24                                           GAO/NSIAD-99-162 Best Practices
                        Chapter 2
                        Maturity of Technology at Program Start Is
                        an Important Determinant of Success




                        development until the technology is matured to a high enough readiness
                        level or (2) reduce the product’s requirements so that a less advanced, but
                        more mature, technology can suffice. If it is perceived that the
                        requirements of the product cannot be lowered and the product launch
                        cannot be delayed until the requisite technology is of a sufficient readiness
                        level, then the remaining option is to launch the product development with
                        the immature technology. If this option is chosen, then the success of the
                        product development will depend heavily on the product manager’s ability
                        to simultaneously close the technology maturity gap and develop the
                        product for manufacture, which is a very challenging task.

                        TRLs do not represent strictures that must be adhered to without
                        exception. According to the people in DOD who have used TRLs, there are
                        occasions when a lower than expected TRL can be accepted, such as when
                        the product development’s schedule and resources are generous enough
                        that the technology will have enough time to mature. In other instances, a
                        higher than expected TRL may be required, such as if the technology in
                        question is the linchpin for the entire product. Nonetheless, we found that
                        TRLs ably reconciled the different maturity levels and product experiences
                        of the 23 technologies reviewed.



Technologies With       The 23 technologies reviewed spanned a wide range of readiness levels at
                        the time they were included in product development programs. The least
High Readiness Levels   mature reached TRL 2 at the time it was included in a product
at Launch Were Better   development, while the most mature had reached TRL 9 at the point of
                        inclusion. We observed a general relationship between TRLs and the
Able to Meet Product    technologies’ inclusion on the intended product developments. Those
Objectives              products whose technologies reached high TRLs at the time they were
                        included were better able to meet cost, schedule, and performance
                        requirements. In fact, commercial firms informed us that maturing the
                        technology separately from and ahead of the product was a main reason
                        they were able to reduce cycle times on their products. An official from one
                        of the firms termed the approach as “moving discovery to the left.”

                        Those technologies with low TRLs at inclusion encountered maturation
                        difficulties and contributed to problems the products experienced. Other
                        problems, such as funding and schedule changes unrelated to the
                        technologies, also contributed to problems in the product developments.
                        Figure 2.2 shows the TRLs when each of the 23 technologies was included
                        in a product design, whether at product development launch (for
                        commercial technologies) or at program launch (for DOD technologies).



                        Page 25                                         GAO/NSIAD-99-162 Best Practices
                                                        Chapter 2
                                                        Maturity of Technology at Program Start Is
                                                        an Important Determinant of Success




Figure 2.2: Readiness Levels of Technologies at the Time They Were Included in Product Designs

        Non-penetrating periscope

            Adaptive cruise control

                       Night vision

            Voice activated control

                    Solar cell array

          High-speed planing craft

               High power water jet

           Weapon ejection system

                      Diesel engine

                   Helicopter rotor

      Lightweight composite armor

                  Helicopter engine

       Moving map and navigation

        Hemical oxygen iodin laser

              Beam control system

           Helmet mounted display

 Helicopter forward linking infrared

                Integrated avionics

                    Data processor

         Inertial measurement unit

                          Warhead

                    Infrared seeker

        Acousting targeting sensor


                                       1            2             3              4        5          6     7          8           9
                                       Technology Readiness Levels


                                                              Commercial technologies

                                                              DoD technologies




                                                        Page 26                                          GAO/NSIAD-99-162 Best Practices
Chapter 2
Maturity of Technology at Program Start Is
an Important Determinant of Success




The cost and schedule experiences of some of the products or programs
that inherited the technologies are shown in table 2.1.



Table 2.1: Cost and Schedule Experiences on Product Developments

                                                                Product development
                                          TRL at
Product development and                 program
associated technologies                   launch       Cost growth          Schedule slippage
                                                                     a
Comanche helicopter                                    101 percent          120 percenta
 Engine                                          5
 Rotor                                           5
 Forward looking infrared                        3
 Helmet mounted display                          3
 Integrated avionics                             3
BAT                                                    88 percent           62 percent
 Acoustic sensor                                 2
 Infrared seeker                                 3
 Warhead                                         3
 Inertial measurement unit                       3
 Data processors                                 3
Hughes HS-702 satellite                                None                 None
 Solar cell array                                6
Ford Jaguar                                            None                 None
 Adaptive cruise control                         8
 Voice activated controls                        8
a
 The Comanche, in particular, has experienced a great deal of cost growth and schedule slippage for
many reasons, of which technology immaturity is only one. Other factors, such as changing the scope,
funding, and pace of the program for affordability reasons, have also contributed.


Data for three weapon system development programs, the Virginia class
attack submarine, AAAV, and the ABL, were not included in the table
because they had not been in the product development phase long enough
to report actual consequences. To date, AAAV and the submarine have
stayed within 15 percent of their cost and schedule estimates for
development. The ABL, for which key technologies were much less mature
at program launch, still faces challenges with these technologies. Ford’s
night vision technology was excluded because the firm decided not to
include the technology on a product. Details on the Comanche, BAT, the
Virginia class attack submarine, and Ford technology and product
experiences follow.




Page 27                                                        GAO/NSIAD-99-162 Best Practices
                                          Chapter 2
                                          Maturity of Technology at Program Start Is
                                          an Important Determinant of Success




Technology and Product                    The key technologies for the Ford Jaguar and the Virginia class attack
Experiences on Ford and                   submarine followed the pattern of increasing TRLs until they demonstrated
                                          a low risk for transition to the product. Two examples are Ford’s voice
Virginia Class Attack
                                          activated controls development and DARPA’s nonpenetrating periscope
Submarine                                 development for the submarine. In both cases, the technologies were
                                          validated, operational prototypes demonstrated, and the technologies had
                                          demonstrated the form, fit, and function of the final article by the beginning
                                          of product development.

                                          Ford’s voice activated controls technology, which allows a driver to control
                                          certain functions such as windows and the radio through verbal
                                          commands, was under development in the technology base for over
                                          10 years, being pushed by the firm’s technology leaders. It was not until
                                          1993 that Ford found that (1) other complementary technologies, such as
                                          processor speeds and low cost memory, had become available and
                                          (2) customers wanted more features and functions but less distractions
                                          from driving. Given this market information, Ford decided to pursue voice
                                          technology as a strategic technology in terms of product differentiation,
                                          recognizing the importance of being first to market with this enabling
                                          technology. Figure 2.3 shows the time line for developing this technology.


Figure 2.3: Time Line for Ford’s Development of Voice Activated Controls Technology


              1983                                    1993                         1995                          1999




     Ford decides to pursue                 Technology is linked         Technology is ready to           Technology featured
     voice activated controls               to a specific vehicle.       transition into a product        on model year 1999
     technology. Technology                 Cost and performance         development program.             Jaguar designs.
     under early development                requirements are             Technology meets all
     in technology base.                    defined.                     cost and schedule targets
                                                                         for the product.

            TRL 3 - 5                               TRL 6 - 7                     TRL 8                          TRL 9




                                          Between 1993 and 1994, based on discussions with customers, Ford
                                          developed cost and performance requirements for the technology. Ford has



                                          Page 28                                                    GAO/NSIAD-99-162 Best Practices
                     Chapter 2
                     Maturity of Technology at Program Start Is
                     an Important Determinant of Success




                     never relaxed them. By September 1995, when Ford allowed the technology
                     into the development program for a new Jaguar design, voice activated
                     controls had been demonstrated as an integrated system in the appropriate
                     form and fit for the Jaguar. Ford officials stated that the product has met all
                     cost and cycle time targets established at the outset of its development.
                     Figure 2.4 shows the Jaguar.


Figure 2.4: Jaguar




                     Ford demonstrated voice activated control technology in the appropriate form and fit before
                     incorporating it into the Jaguar.
                     Source: Ford Motor Company.


                     DARPA began developing the nonpenetrating periscope technology as part
                     of its submarine technology development efforts after recognizing, in 1988,
                     along with the Navy, that the nonpenetrating technology would enhance
                     operator visibility, provide greater submarine design flexibility, and be
                     stealthier than conventional masts and periscopes. At the time, the Virginia
                     class attack submarine program had not been initiated. Once the decision
                     was made to include the nonpenetrating periscope, it became a key feature
                     of the submarine and was a major design driver for the submarine’s overall
                     configuration.

                     Nonpenetrating refers to the fact that the periscope is essentially a group of
                     sensors that are linked to the submarine via fiber optic and other cables.
                     This technology uses infrared imaging and advanced sensors to replace
                     conventional periscopes and frees up physical space compared with a
                     conventional periscope. A conventional periscope relies on a series of
                     telescoping shafts and reflecting surfaces to see above the water’s surface.



                     Page 29                                                         GAO/NSIAD-99-162 Best Practices
                                          Chapter 2
                                          Maturity of Technology at Program Start Is
                                          an Important Determinant of Success




                                          When the periscope is retracted, the shafts take up a column of space from
                                          the top of the submarine to the bottom, through all decks. Its location
                                          virtually dictates the design and placement of the control and other rooms.
                                          If the nonpenetrating periscope technology did not become available, then
                                          the submarine would have to be drastically redesigned to accommodate the
                                          space required by a conventional periscope.

                                          The new nonpenetrating periscope and photonics mast technology
                                          underwent land testing in 1991—a TRL 5. The Navy actually tested the new
                                          technology at sea on the U.S.S. Memphis in 1992 and 1993. According to
                                          program officials, these sea trials demonstrated the highest level of
                                          technology readiness: proving the actual system through successful
                                          mission operations. This readiness equated to a TRL 9. Yet, this technology
                                          was not included in the Virginia class attack submarine requirements until
                                          1995. Figure 2.5 shows an artist’s concept of the Virginia class attack
                                          submarine.


Figure 2.5: Virginia Class Attack Submarine




                                          The Navy demonstrated a key technology at the highest readiness level before including it as a
                                          requirement for the Virginia class attack submarine.
                                          Source: DOD.


                                          The high readiness level of the nonpenetrating periscope afforded the Navy
                                          the opportunity to develop an improved version of the periscope to a
                                          TRL 9. This was a relatively low-risk endeavor as the baseline periscope
                                          was sufficient to meet the submarine’s requirements. Program officials
                                          believe that having knowledge about key technologies, such as the
                                          nonpenetrating periscope, for the Virginia class attack submarine at
                                          program launch made a short program definition and risk reduction phase
                                          possible. This phase for the Virginia class attack submarine was about
                                          75 percent shorter than those of previous acquisition programs. Based on
                                          its demonstrated maturity, we anticipate that the nonpenetrating periscope
                                          to be less likely to impact the cost and schedule of the submarine’s
                                          development program. There are, however, several other technologies that



                                          Page 30                                                        GAO/NSIAD-99-162 Best Practices
                         Chapter 2
                         Maturity of Technology at Program Start Is
                         an Important Determinant of Success




                         are critical to the submarine program. We did not examine these
                         technologies and cannot predict their likely outcomes.


BAT and Comanche Cases   Key technologies for the BAT and Comanche programs had much lower
                         readiness levels at the time the product developments were launched.
                         Consequently, they did not reduce the gap between their demonstrated
                         maturity and the maturity needed to meet product requirements until after
                         program launch. For some technologies, the gap was not closed until well
                         into the product development program. For others, the gap has still not
                         been closed.

                         Five key technologies included in the Army’s BAT program had low TRLs
                         when they were included on the program. The level of readiness for most of
                         these technologies at program launch was characterized by the program
                         office as experimental in nature but with major uncertainty remaining—a
                         TRL 3. The acoustic targeting technology was the most important enabling
                         technology needed to meet the weapon’s performance requirements. This
                         technology provides BAT the capability to locate targets from great
                         distances based on the sounds generated by the target, such as moving
                         tanks and vehicles. At the time the program was launched, the Army knew
                         little about the feasibility of using this technology on this program. In fact,
                         the technology was still being defined in paper studies—a TRL 2. The
                         Army did not prototype this technology until after the program had entered
                         the engineering and manufacturing development phase, more than 6 years
                         after program launch. As of December 1998, the BAT had experienced
                         significant development cost and schedule increases, which program
                         officials attribute at least, in part, to unknowns about the new technologies.
                         Figure 2.6 shows the BAT.




                         Page 31                                           GAO/NSIAD-99-162 Best Practices
                                          Chapter 2
                                          Maturity of Technology at Program Start Is
                                          an Important Determinant of Success




Figure 2.6: Brilliant Anti-Armor Submunition




                                          At the time the program was launched, the Army knew relatively little about the performance of several
                                          key technologies for the BAT
                                          Source: DOD.


                                          Two technologies key to meeting the Comanche helicopter’s requirements–
                                          integrated avionics and forward-looking infrared (FLIR) technologies—
                                          were included on the program when they were still conceptual in nature.
                                          The integrated avionics technology replaces individual radios, navigation,
                                          and other communication equipment with a modular system that shares a
                                          common processor. The FLIR is a second-generation version that uses
                                          infrared sensors to improve the pilot’s ability to see at night and in bad



                                          Page 32                                                         GAO/NSIAD-99-162 Best Practices
                                  Chapter 2
                                  Maturity of Technology at Program Start Is
                                  an Important Determinant of Success




                                  weather. Program officials stated that both had TRLs of 3 when the
                                  helicopter program was started. Despite the low readiness levels of the
                                  technologies, the Army included the technologies on the program to meet
                                  weight, cost, and performance requirements.

                                  The development of these technologies has taken longer than the Army
                                  expected it would. The contractor for the integrated avionics has had
                                  difficulties in getting the multiple avionics modules to work simultaneously
                                  within required size and weight parameters, and the FLIR technology has
                                  undergone several design and performance requirement changes. As of
                                  September 1998—approximately 10 years after program launch—neither
                                  the integrated avionics nor the FLIR technology had advanced past a TRL 5.
                                  Problems with the maturation of these technologies have contributed to
                                  the program’s cost and schedule increases. In contrast, the advanced rotor
                                  and engine technologies, which were the most mature of the Comanche
                                  technologies we reviewed, have experienced fewer problems in maturation
                                  and have not contributed significantly to the program’s cost and schedule
                                  increases. Figure 2.7 shows the Comanche.


Figure 2.7: Comanche Helicopter




                                  The Army included two key technologies in the Comanche when they were still considered conceptual
                                  to meet weight, cost, and performance requirements
                                  Source: DOD.




                                  Page 33                                                      GAO/NSIAD-99-162 Best Practices
Chapter 3

Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed                                                                                      Chapte3
                                                                                                   r




               Closing the gap between technology maturity and product requirements
               before a product is launched—and baselines are set—distinguished the
               more successful cases. Notably, closing the gap before product launch was
               a managed result; it put product managers in a better position to succeed.
               Two conditions were critical to achieving this kind of result. First was an
               environment that put the primary responsibility for maturing technology in
               the hands of S&T managers and provided them considerable flexibility to
               make decisions. Second was having the quality information and standards
               needed to make good technology handoff decisions, coupled with giving
               the product manager the authority to refuse new technology that did not
               meet product requirements. When these conditions were not present, the
               handoff to the product manager was compromised, with negative
               consequences for both technology and product.

               In each of the successful cases, S&T organizations played major roles in
               bridging the gap between technology maturity and product requirements.
               Flexibility provided by requirements communities and resource providers
               enabled S&T and product managers to delay the inclusion of technology if
               it was not ready or to reduce product requirements to match what mature
               technology could deliver. This environment was better suited to the
               unexpected results and delays that accompany technology development.
               Moreover, technology maturation was managed within a disciplined
               process that provided good information to be judged against clear and high
               standards, like TRLs. Armed with the tools and the authority to make
               technology inclusion decisions, both S&T and product managers
               functioned as gatekeepers to safeguard the product development.

               In the more problematic cases, S&T organizations disengaged much earlier,
               and product managers had little choice but to accept immature
               technologies. Accordingly, less information about the technologies was
               available at the point of inclusion. Often, the tools used to assess the
               technologies’ status failed to identify high risks. In retrospect, TRLs
               indicated that risks were in fact high and perhaps unacceptable from a
               product standpoint. Also, pressures to meet cost and schedule estimates in
               product development provided a less forgiving environment for
               technologies in the discovery process.




               Page 34                                        GAO/NSIAD-99-162 Best Practices
                          Chapter 3
                          Controllable Conditions Affect How Well a
                          Technology’s Inclusion on a Product Can Be
                          Managed




Providing the Right       While most new technologies—commercial and military—are initially
                          managed by the S&T community, the more successful cases we reviewed
Environment Is Critical   continued to be managed by S&T organizations until they reached at least
to the Successful         TRL 6 and more often TRL 8 or higher. These technologies were provided
                          the environmental advantages an S&T project has over a product
Maturation of             development. This environment availed S&T managers and product
Technology                managers of the less risky options of waiting or trading to get the match
                          between technology maturity and product requirements—rather than
                          forcing the product launch and gambling on the completion of technology
                          maturity. In contrast, the more problematic technologies did not have as
                          benign an environment. Often, the technologies were handed off early by
                          S&T organizations because inflexible performance requirements for the
                          product demanded their inclusion. Product development managers
                          launched the product development and hoped that the technology
                          development would succeed.

                          Once in a product development environment, external pressures to keep
                          the program moving become dominant, such as preserving cost and
                          schedule estimates to secure budget approval. For example, DOD policies
                          require that a program be funded in the current year and that funds be
                          made available over the next 6 years in the DOD planning cycle. If, during
                          the program definition phase, a program manager were to decide that an
                          additional year was needed to overcome unexpected technology problems
                          to reach the desired level of maturity, the delay could push the start of
                          engineering and manufacturing development back. This delay could
                          jeopardize the funding for that phase, thus risking the funding support for
                          the entire program. Consequently, the program manager may be more likely
                          to accept the risk of not getting the technology to the desired level of
                          maturity and starting the engineering and manufacturing development
                          phase as planned, rather than risk the rest of the program. These conditions
                          compete with and detract from the needs of technology development. One
                          acquisition official stated that these conditions cause the weapon system
                          program “to pull double duty,” inventing new technology while integrating
                          it into a product. In general, he believed there is an equal amount of
                          difficulty in both tasks.


Technologies Matured by   In the most successful cases that we reviewed, S&T organizations bridged
S&T Organizations Made    the gap between immature technology and the maturity needed for either
                          program start in DOD (TRL 6) or product development (TRL 7 or higher).
Smooth Transitions into
                          These cases and the responsible organizations are shown in table 3.1.
Product Developments


                          Page 35                                         GAO/NSIAD-99-162 Best Practices
Chapter 3
Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed




Table 3.1: TRLs of Technologies Managed by S&T Organizations

                            TRL at   Responsible              Receiving product
Technology                 handoff   S&T organization         development program
Nonpenetrating                   9   DARPA                    Virginia class attack
periscope                                                     submarine program
Adaptive cruise control          8   Ford Advanced Vehicle    Jaguar vehicle team
                                     Technology Office
Voice activated controls         8   Ford Advanced Vehicle    Jaguar vehicle team
                                     Technology Office
Solar cell array                 6   Hughes Laboratories      HS-702 satellite
                                                              program
Weapon ejection                  6   Office of Naval          Virginia class attack
system                               Research                 submarine program
Diesel powered engine            6   Office of Naval          AAAV program
                                     Research
High-speed planing               6   Office of Naval          AAAV program
craft                                Research
High-power water jet             6   Office of Naval          AAAV program
                                     Research

Despite the different circumstances between the commercial and DOD
sectors and among the DOD cases themselves, the results were similar:
having S&T organizations bridge the maturity gap reduced
technology-related problems in the products. For the leading commercial
firms we visited, it is standard practice to have S&T organizations
responsible for the bridge. In the DOD cases shown in table 3.1, the S&T
organizations played atypical roles in managing the bridge between
technology and product by delivering the technology to a TRL 6 or higher.
Different pressures and incentives that are brought to bear on the
commercial and DOD product developments explain why DOD product
managers become responsible for more technology development than their
commercial counterparts. These influences are discussed in chapter 4.

Having an S&T organization manage a technology to maturation means
more than just having a different group of people involved than a product
development. S&T projects operate in a different environment than
product developments. The process of developing technology culminates
in discovery and must, by its nature, allow for unexpected results. S&T
provides a more forgiving environment in which events—such as test
“failures,” new discoveries, and delays in the attainment of knowledge—are
considered normal. It is also a less costly environment, making external
pressures to develop knowledge on a schedule less keenly felt. On the



Page 36                                                 GAO/NSIAD-99-162 Best Practices
                            Chapter 3
                            Controllable Conditions Affect How Well a
                            Technology’s Inclusion on a Product Can Be
                            Managed




                            other hand, the process of developing a product culminates in delivery, and
                            thus gives great weight to design and production. The same events and
                            unexpected results that are considered normal for technology development
                            represent problems in the product environment; they can jeopardize
                            achievement of cost and schedule objectives and draw criticism to the
                            product. The ups and downs and the resource changes associated with the
                            technology discovery process do not mesh well with a program’s need to
                            meet cost, schedule, and performance goals. This situation has been
                            described as attempting to “schedule inventions.”


Successful Cases Afforded   In the early 1980s, Hughes Space and Communications began developing
Flexibility to              dual junction solar cell technology that had the potential of greatly
                            increasing the electrical power on satellites. By 1985, a Hughes laboratory
Decisionmakers
                            had demonstrated the technology by ground testing prototypes, a TRL 6,
                            which is considered an acceptable level of demonstration for space-based
                            technology. Nonetheless, Hughes was not satisfied that the supporting
                            infrastructure (materials, reactors, and test equipment) was mature enough
                            to sustain development and production of the new technology on a
                            satellite. The infrastructure was seen as critical to meeting the cost and
                            schedule requirements of a product. As a result, Hughes did not hand off
                            the technology to a product. Instead, the firm kept it in a research
                            environment, away from cost and schedule pressures.

                            In the early 1990s, Hughes established requirements for a new satellite—
                            the HS-702—that would use the solar cell technology to leapfrog the
                            competition. After a laboratory demonstration in 1993, Hughes successfully
                            used the new technology on a high-powered version of its existing HS-601
                            satellite before it began product development on the HS-702 satellite. By
                            1994, it had determined that the business base was available to sustain
                            development and production of the HS-702 satellite. In all, the firm waited
                            10 years for the demonstrated technology to meet the requirements. This
                            experience closely resembled that of Ford’s voice activated control
                            technology because, in both cases, the new technology took 10 years to
                            mature enough for product readiness. Thus, the firms’ approach was not to
                            accelerate technology development but to shorten product development by
                            maturing the technology first. Figure 3.1 shows the solar cell arrays
                            installed on the HS-702.




                            Page 37                                        GAO/NSIAD-99-162 Best Practices
                                       Chapter 3
                                       Controllable Conditions Affect How Well a
                                       Technology’s Inclusion on a Product Can Be
                                       Managed




Figure 3.1: Hughes Solar Cell Arrays




                                       Hughes successfully proved solar cell array technology on a predecessor satellite before beginning
                                       product development of the HS-702
                                       Source: Hughes Space and Communications.




                                       Page 38                                                        GAO/NSIAD-99-162 Best Practices
                             Chapter 3
                             Controllable Conditions Affect How Well a
                             Technology’s Inclusion on a Product Can Be
                             Managed




                             The Navy made trade-offs in choosing a technology for the weapon ejection
                             system, which is used to deploy weapons like torpedoes, of the Virginia
                             class attack submarine. Because of quietness, weight, and cost
                             requirements, the Navy preferred a new elastomeric (rubber-based)
                             technology. However, this technology failed endurance testing, and product
                             managers determined that the technology was too risky to be included in
                             the first product. Product managers could have declined this technology
                             and its attendant risk without delaying the submarine’s schedule because
                             the Navy accepted marginal increases to the cost and weight requirements
                             for the system so that the proven Seawolf ejection technology could be
                             used as a substitute. Using proven technology on the first submarine has
                             allowed the Navy S&T community to continue developing the elastomeric
                             technology, which is to be incorporated into the new system on the fourth
                             production submarine. As discussed earlier, decisionmakers also had the
                             flexibility to wait for the nonpenetrating periscope technology to reach
                             TRL 9 before including it in the submarine’s requirements.


Problematic Cases Provided   According to Army officials, the FLIR and integrated avionics technologies
Little Flexibility to        required for the Comanche helicopter were critical for providing an
                             increased operational capability over existing Army helicopters. The
Managers
                             advanced FLIR technology was needed to meet the user’s requirements for
                             increased targeting range and for improved piloting capabilities in bad
                             weather and at night. It represented a quantum leap from existing
                             capabilities. Integrated avionics technology was expected to replace
                             separate radios, navigation systems, and other communication equipment
                             on the helicopter with a modular system that uses central processors.

                             These technologies were needed to meet weight and size requirements for
                             the aircraft as well as improve communications. Both were critical
                             elements of a mission equipment package that was supposed to reduce the
                             pilot’s workload while improving capabilities. Requirements were
                             inflexible. Thus, requirements managers informed us they were unwilling
                             to accept the product manager’s request to trade requirements that was
                             prompted by his concerns that the technologies could not advance in time
                             to meet the program’s schedule. They believed the product manager to be
                             too risk averse and said they would not take no for an answer. Not only did
                             the user consider the technologies nontradable, they became even more
                             confined by weight and cost restrictions that were placed on the program.
                             For example, a more mature FLIR technology that could possibly meet
                             performance requirements but also weighed more was rejected.




                             Page 39                                        GAO/NSIAD-99-162 Best Practices
Chapter 3
Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed




The technological solutions that could meet the strict requirements were
limited. According to Army officials, the only viable option was to develop
the new technologies, which were in a very immature state, to the required
performance levels because no suitable back-up technologies existed.
When the Comanche acquisition program was launched, the FLIR and
integrated avionics technologies had a TRL 3, barely demonstrated in a
laboratory. This level placed the burden on the Comanche program
manager to complete their development during the acquisition program.
The only ways for the program manager was to slip the schedule or
increase development costs. Figure 3.2 shows an early model of the
integrated avionics component for the Comanche.


Figure 3.2: Integrated Avionics for Comanche Helicopter




The Army launched the Comanche program with immature technologies, placing the burden on the
program manager to complete technology development
Source: DOD.




Page 40                                                    GAO/NSIAD-99-162 Best Practices
                         Chapter 3
                         Controllable Conditions Affect How Well a
                         Technology’s Inclusion on a Product Can Be
                         Managed




                         Similarly, the acoustic sensor technology on the BAT was critical to the
                         submunition’s performance because it provided breakthrough
                         improvements in the capability for precision attack of targets at ranges of
                         up to 500 kilometers and in most weather. There was no flexibility for the
                         program manager to ease requirements to substitute a more mature
                         technology because the Army had no existing capability to perform this
                         mission. Thus, the technology, which had a TRL 2 at program launch, was
                         the only solution for locating and acquiring targets. Its feasibility was based
                         on an engineering analysis in the form of studies. Key challenges for the
                         acoustic sensor were to reduce noise to an acceptable level, develop
                         microphones with sufficient range, and reduce the size of the sensor so it
                         would fit into the BAT delivery system. The technology development that
                         was necessary to have the sensor meet requirements had to be
                         accomplished during the schedule-driven, delivery-oriented product
                         development program. The development program encountered technical
                         problems that left the program manager with no choice but to slip the
                         schedule and increase the cost. By the start of the engineering and
                         manufacturing development phase, program officials stated that the
                         acoustic sensor had a TRL 5—still a high risk using the Air Force Research
                         Laboratory’s criteria.



Good Technology          With the right environment as a precondition, managers on the successful
                         cases benefited from disciplined technology development processes that
Handoff Decisions        linked the technologies to products and provided credible information on
Depend on the Tools      the status of technologies. They also had standards that were both clear
                         and high for assessing readiness. Once a technology’s feasibility and
and Authority Given to   usefulness were demonstrated, it was linked to a product through an early
Managers                 agreement with the product developer to use it if it could be fully
                         developed. Ideally, as technologies approached the higher readiness levels
                         associated with the bridge, S&T managers and receiving product managers
                         agreed to more specific terms for accepting or rejecting a technology.
                         These agreements were early links to the product that were needed for the
                         technology to succeed. If a product manager was not willing to make such
                         an agreement, then the investment to bring the technology to higher
                         readiness levels might not be made. S&T managers were responsible for
                         ensuring that information at key junctures was sufficient and that the
                         technology was ready for inclusion on a product. They saw their role as to
                         screen and develop technologies to standards acceptable to product
                         managers. Product managers were responsible for ensuring that the
                         product could be developed and brought to market within cost and
                         performance targets. They saw as their role to encourage the successful



                         Page 41                                           GAO/NSIAD-99-162 Best Practices
                             Chapter 3
                             Controllable Conditions Affect How Well a
                             Technology’s Inclusion on a Product Can Be
                             Managed




                             development of new technology but to decline the handoff if it did not meet
                             product performance, cost, and schedule requirements.

                             When an S&T organization disengages from a technology at a low TRL, the
                             S&T manager gives up much of the ability to be a gatekeeper. In the event
                             that unyielding requirements or other pressures force product managers to
                             accept technologies before they have matured, they are weakened in their
                             ability to safeguard the product development from technology risks. For
                             the cases in which technologies had problems transitioning to products,
                             decisionmakers were disadvantaged by the incomplete information
                             available to them, yet were not empowered to say no to the handoff. Their
                             situation was further degraded by risk assessments that embodied lower
                             standards for accepting undemonstrated technology readiness. In the case
                             of the BAT, the S&T community was bypassed altogether, as the weapon
                             system and its enabling technologies were proposed by a contractor and
                             assigned directly to a program manager.


Successful Cases Benefited   All new technologies at Ford, regardless of whether they are proposed by
From Strong Gatekeepers,     inside or outside sources, take essentially the same path and gates into
                             products. Initially, technology proposals pass through a process that
Disciplined Processes, and
                             prioritizes them according to customer needs. The proposals are then
High Maturity Standards      passed on to the Advanced Vehicle Technology Office, an S&T organization
                             that determines the readiness of the proposed technology and fits it into
                             Ford’s path of technology demonstration. Once approved, the technology
                             follows a structured process that includes two development phases:
                             concept ready and implementation ready. This process results in a smooth
                             transition from the technology development environment into a product,
                             once the technology is mature. Ford’s adaptive cruise control technology
                             went through this process, as shown in figure 3.3.




                             Page 42                                        GAO/NSIAD-99-162 Best Practices
                                              Chapter 3
                                              Controllable Conditions Affect How Well a
                                              Technology’s Inclusion on a Product Can Be
                                              Managed




Figure 3.3: Process for Closing the Gap Between the Readiness of Adaptive Cruise Control Technology and Jaguar
Requirements

                               Technology                 Bridge between technology                 Product
                                feasibility                       and product                     development




               Advanced vehicle technology office                                      Jaguar vehicle team



  Enabling technologies         Interest expressed      Agreement signed         Product launch              Model introduced
   pursued in the technology      by vehicle team       between S&T and
             base                                         vehicle team

                                      TRL 5                  TRL 7                    TRL 8
         1993                         1995                   1996                     1997                        1999


                                              According to Ford officials, technologies for adaptive cruise control
                                              existed as separate projects in the technology base from about 1993 to
                                              1995, when the Jaguar vehicle team identified a strong demand for the
                                              capability. Ford’s S&T community inventoried the ongoing projects and
                                              demonstrated the technology as a laboratory breadboard—a TRL 5. By
                                              August 1996, the technologists had built a prototype that could
                                              demonstrate the technology in a relevant environment—a TRL 7. This work
                                              comprised the concept ready phase, in which the technology was taken
                                              from concept to where its feasibility was demonstrated to potential users.
                                              At the end of this phase, S&T representatives proved that it could work,
                                              and cost, schedule, and performance targets were established. Also, a
                                              target product and sponsor were identified, linking the technology to a
                                              product. The sponsor agreed that it would accept this technology if specific
                                              cost, quality, schedule, and performance targets were met. The medium for
                                              this acceptance was the Deliverables Agreement Log (DEAL), which was
                                              signed in September 1996 by Jaguar’s chief engineer based on the prototype
                                              demonstration.

                                              Ford uses the DEAL as a tool to maintain visibility over a new technology
                                              as it progresses through the development process and to assess its
                                              readiness and acceptability for inclusion in a vehicle program before
                                              handing it to a sponsor, the vehicle center, or team. The DEAL formalizes



                                              Page 43                                               GAO/NSIAD-99-162 Best Practices
Chapter 3
Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed




the content of the two development phases and establishes agreements
between the technologists managing the project and those with authority to
accept the technology into a product. According to Ford officials, the DEAL
is important to this process because it is a contract between the parties that
addresses the technology’s performance, cost, quality, weight, producibility,
and maintainability targets that must be met before the end of each phase.
It has been invaluable in getting parties to agree on what is expected by the
giver and receiver of a technology during the process.

Once these targets are established, the technology moves to the
implementation ready phase. For the Jaguar, the Advanced Vehicle
Technology Office matured the technology to a high level of readiness by
prototyping it in demonstrator vehicles—a TRL 8. The technology passed
the implementation ready milestone in February 1997. At that point, the
vehicle team accepted the technology for inclusion on a Jaguar product
development.

Ford used this decision-making process to develop the night vision
technology, but with a different result. Since 1991, Ford has been working
on this technology to provide a wide field of view and depth perception for
the driver at night, similar to that provided by a FLIR. By 1998, the
Advanced Vehicle Technology Office brought the technology to a TRL 8.
However, the vehicle center did not agree to include the technology on a
product because the technology did not meet the cost targets established in
the DEAL.

Other companies we visited had similar practices for supporting
technology inclusion decisions. For example, 3M takes technology from its
technology base when it believes it has a customer need. The gatekeeper
responsible for moving technology into a concept phase—analogous to
TRL 3 or 4—is the S&T organization of a business unit. That business unit
monitors the technology’s progress until a new product requirement is
identified and decides whether there is interest from a product center to
“pull” it. If an interest exists, it begins a feasibility phase that refines
requirements through quality functional deployment and builds working
prototypes of the new product—a stage that would be analogous to TRL 7
or 8. This phase culminates with an agreement between the technologists
and the product developers—the receivers—as to the specific cost and
schedule targets that must be met for the technology to be included into a
product. To help facilitate the transition, 3M establishes a product
development team that includes people from research and development,




Page 44                                          GAO/NSIAD-99-162 Best Practices
Chapter 3
Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed




marketing, manufacturing, and other functions that transfer with the new
technology and ensure it is integrated into the new product.

3M also has high standards for measuring the readiness of a technology
before the product developer accepts it. For example, 3M officials told us
that they are developing a fuel cell technology for which they have built 15
prototypes for testing purposes—a TRL 7 or higher. However, because the
technology has not yet met all of the cost, schedule, and performance
targets for product development, they have not allowed it to be included on
a new product, despite demand from the marketplace.

Among the DOD cases, the process followed and the roles played on the
AAAV program had several features that enabled good technology inclusion
decisions. For almost 3 decades, the Marine Corps has stated a need for an
amphibious vehicle with far greater capabilities than the current vehicle.
Specifically, the requirement to achieve a speed of 20 to 25 knots in the
open ocean made advances in propulsion technology key enablers for the
AAAV program. For a vehicle of the planned size and weight of the AAAV,
this requirement meant achieving 2,700 horsepower with a relatively
compact engine that must operate on land and in water. The Corps had
been exploring propulsion technologies for such a vehicle in its technology
base for many years. Despite this, the Office of Naval Research, an S&T
organization, assessed the propulsion technology and advised that it was
not mature enough to warrant inclusion on a program. Based on this
assessment, Marine Corps and Navy decisionmakers delayed program
launch from 1991 to 1995, until the technology could be brought to higher
readiness levels. Figure 3.4 illustrates the process used to transition this
technology.




Page 45                                         GAO/NSIAD-99-162 Best Practices
                                        Chapter 3
                                        Controllable Conditions Affect How Well a
                                        Technology’s Inclusion on a Product Can Be
                                        Managed




Figure 3.4: Process for Closing the Gap Between the Readiness of Propulsion Technologies and AAAV Requirements



                     Technology                   Bridge between                          Product
                      feasibility             technology and program                    development



            Office of Naval Research                                                 AAAV program office



     Engine proof             Agreements reached between           Program launched in                       Prototype
      of concept                S&T and program office              program definition                        engine
     demonstrated                                                         phase                            demonstrated

         TRL 3                                                            TRL 6                               TRL 7

        1988                             1991                             1995                                1999


                                        The S&T community and the product managers agreed on what had to be
                                        done before the program could be launched. The S&T community then
                                        took the lead in maturing the engine to a TRL 6—a level the Air Force
                                        Research Laboratory considers acceptable for starting the program
                                        definition and risk reduction phase. Thus, the assessment by the Office of
                                        Naval Research provided both the information and the criteria that enabled
                                        decisionmakers to say no to launching the program given the low readiness
                                        of the propulsion technology. This was coupled with the flexibility to wait
                                        for the technology to mature and the decision to give an S&T organization
                                        responsibility for managing the bridge to product readiness. Figure 3.5
                                        shows the AAAV.




                                        Page 46                                               GAO/NSIAD-99-162 Best Practices
                   Chapter 3
                   Controllable Conditions Affect How Well a
                   Technology’s Inclusion on a Product Can Be
                   Managed




Figure 3.5: AAAV




                   The Marine Corps and Navy delayed program launch by 4 years to develop key technologies to a
                   higher readiness level
                   Source: DOD.


                   Even with an urgent need for the AAAV, the Marine Corps remained
                   disciplined in its development approach, allowing the technology to mature
                   to the level of the requirement. Two years before program launch, a Navy
                   S&T organization demonstrated the technology in a full-scale prototype
                   engine. By program launch in 1995, the required 2,700 horsepower was
                   demonstrated by a near prototype engine—a TRL 6. The remaining risk was
                   limited to marginal weight and size reductions, although the demonstrator
                   engine could be used as a backup if the size and weight reductions could
                   not be obtained. In early 1999, the AAAV program office demonstrated a




                   Page 47                                                      GAO/NSIAD-99-162 Best Practices
                                         Chapter 3
                                         Controllable Conditions Affect How Well a
                                         Technology’s Inclusion on a Product Can Be
                                         Managed




                                         prototype engine at 2,700 horsepower that met size and weight
                                         requirements—a TRL 7.


Technology Handoffs Were                 In the BAT program, neither the S&T community nor the product manager
Compromised When                         had the opportunity to act as a gatekeeper between product requirements
                                         and the maturity of enabling technologies. All of the technologies for the
Managers Had Limited
                                         BAT came to the program after the contractor, in 1985, had proposed a
Information and Authority                weapon concept for carrying out unmanned, deep strike missions to attack
                                         enemy armored vehicles. Army leadership accepted the concept and
                                         drafted requirements for the BAT, and the acquisition program was
                                         launched after the proposal was accepted. Thus, the technology for the
                                         weapon came directly from the contractor’s technology base into the
                                         acquisition program, with little or no review by the Army’s S&T
                                         organization. The process, information, and standards that were critical to
                                         successful technology inclusion decisions in other cases were not
                                         employed on the BAT. The process followed is shown in figure 3.6.


Figure 3.6: Assimilation of New Technology Into the BAT Program

                                                                 Technology maturation
         Technology concept                                     and product development




       Contractor technology base                                     BAT program office



                               Program launched in                                Start engineering,
                             program definition phase                            manufacturing and
                                                                                 development phase
                                      TRL 2-3                                          TRL 5
                                      1985                                             1991


                                         The program office accepted the acoustic sensor, infrared seeker, and
                                         navigation technologies included on the BAT program. In retrospect, the
                                         levels of demonstration at the time posed high risks to the product
                                         development because the acoustic sensor technology had a TRL of 2 and



                                         Page 48                                               GAO/NSIAD-99-162 Best Practices
Chapter 3
Controllable Conditions Affect How Well a
Technology’s Inclusion on a Product Can Be
Managed




the infrared seeker and navigation technologies had TRLs of 3. Program
officials stated that a significant amount of technology development was
required during product development due to the lack of visibility over
technology readiness before program launch. As a result, the development
program’s cost and schedule significantly increased over original estimates.

An interesting sidelight to the BAT experience concerns the inertial
measurement unit, a navigation component of the submunition. When the
contractor first proposed the BAT concept, the design included a mature
inertial measurement unit in production on other systems. However, after
the program was launched, the contractor substituted a new quartz rate
technology. At the request of the BAT program manager, the Army’s Missile
Research and Development Engineering Center, an S&T organization,
assessed the maturity of the quartz rate technology. The Center concluded
that the new technology had not demonstrated a high enough level of
readiness and recommended that a more proven existing technology be
used in the program. Eventually, the new technology was dropped, and an
existing technology that was at a higher readiness level was used.

We observed additional cases in which decisionmakers relied on
comparatively low standards for including technologies. The Army
assessed the FLIR, integrated avionics, and helmet mounted display
technologies as having moderate risk when they were included in the
Comanche program. Army officials stated that they required only the
existence of an ongoing S&T technology project as acceptable, as long as
the technology was projected to be ready by the engineering and
manufacturing development phase. According to program officials,
demonstrated maturity was considered but not required; proof that the
projects were progressing as scheduled was enough. These technologies,
however, had TRLs of 3 at the time of launch—a high risk for the program
definition and risk reduction phase. This risk assessment is more
consistent with the actual experience of the technologies’ maturation in the
program.

The standards used for accepting the laser technology into the ABL
program also appeared low when compared with the standards used on the
more successful cases. While the Air Force had established demonstration
standards for the laser to meet prior to program launch, these standards
were met if scale models of the laser technology in a laboratory
demonstrated they had the potential to produce the energy needed for an
operational system. This level of technical demonstration equated to a TRL
of 4, representing a high risk for inclusion into an acquisition program.



Page 49                                         GAO/NSIAD-99-162 Best Practices
Chapter 4

Impediments to Adopting Best Practices for
Technology Inclusion in DOD Are
Surmountable                                                                                           Chapte4
                                                                                                             r




                         Although product developments—commercial or defense—fare better
                         when key technologies are matured before they are included in the product
                         design, the more traditional approach within DOD is to mature technology
                         during a product’s development. Rational explanations are behind this
                         tradition. S&T organizations, operating within fixed budget levels, are not
                         necessarily accustomed or equipped to manage the bridge between
                         technology feasibility and product readiness. Programs are more able to
                         command the large budgets necessary for reaching higher levels of
                         technology readiness than S&T projects. Also, pressures are exerted on
                         new programs to offer unique performance and acceptable cost and
                         schedule projections, which encourage premature acceptance of unproven
                         technologies.

                         The Under Secretary of Defense for Acquisition and Technology not only
                         supports shorter cycle times and a more aggressive pursuit of technology
                         outside of programs, but also use of commercial best practices to get these
                         results. DOD has several initiatives underway that could make conditions
                         more favorable for S&T organizations to mature a technology further
                         before it is included in a product development. One Army project calls for
                         an S&T organization to manage all technology maturation and integration
                         tasks for a new combat vehicle up to the engineering and manufacturing
                         development phase. Other initiatives may make the S&T community a more
                         integral participant in matching user requirements with technology and
                         tying S&T projects more closely to product development paths. Whether
                         these efforts are effective and can be applied on a broader scale remains to
                         be seen.



Several Factors Make     Budgetary, organizational, and other factors within DOD make it difficult to
                         bring technologies to high readiness levels before being included in
It Difficult to Mature   weapon systems. These factors encourage S&T organizations to disengage
Technologies Before      from technology development too soon and weapon system program
                         managers to accept immature technology. Factors other than these
They Are Included on     encourage leading commercial firms to keep technology development out
Weapon Systems           of the product developers’ hands and in those of S&T organizations. The
                         differences in these factors and in the management of technology
                         development stem from differences in what helps commercial and DOD
                         programs to succeed. They do not stem from capabilities commercial firms
                         possess that DOD does not.




                         Page 50                                         GAO/NSIAD-99-162 Best Practices
                            Chapter 4
                            Impediments to Adopting Best Practices for
                            Technology Inclusion in DOD Are
                            Surmountable




Budget and Organizational   Budget realities within DOD—the fact that weapon system programs
Factors                     attract higher levels of funding than S&T projects—make these programs a
                            more advantageous setting for funding technology development to the
                            higher readiness levels. As a practical matter, it is often necessary to move
                            immature technology to a weapon system program to get needed funds and
                            management support for maturation. Normally, DOD S&T organizations do
                            not see their role as going beyond demonstrating the feasibility of a
                            technology for generic—versus product specific—application (a TRL 5).
                            However, as seen in several of the cases we reviewed, even this level often
                            is not reached before a product development organization takes over. The
                            S&T organizations that helped to bridge the gap from technology feasibility
                            to product readiness on the more successful cases had gone beyond their
                            typical role.

                            One of the reasons that S&T organizations disengage relatively early is that
                            S&T work is traditionally funded as a percentage of the overall DOD
                            research and development budget. S&T organizations receive about
                            $8 billion annually, or about 20 percent, of DOD’s research and
                            development budget. This money funds several thousand projects,
                            providing less than $1 million per project on average. As a result, a project
                            needing $100 million or more to mature technology to higher readiness
                            levels than normal—not unreasonable sums—would command a fairly
                            large share of an S&T organization’s budget, thereby reducing funds
                            available for other projects. Under the current scenario, the remaining
                            80 percent of DOD’s research and development funds, approximately
                            $30 billion, is spread out over a much smaller number of specific weapon
                            programs. A typical weapon system program can receive several hundred
                            million dollars annually and occasionally over $1 billion to fund
                            development. A major program, such as the F-22, can command $15 billion
                            or more in total for product development, receiving sometimes more than
                            $2 billion in a year.

                            Events on the Air Force’s ABL program illustrate these realities. Originally,
                            the Air Force had planned the ABL as a technology development project to
                            be managed to high readiness levels by an S&T organization. The project
                            was started in 1992 as an advanced technology transition demonstration to
                            design, fabricate, and test a single demonstrator weapon system and was to
                            take 8 years to complete. The pacing technologies, the laser and the beam
                            control, were to be matured to a high level—equivalent to TRL 6 or 7—
                            before being included in a product development program. Requirements
                            had not been fixed. In other words, the planned approach resembled what




                            Page 51                                         GAO/NSIAD-99-162 Best Practices
Chapter 4
Impediments to Adopting Best Practices for
Technology Inclusion in DOD Are
Surmountable




we have described as the more successful cases in our review. Figure 4.1
shows the ABL.


Figure 4.1: Airborne Laser




The Air Force used relatively low readiness level standards to include a key technology into ABL
Source: DOD.


In 1996, the Air Force abandoned this approach and decided to launch ABL
as a weapon system development program, not because technologies were
sufficiently mature but because of funding and sponsorship concerns. At
this time, the two key technologies were at TRLs 3 and 4. According to the
retired manager of the S&T project, a product development program was
deemed necessary to make the technology development effort appear real
to the users and not a scientific curiosity. Within the Air Force, the
perceived lack of support by the users placed the project in a constant state
of funding jeopardy. This perception was important because the S&T
project was costly, with a total estimated cost of $800 million, with some
annual funding requirements approaching $200 million. The annual funding
requirements would encompass a large percentage of the Air Force’s
S&T budget unless additional funds were made available from weapon
system budgets or elsewhere. By transitioning to a weapon system program
linked to user requirements, the ABL was more likely to get these funding
levels.

This approach was successful—the program won user support and the
desired funding. However, sacrifices were made in technology
development. According to the former project manager, the new program
focused less on the elemental technology hurdles and more on meeting all



Page 52                                                         GAO/NSIAD-99-162 Best Practices
                   Chapter 4
                   Impediments to Adopting Best Practices for
                   Technology Inclusion in DOD Are
                   Surmountable




                   user requirements. More expensive demonstrations were necessary to
                   meet these broader requirements without necessarily doing more to
                   demonstrate basic technology readiness. It became a more traditional
                   program with technology and product development proceeding at the same
                   time, with attendant higher risks. In March 1999, we reported that, while
                   the ABL has made progress in developing these technologies, it still faced
                   technical challenges.1


Other Incentives   Pressures exerted on weapon system programs can make it advantageous
                   to include in their design immature technologies that offer significant
                   performance gains. One traditional source has been the perceived threat.
                   Users can demand performance improvements that necessitate the
                   application of unproven technologies, particularly when a fielding date is
                   mandated, to stay ahead of the threat. Another source is technologists,
                   whether from S&T organizations or contractors, who see a new weapon
                   system as an opportunity to apply a new technology. Also, the competition
                   for funds can encourage performance features—and requisite
                   technologies—that distinguish the new weapon system from competitors.

                   The F-22 was justified as being faster, stealthier, and more lethal than other
                   fighters, such as the F-15 and F-117, were. As a result, the F-22 is being
                   designed with several advanced technologies, including a very
                   sophisticated suite of avionics that is critical to its performance features
                   that distinguish it from the other fighters. However, at the time the F-22
                   program was launched in 1986, the avionics technologies were immature;
                   they have since been a source of problems on the program. We recently
                   reported that the development of the F-22’s integrated avionics systems
                   continues to experience cost growth and schedule delays, more than
                   12 years into the program.2

                   A different set of incentives causes leading commercial firms to make their
                   S&T organizations responsible for maturing technologies to higher
                   readiness levels. Commercial firms are aware of the risks associated with
                   the high investment that product development requires. They have a strong
                   incentive in the realization that if a product is late, costs more, or performs


                   1
                   Defense Acquisitions: DOD Efforts to Develop Laser Weapons for Theater Defense (GAO/NSIAD-99-50,
                   Mar. 31, 1999).
                   2
                    F-22 Aircraft: Issues in Achieving Engineering and Manufacturing Development Costs
                   (GAO/NSIAD-99-55, Mar. 15, 1999).




                   Page 53                                                       GAO/NSIAD-99-162 Best Practices
                        Chapter 4
                        Impediments to Adopting Best Practices for
                        Technology Inclusion in DOD Are
                        Surmountable




                        less than expected, the customer could walk away from the product and
                        the investment would be lost. Minimizing the possibility of technology
                        being the cause of such problems is thus a top priority. Having their S&T
                        organizations reduce those risks is essential to putting product
                        developments in the best position to succeed. DOD does not have the same
                        incentives. DOD programs are not penalized if a product is late, costs more,
                        or performs less than expected, because the customer does not walk away.



Services Encouraged     Over the past several years, DOD has encouraged the services to use best
                        practices to streamline the current process for acquiring new weapon
to Use Best Practices   systems in order to make them faster, cheaper, and better. Shorter
                        acquisition cycle times are seen as critical to making the best use of
                        advances in technology. To encourage change, DOD has set a goal to reduce
                        the average acquisition cycle time for all program starts in fiscal year 1999
                        and beyond by 50 percent over historical averages. DOD has several
                        initiatives to improve its technology development process and to move
                        technologies to the warfighter faster and less expensively than the
                        traditional means. The initiatives also attempt to put the organizations and
                        funding in place to bring technologies to higher readiness levels before they
                        are included in programs. These initiatives—defense technology
                        objectives, advanced technology demonstrations, and advanced concept
                        technology demonstrations—call for S&T organizations to play a bigger
                        role in managing technologies closer to the point of product readiness,
                        matching requirements to technology projects, and making better use of
                        demonstration standards.


Defense Technology      Defense technology objectives (DTO) are used to bring more discipline to
Objectives              S&T projects and to link them more closely with weapon system
                        development programs. A DTO typically involves a particular technology
                        advance, such as high temperature materials for turbine engines and high
                        fidelity infrared sensors. It can also group several technologies into a larger
                        demonstration. Each DTO identifies a specific technology advancement
                        that will be developed or demonstrated, the anticipated date of the
                        technology availability, the ultimate customer, and the specific benefits
                        resulting from the technology. It places a corporate attention and
                        commitment on the technology project by having the technologists,
                        product developer, and customer involved in the project.

                        According to DOD, the focus of its S&T investment is enhanced and guided
                        through DTOs. Each DTO must go through a formal review and approval


                        Page 54                                           GAO/NSIAD-99-162 Best Practices
                            Chapter 4
                            Impediments to Adopting Best Practices for
                            Technology Inclusion in DOD Are
                            Surmountable




                            process within DOD and must be directly related to advancing the
                            operational concepts depicted in DOD’s “Joint Vision 2010” planning
                            document. According to DOD officials, those requirements have helped to
                            eliminate instances in which technologists work on projects of particular
                            interest to them, but with no military application, because the projects
                            should be linked to a specific warfighter need. For fiscal year 1999, DOD
                            established approximately 350 DTOs, which accounted for $3 billion, or
                            less than 50 percent, of the funds DOD had allocated to S&T projects. The
                            remaining funds were allocated to projects under the jurisdiction of each
                            military service or other defense agencies and did not go through the same
                            review and approval process.


Advanced Technology         Advanced technology demonstrations (ATD) are intended to more rapidly
Demonstrations              evolve and demonstrate new technologies so they can be incorporated into
                            a product, if warranted. An ATD has four characteristics that distinguish it
                            from a conventional S&T project. They (1) require large-scale resources;
                            (2) involve the user; (3) use specific cost, schedule, and performance
                            metrics; and (4) identify a target product for inclusion. An ATD is managed
                            by an S&T organization and should conclude with an operational
                            demonstration of the potential capabilities of the technology, equating to a
                            TRL 5 or 6. The original approach to the ABL was essentially an ATD
                            approach. Most ATDs use laboratory hardware to demonstrate the
                            potential capability of nonproduct specific technologies and not prototype
                            hardware. If the technology is determined to be feasible and provides some
                            military use, then it may proceed to the program definition and risk
                            reduction phase of an acquisition program. From that point, the product
                            developer completes the technology development for a specific product.


Advanced Concept            In 1994, DOD initiated Advanced Concept Technology Demonstrations
Technology Demonstrations   (ACTD) to help expedite the transition of mature technologies from the
                            developers to the warfighters. ACTDs are intended to help the DOD
                            acquisition process adapt to budget constraints while developing
                            technology more rapidly. The purpose of an ACTD is to assess the military
                            use of a capability, such as a weapon, comprised of mature technologies.
                            Typically, ACTDs last 2 to 4 years and consist of building and
                            demonstrating a prototype to provide a warfighter the opportunity to
                            assess a prototype’s capability in realistic operational scenarios. From this
                            demonstration, the warfighter can refine operational requirements, develop
                            an initial concept of operation, and determine the military use of the
                            technology before it proceeds to the product development process.



                            Page 55                                         GAO/NSIAD-99-162 Best Practices
                           Chapter 4
                           Impediments to Adopting Best Practices for
                           Technology Inclusion in DOD Are
                           Surmountable




                           According to DOD, ACTDs, which are managed by S&T organizations, will
                           be a key mechanism to ensure technology development is separated from
                           product development. In related work on unmanned air vehicles, we found
                           that ACTDs provided decisionmakers credible data that they used to
                           terminate efforts or transition the demonstrator to an acquisition program.
                           In these cases, ACTDs put decisionmakers in a better position to be
                           gatekeepers. However, we have reported that the ACTD program needs to
                           be improved.3 We found that DOD’s process for selecting program
                           candidates does not include adequate criteria for assessing the maturity of
                           proposed technology and has resulted in the approval of projects that
                           included immature technologies. We found that the use of specific criteria
                           for determining maturity was a best practice in the most successful
                           technology development cases we examined.



Two Unique DOD             Two DOD projects are using S&T organizations to manage technology
                           development to higher readiness levels. One, the Army’s Future Scout and
Projects May Provide       Cavalry System, is using a modified ATD to mature technologies and make
Lessons on How to          performance trade-offs in the more flexible environment provided by
                           S&T. The other is a joint government and industry program, which the Air
Enable S&T                 Force Research Laboratory is managing to reduce the risks associated with
Organizations to           new jet engine technologies. These projects may provide insights on how
Manage Technology          S&T organizations could routinely play a bigger role in maturing
                           technologies enough for safe inclusion on weapon system programs. They
Further                    may also clarify the concern that playing a bigger role in technology
                           maturation could cause S&T organizations to do less basic research and
                           technology development.


Future Scout and Cavalry   In fiscal year 1997, the Army began piloting a variation of an ATD that is
System                     designed to help bridge the gap between technology development and
                           product development by expanding the S&T community’s role in managing
                           technologies further into the development cycle. The Army’s initiative,
                           called Fast Track, is intended to reduce cost and cycle time by bypassing
                           the program definition and risk reduction phase of the DOD acquisition
                           process. The Army is testing this concept with its Future Scout and Cavalry
                           System project. In this project, the Army will design, develop, and build a
                           demonstrator vehicle to show the technical feasibility of the weapon. All of

                           3
                            Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved
                           (GAO/NSIAD-99-4, Oct. 15, 1998).




                           Page 56                                                    GAO/NSIAD-99-162 Best Practices
                                           Chapter 4
                                           Impediments to Adopting Best Practices for
                                           Technology Inclusion in DOD Are
                                           Surmountable




                                           these tasks will be done under the management of the Army’s S&T
                                           organization. The Army believes that a more extensive S&T project will
                                           make the program definition phase unnecessary and estimates that this
                                           concept will reduce the development process by as much as 4 years and
                                           save about $400 million. We did not review the Future Scout project in
                                           terms of its affordability, feasibility, or any impacts it may have on the
                                           Army’s S&T budget. Figure 4.2 compares the Fast Track development
                                           process with the traditional approach.


Figure 4.2: Comparison of Traditional Technology Development Process With the Army’s Fast Track Approach

                                                                            Technology maturation
                            Technology feasibility                         and product development




          Traditional        S&T organization                            Weapon system program office
          approach
                                                                                     9 years

                                              Program launched in                                             Production
                                            program definition phase

                                                      TRL 3-5



                                                             Bridge built to
                             Technology feasibility                                     Technology maturation
                                                          increase technology
                                                                maturity               and product development




          Future Scout
                              Army S&T organization                           Weapon system program office
          and Cavalry
          System                                                                               4.5 years

                                                 Program office          Program launch                          Production
                                                 staff joins S&T         in engineering,
                                                  project office        manufacturing and
                                                                       development phase.

                                                                             TRL 6



                                           While we do not necessarily agree that the first phase of the acquisition
                                           cycle can be omitted, so far the Future Scout project is emulating
                                           technology development practices like those we observed in the successful
                                           cases. First, it has established demonstration criteria that must be met
                                           before the technology enters product development. Second, it has also


                                           Page 57                                                         GAO/NSIAD-99-162 Best Practices
Chapter 4
Impediments to Adopting Best Practices for
Technology Inclusion in DOD Are
Surmountable




established forums that involve key players on the technology path to keep
them informed of the technology’s development progress. For example, the
acquisition program manager will be integrated into the development
project during the final 1.5 years of the S&T program. This should provide a
good link between the technology development and product development,
allowing the program manager to fully understand the technology before
product development begins. Finally, by allowing an S&T organization the
flexibility to manage technologies further into the development cycle,
Army officials believe they will be able to make trade-offs among cost,
schedule, and performance requirements before program launch, without
raising concerns about the state of the project or breaching baselines that
had been set without enough knowledge.

While this concept comes closer to the most successful technology
development cases we reviewed, it still embodies greater technical risk.
The Army expects to demonstrate some performance capabilities of the
vehicle before the product development phase begins. However, the
demonstrator vehicle will only be about 75 percent of a complete
prototype, which means some key technologies will not be demonstrated
to high readiness levels before that phase begins. Nonetheless, the project
manager equated the expected overall technical maturity of the vehicle at
transition to a TRL 6. The Army considers this a medium or acceptable
level of risk, and it is willing to enter product development with some
immature technologies. If, however, product development begins at
engineering and manufacturing development, this risk could be assessed as
high, based on TRLs. Figure 4.3 shows an artist’s concept of the Future
Scout and Cavalry System.




Page 58                                         GAO/NSIAD-99-162 Best Practices
                            Chapter 4
                            Impediments to Adopting Best Practices for
                            Technology Inclusion in DOD Are
                            Surmountable




                            Figure 4.3: Future Scout and Cavalry System




                            An Army S&T organization is maturing technologies and proposing performance trade-offs for the
                            Future Scout and Cavalry System before program launch
                            Source: DOD.




Integrated High             The Integrated High Performance Turbine Engine Technology program—a
Performance Turbine         joint government and industry effort—is focused on developing
                            technologies for more affordable and higher performance turbine engines
Engine Technology Program
                            for both missiles and aircraft. It is a technology validation program and is
                            managed by an S&T organization to perform demonstrations of various
                            engine technologies to higher readiness levels than most S&T projects.
                            After the demonstrations, the technologies enter a product development
                            program. The program takes the technology through a series of tests that
                            range from individual component tests to full-scale engine demonstrations.
                            The program has established strong links with the acquisition programs for
                            which the technologies are intended. For example, Air Force Research
                            Laboratory officials informed us that they established formal technology
                            transition plans with the F-22 and Joint Strike Fighter programs that
                            document agreements on what technology development activities will be
                            performed to support the programs. Representatives from each program
                            office are invited to all technology demonstrations and are kept informed
                            about demonstrated progress.




                            Page 59                                                       GAO/NSIAD-99-162 Best Practices
Chapter 4
Impediments to Adopting Best Practices for
Technology Inclusion in DOD Are
Surmountable




As part of the program, the Air Force developed a set of standards to assess
the readiness levels of technologies similar to NASA’s TRLs. According to
the Air Force, the S&T organization uses these standards to determine
when the project has been completed. These standards were the first
application of readiness levels by the Air Force Research Laboratory. There
are five technology readiness levels ranging from component-level tests in
a laboratory to the highest level involving actual flight tests of engines. The
program typically does not take technologies to the highest readiness level
(flight test) because of the high cost. The program stops when it has been
determined the technology is well defined within acceptable boundaries
and a good correlation exists between test results and engineering
predictions. This readiness level would translate to a TRL of 5 or 6, as used
in this report. The final step of the technology development is left to the
product developer who determines if the technology can be packaged and
integrated into the final product.




Page 60                                           GAO/NSIAD-99-162 Best Practices
Chapter 5

Conclusions and Recommendations                                                               Chapte5
                                                                                                    r




Conclusions   Clearly, DOD’s continued advancement of new technologies is essential to
              the continued superiority of its weaponry. The leading edge military
              capabilities the United States possesses today, such as stealth aircraft,
              precision munitions, and intelligence-gathering satellites, bear witness to
              the effects of such technical advances. At the same time, the incorporation
              of advanced technologies before they are mature has been a major source
              of cost increases, schedule delays, and performance problems on weapon
              systems. As DOD contemplates increasing its annual investment in new
              weapons to $60 billion, the expectations on program managers are great:
              they must develop and field weapons of superior capability more quickly
              and less expensively than in the past. The way advanced technologies are
              matured and included in weapon systems will play a central role in meeting
              these expectations. Although different ways to better assimilate new
              technologies into weapons are legitimate topics for debate, that it has to be
              done better is not.

              The leading commercial firms’ practices have produced results that
              resemble those sought by DOD: more technically advanced, higher quality
              products, developed in significantly less time, and less expensively than
              their predecessors. Managing the development of advanced technology
              differently--and separately--from the development of a product has been
              key to these results. The firms insist that advanced technology reach a high
              level of maturity, the point at which the knowledge about that technology is
              essentially complete, before allowing it into a product development. By
              separating the two, the firms lessen the product manager’s burden and
              place that person in a better position to succeed in delivering the product.
              These practices may not necessarily accelerate the pace at which
              technology matures. In fact, several of the commercial technologies we
              reviewed took 10 years or more to get to market. The clear beneficiaries of
              the practices are the product developments, for which the investments are
              much larger, and time translates into significantly more resources than in a
              technology project. Adapting these practices on its weapon system
              programs can help DOD to reduce costs and the time from product launch
              to fielding, and use technology advances as they become available more
              frequently.

              Separating technology development from product development calls for a
              new approach to managing technology development. Two conditions are
              essential to such an approach. First, the right environment for maturing
              technologies must exist. A practice that is instrumental in providing this
              environment is maturing technology to achieve product readiness before it



              Page 61                                         GAO/NSIAD-99-162 Best Practices
Chapter 5
Conclusions and Recommendations




is constrained by the rules of an acquisition program. In the successful
DOD cases we reviewed, this environment was provided by S&T
organizations or a team of S&T and product developers who managed
technologies to high readiness levels before they were included in an
acquisition program. These organizations provided an environment more
conducive to the ups and downs normally associated with the discovery
process. A corollary practice is agreeing on what level of knowledge is
needed about a new technology before it is considered for inclusion in a
product design. When that knowledge level does not exist, the flexibility for
S&T organizations and product managers to either take the time to mature
the technology or trade off product requirements until they can be met with
mature technology is essential. It is a rare program that can proceed with a
gap between product requirements and maturity of key technologies and
still be delivered on time and within costs. Second, S&T and product
managers must be provided with the disciplined processes, information,
standards, and authority to make good handoffs of technology to product.
Prepared with the tools and authority to make sound handoff decisions,
both S&T and product managers can function as gatekeepers to safeguard
the product development from undue technology risks.

Leading commercial firms have adopted this approach as a matter of
necessity and have used the organizations, tools, and other practices to
foster technology development and improve the outcomes of product
developments. The high stakes stemming from the large investment
required for a new product and the risks if the product does not meet
customer needs reinforce this approach in leading commercial firms. The
DOD cases that followed a similar approach were realizing better program
outcomes, at least in the sense that the programs avoided key technology
development problems. Yet, these cases are not the norm for DOD
programs for several reasons.

• More typically, the commitment to develop and produce a weapon
  system is made before a match between technology and weapon system
  requirements exists.
• DOD programs operate under different conditions that make it more
  difficult—and less rewarding—to separate technology development
  from product development.
• Budget realities make it more difficult for S&T organizations to carry
  technologies to the high readiness levels needed to meet product
  requirements; such resources are more available within product
  developments.




Page 62                                         GAO/NSIAD-99-162 Best Practices
                  Chapter 5
                  Conclusions and Recommendations




                  • The pressures to show the unequalled performance necessary to win
                    funding encourage including promising, but immature, technologies in
                    weapon system designs.

                  It will take procedural, organizational, and cultural changes within DOD’s
                  acquisition process to foster an environment in which (1) technologies can
                  be successfully matured outside the purview of weapon system programs,
                  (2) programs can be relieved of the pressures to include immature
                  technologies and the unrealistic expectations that the technologies will
                  conform to tight cost and schedule projections, and (3) technology
                  advances will not stall due to inadequate funding or lack of identification
                  with a product in the later, more expensive stages of demonstration.

                  Experience has shown that such an approach can work within DOD on
                  individual cases. DARPA played a primary role in managing the transition
                  of the nonpenetrating photonics mast technology to the Virginia class
                  attack submarine. The Integrated High Performance Turbine Engine
                  Technology program has carried advanced jet engine technologies to TRLs
                  of between 5 and 6 for successful inclusion into programs. In the Future
                  Scout program, an Army S&T organization, augmented by product
                  development staff, is managing an ATD to lower the risk of key
                  technologies before a product development program is launched. However,
                  it remains to be seen whether the Army will be successful in using large and
                  expensive S&T projects, such as the Future Scout program, without
                  affecting other Army S&T projects. A challenge for DOD will be whether
                  the lessons learned from these individual cases offer an approach that has
                  DOD-wide application. Meeting this challenge is essential to fielding
                  technologically superior weapons more quickly and within predicted costs.



Recommendations   We have previously recommended that DOD separate technology
                  development from weapon system programs. That recommendation was
                  made without prejudice toward the necessity of technology development
                  but rather with the intent that programs could be better managed if such
                  development was conducted outside of a program manager’s purview.
                  Similarly, the recommendations that follow are made without prejudice
                  toward—or the intention of compromising—the basic research and other
                  activities that S&T organizations perform. We recognize that
                  implementation of these recommendations will have organizational,
                  funding, and process implications and will require the cooperation of the
                  Congress.




                  Page 63                                        GAO/NSIAD-99-162 Best Practices
Chapter 5
Conclusions and Recommendations




To help ensure that new technologies are vigorously pursued and
successfully moved into weapon system programs, we recommend that the
Secretary of Defense adopt a disciplined and knowledge-based method for
assessing technology maturity, such as TRLs, DOD-wide. This practice
should employ standards for assessing risks of handoff to program
managers that are based on a technology’s level of demonstration and its
criticality to meeting the weapon system’s requirements.

With these tools in hand, we recommend that the Secretary (1) establish
the place at which a match is achieved between key technologies and
weapon system requirements as the proper time for committing to the cost,
schedule, and performance baseline for developing and producing that
weapon system and (2) require that key technologies reach a high maturity
level—analogous to TRL 7—before making that commitment. This would
approximate the launch point for product development as practiced by
leading commercial firms.

We recommend that the Secretary find ways to ensure that the managers
responsible for maturing the technologies and designing weapon systems
before product development are provided the more flexible environment
that is suitable for the discovery of knowledge, as distinct from the delivery
of a product. Providing more flexibility will require the cooperation of
requirements managers and resource managers so that rigid requirements
or the threat of jeopardizing the funding planned to start product
development will not put pressure on program managers to accept
immature technologies. Such an environment may not be feasible if the
program definition and risk reduction phase remains the effective launch
point for an entire weapon system program.

An implication of these recommendations is that S&T organizations will
have to play a greater role in maturing technologies to higher levels and
should be funded accordingly. Therefore, we recommend that the Secretary
of Defense evaluate the different ways S&T organizations can play a greater
role in helping technologies reach high levels of maturity before product
development begins. For example, given that a technology has sufficient
potential for application to a weapon system, at a minimum, an S&T
organization should be responsible for taking a technology to TRL 6 before
it is handed off to a program office at the program definition and risk
reduction phase. During this phase, the program manager would be
responsible for maturing the technology to TRL 7 before it is included in an
engineering and manufacturing development program. In a situation where
a single, design-pacing technology is to be developed for a known



Page 64                                          GAO/NSIAD-99-162 Best Practices
                      Chapter 5
                      Conclusions and Recommendations




                      application—like the nonpenetrating periscope—an S&T organization
                      should be required to mature that technology to TRL 7 before it is turned
                      over to a product development manager. S&T organizations could play a
                      similar role when a significant new technology is being prepared for
                      insertion into an existing weapon system. Finally, when multiple new
                      technologies are to be merged to create a weapon system, S&T
                      organizations should be required to bring key technologies to TRL 6 and
                      then become part of a hybrid organization with product developers to
                      integrate the technologies and bring them to TRL 7 before handing full
                      responsibility to a product development manager.

                      To help guard against the possibility that the more basic research and
                      technology development activities would be compromised by having S&T
                      organizations routinely take key technologies to TRL 6 or higher, we
                      recommend that the Secretary extract lessons from the nonpenetrating
                      periscope, the AAAV, and the Army’s Future Scout programs, and other
                      ATD and ACTD programs. Specifically, the Secretary should assess whether
                      the resources needed to enable S&T organizations to play a leading role in
                      the development of technologies and, in some cases, preliminary system
                      design, detracted from or displaced more basic research and technology
                      development programs.

                      Finally, we recommend that the Secretary empower managers of product
                      development programs to refuse to accept key technologies with low levels
                      of demonstrated maturity. The Secretary can encourage this behavior
                      through supportive decisions on individual programs, such as by denying
                      proposals to defer the development of key technologies and by favoring
                      proposals to lengthen schedules or lessen requirements to reduce
                      technological risk early.



Agency Comments and   DOD generally concurred with a draft of this report and its
                      recommendations, noting that the traditional path to new weapon system
Our Evaluation        development is no longer affordable or necessary (see app. I). DOD stated
                      that it has embarked upon a “Revolution in Business Affairs” that will
                      enable new technologies to be developed more efficiently and effectively. It
                      believes that the first steps in this direction have already been taken but
                      agrees that more progress needs to be made. DOD agreed that TRLs are
                      necessary in assisting decisionmakers in deciding on when and where to
                      insert new technologies into weapon system programs and that weapon
                      system managers should ensure that technology is matured to a TRL 7
                      before insertion occurs. DOD concurred that S&T organizations should be



                      Page 65                                         GAO/NSIAD-99-162 Best Practices
Chapter 5
Conclusions and Recommendations




involved in maturing technologies to high levels, such as TRL 6, before
transitioning to the engineering and manufacturing development phase and
agreed to assess the impact of this involvement on other S&T resources.
We note that the best practice is to mature technology to at least a TRL 7
before starting the engineering and manufacturing development phase,
whether the technology is managed by an S&T organization, a weapon
system program manager, or a hybrid of the two organizations.

DOD noted that while TRLs are important and necessary, the increasing
projected life for new weapon systems, total ownership costs, and urgency
based upon threat assessments are also important considerations for
system development decisions. We agree and note that our
recommendations are not intended to cover all aspects of weapon system
development decisions or to suggest that technology maturity is the only
factor in such decisions. Rather, the recommendations are in keeping with
the purpose of the report, “to determine whether best practices offer
methods to improve the way DOD matures new technology so that it can be
assimilated into weapon system programs with less disruption.” We believe
that a knowledge-based approach to maturing technology, such as TRLs,
can benefit other considerations as well. For example, decisions on what
technologies to include in a weapon system and when to include them can
have a significant bearing on its total ownership costs.

DOD stated that there should be an established point for the transition of
technologies and that it plans to supplement its milestone review process
with additional guidance in the next revisions to DOD Directive 5000.2R. It
also stated that its policy on the evolutionary approach to weapon
acquisitions should be developed in consonance with the technology
transition strategy. We cannot comment on the revisions to the directive or
the evolutionary acquisition policy because they have yet to be published.
However, under the current milestone review process, the pressures placed
on a program during the program definition and risk reduction phase—
when much technology development occurs—can operate against the
flexibility and judgments that are needed to mature technologies. If the
revisions to the directive supplement the current milestones without
relieving the pressures brought to bear on programs as they are launched in
the program definition and risk reduction phase, it will remain difficult to
discourage the acceptance of immature technologies in the design of new
weapon systems. To relieve these pressures, we encourage DOD, as it
develops the directive and the evolutionary acquisition policy, to separate
technology development from product development and to redefine the
launch point for a program as the point at which enough knowledge has



Page 66                                         GAO/NSIAD-99-162 Best Practices
Chapter 5
Conclusions and Recommendations




been gained to ensure that a match is reached between the maturity of key
technologies and weapon system requirements.

DOD also stated that program managers already have the ability to reject
inappropriately mature technologies, and to the extent technology
immaturity affects acquisition baselines, to advise acquisition executives of
feasible alternatives. We did not find this to be the case in our review.
Rather, we found that the program managers’ ability to reject immature
technologies is hampered by (1) untradable requirements that force
acceptance of technologies despite their immaturity and (2) reliance on
tools for judging technology maturity that fail to alert the managers of the
high risks that would prompt such a rejection. As noted in the report, once
a weapon system program begins, the environment becomes inflexible and
deviations to program baselines can attract unwanted attention. This
reality limits the program managers’ ability to reject immature
technologies.




Page 67                                         GAO/NSIAD-99-162 Best Practices
Appendix I

Technology Readiness Levels and Their
Definitions                                                                                                                                      Appenx
                                                                                                                                                      Idi




Technology readiness level                            Description
1. Basic principles observed and reported.            Lowest level of technology readiness. Scientific research begins to be translated into
                                                      applied research and development. Examples might include paper studies of a
                                                      technology’s basic properties.
2. Technology concept and/or application              Invention begins. Once basic principles are observed, practical applications can be
formulated.                                           invented. The application is speculative and there is no proof or detailed analysis to
                                                      support the assumption. Examples are still limited to paper studies.
3. Analytical and experimental critical function      Active research and development is initiated. This includes analytical studies and
and/or characteristic proof of concept.               laboratory studies to physically validate analytical predictions of separate elements of
                                                      the technology. Examples include components that are not yet integrated or
                                                      representative.
4. Component and/or breadboard validation in          Basic technological components are integrated to establish that the pieces will work
laboratory environment.                               together. This is relatively “low fidelity” compared to the eventual system. Examples
                                                      include integration of “ad hoc” hardware in a laboratory.
5. Component and/or breadboard validation in          Fidelity of breadboard technology increases significantly. The basic technological
relevant environment.                                 components are integrated with reasonably realistic supporting elements so that the
                                                      technology can be tested in a simulated environment. Examples include “high fidelity”
                                                      laboratory integration of components.
6. System/subsystem model or prototype                Representative model or prototype system, which is well beyond the breadboard
demonstration in a relevant environment.              tested for TRL 5, is tested in a relevant environment. Represents a major step up in a
                                                      technology’s demonstrated readiness. Examples include testing a prototype in a high
                                                      fidelity laboratory environment or in simulated operational environment.
7. System prototype demonstration in an               Prototype near or at planned operational system. Represents a major step up from
operational environment.                              TRL 6, requiring the demonstration of an actual system prototype in an operational
                                                      environment, such as in an aircraft, vehicle or space. Examples include testing the
                                                      prototype in a test bed aircraft.
8. Actual system completed and “flight qualified”     Technology has been proven to work in its final form and under expected conditions.
through test and demonstration.                       In almost all cases, this TRL represents the end of true system development.
                                                      Examples include developmental test and evaluation of the system in its intended
                                                      weapon system to determine if it meets design specifications.
9. Actual system “flight proven” through              Actual application of the technology in its final form and under mission conditions,
successful mission operations.                        such as those encountered in operational test and evaluation. In almost all cases, this
                                                      is the end of the last “bug fixing” aspects of true system development. Examples
                                                      include using the system under operational mission conditions.




                                                   Page 68                                                   GAO/NSIAD-99-162 Best Practices
Appendix II

Comments From the Department of Defense                     AppenIx
                                                                  di




              Page 69         GAO/NSIAD-99-162 Best Practices
                          Appendix II
                          Comments From the Department of Defense




Now on pp. 7 and 63-64.




Now on pp. 7 and 63-64.




Now on pp. 7 and 63-64.




                          Page 70                                   GAO/NSIAD-99-162 Best Practices
                          Appendix II
                          Comments From the Department of Defense




Now on pp. 7 and 63-64.




                          Page 71                                   GAO/NSIAD-99-162 Best Practices
                       Appendix II
                       Comments From the Department of Defense




Now on pp. 7 and 65.




Now on pp. 7 and 65.




                       Page 72                                   GAO/NSIAD-99-162 Best Practices
Appendix III

GAO Contacts and Staff Acknowledgments                                                      AppeInIx
                                                                                                   di




GAO Contacts      Louis Rodrigues, (202) 512-4841
                  Paul Francis, (202) 512-2811



Acknowledgments   In addition to those named above, Michael Sullivan, Jeffrey Hunter,
                  Matthew Lea, Maria Santos, Rae Ann Sapp, and Katrina Taylor made key
                  contributions to this report.




                  Page 73                                     GAO/NSIAD-99-162 Best Practices
Page 74   GAO/NSIAD-99-162 Best Practices
Page 75   GAO/NSIAD-99-162 Best Practices
Related GAO Products


                   Best Practices: Commercial Quality Assurance Practices Offer
                   Improvements for DOD (GAO/NSIAD-96-162, Aug. 26,1996).

                   Major Acquisitions: Significant Changes Underway in DOD’s Earned Value
                   Management Process (GAO/NSIAD-97-108, May 5, 1997).

                   Best Practices: Successful Application to Weapon Acquisitions Requires
                   Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998).

                   Best Practices: DOD Can Help Suppliers Contribute More to Weapon
                   System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).

                   Defense Acquisition: Improved Program Outcomes Are Possible
                   (GAO/T-NSIAD-98-123, Mar. 18, 1998).

                   Defense Acquisitions: Best Commercial Practices Can Improve Program
                   Outcomes (GAO/T-NSIAD-99-116, Mar. 17, 1999).




(707336)   Leter   Page 76                                       GAO/NSIAD-99-162 Best Practices
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested