oversight

Defense Acquisition: Advanced SEAL Delivery System Program Needs Increased Oversight

Published by the Government Accountability Office on 2003-03-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Committee on Armed
             Services, U.S. Senate



March 2003
             DEFENSE
             ACQUISITIONS
             Advanced SEAL
             Delivery System
             Program Needs
             Increased Oversight




GAO-03-442
                                                March 2003


                                                DEFENSE ACQUISITIONS

                                                Advanced SEAL Delivery System
Highlights of GAO-03-442, a report to the       Program Needs Increased Oversight
Committee on Armed Services, U.S.
Senate




The Advanced SEAL Delivery                      While progress has been made in addressing technical difficulties with the
System (ASDS) is a mini-submarine               first boat, some problems must yet be solved and other capabilities
that is one of the U.S. Special                 demonstrated before the ASDS can meet all of its key performance
Operations Command’s largest                    requirements. For example, the Navy has not yet been able to develop an
investments. The program is                     adequate propulsion battery. In addition, the ASDS’ final design will remain
approaching the end of a difficult
development and must undergo key
                                                uncertain until technical problems are solved and testing is completed. The
testing before decisions are made               degree of uncertainty will be important as the U.S. Special Operations
to proceed beyond the first boat.               Command could decide that the ASDS is ready to conduct missions and
Over the past several years, the                commit to buying more boats after the operational evaluation scheduled for
Congress has raised concerns                    April 2003.
about technical difficulties,
schedule delays, cost growth, and               The ability of the ASDS to meet cost and schedule projections is
management oversight. The Senate                problematic. The program has experienced major schedule delays and cost
Armed Services Committee                        increases. The program is 6 years behind its original schedule, and, by
requested that GAO review the                   GAO’s estimates, costs have more than tripled. Cost and schedule estimates
status and problems facing the                  were being formally revised at the time of this report, but even their
program. Specifically, this report
examines the ASDS program’s (1)
                                                accuracy will be uncertain because of unresolved, known problems; the
progress towards meeting                        potential for discovering new problems in upcoming testing; and the
requirements and technical                      difficulty of estimating costs for future boats based on the first boat’s aging
challenges, (2) ability to meet                 data.
schedule and cost projections, and
(3) underlying factors contributing             Several underlying factors have contributed to the ASDS’ difficult
to program problems.                            development. In retrospect, the capabilities required of the boat outstripped
                                                the developer’s resources in terms of technical knowledge, time, and money.
                                                Key problems, such as the battery and the propeller, were discovered late—
                                                in testing on the first boat—rather than in component or subsystem level
GAO is recommending that, before                testing. Finally, the program suffered from insufficient management
the operational evaluation is held,
                                                attention on the part of both the government and the contractor, which led
DOD ensure that the overall ASDS
test and evaluation master plan and             to missed opportunities for righting the program as it proceeded. Moreover,
the specific test plan for the                  the management attention that was exercised has been hampered by
operational evaluation are both                 outdated information.
sufficient in scope and approved.
GAO is also recommending that                   ASDS Mated to USS Greeneville (SSN 772) Off the Coast of Hawaii, September 2002
DOD elevate the level of
management attention and hold a
formal milestone review before
buying additional boats. DOD
concurred with most of GAO’s
recommendations.



www.gao.gov/cgi-bin/getrpt?GAO-03-442.

To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Paul Francis at
(202) 512-4841 or francisp@gao.gov.
Contents


Letter                                                                                                  1
             Results in Brief                                                                           1
             Background                                                                                 3
             Progress Made in Meeting System Requirements and Resolving
               Technical Problems, but Difficulties Remain                                              4
             Difficulties Remain in Making Credible Schedule and Cost
               Projections                                                                              8
             Several Underlying Factors Contributed to the ASDS’s Difficult
               Development                                                                             11
             Conclusions                                                                               17
             Recommendations for Executive Action                                                      17
             Agency Comments and Our Evaluation                                                        18
             Scope and Methodology                                                                     19

Appendix I   Comments from the Department of Defense                                                   21



Tables
             Table 1: Status of the ASDS Key Performance Parameters, Boat 1                             5
             Table 2: ASDS Program Costs for Six Boats and Two Facilities                              10


Figure
             Figure 1: Estimated Delivery of the First Boat                                             9




             This is a work of the U.S. Government and is not subject to copyright protection in the
             United States. It may be reproduced and distributed in its entirety without further
             permission from GAO. It may contain copyrighted graphics, images or other materials.
             Permission from the copyright holder may be necessary should you wish to reproduce
             copyrighted materials separately from GAO’s product.




             Page i                                                  GAO-03-442 Defense Acquisitions
United States General Accounting Office
Washington, DC 20548




                                   March 31, 2003

                                   The Honorable John Warner
                                   Chairman
                                   The Honorable Carl Levin
                                   Ranking Minority Member
                                   Committee on Armed Services
                                   United States Senate

                                   Key decisions lie ahead for the Advanced SEAL Delivery System (ASDS), a
                                   mini- submarine that is one of the U.S. Special Operations Command’s
                                   largest investments. The Department of Defense (DOD) is increasingly
                                   relying on special operations forces to accomplish its missions, especially
                                   in the current national security environment of the fight against terrorism.
                                   The ASDS, with a potential cost on the order of $2 billion, is a major
                                   development effort to enhance the capabilities of the U.S. Special
                                   Operations Command’s naval special forces. The mini-submarine is
                                   designed for clandestine delivery and extraction of Navy SEAL teams and
                                   equipment performing operational missions in high-threat environments.
                                   The program, which is being managed by the Navy, is approaching the end
                                   of a difficult development and the first boat must undergo key testing
                                   before a decision is made to proceed with additional boats.

                                   During the past several years, the Congress has raised concerns about the
                                   technical difficulties, schedule delays, cost growth, and management
                                   oversight of the ASDS program. Senate Report 107-62, which accompanied
                                   the National Defense Authorization Act for Fiscal Year 2002, called for us
                                   to review the status and problems facing the program. We briefed the
                                   defense committees in April and May 2002 on the results of our review. At
                                   that time, you asked us to continue reviewing the program’s progress.
                                   Accordingly, this report examines the ASDS program’s (1) progress
                                   towards meeting requirements and technical challenges, (2) ability to meet
                                   schedule and cost projections, and (3) underlying factors that have
                                   contributed to program problems.


                                   While progress has been made in addressing technical difficulties with the
Results in Brief                   first boat, some problems must yet be solved and other capabilities
                                   demonstrated before the ASDS can meet all of its key performance
                                   requirements. For example, the Navy has not yet been able to develop an
                                   adequate propulsion battery and the first boat is not quiet enough to meet


                                   Page 1                                         GAO-03-442 Defense Acquisitions
acoustic stealth requirements. In addition to solving known technical
problems, the potential for discovering new ones is significant because the
first boat is slated to begin operational evaluation in April 2003. The
ASDS’s final design will remain uncertain until technical problems are
solved and testing is completed. The degree of uncertainty will be
important because the U.S. Special Operations Command could, following
the operational evaluation, decide that the ASDS is ready to conduct
missions and commit to buying more boats.

The ability of the ASDS to meet schedule and cost projections is
problematic. The program has experienced major schedule and cost
increases since it was started, and these increases have continued since
our April 2002 briefing. The program is 6 years behind its original
schedule, and, by our estimates, costs have more than tripled. Current
schedule and cost estimates have not been updated since 1999. These
estimates were being formally revised at the time of this report, but they
will be subject to change because of unresolved, known problems; the
potential for discovering new problems in upcoming testing; and the
difficulty of estimating costs for future boats based on the first boat’s
aging data.

Several underlying factors have contributed to the ASDS’s difficult
development. In retrospect, the capabilities required of the boat
outstripped the developer’s resources in terms of technical knowledge,
time, and money. Key technical problems, such as the battery and the
propeller, were discovered late—during testing on the first boat—rather
than in component or subsystem level testing. Finally, the program
suffered from insufficient management attention on the part of both the
government and the contractor, which led to missed opportunities for
righting the program as it proceeded. Moreover, the management attention
that was given has been hampered by outdated information.

We are making recommendations aimed at improving the quality of
information available and enhancing DOD’s oversight of the ASDS
program. In its comments on a draft of this report, DOD concurred with
two of our recommendations and partially concurred with our
recommendation to elevate the level of management attention.
Specifically, DOD concurred with all but a part of one of the particulars of
this recommendation concerning the development of an independent cost
estimate.




Page 2                                         GAO-03-442 Defense Acquisitions
             The ASDS is a battery-powered mini-submarine about 65 feet long and 8
Background   feet in diameter with a dry interior. It is operated by a two-person crew
             and can carry equipment and SEAL personnel. The vehicle has a
             hyperbaric recompression chamber with a lower hatch that can be opened
             and closed underwater to allow divers to exit and reenter the vehicle
             (referred to as lock in/lock out) at various depths.1 The ASDS is expected
             to have increased range, speed, and capacity over the current underwater
             SEAL delivery vehicle, which is an open, wet submersible that transports
             SEALs wearing scuba gear and thus exposes them to ocean water
             temperatures. The ASDS’s main advantage over existing SEAL delivery
             systems is its ability to transport forces in a dry environment, which
             reduces the SEALs’ exposure to cold-water as well as their physical and
             mental fatigue. Use of the ASDS is not limited to delivery of Navy SEALs. It
             can be used for intelligence collection, surveillance and reconnaissance,
             combat search and rescue, sabotage and diversionary attacks, forward
             observation for fire direction, underwater ship attack, and offensive mine
             operations.

             Several organizations are involved with the ASDS program. The U.S.
             Special Operations Command is funding the program, and its Naval
             Special Warfare Command set the requirements and will be the user of the
             system. The Assistant Secretary of the Navy for Research, Development,
             and Acquisition is responsible for approving each phase of the ASDS
             acquisition process. The Naval Sea Systems Command is the acquisition
             program manager and is responsible for overseeing the prime contractor,
             Northrop Grumman.

             The program’s last official baseline—from 1999—calls for building six
             boats and two facilities. The plans also include the ability to transport the
             ASDS boat using a variety of methods, including undersea (“piggy-back”
             on a 688-class attack submarine), by air (aboard C-5 aircraft), and by road
             (on a large flat-bed trailer). The Naval Sea Systems Command awarded a
             contract to Northrop Grumman for detailed design and manufacturing
             development in September 1994.2 In August 2001, the Navy program office
             took what it calls “conditional” preliminary acceptance of the first boat
             from Northrop Grumman under an agreement that all contractual


             1
              A chamber used to treat divers suffering from decompression sickness, which can be
             caused by descending below sea level.
             2
              The contract was originally awarded to Westinghouse Electric Corporation, which was
             subsequently bought by Northrop Grumman in 1996.




             Page 3                                                GAO-03-442 Defense Acquisitions
                           requirements needed for final government acceptance would be
                           completed within 1 year. However, Navy officials told us that the
                           contractor has not satisfactorily completed the contract requirements
                           within this period, and the first boat is still not ready for final government
                           acceptance.

                           The first ASDS boat is scheduled to undergo an operational evaluation in
                           April 2003. An operational evaluation is a field test conducted by the war
                           fighter under realistic conditions for the purpose of determining the
                           effectiveness and suitability of the boat for use in combat. The operational
                           evaluation will be a major factor in deciding whether or not to declare an
                           initial operating capability—the point at which the first boat can be used
                           to conduct missions.


                           During the past year, the ASDS program has made considerable progress
Progress Made in           in addressing technical difficulties. However, the first boat has not yet
Meeting System             demonstrated that it can meet all key requirements. Several technical
                           challenges still need to be addressed, and further technical and
Requirements and           operational testing is required before all key performance requirements
Resolving Technical        can be demonstrated and the first boat can be considered fully
                           operational.
Problems, but
Difficulties Remain

The ASDS Has Not Met All   The first boat produced has not demonstrated the ability to meet all of the
Requirements               program’s key performance parameters. Key performance parameters
                           represent those critical performance parameters so significant that a
                           failure to meet a minimum value of performance can call into question a
                           system’s ability to perform missions. Each key performance parameter is
                           made up of individual subordinate requirements that must be met to
                           demonstrate the parameter.3 At the time of our review, Naval Sea Systems
                           Command officials judged that 11 of the first boat’s 16 key performance
                           parameters had been met, 4 were still in process, and 1 required action.
                           However, the status of the subordinate requirements, as shown in table 1,
                           makes the assessment of the key performance parameters less clear.




                           3
                            These subordinate individual requirements are specified in the contract and the user’s
                           Operational Requirements Document.




                           Page 4                                                  GAO-03-442 Defense Acquisitions
Table 1: Status of the ASDS Key Performance Parameters, Boat 1

                                                                                     Status
                                                                         Demonstrated,
                                              Demonstrated, all        some subordinate
 Key performance     Key performance                subordinate           requirements
 parameter number    parameter                requirements met                  not met                  In process        Action required
 1                   Maximum combat                                                   3
                     range
 2                   Maximum cruise                                                       3
                     speed
 3                   Transport depth                                                      3
                     (submarine host)
 4                   Transport speed                                                      3
                     (submarine host)
 5                   Concurrent lock                                                      3
                     in/lock out
 6                   Operating                                                                                     3
                     temperature
 7                   Storage temperature                         3
 8                   Survivability                                                                                 3
 9                   Vibration                                                            3
 10                  Crew                                        3
 11                  Passengers                                  3
 12                  Endurance                                                            3
 13                  External payload                                                                              3
 14                  Transportability                                                     3
 15                  Vehicle signatures                                                                                                     3
 16                  Interoperability                                                                              3
 Total                                                            3                        8                       4                        1
Source: U.S. Navy.

                                           Note: Naval Sea Systems Command, ASDS Top Level Requirement Verification Matrix, January 2003.




                                           As indicated in table 1, all subordinate requirements for the first boat have
                                           not been met for eight key performance parameters that were judged as
                                           demonstrated. For example, the third parameter—transport depth
                                           (attached to the submarine host)—has three subordinate requirements.
                                           One has been demonstrated, but two are still in process. Similarly, the fifth
                                           parameter—concurrent lock in/lock out—has 12 subordinate
                                           requirements. Four have been demonstrated, but five are still in process
                                           and three require action.

                                           Some requirements have also been delayed, reduced, and eliminated by
                                           the U.S. Special Operations Command. For example, the acoustic, or noise
                                           level, requirement, which is part of the vehicle signatures key performance



                                           Page 5                                                       GAO-03-442 Defense Acquisitions
                       parameter, has been deferred until the second boat. The transportability
                       parameter—although now considered demonstrated by the Naval Sea
                       Systems Command—was also reduced. It no longer includes transport by
                       C-17 aircraft, amphibious ships, and the SSN-21 submarine. Also, a
                       degaussing system needed to lower the vehicle’s magnetic signature has
                       been delayed and designated as a preplanned product improvement.4
                       Although a degaussing system was originally included in the ASDS design,
                       the program used the funds for this system to cover other program
                       expenses. If the vehicle has a large magnetic signature, it will have
                       increased vulnerability to mines. Nonetheless, even with these reduced
                       requirements, Navy and U.S. Special Operations Command officials
                       believe that the ASDS is still a cost-effective capability that provides an
                       improvement over existing vehicles.


Problems in Critical   In the past year, the ASDS program has made progress in resolving
Components Remain      technical issues. Achievements include a successful new anchor design,
Unsolved               improved battery design, sonar systems upgrades, improvements in
                       configuration management control, renewed focus on ASDS logistics
                       needs, completion of safety-critical software testing, and substantial
                       progress in developing engineering drawings. For example, the original
                       anchors have been redesigned and tested successfully to hold the ASDS
                       level enough to provide a stable dive platform during ocean swells. The
                       program has also succeeded in decreasing the operating temperatures of
                       the silver-zinc batteries, which has reduced the frequency of electrical
                       shorts and improved the batteries’ performance.

                       Nonetheless, there are still unresolved issues that prevent the vehicle from
                       meeting its operational requirements. Battery reliability and acoustics are
                       currently the most critical issues facing the program. The silver-zinc
                       propulsion battery has limited the performance of the ASDS system. The
                       first attempts to use silver-zinc batteries in the ASDS resulted in
                       unexpected shorting and premature failure. One of the key reasons for the
                       battery shorting was because of the high-temperature environment in
                       which the battery operates. Through ongoing assessment and
                       modifications, the Navy has been able to extend the endurance of a fully
                       charged battery. Program officials have not determined whether the


                       4
                        Degaussing cables and other elements of a degaussing system were installed and
                       electrical hull penetrators were added during construction of the first ASDS vehicle to
                       shorten the time needed to create a functional degaussing system in the future. A power
                       supply is the major part of the degaussing system that needs to be created.




                       Page 6                                                  GAO-03-442 Defense Acquisitions
                    battery’s endurance can be extended to support all missions. In addition,
                    the battery’s demonstrated life—the number of times it can be recharged
                    before requiring replacement—is much shorter than expected. Currently,
                    the battery can only be recharged two to three times before failing,
                    whereas 20 recharge cycles were expected. If the battery cannot last
                    through the expected recharge cycles, the impact on the submarine’s
                    availability and operation and support costs will be significant because
                    replacing the battery requires the boat to return to its base facility, be
                    taken out of the water, and partially dismantled.

                    Although the Navy continues to mature the silver-zinc battery for the first
                    boat, it is developing a lithium-ion battery as a replacement on the first
                    boat and any additional boats. Program officials expect the lithium-ion
                    battery to be developed by the summer of 2004. Lithium-ion battery
                    technology, like silver-zinc, is not new; however, the challenge lies in
                    adapting the technology to ASDS’s size and environment.

                    To meet the acoustics portion of the vehicle signatures key performance
                    parameter, the boat must be quiet enough to (1) evade detection while
                    performing its mission of inserting SEALs into hostile territory and (2) not
                    give away the location of the host submarine. However, the first boat
                    makes too much noise and does not meet this acoustic requirement. The
                    most significant noise offender at this point is the propeller. The program
                    manager assembled a team of government and private experts to redesign
                    the propeller by March 2003 in preparation for the operational evaluation
                    in April 2003. However, more propeller work may be needed, and other
                    acoustic problems may have to be addressed in order to meet the
                    requirement. Therefore, the U.S. Special Operations Command deferred
                    the acoustic requirement until delivery of the second boat—in several
                    years—and will accept the noise level that the first boat achieves.


Design and Test     The final design of the boat is still evolving, pending the resolution of
Challenges Remain   existing problems and remaining testing—notably the operational
                    evaluation. The program has made some progress in finalizing the
                    drawings of the boat by catching up on engineering drawing updates. As of
                    January 2003, the program had completed about 76 percent of
                    approximately 12,000 revisions to the engineering drawings. Program
                    officials expect to eliminate the remaining backlog of 2,846 revisions by
                    August 2003.

                    Unresolved technical problems could have implications for the design of
                    the ASDS and require further revisions to the boat’s engineering drawings.


                    Page 7                                         GAO-03-442 Defense Acquisitions
                         The ASDS boat includes an outer shell, or exostructure, and an inner, or
                         “pressure,” hull. Many of the boat’s critical systems, such as the battery
                         system, sonars, and anchors, are located between the outer shell and the
                         inner hull. Future changes to any of these systems may require
                         modifications to the design of the outer shell or inner hull. For example,
                         the battery system is mounted to the inner hull. Replacing the silver-zinc
                         battery system with a lithium-ion battery system will likely require
                         modifications to the boat’s design.

                         Another factor that may affect the design of the ASDS is the statutory
                         requirement to conduct realistic survivability testing. A key element of
                         survivability is live-fire testing, which evaluates how vulnerable the boat’s
                         design is to the shock of being under fire and assesses crew safety. These
                         tests will need to be conducted both on the first boat alone and while the
                         boat is attached to the host submarine. However, the program office has
                         requested that the Director, Operational Test and Evaluation, waive full-
                         up, system-level live-fire testing of the first boat. The law allows an
                         alternative approach to full-up, system-level live-fire testing if it is
                         impractical or overly costly, which is typically the case on submarines
                         because live-fire testing would significantly damage or possibly destroy
                         the boat. To date, an alternative approach has not been approved. Until the
                         ASDS is tested in some manner to satisfy the live-fire requirement, the
                         possibility of discovering the need for design modifications and upgrades
                         will continue.


                         Consistent and substantial schedule delays and cost increases have
Difficulties Remain in   characterized the ASDS program since its beginning. The program
Making Credible          originally projected that the first boat could be delivered in fewer than 3
                         years; 9 years later, the first boat is still not fully operational. A variety of
Schedule and Cost        technical challenges and problems have contributed to these delays. In
Projections              addition, according to the initial approved program baseline, adjusted to
                         reflect six boats and two facilities, the program would cost $527 million in
                         fiscal year 2003 dollars. Currently, we project that the program will cost
                         over $2 billion. Continuing technical problems, obsolete estimates, and
                         upcoming tests and demonstrations make it difficult to assess conclusively
                         whether or not the ASDS program is stable or will incur additional delays
                         and cost increases.




                         Page 8                                            GAO-03-442 Defense Acquisitions
Schedule Delays and Cost                  Delivery of the first boat is now 6 years behind schedule, as shown in
Increases                                 figure 1.

Figure 1: Estimated Delivery of the First Boat




                                          The original program schedule called for delivery of the first boat in July
                                          1997. The 1999 schedule called for delivery of the first boat in February
                                          2000, almost 3 years late. This last program schedule has not been revised
                                          since June 1999.5 Although the first boat was conditionally accepted in
                                          August 2001, Navy officials now expect delivery of a fully acceptable boat
                                          from the contractor in June 2003, pending the completion of operational
                                          evaluation.




                                          5
                                           Navy officials informed us that a revised program schedule was recently drafted and is
                                          being reviewed by DOD officials.




                                          Page 9                                                  GAO-03-442 Defense Acquisitions
As shown in table 2, the ASDS’s cost increases essentially parallel the
schedule delays.

Table 2: ASDS Program Costs for Six Boats and Two Facilities

    Fiscal year 2003 dollars in millions
                                                                          Last
                                            First                  acquisition      GAO           GAO
                          Development rebaseline                      program projectionc   projectiond
    Budget                    baseline    (Sept.                    rebaseline     (April     (January
                                      b
    category               (Oct. 1994)     1998)                  (June 1999)      2002)         2003)
    Research,
    development,
    test and
    evaluation                      $131.4               $244.2       $310.8       $437.6        $467.7
    Procurement                     $362.7               $452.2       $675.0     $1,258.6      $1,823.7
    Military
    construction                     $33.0                $37.4         $36.8       $51.7         $51.7
          a
    Total                           $527.1               $733.7      $1,022.6    $1,747.8      $2,343.0
Source: U.S. Navy and U.S. Special Operations Command.

Note: GAO analysis of Navy and U.S. Special Operations Command data.
a
    Totals may not add due to rounding.
b
 The development baseline was for 3 boats and 1 facility. To put this estimate on the same footing as
the 1998 and 1999 estimates, GAO projected what the development baseline would equal for 6
ASDS and 2 facilities.
c
 GAO projected the acquisition costs based on the U.S. Special Operations Command’s actual
funding for ASDS through fiscal year 2001 and projections through fiscal year 2007. Our projections
are straight-line estimates and do not include any learning curve, economies of scale, or nonrecurring
cost effects.
d
 GAO projected the acquisition costs based on the U.S. Special Operations Command’s actual
funding for ASDS through fiscal year 2002 and projections through fiscal year 2009. Our projections
are straight-line estimates and do not include any learning curve, economies of scale, or nonrecurring
cost effects.


By the last approved acquisition program baseline in June 1999, total costs
had almost doubled, and research, development, test and evaluation costs
had more than doubled. The June 1999 program baseline has not been
updated since, and updated cost estimates are not available.6 However, at
the time of our briefing in April 2002, we projected that both of these costs
had more than tripled.




6
    Navy officials informed us that the acquisition program baseline was being revised.




Page 10                                                                 GAO-03-442 Defense Acquisitions
Several Factors Make           Uncertainties about the schedule and cost of the ASDS program remain,
Future Cost and Schedule       making it difficult to predict future performance. The program faces
Performance Uncertain          additional cost and schedule risks as outlined below:

                           •   The program has experienced continual delays in preparing and getting
                               estimates approved. Specifically, none of the following has been finalized
                               or approved: the acquisition program baseline, the test and evaluation
                               master plan, the test plan for the operational evaluation, or the cost
                               estimate.
                           •   The solutions to several known technical problems need to be
                               demonstrated through testing, and the results of this testing may reveal the
                               need for additional changes.
                           •   The operational evaluation could expose new problems, which may
                               require redesign and other solutions. Further, if the operational evaluation
                               is not rigorous enough, or the scope is too narrow, the program risks
                               missing problems. For instance, any requirements deferred to the second
                               boat will not be included in the operational evaluation. Some scope
                               reduction has already occurred with the deferral of the propeller and the
                               degaussing system. Moreover, we could not determine the rigor and scope
                               of the operational evaluation due to the lack of an approved test plan.
                           •   The wide, but now necessary, gap between the construction of the first
                               two boats makes cost projections for the second and future boats more
                               difficult due to loss of production base as well as obsolescence of certain
                               technologies and systems that have been incorporated into the first boat.


                               Developing the ASDS was clearly a difficult undertaking—a challenge
Several Underlying             under any circumstances. However, several factors either made the
Factors Contributed            development effort more difficult than necessary or limited opportunities
                               for responding to problems early. These include discovery of problems in
to the ASDS’s Difficult        system-level testing, a mismatch between requirements and resources at
Development                    program start, and insufficient management attention. In retrospect, some
                               of the ASDS’s difficulties could have been foreseen and their effect
                               lessened.


Discovery of Key Problems      While technical challenges, such as the battery and propeller, have caused
in System-Level Testing        schedule delays and cost increases, the effect of technical challenges and
                               problems has been magnified because critical problems were not
                               discovered until tests of the full ASDS system. Ideally, system-level testing
                               occurs after components have successfully completed laboratory and
                               subsystem testing. In a 2000 report on test and evaluation, we found that a
                               best practice was to expose problems early in component and subsystem



                               Page 11                                         GAO-03-442 Defense Acquisitions
                           level tests so that they could be corrected for less cost.7 Conversely, we
                           found that when tests of a full system became the vehicle for discovering
                           problems that could have been found out earlier, additional—and
                           unanticipated—time, money, and effort had to be expended to overcome
                           the problems. One firm referred to this phenomenon as “late cycle churn.”

                           It appears that the ASDS program is experiencing late cycle churn with the
                           battery and acoustics problems. Early silver-zinc battery tests were
                           performed under very limited, unrealistic environmental conditions.
                           Consequently, the problems with the battery were not discovered until
                           shortly after the first set of batteries was installed on the boat in
                           December 2000. Similarly, acoustic tests of the propeller were not
                           performed until February 2002—again, on the first boat. Had the battery
                           and propeller acoustics problems been discovered earlier in more realistic
                           component or subsystem level testing, their effect on schedule and costs
                           might have been minimized.


Mismatched Requirements    It is now clear that when the ASDS program began, the capabilities
and Resources at Program   required of the boat outstripped the developer’s resources in terms of
Start                      technical knowledge, time, and money. Our work on best practices has
                           shown that when such a mismatch occurs at the outset of product
                           development, a program is put in a poor position to succeed.8 Cost
                           increases, schedule delays, and performance shortfalls are the typical
                           consequences of such a mismatch.

                           Before product development begins, successful programs achieve a match
                           between the product performance desired by the customer and the ability
                           of the developer to marshal the resources necessary to develop such a
                           product. It is essential that both parties understand the demands that the
                           customer is making of the product and the challenges these pose for the
                           developer before the parties commit to product development. Importantly,
                           achieving this match is a managed outcome—on successful programs, it is
                           done deliberately, using metrics for assessing technology and design risks.



                           7
                            U.S. General Accounting Office, Best Practices: A More Constructive Test Approach Is
                           Key to Better Weapon System Outcomes, GAO/NSIAD-00-199 (Washington, D.C.: July 31,
                           2000).
                           8
                            U.S. General Accounting Office, Best Practices: Better Matching of Needs and Resources
                           Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8,
                           2001).




                           Page 12                                               GAO-03-442 Defense Acquisitions
                           In particular, the maturity of technology is an important weathervane for
                           achieving a match between requirements and resources. On successful
                           product development programs, developers will not allow immature
                           technologies—those that require further development—to be included in
                           the product design. Immature technologies make it very difficult to
                           estimate a product development’s schedule and cost accurately.

                           If the developer does not have the requisite technologies, engineering and
                           design knowledge, and sufficient time and money to deliver the desired
                           product when it is needed, tradeoffs must be made. These include
                           (1) lowering product requirements to match the developer’s resources or
                           (2) deferring the program until the developer can make the additional
                           investments to meet the customer’s requirements. When a match between
                           requirements and resources is not achieved at the outset of product
                           development, the ensuing program is much more susceptible to
                           performance shortfalls, cost increases, and schedule delays. The
                           competition for funds often makes the situation worse by enticing
                           managers to be optimistic about the time and money needed to complete
                           development.

                           The ASDS’s experience, as detailed in the preceding sections, has followed
                           this path. Product requirements have been lowered, dropped, or deferred
                           in an effort to match what the developer could deliver—tradeoffs that
                           could perhaps have been made before product development began. Cost
                           increases and schedule delays evidence the struggle of the developer to
                           mature key technologies, such as the battery, and solve design problems,
                           such as the propeller, while producing the first boat. The experience of the
                           ASDS underscores the need for nascent and future weapon system
                           programs to manage customer needs and developer resources so that a
                           match is achieved before product development is approved.


Management Attention Has   Weaknesses in the ASDS’s management compounded the problems
Been Insufficient          resulting from the mismatch between user requirements and the
                           developer’s resources. Management of the program on the part of both the
                           government and the contractor has been insufficient; consequently, early
                           opportunities to act on problems were missed. Moreover, direction by the
                           Congress to elevate the oversight of the program to include a higher level
                           of DOD review has not been followed fully, although the program has
                           received heightened managerial attention. Recent steps taken to improve
                           management of the program will help, but they have come very late in the
                           product development process.



                           Page 13                                        GAO-03-442 Defense Acquisitions
    In August 1994, before the ASDS began product development, the DOD
    Inspector General reported serious problems with the program, including
    noncompliance with mandatory DOD acquisition guidance, and
    recommended increased senior-level DOD oversight and better
    coordination with the Joint Staff, the services, and defense agencies.
    However, the acquisition executive at the time disagreed, based on input
    from other sources, including the Naval Sea Systems Command’s
    assessment that the program was technically sound and executable.
    Consequently, the Navy began product development with the award of the
    engineering and manufacturing development contract to Northrop
    Grumman as planned on September 29, 1994.9

    In 1997 and 1999, two Navy independent review teams identified
    continuing problems with the ASDS program, including cost growth,
    schedule delays, and—perhaps most importantly—a lapse in effective
    program management by both the government and the contractor.
    Collectively, these problems necessitated developing a new baseline. Navy
    reviews identified several causes for the lapse in effective program
    management. These included:

•   a lack of contractor experience in submarine design and construction;
•   the government’s lack of influence or visibility into problems between the
    contractor and the subcontractors;
•   a focus on technical rather than management aspects of the program by
    both the program office and the contractor;
•   ineffective oversight by the program office and little attention to the
    financial performance of the contractor; and
•   frequent changes in the contractor’s project management team.

    As a result, the Navy created a management integrated product team
    comprised of the Naval Sea Systems Command’s Program Manager for the
    Deep Submergence Program Office; a Northrop Grumman Senior Vice
    President; the U.S. Special Operations Command’s Program Executive
    Officer, Maritime and Rotary Wing; and the Naval Special Warfare
    Command’s Assistant Chief of Staff for Resources, Requirements, and
    Assessments to help deal with ASDS program problems.




    9
     The contract was originally awarded to Westinghouse Electric Corporation, which was
    subsequently bought by Northrop Grumman in 1996.




    Page 14                                               GAO-03-442 Defense Acquisitions
In August 1999,10 the Congress expressed its continuing concern over cost
growth, development and testing activities, and level of oversight. It
established the ASDS as an item of special interest that it would monitor
closely. It also requested that—although ASDS may not meet the normal
dollar threshold for automatic elevation to a major defense acquisition
program (acquisition category I)—the program be elevated to an
equivalent level of DOD review because of the “troubled history” and
“concern that this program may not be out of difficulty yet.”11 Programs
designated as acquisition category I programs must meet certain statutory
and DOD requirements applicable to such programs, including regular
reporting to Congress; establishment of a firm baseline for measuring the
program; a mechanism for addressing cost and schedule variances;
establishment of cost, schedule, and performance goals; development of
an independent life-cycle cost estimate by the Secretary of Defense’s Cost
Analysis Improvement Group; and an independent operational test and
evaluation. Further, elevation of the program to this higher acquisition
category would result in a more disciplined program management
approach under DOD’s acquisition system guidance, including following a
prescribed process for making major decisions, providing documentation
such as test results for those decisions, and holding formal reviews before
making those decisions. This process, while intended to facilitate the
management of major programs, also provides the mechanisms and
opportunities for exercising oversight. In early 2001, the U.S. Special
Operations Command informed the Navy that ASDS cost projections were
approaching the acquisition category I program threshold and proposed
elevating the status of the program.

In both instances, DOD declined to designate the ASDS as a major defense
acquisition program. Rather, to increase management attention, DOD
established new top-level overarching integrated product team reviews
and placed the program on the oversight list of the Director, Operational
Test and Evaluation. In fact, however, while the top-level overarching
integrated product team was slated to meet twice each year to review the


10
 House of Representatives Conference Report 106-301, pages 585-586, accompanying the
National Defense Authorization Act for Fiscal Year 2000.
11
  The conferees presumably intended for DOD to designate the program as a major
defense acquisition program, known as acquisition category I. A program is considered to
be a major defense acquisition program if its total research and development expenditures
are estimated to be at least $365 million (in constant fiscal year 2000 dollars). In addition to
the monetary threshold, programs can also be designated as category I under the discretion
of the Secretary of Defense (for example, because of congressional interest).




Page 15                                                    GAO-03-442 Defense Acquisitions
ASDS’s progress, it has only met two times, and has not met since May
2001. The lower-level integrating team has met more often. In any event,
the program office has not developed a new program baseline, including
an updated cost estimate, since 1999, which denied the teams current
information even if they had made a more rigorous attempt to provide
oversight.

Despite these and the earlier attempts at improving management attention,
lapses in effective program management have continued. Most recently,
Navy officials informed us that they have had to require the contractor to
redo all of the required safety-critical software testing because the
contractor did not provide documentation that this testing had been
performed. This rework has contributed to recent schedule delays and
cost increases. Program officials also informed us that in early 2002 they
had to hire another contractor to investigate and develop solutions for the
battery problems. This also has contributed to recent schedule delays and
cost increases.

In November 2002, Congress directed the Secretary of Defense to conduct
a complete review of the requirements, mission, management, and cost
structure of the ASDS program and report to the congressional defense
committees before obligating more than 50 percent of fiscal year 2003
ASDS procurement funding.12 This review is in progress, and results are
expected by late March 2003. Congress again intervened during the fiscal
year 2003 congressional budget review. As a result of the review, DOD
agreed that the first boat should be fully operational and meet the user’s
requirements before it commenced with the procurement of additional
boats. Under the condition that the U.S. Special Operations Command
would agree to resolve the technical issues with the first boat before
declaring initial operational capability, Congress approved additional
funding for the program.

At the program level, several management improvements have been made
recently. While they will not necessarily address oversight, they should
facilitate solving technical problems and improve the quality of program
information. The program office and the U.S. Special Operations
Command augmented their staffs and expertise in 2002 to meet the needs
of the ASDS program. In addition, the ASDS program manager has enlisted



12
 House of Representatives Conference Report 107-772, page 436, accompanying the
National Defense Authorization Act for Fiscal Year 2003.




Page 16                                             GAO-03-442 Defense Acquisitions
                      outside experts from Battelle and Penn State University to investigate
                      technical problems with the battery, sonars, and the propeller. The
                      program office is also revising the Acquisition Strategy, the Approved
                      Program Baseline, and the Test and Evaluation Master Plan. According to
                      program officials, the new baseline will allow for more testing and
                      information to be gathered before commitments are made to purchase
                      additional boats.


                      After a troubled history, the ASDS program has made tangible progress,
Conclusions           particularly in resolving technical problems. Nevertheless, as the program
                      prepares to begin the April 2003 operational evaluation of the first boat,
                      the ASDS still has not met all key performance requirements and must still
                      solve significant technical problems. The challenge in solving known
                      problems, coupled with the possibility of discovering new ones in
                      upcoming tests, pose risks for achieving initial operational capability as
                      planned and for having sound cost and schedule estimates. While every
                      reasonable effort should be made to overcome the first boat’s shortfalls
                      and have a successful operational evaluation, decisions on investing in
                      additional boats must be based on both sound information and a sound
                      process for decision making.

                      The information decision makers will need includes demonstrable
                      knowledge that (1) key design problems have been resolved, (2) the
                      resulting mission performance of the ASDS is worthwhile, and (3) credible
                      cost and schedule estimates for building follow-on boats, facilities, and
                      operations and support are developed. It is important that the activities
                      that will provide this information, such as improvements to the battery and
                      propeller and the operational evaluation of the first boat, have sufficient
                      scope and take place before key decisions are made. Good information,
                      when it becomes available, must be used effectively. Thus, it is equally
                      important that a formal process be followed for evaluating this
                      information and making decisions. In particular, DOD decision makers
                      should have the benefit of a formal, informed, transparent decision
                      meeting before proceeding with purchases of additional boats.


                      Before the operational evaluation is held, we recommend that the
Recommendations for   Secretary of Defense ensure that the overall ASDS test and evaluation
Executive Action      master plan and the specific test plan for the operational evaluation are
                      both sufficient in scope and approved.




                      Page 17                                        GAO-03-442 Defense Acquisitions
                         Before a decision to purchase additional boats is made, we recommend
                         that the Secretary of Defense ensure that:

                     •   the ASDS operational evaluation is completed as planned;
                     •   solutions to key technical and performance problems are demonstrated;
                     •   the most likely performance of the ASDS is reassessed on the basis of the
                         operational evaluation and demonstrated solutions to problems;
                     •   the ASDS program is designated a major defense acquisition program
                         (acquisition category I);
                     •   a formal milestone C decision, in accordance with DOD acquisition
                         guidance, is held;
                     •   the Cost Analysis Improvement Group develops an independent cost
                         estimate for milestone C, based on the acquisition plan and planned
                         product improvements;
                     •   the program is funded to the level of the independent cost estimate; and
                     •   the worthiness of proceeding with additional purchases is assessed against
                         both (1) the ability of the ASDS to perform missions and be sustained and
                         (2) the opportunity costs of investing in the ASDS versus other special
                         operations needs.

                         If a decision to proceed with the purchase of additional boats is
                         warranted, we recommend that the Secretary of Defense ensure that a
                         follow-on test and evaluation of the second boat is planned and funded to
                         demonstrate that remaining deficiencies have been corrected.


                         DOD provided us with written comments on a draft of this report. The
Agency Comments          comments, along with our responses, appear in appendix I.
and Our Evaluation
                         DOD concurred with our recommendation that, before an operational
                         evaluation is held, DOD should ensure that the overall ASDS test and
                         evaluation master plan and the specific test plan for the operational
                         evaluation are both sufficient in scope and approved. DOD also concurred
                         with our recommendation that, if a decision is made to purchase
                         additional boats, a follow-on test and evaluation of the second boat is
                         planned and funded to demonstrate that remaining deficiencies have been
                         corrected. DOD partially concurred with our recommendation to elevate
                         the level of management attention. Specifically, DOD concurred with all
                         but a part of one of the particulars of this recommendation; that is, DOD
                         has not yet determined the level of Cost Analysis Improvement Group
                         involvement necessary for developing an independent cost estimate for
                         milestone C.




                         Page 18                                       GAO-03-442 Defense Acquisitions
              DOD provided several comments that offered a more optimistic view of
              ASDS’s status than we reported. Specifically, DOD commented that
              (1) while management attention had been lacking in the early part of the
              program, it has improved in recent years; (2) program cost and schedule
              performance have stabilized; and (3) 14 of 16 key performance parameters
              have been achieved.

              We agree that management attention on the program has improved and
              noted this in the draft report. However, the difficulties the program has
              continued to experience in recent years, including the unavailability of
              current cost and schedule estimates, warrant increased attention. We do
              not share DOD’s view that cost and schedule performance have stabilized.
              New estimates appear imminent for the first time since 1999, but their
              release alone will not provide stability—this will come from demonstrating
              that key requirements have been met and problems have been overcome.
              Regarding the achievement of 14 requirements, it is possible that DOD has
              completed more test reports since our draft report, but it has not provided
              such evidence.

              DOD also provided technical comments, which we have incorporated as
              appropriate.


              During our review, we met with officials from the U.S. Special Operations
Scope and     Command; the Naval Sea Systems Command; the Naval Special Warfare
Methodology   Command, Navy SEAL Delivery Vehicle Team One; Submarine Force, U.S.
              Pacific Fleet; the Assistant Secretary of the Navy’s Office of Research,
              Development, and Acquisition; the Office of the Under Secretary of
              Defense, Acquisition, Technology & Logistics, Naval Warfare; and the
              Office of the Director, Operational Test and Evaluation.

              To determine the progress toward meeting requirements and technical
              challenges, we examined the Operational Requirements Document, the
              Acquisition Program Baseline, the ASDS Acquisition Strategy, program
              status documents, test results, and technical reports. We also discussed
              requirements and mission needs with the former Commander, Naval
              Special Warfare Command, and other key Navy and U.S. Special
              Operations Command officials.

              To determine the ASDS program’s ability to meet schedule and cost
              projections, we examined the U.S. Special Operations Command’s budget
              requests, ASDS funding profiles, and other ASDS cost data. We compared
              the amounts that DOD requested in its budget submissions with amounts


              Page 19                                       GAO-03-442 Defense Acquisitions
approved by Congress. We reviewed documents from two Independent
Review Team assessments, internal Naval Sea Systems Command Reports,
legislative actions, contract documents, ASDS program status briefs, and
presentations and responses to congressional staff.

To determine the underlying factors contributing to program problems, we
reviewed numerous historical documents, including a 1994 DOD Inspector
General report, and the 1997 and 1999 Independent Review Team
assessments. We also drew upon our previous work on best practices for
developing products.

We conducted our review from May 2002 to January 2003 in accordance
with generally accepted government auditing standards.


We are sending copies of this report to the Secretary of Defense; the
Secretary of the Navy; the Commander, U.S. Special Operations Command;
the Director of the Office of Management and Budget; and interested
congressional committees. We will also make copies available to others
upon request. In addition, the report will be available at no charge on the
GAO Web site at http://www.gao.gov. Major contributors to this report
were Catherine Baltzell, Mary Quinlan, Charles Cannon, Robin Eddington,
Gary Middleton, Charles Perdue, and Adam Vodroska. If you have any
questions regarding this report, please call me at (202) 512-4841.




Paul L. Francis
Director
Acquisition and Sourcing Management




Page 20                                       GAO-03-442 Defense Acquisitions
                            Appendix I: Comments from the Department
Appendix I: Comments from the Department
                            of Defense



of Defense

Note: GAO comments
supplementing those in
the report text appear at
the end of this appendix.




                            Page 21                                    GAO-03-442 Defense Acquisitions
Appendix I: Comments from the Department
of Defense




Page 22                                    GAO-03-442 Defense Acquisitions
                 Appendix I: Comments from the Department
                 of Defense




See comment 1.




                 Page 23                                    GAO-03-442 Defense Acquisitions
                 Appendix I: Comments from the Department
                 of Defense




See comment 2.




See comment 3.




                 Page 24                                    GAO-03-442 Defense Acquisitions
Appendix I: Comments from the Department
of Defense




Page 25                                    GAO-03-442 Defense Acquisitions
               Appendix I: Comments from the Department
               of Defense




               The following are GAO’s comments on the Department of Defense’s letter
               dated March 10, 2003.


               1. DOD did not provide any new evidence that 14 of ASDS’s 16 key
GAO Comments   performance parameters have been met. As we discussed in our report,
               program officials had earlier judged that 11 of the first boat’s key
               performance parameters had been met, 4 were still in process, and 1
               required action. We did not count those in process as being met, because
               test reports were not yet completed. It is possible that some of these
               reports have since been completed, but DOD has not provided this
               information.

               DOD also did not provide any new evidence that outstanding subordinate
               requirements either exceed required values or apply only in the future. As
               we discuss in our report, as of January 30, 2003, DOD documentation
               showed that numerous subordinate requirements for the first boat—which
               must be met to demonstrate the key performance parameters—had not yet
               been fully demonstrated. We have noted that in some instances, future
               requirements were actually planned for the first boat, but deferred.

               2. We continue to believe that uncertainties about the schedule and cost of
               the ASDS program remain and make it difficult to develop credible
               projections. As we discuss in this report, progress has been made in
               correcting various technical problems with the ASDS. However, serious
               technical problems and significant uncertainty remain. Operational testing
               has not yet begun and may reveal additional problems, which could
               require redesign and other solutions. In addition, the program has
               experienced continual delays in preparing and getting schedule and cost
               estimates approved. While officials have told us that the acquisition
               program baseline with an updated schedule is currently being revised, the
               baseline has not yet been completed and approved. The ASDS program is
               still operating with the June 1999 acquisition program baseline, which is
               now considerably out of date. Consequently, credible criteria for
               measuring program cost and schedule stability—and whether or not the
               program is on track—are still lacking. Finally, the problems of loss of
               production base and obsolescence of certain technologies remain for the
               second boat.

               The Navy did recently provide GAO with several briefing slides that were
               based on an independent cost estimate, but they are not the actual
               estimate. Specifically, the briefing slides show some—but not all—costs



               Page 26                                        GAO-03-442 Defense Acquisitions
           Appendix I: Comments from the Department
           of Defense




           for the second boat only. They did not provide details about estimation
           methodology or about what costs are included and excluded.

           3. As we discuss in this report, we recognize that management attention
           has increased in recent years. Nevertheless, we continue to believe that
           the ASDS program needs additional management attention, particularly at
           higher DOD levels. Our conclusion is based on the current status of the
           ASDS program itself, including the challenges and risks it faces, and the
           significant investment it now represents. DOD’s statement that the ASDS
           program has been reviewed more often than a majority of acquisition
           category I programs is difficult to evaluate without seeing evidence.
           Nonetheless, the number of times a program is reviewed does not
           necessarily equate to the right kind of management attention.




(120155)
           Page 27                                        GAO-03-442 Defense Acquisitions
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548