oversight

Embedded Computers: Navy Not Ready to Buy Avionics Computers for Its LAMPS Mk I Helicopters

Published by the Government Accountability Office on 1990-05-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

           United   States   General   Accounting   Office
           Report to Congressional Requesters
GAO

May 1990
           EMBEDDED
           COMPUTERS
           Navy Not Ready to
           Buy Avionics
           Computers for Its
           LAMPS Mk I
           Helicopters

                                                      ..




           RESTRICTED --Not      to be released outside the
           General Accounting office unless speciik&y
           approved by the Offlee of Congressional
           Relations.
      united
        states
GAO   General Accounthg
      Wwhingtan,
                           Office
                   D.C. 20648

      Information     Management         and
      Technology     Division

      B238826
      May 31,lQQO
      The Honorable John Conyers, Jr.
      Chairman, Subcommittee on Legislation
        and National Security
      Committee on Government Operations
      House of Representatives

      The Honorable Frank Horton
      Ranking Minority Member, Subcommittee
        on Legislation and National Security
      Committee on Government Operations
      Houseof Representatives
      The Navy plans to spend $66.7 million for 160 avionics1computer sys-
      tems as part of a $896 mllllon overall upgrade to its Light Airborne Mul-
      tipurpose System (LAMPS)   Mk I helicopter. This helicopter is used for
      antisubmarine warfare, and it will rely extensively on the new com-
      puter-based avionics system to perform this mission. This avionics com-
      puter system will allow in-fllght processingof data from sensorsthat
      detect submarines. Currently, helicopters send the sensor data to ship
      board computer systems, wait for the ship to analyze the data, and
      transmit it back to the helicopter.

      The Navy has already spent $6.3 mlllion to buy 12 of the avionics com-
      puter systems, and ln June 1990 it will decide whether to buy another
      16 costing $6.6 million. While the cost of all avionics computer systems
      to be bought is relatively small, these systems directly affect the mission
      effectiveness of an $896 million program to upgrade the helicopters’ air-
      frames and avionics suites.

      This report responds to your office’s October 1989 request to review the
      Navy’s acquisition of computer systems embeddedin selected antlsub-
      marine warfare systems, and ls part of your overall request to review
      computer systems that are embeddedin Defenseweapon systems. Our




      ‘Avionica (i.e., aviation electronics) computers receive data from sensors (e.g., sonobuoys, radar,
      mdcpit switches, tI3nperahll.e probea, accelerometers, rate gyros, etc.), pl-ocesathe data on a “real
      time” basis and send the pnxessed data to other systems (e.g., cockpit displays, aeronautical control
      surf-,    other computers, etc.).



       page 1                 GAO-                 Navy Not Eeady to Buy compntus        for Its Helicoptenr
             testing requirements for the avionics computer system becauseof its
             early perception that the system was just a minor modification of an
             existing computer system. In light of the significant software and hard-
             ware changesthe system requires and the program office’s limited test
             plans, we believe that this oversight office should immediately begin
             supervising the program and ensuring that the requisite testing is
             performed.


             The Navy’s LAMPS     Mk I ls a ship-basedhelicopter that will rely exten-
Background   sively on computer systems embeddedin its avionics to perform its basic
             mission of s&submarine warfare (i.e., locating, identifying, following,
             and engaglng enemy submarines). The computer-basedavionics system
             will keenly sharpen the helicopter crew’s ability to find, recognize,and
             target submarines through the use of sensors,submarine identification
             software, and data display devices. The effectiveness of these computer-
             ized systems is a critical variable ln determlnlng how well the Navy’s
             billion dollar fleet of LAMPS  Mk I helicopters can meet its mission
             requirements.

             The Navy introduced the LAMPS    Mk I helicopter into its fleet in 1973 to
             collect sensor data on submarine activity and send it to the helicopter’s
             home ship for analysis and display. In 1986, the Director of Naval
             Reserve,Chief of Naval Operations, changedthe helicopter’s operational
             requirement to include in-fllght processing and display, thus reducing its
             reliance on the communication llnk between the helicopter and its ship.
             The reason for this expanded requirement was to improve the Navy’s
             mission effectiveness by (1) saving critical secondsin detecting, identi-
             fying, and countering enemy submarines, and (2) allowing the helicopter
             to perform its mission beyond data-llnk range with its ship.

             To meet the new requirement, the Navy ls developing and installing a
             new avionics computer system-designated the AN/ASN-16OA-as
             part of an overall program to upgrade the helicopters4 The Navy plans
             to modify 103 hellcopters9,6
                                        at an estimated cost of $896 mllllon.6 Of this,


             ‘In addition to the avionics computer snd related equipment, LAMPS Mk I helicopters will receive
             new engines, and some will also get airframe modifications and rotor blades.

             61ncl~des42 new helicopters under contract and 61 active helicopters currently in the fleet,

             %xtrapoleted from program manager’s recurkg cost e&mate of $8.7 million for each helicopter.
             Accordingtothep~manager,thetotalmodificationp~costisanestimatebecausethe
             effort has not been programm ed nor funded.



             Pyle 3                  GAO-                Navy Not Eeady to Buy Camputera for Ita Helicoptera
                           E238828




                           when they are cheaper to correct. Both developmental and operational
                           testing are critical to demonstrate that a system is ready for production.


Navy DoesNot Plan to       Despite its own requirements to operationally test systems and subsys-
Operationally Test the     tems before either a limited production (milestone IIIA) or full-rate
                           production (milestone BIB) decision? the Navy doesnot plan to opera-
System Before Production   tionally test the AN/ASN-16OA avionics computer system before a June
                           1990 decision to buy 16 of them and initially produce the modified hell-
                           copter. While the program manager admits that this testing approach
                           falls short of Navy requirements and introduces increased risk into the
                           program, this official stated that it strikes a realistic balance between a
                           prudent system engineering approach and a need to obligate funds
                           already appropriated by the Congress.
                           According to the current program manager, development of the AN/
                           ASN-16OAcomputer system was initially viewed as a low-risk modifica-
                           tion of a computer system already approved for use in another aircraft.
                           Accordingly, in April 1989, the previous program manager requested an
                           extension of application9 for the computer system, and the Director,
                           Researchand Development, Test and Evaluation, Office of the Chief of
                           Naval Operations, approved the extension ln May 1989.

                           In our opinion, the Navy’s portrayal of the AN/ASN-16OA as a low-risk,
                           minor modification of an existing and approved computer system was
                           not accurate, and has placed the Navy in its current dilemma of whether
                           or not to obligate funds already appropriated, buy the computer sys-
                           tems, and initially produce the helicopters, before it has first operation-
                           ally tested them We found that the AN/ASN-160A is significantly
                           different from the aheady approved computer system. Specifically, the
                           former computer system performed solely tactical navigation functions.
                           In contrast, the modified computer system will perform not only naviga-
                           tion functions, but also in-flight sensorprocessingfunctions associated


                           sOffice of the Chief of Naval Operations Instruction 3!360.1OC,Test and hrahmtion, states that
                           upgades to production systems that incorporate new computer systems are to be operationally test.4
                           in order to ensure that system performance has been maintained or improved and meets Navy
                           requirements. Further, In&u&on 6000.42, Raearch, Development, and Acquisition Prcadures,
                           xvWires successful operational testing of subsystau, like the AN/AS+160 avionics computer, before
                           authority to buy the subsystem is granted.

                           Wae extension of appkation is an exemption from the operational testing requkment. It is allowed
                           for subsystems already in use for similar appkations in other aircraft and when “‘there are no com-
                           plicaalg cirmmstances such as Saware czha@s.”



                           Page 5                 GAODBWJSMM           Navy Not Ready to Hny Compatem for Its Hekoptera
Developmental Testing         Even though the Navy has performed developmental testing, including
Revealed Software             laboratory integration testing, this testing is not a substitute for opera-
                              tional testing. Moreover, completed integration tests have revealed 250
Problems and Is Not a         uncorrected software discrepancies,20 of which could severely affect
Substitute for Operationnal   the avionics computer system’s ability to accurately locate a moving tar-
Testing                       get and launch a torpedo.
                              The Navy has elected to treat the AN/ASN-15OA as an extension of
                              application and baseits upcoming procurement decision solely on devel-
                              opmental testing in a laboratory. In our opinion, this developmental test-
                              ing doesnot substitute for operational testing. Navy instructions require
                              operational testing by an independent test activity, and nothing in these
                              instructions indicates that developmental tests conducted by the con-
                              tractor may be used in its place. Additionally, the official in the Navy’s
                              Operational Test and Evaluation Force who is responsible for coordinat-
                              ing operational testing of this program said that developmental testing
                              in a laboratory is not an adequate substitute becauseit cannot fully sim-
                              ulate true operational conditions; and according to the program’s test
                              and evaluation master plan, one of the critical issuesto be resolved for
                              this system is whether the integrated avionics system (i.e., the computer
                              and sensors)can detect a submarine under actual mission conditions.
                              Additionally, the contractor, overseenby the program office, has per-
                              formed all developmental testing to date. An independent test activity
                              has not yet conducted any testing, and Navy instructions require inde-
                              pendent testing prior to a production decision.

                              Moreover, we found that the Navy’s laboratory integration testing on
                              the AN/ASN-150A avionics computer system has in fact not realistically
                              simulated the strain the software will be under while actually tracking
                              submarines. That is, the laboratory stress and performance testing has
                              not subjected the avionics system’s software programs to the volumes of
                              data that would be received simultaneously from three different types
                              of sensorswhile tracking submarines under combat conditions. The rea-
                              son is that the Navy does not have the capability to simulate all the
                              sensorinputs, let alone simulate them simultaneously. Specifically, only
                              data inputs from the acoustic sensor (device dropped from the helicop-
                              ter into the water to detect underwater sounds) and the magnetic anom-
                              aly detector (sensing device towed behind the helicopter to detect the
                              magnetic density of submarines) have been simulated. The radar data
                              inputs have not. According to program officials, the radar inputs have
                              not been evaluated becauseit would be too costly to expand develop-
                              mental testing beyond what is already planned. In our opinion, such lim-
                              itations to integration testing are all the more reasonto ensure that


                              Page 7            GAO/lMTECSo54   Navy Not Beady to Buy compntem for Ite Helko~tem
              existing computer system. This office did, however, require that opera-
              tional testing be conducted before deciding whether to deploy the modi-
              fied helicopter, a decision currently planned for March 1991. As stated
              earlier, we believe that the Office of the Assistant Chief of Naval Opera-
              tions (Air Warfare) decision to grant the extension of application and
              waive operational testing before a production decision was inherently
              risky and inappropriate.

              We also found that the Office of the Assistant Chief of Naval Operations
              (Air Warfare) did not ensure there was specified performance criteria in
              the AN/ASN-15OA development contract before allowing the program to
              proceed. Specifically, DefenseStandard 2167A, DefenseSystem
              Software Development, states that system specifications are provided to
              contractors as a requirements baseline for software development; and
              Navy Instruction 3960.10, Test and Evaluation, requires that software
              performance criteria be established and used as performance measures
              in testing. According to a Defensereport on software development
              problems, if such criteria are not specified, the cost, schedule,and qual-
              ity of the entire software development effort will bejeopardized.12

              The Navy contracted for the LAMPS   Mk I modification in February 1987
              without a system specification that defined AN/ASN-15OA performance
              requirements. In lieu of such a specification, the software subcontractor
              used a preliminary specification, and the Navy began testing the
              software in September 1989 without such criteria. In December1989,
              the Navy finally incorporated the specification into the modification
              contract. However, these add-on requirements cost the Navy an extra
              $2.1 million, and causedthe modification program to slip 4 months.

              The Navy will soon face pivotal decisions on whether to buy more avi-
Conclusions   onics computer systems and begin modifying its billion dollar fleet of
              LAMFJS Mk I helicopters. These helicopters will heavily depend on their
              new avionics computer systems, which will ultimately cost the Navy
              $65.7 million to buy and install on these aircraft as part of a $896 mil-
              lion LAMPS  Mk I modification program.

              Clearly, making such a decision requires, at a minimum, sufficient test-
              ing to know whether the integrated avionics system will perform as
              intended. In our opinion, this information does not currently exist, and

              “B           of the Joint logistics Commanders Joint Policy coordinating Group on Computer
              Resource Management (Aug. 1979).



              page. 9               GAO/IMTEC~          Navy Not Ready to Ehxycompoten, for Ita Eelicopte~
As arranged with your offices, unless you publicly announcethe con-
tents of this report earlier, we plan no further distribution until 30 days
from the date of this letter. At that time, we will send copies to the
Chairman, Senate and House Appropriations Committees; the Secretar-
ies of Defenseand the Navy; and to other interested parties. We will also
make copies available to others upon request. This report was prepared
under the direction of Samuel W. Bowlin, Director, Defenseand Security
Information Systems, who can be reached at (202) 2754649. Other
major contributors are listed in appendix III.




Ralph V. Carlone
Assistant Comptroller General




Page 11          GAO-           Navy Not Ready to Bay Computem for Ita Helhpters
Page 13   GAO/IMTEGgod4   Navy Not Beady to Buy Camptaem for Its Helifopters
Appendix II

Near-Term F’undingRequirementsfor
Modification Kits

              In millions of dollars
              Fiscal Year                                                   Entire Kits            AN/ASN-1SOA
              1909                                                                $45.0                     $2.6
              1990                                                                 55.3                      2.6
              1991                                                                 54.0                      2.6
              Is92                                                                 53.4                      2.6
              1gxl                                                                 40.8                      2.6
              1994                                                                  3.1                       1.3
              Total                                                              $287.4                    514.3
              Qoes not include funding requirements for spares Including spares. the cornblned fiscal yeal
              1969/1990 funding requirements for the AN/ASN-1MA would be $6 6 million.




              Page 16
.




    Requests    for copies of GAO reports   should be sent to:

    U.S. General Accounting     Office
    Post Office Box 6015
    Gaithersburg,  Maryland     20877

    Telephone    202-275-6241

    The Fit five copies of each report      are free. Additional   r~#&z:)      .’
    $2.00 each.

    There is a 25% discount     on orders for 100 or more copies    I (k I . _ _.-
    single address.

    Orders must be prepaid by cash or by check or money order           tI v II r
    out to the Superintendent of Documents.




                                                                                     ii’
United States                                First-Class Mail
General Accounting    Office               Postage & Fees #T
Washington,   D.C. 20548                            GAO
                                        --   Permit    No. GlOO
Official   Busmess                                              ,,..
Penalty    for Private   Use $300


                                    -
Appendix III

Major Contributors to This Report


                        John Stephenson,Assistant Director
Information             Randolph C. Hite, Assignment Manager
Managementand
Technology Division,
Washington,D.C.
                        Daniel R. Garcia, Regional ManagementRepresentative
Philadelphia Regional   Norman C. Berman, Evaluator-in-Charge
Office                  William E. Petrlck, Evaluator




 (610678)                PI@ 16
Objectives,Scope,and Methodology


              In October 1989, the Subcommittee on Legislation and National Security,
              House Committee on Government Operations, expressedinterest in the
              Navy’s plans to acquire embeddedcomputer systems for selectedanti-
              submarine warfare systems, and requested that we determine whether
              (1) the Navy plans to conduct adequate developmental and operational
              testing of these embeddedcomputer systems before buying them, and
              (2) Navy managementis overseeingthe acquisition of them. This
              request relates to an overall request from the Chairman and the Sub-
              committee’s Ranking Minority Member to review computer systems that
              are embeddedin Defenseweapon systems.
              To accomplish our objectives, we reviewed Defenseand Navy instruc-
              tions and standards governing the development, testing, and manage-
              ment oversight of embeddedcomputer systems for weapon systems. We
              also reviewed relevant program documentation addressing the acquisi-
              tion strategy, test plans and results, and schedule and funding require-
              ments for both the AN/ANSlSOA avionics computer system as well as
              the entire helicopter modification program. Additionally, we inter-
              viewed program officials responsible for managing various aspectsof
              the program, including the avionics computer system’s development and
              testing. We also interviewed Chief of Naval Operations officials respon-
              sible for overseeingthe program, and reviewed documentation relating
              to the discharge of this oversight responsibility. Further, we discussed
              test plans and results with officials from the Navy test activities
              involved in developmental (functional and integration) and operational
              testing, and we observed the execution of integration testing at the con-
              tractor’s facilities.

              We performed our work between October 1989 and March 1990, prima-
              rily at the LAMPSMk I program office within the Naval Air Systems Com-
              mand ln Arlington, Virginia, and the Naval Air Development Center in
              Warminster, Pennsylvania. We also visited the Navy’s Operational Test
              and Evaluation Force in Norfolk, Virginia, and the contractor’s labora-
              tory in Northridge, California.

              As requested by the Chairman’s office, we did not obtain official agency
              comments on a draft of the report. However, we discussedits contents
              with Navy and Office of the Secretary of Defenseofficials, and have
              incorporated their comments where appropriate. We conducted our
              review in accordancewith generally acceptedgovernment auditing
              standards.




              Page 14          GAO-           Navy Not Ready to Buy Comp~~tem for Its   Eeiicoptere
Contents


Letter
Appendix1
Objectives,Scope,and
Methodology
Appendix11
Near-TermFunding
Requirementsfor
Modification Kits
Appendix111                                                                  16
Major Contributorsto
ThisReport




                       Abbreviations

                       GAO   General Accounting Office
                       IMTM: Information Managementand Technology Division
                       LAMPS Light Airborne Multipurpose System
                  the Navy’s plans for testing prior to its computer system purchase and
                  initial production decisionswill not provide it. Further, the results of
                  laboratory integration testing have revealed software discrepanciesof
                  such magnitude that the Navy cannot justify going beyond the labora-
                  tory test environment at this time, much less make a decision to buy
                  more computer systems and initially produce the modified aircraft.
                  Unless the Navy changesits approach to testing, discovery of AN/ASN-
                   16OAperformance problems wlll likely be delayed to a time when their
                  correction will be more costly than necessary.In fact, estimates show
                  that software problems found late in the development processcan cost
                  six to ten times more to correct than if found early in the process.More-
                  over, they may prevent the system from ever performing as originally
                  intended.
                  The Office of the Assistant Chief of Naval Operations (Air Warfare), the
                  office responsible for overseeingthe avionics computer system’s devel-
                  opment and testing, has not assured compliance with embeddedcom-
                  puter system development and testing requirements becauseit agrees
                  with the program office’s initial position that the computer system is a
                  minor modification. In our opinion, this position is inappropriate, and
                  attention by the Office of the Assistant Chief of Naval Operations (Air
                  Warfare) to the program office’s approach to development and testing of
                  the AN/ASN-16OA may prevent further deviations from Navy
                  requirements.


                  We recommend that the Secretary of the Navy direct the LAMPS   Mk I
Recommendations   program office to defer buying the AN/ASN-160A computer systems
                  until complete and thorough operational testing demonstrates that the
                  avionics systems will satisfy mission requirements. We further recom-
                  mend that the Secretary direct the Assistant Chief of Naval Operations
                  (Air Warfare) to take a more active role in overseeingthe program and
                  ensure that Navy requirements for AN/ASN-16OA operational testing
                  are met.

                  As requested by your offices, we did not obtain official agency com-
                  ments on a draft of this report. However, we discussedits contents with
                  Navy and Office of the Secretary of Defenseofficials, and have incorpo-
                  rated their comments where appropriate. Cur work was performed ln
                  accordancewith generally acceptedgovernment auditing standards,
                  between October 1989 and March 1990.



                  Page 10          GAO-           Navy Not Ready to Buy Compmern for Its Hekopters
                       operational testing precedesa decision to buy any more avionics com-
                       puter systems to initially modify the helicopters.

                       Laboratory integration testing conducted by the program test team in
                       October 1989 revealed hundreds of software discrepancies.In February
                       1990, we observed retests and found that while some discrepancieshad
                       been corrected, additional discrepancieswere found. As of March 1,
                       1990,260 of 636 cumulative discrepanciesstill were unresolved, and the
                       Navy designated 20 of these as critical (i.e., they significantly degrade
                       mission effectiveness). For example, data from one of the sensorscould
                       not be processedand displayed quickly enough to permit targeting of a
                       submarine. Additionally, the computer system did not have to contend
                       with concurrent inputs from multiple sensorsduring this testing scena-
                       rio. Such multiple inputs could exacerbatethis processingand display
                       problem.

                       On at least two occasions,Navy managementoversight authorities have
Navy Oversight         not acted to ensure that the LAMPSMk I modification complies with Navy
Authorities Have Not   system development and testing requirements. This inaction contributed
Assured Compliance     to cost increasesand schedule delays, and has allowed the program to
                       assumean increased level of risk associatedwith not operationally test-
With Requirements      ing the system before a production decision.

                       According to Office of the Chief of Naval Operations Instruction
                       5000.42, Research,Development, and Acquisition Procedures,the pro
                       gram sponsorswithin the Office of the Chief of Naval Operations are
                       responsible for providing overall direction to the program and assuring
                       that Navy requirements for development and testing of embeddedcom-
                       puter systems are properly implemented. The level of oversight respon-
                       sibility is determined by the cost of the program and other
                       considerations, such as development risks. For the LAMPS  Mk I modifica-
                       tion program, management oversight responsibility resides with the
                       Assistant Chief of Naval Operations (Air Warfare).

                       We found that the Office of the Assistant Chief of Naval Operations (Air
                       Warfare) has elected not to focus on development and testing of the AN/
                       ANS-16OAavionics computer system becauseit was portrayed as a
                       modification of an existing system. According to the official responsible
                       for ntonltoring this LAMPSMk I modification, the AN/ASN-16OA com-
                       puter system was exempted from operational testing by an extension of
                       application becauseit was believed to be a low-risk modification of an



                       Page 8           GAO-           Navy Not Ready to Buy Complltus   for Its Hekoptem
B-238826




with the antisubmarine warfare mission, which were previously per-
formed on the baseship. To provide for these new functions, 16 of the
computer system’s 23 circuit board cardsJowere changed,and major
changesto the software were made. In fact, only about 67 percent of the
former system’s 370,000 lines of software code remained intact-7 per-
cent, or about 26,000 lines of code, was changed;4 percent, or about
16,000 lines of code, was added;, and 22 percent, or about 82,000 lines
of code,was deleted.” Additionally, the 7 percent of software that was
changedperforms critical tactical functions such as m-flight processing
of sensor data. Moreover, the Navy’s test plan describessystem
software integration as a “critical issue.” Also, the results of recently
completed integration testing (seenext section of the report) confirm
that the modification is not a low-risk endeavor.

The current program manager agreed with our assessmentthat the AN/
ASN-16OAavionics computer system is not a minor software modifica-
tion. According to this official, the avionics computer system is a com-
plex software development effort requiring extensive software
integration. However, this official also stated that the current testing
approach does not introduce enough risk into the program to justify
postponing a limited production decision until after operational testing.
This testing is currently scheduledto begin in September 1990. The pro-
gram manager added that (1) funds have already been appropriated for
buying 16 avionics computer systems and modifying 12 more helicop-
ters, and (2) establishing the production line requires over 2 years,
which ls sufficient lead time to correct any computer system problems
discovered during operational testing. In our opinion, the mere availabil-
ity of appropriated funds does not justify prematurely obligating them.
Further, the sufficiency of lead time to correct computer system
problems at this point in time is purely speculative, and cannot be accu-
rately determined until complete and thorough testing disclosesthe
extent of any problems.



‘°Ciit     boards and cards are part of the computer system’s hardware. Colkctively, they provide the
vital internal circuitry for the computer, and consist of the mounting boards and the components that
are mounted on the bmrds. The cards are speck&M gmup of components that pmvide some spe
ciaked function, such as central processing, internal memory, and display screen control and
Fesolution.

“The former system included software for processing data from a dipping sonar sensor, which is not
being used on the LAMF’S Mk I. Therefore, this code was not needed. Acarding to the avionica system
pmject officer, such changes in functionality account for the redwed number of lines of code in the
modified system.



Page 6                 GAO-                 Naw Not Ready to Boy CYkmpoters for Its Helicopten,
                      $66.7 million will be spent to buy 160 avionics computer systems.’ To
                      date, the Navy has bought 12 of the new computer systems costing $5.3
                      million. Appendix II contains the funding requirements for the modifica-
                      tion program, as well as the avionics computer system portion of the
                      program, through fiscal year 1994.
                      In June 1990, the Navy plans to decide whether to begin the limited pro-
                      duction phase of the modification program. This entails deciding
                      whether to buy 15 more computer systems (12 for immediate installa-
                      tion and 3 as spares) costing $6.6 million, and begin installing these com-
                      puter systems, along with new engines,rotor blades, and other avionics
                      equipment on 12 aircraft. In March 1991, the Navy plans to decide
                      whether to buy the remainder of the 160 computer systems and begin
                      fully producing the modified helicopter.

                      The Navy plans to decide whether to purchase 15 more AN/ASN-150A
Navy’s Planned        avionics computer systems and begin initially producing the modified
Testing of the        helicopter before operationally testing the system. By doing so, the Navy
Avionics Computer     is disregarding its own software development requirements, and greatly
                      increasing the risk that operational problems-those that cannot be
SystemIs Inadequate   anticipated and tested for in a simulated environment-will not be dis-
                      covered until later in the system development process,when the Navy
                      will have modified an additional 12 helicopters. As a result, correcting
                      these problems will be more difficult and costly than if they were found
                      before any further computer purchases and helicopter modifications.
                      Moreover, should the system fail to perform as required, the Navy may
                      find that it has spent money unnecessarily.

                      Computer system testing is incremental, with early developmental tests
                      focusing on whether system componentsand subcomponentsperform
                      the functions they are designedto do. Later developmental tests build
                      on early tests, and addressthe ability of the components and subcom-
                      ponents to perform their intended functions as integrated units. Still
                      later developmental testing addresseshow well the integrated units per-
                      form (i.e., how fast, how reliable, how accurate, how often) in a labora-
                      tory that realistically simulates the stress the system will be under.
                      Following developmental testing, the complete system is tested in a true
                      operational setting, with actual users. This testing progression empha-
                      sizesthe benefits of finding problems early in the development process

                      ?Estimate based cm the 103 helicopters to be modified plustheprogram   manager’s estimate of
                      another 47 computers  asspares  for helicopter maintenance and tmining purpcees.



                      Page 4                 GAO/lM’IECSO44      Navy Not Beady to Buy    Compoters
                                                                                                 for Its     Helicopters
                  objectives were to determine whether (1) the Navy plans to conduct ade-
                  quate developmental2and operational3 testing of the helicopters’ embed-
                  ded computer systems before buying them, and (2) Navy managementis
                  overseeingthe acquisition of these embeddedcomputer systems.A
                  detailed explanation of our objectives, scope,and methodology is con-
                  tained in appendix I.


                  In June 1990, the Navy plans to decide whether to buy $6.6 million
Resultsin Brief   worth of upgraded avionics computer systems as part of a program to
                  modify 12 LAMPS   Mk I helicopters. These upgraded systems, although
                  initially thought to only require minor modifications to a system already
                  approved for another helicopter, actually require extensive software
                  and hardware changes.However, the Navy has chosennot to operation-
                  ally test the system before its June 1990 decision.
                  In our opinion, omitting operational testing greatly increasesthe risk
                  that system performance problems will not be detected and resolved
                  until later in the development process.History has shown that the
                  longer a software problem goesundetected, the more expensive it is to
                  fix, assuming that it can be fixed.

                  Developmental testing done to date on the new avionics computer sys-
                  tem revealed significant software problems that cast doubt on whether
                  the Navy should proceed until the problems are corrected. The program
                  manager agreed that operational testing should precede a production
                  decision, but the Navy is faced with the need to quickly obligate funds
                  appropriated by the Congressfor fiscal years 1989 and 1990. We believe
                  that the Navy should not buy any more avionics computer systems until
                  it has completed operational testing.
                  The Office of the Assistant Chief of Naval Operations (Air Warfare) has
                  management oversight responsibility for the modification program.
                  However, this office has not assured compliance with development and

                  2Defense Directive 6ooO.3.Test and Evaluation, defines developmental testing as testing associated
                  with the engineering design and development of a system to verify attainment of performance and
                  functional specifications, objectives, and supportabiity. It adds that this tasting addresw system
                  components and their integration, and involves the use of models, simulations, and testbeds.

                  aDefense Diredive 6000.3, Test and Evaluation, defmes operational testing as field testing, under
                  realistic omditions, of any item or component of a weapons system, to determine its effectiveness and
                  suitability for combat use. It further states that this testing is to be conducted in an envimnment that
                  is operationally ralistic, including having typical users operate and maintain the system in a settiig
                  representative of combat stress conditions.



                  page 2                  GAO/IbfTBG6044       Navy Not Beady to Buy compnters for Its Helicopters