United States General Accounting Office Report to Congressional Requesters GAO May 1990 EMBEDDED COMPUTERS Navy Not Ready to Buy Avionics Computers for Its LAMPS Mk I Helicopters .. RESTRICTED --Not to be released outside the General Accounting office unless speciik&y approved by the Offlee of Congressional Relations. united states GAO General Accounthg Wwhingtan, Office D.C. 20648 Information Management and Technology Division B238826 May 31,lQQO The Honorable John Conyers, Jr. Chairman, Subcommittee on Legislation and National Security Committee on Government Operations House of Representatives The Honorable Frank Horton Ranking Minority Member, Subcommittee on Legislation and National Security Committee on Government Operations Houseof Representatives The Navy plans to spend $66.7 million for 160 avionics1computer sys- tems as part of a $896 mllllon overall upgrade to its Light Airborne Mul- tipurpose System (LAMPS) Mk I helicopter. This helicopter is used for antisubmarine warfare, and it will rely extensively on the new com- puter-based avionics system to perform this mission. This avionics com- puter system will allow in-fllght processingof data from sensorsthat detect submarines. Currently, helicopters send the sensor data to ship board computer systems, wait for the ship to analyze the data, and transmit it back to the helicopter. The Navy has already spent $6.3 mlllion to buy 12 of the avionics com- puter systems, and ln June 1990 it will decide whether to buy another 16 costing $6.6 million. While the cost of all avionics computer systems to be bought is relatively small, these systems directly affect the mission effectiveness of an $896 million program to upgrade the helicopters’ air- frames and avionics suites. This report responds to your office’s October 1989 request to review the Navy’s acquisition of computer systems embeddedin selected antlsub- marine warfare systems, and ls part of your overall request to review computer systems that are embeddedin Defenseweapon systems. Our ‘Avionica (i.e., aviation electronics) computers receive data from sensors (e.g., sonobuoys, radar, mdcpit switches, tI3nperahll.e probea, accelerometers, rate gyros, etc.), pl-ocesathe data on a “real time” basis and send the pnxessed data to other systems (e.g., cockpit displays, aeronautical control surf-, other computers, etc.). page 1 GAO- Navy Not Eeady to Buy compntus for Its Helicoptenr testing requirements for the avionics computer system becauseof its early perception that the system was just a minor modification of an existing computer system. In light of the significant software and hard- ware changesthe system requires and the program office’s limited test plans, we believe that this oversight office should immediately begin supervising the program and ensuring that the requisite testing is performed. The Navy’s LAMPS Mk I ls a ship-basedhelicopter that will rely exten- Background sively on computer systems embeddedin its avionics to perform its basic mission of s&submarine warfare (i.e., locating, identifying, following, and engaglng enemy submarines). The computer-basedavionics system will keenly sharpen the helicopter crew’s ability to find, recognize,and target submarines through the use of sensors,submarine identification software, and data display devices. The effectiveness of these computer- ized systems is a critical variable ln determlnlng how well the Navy’s billion dollar fleet of LAMPS Mk I helicopters can meet its mission requirements. The Navy introduced the LAMPS Mk I helicopter into its fleet in 1973 to collect sensor data on submarine activity and send it to the helicopter’s home ship for analysis and display. In 1986, the Director of Naval Reserve,Chief of Naval Operations, changedthe helicopter’s operational requirement to include in-fllght processing and display, thus reducing its reliance on the communication llnk between the helicopter and its ship. The reason for this expanded requirement was to improve the Navy’s mission effectiveness by (1) saving critical secondsin detecting, identi- fying, and countering enemy submarines, and (2) allowing the helicopter to perform its mission beyond data-llnk range with its ship. To meet the new requirement, the Navy ls developing and installing a new avionics computer system-designated the AN/ASN-16OA-as part of an overall program to upgrade the helicopters4 The Navy plans to modify 103 hellcopters9,6 at an estimated cost of $896 mllllon.6 Of this, ‘In addition to the avionics computer snd related equipment, LAMPS Mk I helicopters will receive new engines, and some will also get airframe modifications and rotor blades. 61ncl~des42 new helicopters under contract and 61 active helicopters currently in the fleet, %xtrapoleted from program manager’s recurkg cost e&mate of $8.7 million for each helicopter. Accordingtothep~manager,thetotalmodificationp~costisanestimatebecausethe effort has not been programm ed nor funded. Pyle 3 GAO- Navy Not Eeady to Buy Camputera for Ita Helicoptera E238828 when they are cheaper to correct. Both developmental and operational testing are critical to demonstrate that a system is ready for production. Navy DoesNot Plan to Despite its own requirements to operationally test systems and subsys- Operationally Test the tems before either a limited production (milestone IIIA) or full-rate production (milestone BIB) decision? the Navy doesnot plan to opera- System Before Production tionally test the AN/ASN-16OA avionics computer system before a June 1990 decision to buy 16 of them and initially produce the modified hell- copter. While the program manager admits that this testing approach falls short of Navy requirements and introduces increased risk into the program, this official stated that it strikes a realistic balance between a prudent system engineering approach and a need to obligate funds already appropriated by the Congress. According to the current program manager, development of the AN/ ASN-16OAcomputer system was initially viewed as a low-risk modifica- tion of a computer system already approved for use in another aircraft. Accordingly, in April 1989, the previous program manager requested an extension of application9 for the computer system, and the Director, Researchand Development, Test and Evaluation, Office of the Chief of Naval Operations, approved the extension ln May 1989. In our opinion, the Navy’s portrayal of the AN/ASN-16OA as a low-risk, minor modification of an existing and approved computer system was not accurate, and has placed the Navy in its current dilemma of whether or not to obligate funds already appropriated, buy the computer sys- tems, and initially produce the helicopters, before it has first operation- ally tested them We found that the AN/ASN-160A is significantly different from the aheady approved computer system. Specifically, the former computer system performed solely tactical navigation functions. In contrast, the modified computer system will perform not only naviga- tion functions, but also in-flight sensorprocessingfunctions associated sOffice of the Chief of Naval Operations Instruction 3!360.1OC,Test and hrahmtion, states that upgades to production systems that incorporate new computer systems are to be operationally test.4 in order to ensure that system performance has been maintained or improved and meets Navy requirements. Further, In&u&on 6000.42, Raearch, Development, and Acquisition Prcadures, xvWires successful operational testing of subsystau, like the AN/AS+160 avionics computer, before authority to buy the subsystem is granted. Wae extension of appkation is an exemption from the operational testing requkment. It is allowed for subsystems already in use for similar appkations in other aircraft and when “‘there are no com- plicaalg cirmmstances such as Saware czha@s.” Page 5 GAODBWJSMM Navy Not Ready to Hny Compatem for Its Hekoptera Developmental Testing Even though the Navy has performed developmental testing, including Revealed Software laboratory integration testing, this testing is not a substitute for opera- tional testing. Moreover, completed integration tests have revealed 250 Problems and Is Not a uncorrected software discrepancies,20 of which could severely affect Substitute for Operationnal the avionics computer system’s ability to accurately locate a moving tar- Testing get and launch a torpedo. The Navy has elected to treat the AN/ASN-15OA as an extension of application and baseits upcoming procurement decision solely on devel- opmental testing in a laboratory. In our opinion, this developmental test- ing doesnot substitute for operational testing. Navy instructions require operational testing by an independent test activity, and nothing in these instructions indicates that developmental tests conducted by the con- tractor may be used in its place. Additionally, the official in the Navy’s Operational Test and Evaluation Force who is responsible for coordinat- ing operational testing of this program said that developmental testing in a laboratory is not an adequate substitute becauseit cannot fully sim- ulate true operational conditions; and according to the program’s test and evaluation master plan, one of the critical issuesto be resolved for this system is whether the integrated avionics system (i.e., the computer and sensors)can detect a submarine under actual mission conditions. Additionally, the contractor, overseenby the program office, has per- formed all developmental testing to date. An independent test activity has not yet conducted any testing, and Navy instructions require inde- pendent testing prior to a production decision. Moreover, we found that the Navy’s laboratory integration testing on the AN/ASN-150A avionics computer system has in fact not realistically simulated the strain the software will be under while actually tracking submarines. That is, the laboratory stress and performance testing has not subjected the avionics system’s software programs to the volumes of data that would be received simultaneously from three different types of sensorswhile tracking submarines under combat conditions. The rea- son is that the Navy does not have the capability to simulate all the sensorinputs, let alone simulate them simultaneously. Specifically, only data inputs from the acoustic sensor (device dropped from the helicop- ter into the water to detect underwater sounds) and the magnetic anom- aly detector (sensing device towed behind the helicopter to detect the magnetic density of submarines) have been simulated. The radar data inputs have not. According to program officials, the radar inputs have not been evaluated becauseit would be too costly to expand develop- mental testing beyond what is already planned. In our opinion, such lim- itations to integration testing are all the more reasonto ensure that Page 7 GAO/lMTECSo54 Navy Not Beady to Buy compntem for Ite Helko~tem existing computer system. This office did, however, require that opera- tional testing be conducted before deciding whether to deploy the modi- fied helicopter, a decision currently planned for March 1991. As stated earlier, we believe that the Office of the Assistant Chief of Naval Opera- tions (Air Warfare) decision to grant the extension of application and waive operational testing before a production decision was inherently risky and inappropriate. We also found that the Office of the Assistant Chief of Naval Operations (Air Warfare) did not ensure there was specified performance criteria in the AN/ASN-15OA development contract before allowing the program to proceed. Specifically, DefenseStandard 2167A, DefenseSystem Software Development, states that system specifications are provided to contractors as a requirements baseline for software development; and Navy Instruction 3960.10, Test and Evaluation, requires that software performance criteria be established and used as performance measures in testing. According to a Defensereport on software development problems, if such criteria are not specified, the cost, schedule,and qual- ity of the entire software development effort will bejeopardized.12 The Navy contracted for the LAMPS Mk I modification in February 1987 without a system specification that defined AN/ASN-15OA performance requirements. In lieu of such a specification, the software subcontractor used a preliminary specification, and the Navy began testing the software in September 1989 without such criteria. In December1989, the Navy finally incorporated the specification into the modification contract. However, these add-on requirements cost the Navy an extra $2.1 million, and causedthe modification program to slip 4 months. The Navy will soon face pivotal decisions on whether to buy more avi- Conclusions onics computer systems and begin modifying its billion dollar fleet of LAMFJS Mk I helicopters. These helicopters will heavily depend on their new avionics computer systems, which will ultimately cost the Navy $65.7 million to buy and install on these aircraft as part of a $896 mil- lion LAMPS Mk I modification program. Clearly, making such a decision requires, at a minimum, sufficient test- ing to know whether the integrated avionics system will perform as intended. In our opinion, this information does not currently exist, and “B of the Joint logistics Commanders Joint Policy coordinating Group on Computer Resource Management (Aug. 1979). page. 9 GAO/IMTEC~ Navy Not Ready to Ehxycompoten, for Ita Eelicopte~ As arranged with your offices, unless you publicly announcethe con- tents of this report earlier, we plan no further distribution until 30 days from the date of this letter. At that time, we will send copies to the Chairman, Senate and House Appropriations Committees; the Secretar- ies of Defenseand the Navy; and to other interested parties. We will also make copies available to others upon request. This report was prepared under the direction of Samuel W. Bowlin, Director, Defenseand Security Information Systems, who can be reached at (202) 2754649. Other major contributors are listed in appendix III. Ralph V. Carlone Assistant Comptroller General Page 11 GAO- Navy Not Ready to Bay Computem for Ita Helhpters Page 13 GAO/IMTEGgod4 Navy Not Beady to Buy Camptaem for Its Helifopters Appendix II Near-Term F’undingRequirementsfor Modification Kits In millions of dollars Fiscal Year Entire Kits AN/ASN-1SOA 1909 $45.0 $2.6 1990 55.3 2.6 1991 54.0 2.6 Is92 53.4 2.6 1gxl 40.8 2.6 1994 3.1 1.3 Total $287.4 514.3 Qoes not include funding requirements for spares Including spares. the cornblned fiscal yeal 1969/1990 funding requirements for the AN/ASN-1MA would be $6 6 million. Page 16 . Requests for copies of GAO reports should be sent to: U.S. General Accounting Office Post Office Box 6015 Gaithersburg, Maryland 20877 Telephone 202-275-6241 The Fit five copies of each report are free. Additional r~#&z:) .’ $2.00 each. There is a 25% discount on orders for 100 or more copies I (k I . _ _.- single address. Orders must be prepaid by cash or by check or money order tI v II r out to the Superintendent of Documents. ii’ United States First-Class Mail General Accounting Office Postage & Fees #T Washington, D.C. 20548 GAO -- Permit No. GlOO Official Busmess ,,.. Penalty for Private Use $300 - Appendix III Major Contributors to This Report John Stephenson,Assistant Director Information Randolph C. Hite, Assignment Manager Managementand Technology Division, Washington,D.C. Daniel R. Garcia, Regional ManagementRepresentative Philadelphia Regional Norman C. Berman, Evaluator-in-Charge Office William E. Petrlck, Evaluator (610678) PI@ 16 Objectives,Scope,and Methodology In October 1989, the Subcommittee on Legislation and National Security, House Committee on Government Operations, expressedinterest in the Navy’s plans to acquire embeddedcomputer systems for selectedanti- submarine warfare systems, and requested that we determine whether (1) the Navy plans to conduct adequate developmental and operational testing of these embeddedcomputer systems before buying them, and (2) Navy managementis overseeingthe acquisition of them. This request relates to an overall request from the Chairman and the Sub- committee’s Ranking Minority Member to review computer systems that are embeddedin Defenseweapon systems. To accomplish our objectives, we reviewed Defenseand Navy instruc- tions and standards governing the development, testing, and manage- ment oversight of embeddedcomputer systems for weapon systems. We also reviewed relevant program documentation addressing the acquisi- tion strategy, test plans and results, and schedule and funding require- ments for both the AN/ANSlSOA avionics computer system as well as the entire helicopter modification program. Additionally, we inter- viewed program officials responsible for managing various aspectsof the program, including the avionics computer system’s development and testing. We also interviewed Chief of Naval Operations officials respon- sible for overseeingthe program, and reviewed documentation relating to the discharge of this oversight responsibility. Further, we discussed test plans and results with officials from the Navy test activities involved in developmental (functional and integration) and operational testing, and we observed the execution of integration testing at the con- tractor’s facilities. We performed our work between October 1989 and March 1990, prima- rily at the LAMPSMk I program office within the Naval Air Systems Com- mand ln Arlington, Virginia, and the Naval Air Development Center in Warminster, Pennsylvania. We also visited the Navy’s Operational Test and Evaluation Force in Norfolk, Virginia, and the contractor’s labora- tory in Northridge, California. As requested by the Chairman’s office, we did not obtain official agency comments on a draft of the report. However, we discussedits contents with Navy and Office of the Secretary of Defenseofficials, and have incorporated their comments where appropriate. We conducted our review in accordancewith generally acceptedgovernment auditing standards. Page 14 GAO- Navy Not Ready to Buy Comp~~tem for Its Eeiicoptere Contents Letter Appendix1 Objectives,Scope,and Methodology Appendix11 Near-TermFunding Requirementsfor Modification Kits Appendix111 16 Major Contributorsto ThisReport Abbreviations GAO General Accounting Office IMTM: Information Managementand Technology Division LAMPS Light Airborne Multipurpose System the Navy’s plans for testing prior to its computer system purchase and initial production decisionswill not provide it. Further, the results of laboratory integration testing have revealed software discrepanciesof such magnitude that the Navy cannot justify going beyond the labora- tory test environment at this time, much less make a decision to buy more computer systems and initially produce the modified aircraft. Unless the Navy changesits approach to testing, discovery of AN/ASN- 16OAperformance problems wlll likely be delayed to a time when their correction will be more costly than necessary.In fact, estimates show that software problems found late in the development processcan cost six to ten times more to correct than if found early in the process.More- over, they may prevent the system from ever performing as originally intended. The Office of the Assistant Chief of Naval Operations (Air Warfare), the office responsible for overseeingthe avionics computer system’s devel- opment and testing, has not assured compliance with embeddedcom- puter system development and testing requirements becauseit agrees with the program office’s initial position that the computer system is a minor modification. In our opinion, this position is inappropriate, and attention by the Office of the Assistant Chief of Naval Operations (Air Warfare) to the program office’s approach to development and testing of the AN/ASN-16OA may prevent further deviations from Navy requirements. We recommend that the Secretary of the Navy direct the LAMPS Mk I Recommendations program office to defer buying the AN/ASN-160A computer systems until complete and thorough operational testing demonstrates that the avionics systems will satisfy mission requirements. We further recom- mend that the Secretary direct the Assistant Chief of Naval Operations (Air Warfare) to take a more active role in overseeingthe program and ensure that Navy requirements for AN/ASN-16OA operational testing are met. As requested by your offices, we did not obtain official agency com- ments on a draft of this report. However, we discussedits contents with Navy and Office of the Secretary of Defenseofficials, and have incorpo- rated their comments where appropriate. Cur work was performed ln accordancewith generally acceptedgovernment auditing standards, between October 1989 and March 1990. Page 10 GAO- Navy Not Ready to Buy Compmern for Its Hekopters operational testing precedesa decision to buy any more avionics com- puter systems to initially modify the helicopters. Laboratory integration testing conducted by the program test team in October 1989 revealed hundreds of software discrepancies.In February 1990, we observed retests and found that while some discrepancieshad been corrected, additional discrepancieswere found. As of March 1, 1990,260 of 636 cumulative discrepanciesstill were unresolved, and the Navy designated 20 of these as critical (i.e., they significantly degrade mission effectiveness). For example, data from one of the sensorscould not be processedand displayed quickly enough to permit targeting of a submarine. Additionally, the computer system did not have to contend with concurrent inputs from multiple sensorsduring this testing scena- rio. Such multiple inputs could exacerbatethis processingand display problem. On at least two occasions,Navy managementoversight authorities have Navy Oversight not acted to ensure that the LAMPSMk I modification complies with Navy Authorities Have Not system development and testing requirements. This inaction contributed Assured Compliance to cost increasesand schedule delays, and has allowed the program to assumean increased level of risk associatedwith not operationally test- With Requirements ing the system before a production decision. According to Office of the Chief of Naval Operations Instruction 5000.42, Research,Development, and Acquisition Procedures,the pro gram sponsorswithin the Office of the Chief of Naval Operations are responsible for providing overall direction to the program and assuring that Navy requirements for development and testing of embeddedcom- puter systems are properly implemented. The level of oversight respon- sibility is determined by the cost of the program and other considerations, such as development risks. For the LAMPS Mk I modifica- tion program, management oversight responsibility resides with the Assistant Chief of Naval Operations (Air Warfare). We found that the Office of the Assistant Chief of Naval Operations (Air Warfare) has elected not to focus on development and testing of the AN/ ANS-16OAavionics computer system becauseit was portrayed as a modification of an existing system. According to the official responsible for ntonltoring this LAMPSMk I modification, the AN/ASN-16OA com- puter system was exempted from operational testing by an extension of application becauseit was believed to be a low-risk modification of an Page 8 GAO- Navy Not Ready to Buy Complltus for Its Hekoptem B-238826 with the antisubmarine warfare mission, which were previously per- formed on the baseship. To provide for these new functions, 16 of the computer system’s 23 circuit board cardsJowere changed,and major changesto the software were made. In fact, only about 67 percent of the former system’s 370,000 lines of software code remained intact-7 per- cent, or about 26,000 lines of code, was changed;4 percent, or about 16,000 lines of code, was added;, and 22 percent, or about 82,000 lines of code,was deleted.” Additionally, the 7 percent of software that was changedperforms critical tactical functions such as m-flight processing of sensor data. Moreover, the Navy’s test plan describessystem software integration as a “critical issue.” Also, the results of recently completed integration testing (seenext section of the report) confirm that the modification is not a low-risk endeavor. The current program manager agreed with our assessmentthat the AN/ ASN-16OAavionics computer system is not a minor software modifica- tion. According to this official, the avionics computer system is a com- plex software development effort requiring extensive software integration. However, this official also stated that the current testing approach does not introduce enough risk into the program to justify postponing a limited production decision until after operational testing. This testing is currently scheduledto begin in September 1990. The pro- gram manager added that (1) funds have already been appropriated for buying 16 avionics computer systems and modifying 12 more helicop- ters, and (2) establishing the production line requires over 2 years, which ls sufficient lead time to correct any computer system problems discovered during operational testing. In our opinion, the mere availabil- ity of appropriated funds does not justify prematurely obligating them. Further, the sufficiency of lead time to correct computer system problems at this point in time is purely speculative, and cannot be accu- rately determined until complete and thorough testing disclosesthe extent of any problems. ‘°Ciit boards and cards are part of the computer system’s hardware. Colkctively, they provide the vital internal circuitry for the computer, and consist of the mounting boards and the components that are mounted on the bmrds. The cards are speck&M gmup of components that pmvide some spe ciaked function, such as central processing, internal memory, and display screen control and Fesolution. “The former system included software for processing data from a dipping sonar sensor, which is not being used on the LAMF’S Mk I. Therefore, this code was not needed. Acarding to the avionica system pmject officer, such changes in functionality account for the redwed number of lines of code in the modified system. Page 6 GAO- Naw Not Ready to Boy CYkmpoters for Its Helicopten, $66.7 million will be spent to buy 160 avionics computer systems.’ To date, the Navy has bought 12 of the new computer systems costing $5.3 million. Appendix II contains the funding requirements for the modifica- tion program, as well as the avionics computer system portion of the program, through fiscal year 1994. In June 1990, the Navy plans to decide whether to begin the limited pro- duction phase of the modification program. This entails deciding whether to buy 15 more computer systems (12 for immediate installa- tion and 3 as spares) costing $6.6 million, and begin installing these com- puter systems, along with new engines,rotor blades, and other avionics equipment on 12 aircraft. In March 1991, the Navy plans to decide whether to buy the remainder of the 160 computer systems and begin fully producing the modified helicopter. The Navy plans to decide whether to purchase 15 more AN/ASN-150A Navy’s Planned avionics computer systems and begin initially producing the modified Testing of the helicopter before operationally testing the system. By doing so, the Navy Avionics Computer is disregarding its own software development requirements, and greatly increasing the risk that operational problems-those that cannot be SystemIs Inadequate anticipated and tested for in a simulated environment-will not be dis- covered until later in the system development process,when the Navy will have modified an additional 12 helicopters. As a result, correcting these problems will be more difficult and costly than if they were found before any further computer purchases and helicopter modifications. Moreover, should the system fail to perform as required, the Navy may find that it has spent money unnecessarily. Computer system testing is incremental, with early developmental tests focusing on whether system componentsand subcomponentsperform the functions they are designedto do. Later developmental tests build on early tests, and addressthe ability of the components and subcom- ponents to perform their intended functions as integrated units. Still later developmental testing addresseshow well the integrated units per- form (i.e., how fast, how reliable, how accurate, how often) in a labora- tory that realistically simulates the stress the system will be under. Following developmental testing, the complete system is tested in a true operational setting, with actual users. This testing progression empha- sizesthe benefits of finding problems early in the development process ?Estimate based cm the 103 helicopters to be modified plustheprogram manager’s estimate of another 47 computers asspares for helicopter maintenance and tmining purpcees. Page 4 GAO/lM’IECSO44 Navy Not Beady to Buy Compoters for Its Helicopters objectives were to determine whether (1) the Navy plans to conduct ade- quate developmental2and operational3 testing of the helicopters’ embed- ded computer systems before buying them, and (2) Navy managementis overseeingthe acquisition of these embeddedcomputer systems.A detailed explanation of our objectives, scope,and methodology is con- tained in appendix I. In June 1990, the Navy plans to decide whether to buy $6.6 million Resultsin Brief worth of upgraded avionics computer systems as part of a program to modify 12 LAMPS Mk I helicopters. These upgraded systems, although initially thought to only require minor modifications to a system already approved for another helicopter, actually require extensive software and hardware changes.However, the Navy has chosennot to operation- ally test the system before its June 1990 decision. In our opinion, omitting operational testing greatly increasesthe risk that system performance problems will not be detected and resolved until later in the development process.History has shown that the longer a software problem goesundetected, the more expensive it is to fix, assuming that it can be fixed. Developmental testing done to date on the new avionics computer sys- tem revealed significant software problems that cast doubt on whether the Navy should proceed until the problems are corrected. The program manager agreed that operational testing should precede a production decision, but the Navy is faced with the need to quickly obligate funds appropriated by the Congressfor fiscal years 1989 and 1990. We believe that the Navy should not buy any more avionics computer systems until it has completed operational testing. The Office of the Assistant Chief of Naval Operations (Air Warfare) has management oversight responsibility for the modification program. However, this office has not assured compliance with development and 2Defense Directive 6ooO.3.Test and Evaluation, defines developmental testing as testing associated with the engineering design and development of a system to verify attainment of performance and functional specifications, objectives, and supportabiity. It adds that this tasting addresw system components and their integration, and involves the use of models, simulations, and testbeds. aDefense Diredive 6000.3, Test and Evaluation, defmes operational testing as field testing, under realistic omditions, of any item or component of a weapons system, to determine its effectiveness and suitability for combat use. It further states that this testing is to be conducted in an envimnment that is operationally ralistic, including having typical users operate and maintain the system in a settiig representative of combat stress conditions. page 2 GAO/IbfTBG6044 Navy Not Beady to Buy compnters for Its Helicopters
Embedded Computers: Navy Not Ready to Buy Avionics Computers for Its LAMPS Mk I Helicopters
Published by the Government Accountability Office on 1990-05-31.
Below is a raw (and likely hideous) rendition of the original report. (PDF)