linited States General Accounting Office Report to the Secretary of Defense GAO May 1990 WEAPONS TESTING DOD Needs to Plan and Conduct More Timely Operational Tests and Evaluation ’ ___---- GAOi’NSIAJNO-107 . , . united StateB GAO General &comting Offke Washington, D.C. 20648 National Security and Intemationai Affairs Division B-222886 May 17,1990 The Honorable Richard B. Cheney The Secretary of Defense Dear Mr. Secretary: The Congress continues to express concern that weapon systems have begun production without operational test and evaluation (d&E). or&E conducted before a system’s “production start-up,“’ is a key internal control to ensure that decisionmakers have the best information availa- ble on a weapon system’s performance to minimiz risks of procuring costly and ineffective systems. We identified many perceived barriers to earlier or&~. Several had been previously reported by us and other agencies, and others were identified during discussions with officials in the Office of the Secretary of Defense (OSD)and the military services. This report provides the results of our analysis of the feasibility of overcoming three major barriers that we believe will allow or&E before production start-up. The three barriers were l lack of prototypes for testing early, l starting production before development is completed-(concurrency), and l the need to start production to obtain or&~ test articles2 We recognize that other factors, such as cost, urgency, and the state of advanced technology may sometimes preclude urr~ before production start-up. However, making ora results available before production start-up could help preclude cost growth, schedule slippages, and per- formance shortfalls that frequently arise during the later phases of a weapon system’s development. Several events make the present systems acquisition environment very receptive to earlier or&~. These include (1) recent DODreports that espouse the “aggressive use of prototyping and testing ro identify and remedy problems before production,” (2) DOD’Srevisior.5 to acquisition and test and evaluation regulations current!y underway, (3) the ‘In ths report, we use the term production start-upto referto thatpointinthe acqtusition process at which a weapon system or its subsystems begin production. “Low-rate initial production [LRIP],” “limited production,” and “pilot pmductlon” are terms that the Depatunent of Defense (DOD) uses to refer to production start-up. ‘Test articles are the units of a weapon system that are used to test a weapon’s performance. Page1 GAO/NSlAD90107 OT&E 5223886 appointment of a new Director of ora, (4) the need to make critical tradeoff decisions in systems’ acquisition, and (5) DOD’S agreement, in response to our May 1989 report discussed on page 4, to reemphasize the need for or&~ as early as possible in the acquisition cycle. Several DOD and congressional initiatives to reduce risks in system acqui- Results in Brief sition and many reports have highlighted weapon system performance and testing shortfalls. Our review of six current systems plus other recent work shows that DOD has made little progress in assuring that earlier UN.E is planned and conducted. The military services generally are not conducting or planning to conduct CT&E on weapon systems until after production start-up. In our view, the barriers we examined do not preclude earlier CT&E.With proper planning, an operational test can and should be accomplished before production start-up.3 We found that although some prototypes have been used for development test and evaluation the military ser- vices generally are not planning to use them for or&~ before production start-up. We believe, however, that with adequate planning, prototypes can be designated for or&~. We also found that concurrency does not preclude earlier OTBE,if such testing is properly planned. Moreover, because test articles are available during the full-scale development phase of the acquisition process, it is generally not necessary to begin production to provide them for CT&E. Background DOD budgets over $100 billion annually for research, development, test, and evaluation and for production of major weapon systems. In view of such large sums, decisionmakers need the best information possible on weapon systems’ operational performance and risks associated with new technology before approving production start-up. CT&Eis intended to provide an independent analysis of a weapon system’s performance against operational requirements and the system’s reliability and ability to be supported in the field. %RkE before producuon start-up may be m addition to later CT&E conducted to support a full-rate producuon decision (often referred to as the “final exam”). m&E before produmon start-up i 11may be on a system or critical subsystem prototype, (2) might not meet alI requirements for effectweness or suitability, and (3) should not be viewed as a “pans/fail” test but can serve as an input to the design procw to ensure problems are addressed before a system begins production. Page 2 GAO/‘NSLA.D9@107 URkE The services are statutorily required to conduct CT&Ebefore making a full-rate production decision. DOD’scurrent test and evaluation poliw not only reflects this, but also states that “UNE shall be structured to provide inputs at each decision point, including major milestones.“5 Yet, DOD’S policy also permits production start-up before or&E is conducted. Studies of the DOD acquisition pn>cess have recommended earlier opera- tional testing, using prototypes, before production start-up. In 1986, the President’s Blue Ribbon Commission on Defense Management (the Pack- ard Commiss’ron) recommended that operational testing of prototype systems begin early in advanced development and continue through full- scale development. DOD’S recent Defense Management Review initiatives, as reflected in the July 1989 DOD Report to the President on Defense Management and the January 1990 DOD Annual Report to the President and the Congress support implementation of the Packard Commission recommendatiod, including aggressive use of prototyping and testing to identify and remedy problems “well prior to commencement of high rate production.” Over the years, we have reported on weapon system performance short- With Proper Planning, comings and the lack of adequate or&~. (See app. III.) Our work on six OT&E Before current systems plus our other recent work (see apps. I and II) has Production Start-Up Is shown that CT&Eis not being conducted until production start-up has been approved. We understand that the military services want to move Feasible weapon systems into production as quickly as possible in order to reduce the likelihood of them being canceled-a “build now and fix problems later” attitude. We believe that crr%~results have not been available to support decisions to start production because of the failure of the military services to plan for earlier CR&E and incorporate it into program acquisition strategies. The barriers often cited for or&~ not being adequately planned for or conducted before production start-up has been approved include (1) prototypes not available for testing early, (2) concurrency, and (3) the need to start production to obtain ULQE test articles. We believe that these barriers do not preclude OT&E prior to production start-up. 4~ DOD Directive 5000.3, Test and Evaluation, dated March 12,1fB6. “he milestones are: I (to dart the concept demonstration/ validation phase), II (to enter the fulLscale development phase), and III (to enter the production phase). Page 3 GAO/‘NSIAD~107 d&J3 In a recent report on test and evaluation,6 we discussed the manner in which DOD used early operational assessments. Early operational assess- ments provide input to decisionmakers on whether a weapon system is ready for the “final exam or&~” and whether operational shortfalls are being identified and corrected early in the development process. These assessments, which usually rely on development test data, are being used to support decisions for full-scale development and production start-up. We reported in May 1989 that DOD’S reliance on early opera- tional assessments when equipment is unavailable for or&~ is a step for- ward in filling the information void, but should not be a substitute for Ur&E. However, we support DOD’S current policy that calls for UME at each decision point, including major milestones, and believe that more emphasis should be given to conducting cm&~before production start-up. This or&~ should not be viewed as a pass/fail test; it would function as an input to decisionmakers to ensure that problems are identified before a system begins production. Prototypes Are Being Built Cur review has shown that even though prototypes have been built or but Not Being Used for are under construction for some systems, the services generally do not plan to use them to conduct or&~. For example, prototypes were built or OF&E under construction for five of the six systems we reviewed (the Air Force’s Advanced Tactical Fighter and Short Range Attack Missile-II, the Army’s Non-Line-Of-Sight Missile,and the Navy’s bng-Range Air Anti-Submarine Warfare Capability Aircraft and Fixed Distributed Sys- tem). However, only the Navy used or has plans to use its prototypes in or&~ before production start-up. This is because only the Navy has regu- lations calling for or&~ before production start-up. The other services’ regulations only require or&~,to support full-rate production decisions, although some field testing or experimentation may be conducted. The Navy used the Fixed Distributed System prototype to conduct or&E before production start-up. Although the Fixed Distributed System was not a mature system with a prototype available for only one of its two major subsystems, the Navy considered the or&~ results to be useful. This CJT&Ewas conducted before full-scale development. We recognize that not all systems can or should have prototypes. How- ever, o’r&~ conducted before the decision to begin production could be performed on critical subsystems if it is not feasible to have a prototype ‘Navy Weapons Testing: Defense Policy on Early OperationalTestmg (GAO/NSIAD-89-98, Maya. lf=Q), Page4 GAO/NSIMMO-107 umE of the full system. The controlling factors would be system/subsystem maturity and the expected risk. For earlier ONE to be conducted, acquisition strategy from the beginning should require that prototypes be built and operationally tested before the decision to begin production is made. This is the responsibility of acquisition managers, not operational testers. Concurrency Does Not As we have previously reported,’ concurrency can be an effective tech- Preclude OF&E Before nique for expediting acquisitions if it is well planned and controlled; however, the practice increases the risk that systems will be produced Production with major flaws. ONE before production start-up is an especially impor- tant safeguard against the increased risks of concurrent programs. On April 17,1990, DOD reported to the Congress that its policy allows varying degrees of concurrency, but mandates the completion of testing and the assessments of results before the production phase.6 Proposed concurrency guidelines call for the aggressive use of prototypes and testing to identify problems well before production. Under normal cir- cumstances, the low-rate initial production decision should be supported by the completion of some early UWE. Most DOD officials we talked with concerning earlier or&~ believe that it is feasible to conduct oral before most production start-up decisions. However, our discussions identified several factors that need to be con- sidered when CT&Eis to be conducted before the production start-up decision: . d&E must be planned as part of the system’s acquisition strategy. . Systems tested early may not meet ail requirements for suitability (i.e., reliability, availability, and maintainability). . Decisionmakers need to be told that or&~ earlier in the acquisition pro- cess should not be viewed as pass/fail; it should be viewed as a way to identify system performance shortfalls that need to be corrected before production begins. With my L&n&d Operational Test and Evalua- ‘As requested by section SO1of the National Defense Authorization Act for Fiscal Years 1990 and 1991,DODwastoreporrontherisk~withconcurrencyin~racquisitionp~ams. P8ge 6 GAO,‘NSLMM@1o7 oT&E B-222886 We believe that these factors are also important to systems that are not categorized as concurrent programs. Production Start-Up Is Production start-up is traditionally considered as a means by which test Generally Not Necessary articles are acquired for or&~. The military services usually use the term low-rate initial production to describe this. However, the Congress and for OT&E Test Articles DOD'S Inspector General have expressed concern that the services’ use of low-rate initial production has sometimes resulted in de facto full-rate production before any or&~ is conducted. To try to control the misuse of low-rate initial production, in November 1989, the Congress passed leg- islation requiring an appropriate DOD official to specify the quantities of test articles needed for or&~ in major programs to decrease the number of items produced before UIXE was conducted. We found that production start-up is generally not necessary to produce or&~ test articles. The DOD budget manual and the military services’ reg- ulations require or at least allow or&E test articles to come out of full- scale development. In addition, we identified many systems currently under development or in production where the services plan to use or have used or&~ test articles produced from full-scale development. (See app. I.) In only one of the six systems we reviewed will the UWE test articles come from a low-rate initial production decision. In that case, according to the Deputy Program Manager, the Lineof Sight-Forward- Heavy was already in production for an allied country before the U.S. Army bought it. Conclusions and Weapon systems are starting production before ora is conducted because DOD is not giving full effect to its policy for d&E at each decision Recommendations point, including major milestones. On numerous occasions we have reported the negative effects of systems that have been produced with- out adequate or&~. The barriers to earlier CWE can be overcome with better planning. To strengthen DOD'S recent reaffirmation of the proper role of CT&E as a key internal control in the systems acquisition process made through the Defense Management Review initiatives, we recommend you ensure that DOD acquisition and testing directives currently under revision l clearly establish the need for or&~ before production start-up; Page g . define when or&~ must occur and when it may be appropriate for deci- sionmakers to rely on operational assessments in lieu of 68tE for produc- tion start-up decisions, as previously recommended in our May 1989 mpofi; . require the services to plan for and conduct earlier or&~; and . ensure thatacquisition strategies for major weapon systems require sys- tem or subsystem prototypes be built where practical and that these prototypes be operationally tested before production start-up. To determine whether perceived barriers precluded early or&~, we per- Scopeand formed work at test and program oversight offices within CISD, military Methodology service headquarters, and operational test agencies, and applicable pro- gram executive and management offices. On the basis of a judgment sample drawn with direct OSD and service input, we examined six cur- rent systems and reviewed our prior and recently reported work on additional systems. (See apps. I, II, and III.) We also performed an exten- sive literature search and review of available documents on defense acquisition, testing, prototyping, and concurrency. We did not obtain written agency comments on this report. However, we discussed the information in this report with DOD officials and incorpo rated their comments as appropriate. DOD officials were very receptive to our findings and conclusions and believed that systems acquisition changes would be helpful to the or&~ community. They pointed out that Defense Management Review initiatives and the Under Secretary of Defense for Acquisition are stressing the need for earlier m&E. We conducted our work from August 1989 through February 1990 in accordance with generally accepted government auditing standards. As you know, 31 USC. 720 requires the head of a federal agency to submit a written statement on actions taken on our recommendations to the House Committee on Government Operations and the Senate Com- mittee on Governmental Affairs not later than 60 days after the date of the report. A written statement must also be submitted to the House and Senate Committees on Appropriations with the agency’s first request for appropriations made more than 60 days after the date of the report. We are sending copies of this report to the Chairmen, House Committees on Government Operations and on Armed Services and Senate Commit- tees on Governmental Affairs and on Armed Services; the Secretaries of Page 7 GAo,msL4D-90107 cm&E _ . Iv223333 the Army, Air Force, and Navy; the Director, or&~; the Deputy Director for Defense Research and Engineering/Test and Evaluation; the Direc- tor, Office of Management and Budget; and the military service opera- tional test agencies. We will also make copies available to other interested parties upon request. This report was prepared under the direction of Mr. Paul F. Math, Direc- tor, Research, Development, Acquisition, and Procurement Issues, who may be reached on (202) 27543400 if you or your staff have questions concerning the report. Major contributors to this report are listed in appendix IV. Frank C. Conahan - Assistant Comptroller General Page 8 GAO/?WAD90107 d&E Page 9 Contents Letter Appendix I 12 Cn’&ESchedulesfor Six WeaponSystems We Reviewed Appendix II Examples of Systems Noted in Our Recent Reports That Did Not Have OT&E Planned Before Production Start-Up Appendix III Examples of Reports Illustrating Where Production Decisions Were Made Before OT&E Was Started or Completed Appendix IV 18 Major Contributors to This Report Abbreviations DOD Department of Defense OSD Office of the Secretary of Defense CT&E operational test and evaluation RWR radar warning receivers Page 10 GAO/‘?USIAD9@l(yI d%E Page 11 Appendix I UP&E Schedulesfor Six WeaponSystems We Reviewed Fircrl yams lnithl m&z plod- source of ol6E teat Senfice/rystem Program 8tut decision mlcle3 AmY Lrne-of Srght-Forward-Heavy (Air Defense Antf- 1987 1990 1987b Low-rate tntttal productron Tank System) Non-Ltne-of-SrghtC (Fiber Opttc Gutded Missile) 1987 1994 1991b Full-scale development Air Force Advanced Tacttcal FrghterC 1981 1997 1994D Full-scale development Short-Range Attack Misstle-IP 1985 1992 1991b Full-scale development Navy Long-Range Anti-Submanne Warfare Advanced 1986 1992 1992b Full-scale development Capabtkty AIrcraW (P-7A) Fixed Dtstnbuted SystemC 1984 19W 19W Demonstratton/valrdatton ‘Without extensrvely revbewlng additlonal systems, we noted that several others used full-scale develop- ment stems for producmg OTIE test articles. These were MK-48 Advanced Capablrty Torpedo, Extremely Low Frequency Ccmmunicatton System, ANSCBS3C Long-Range Sonar, and Arrbcrne Setf- Protection Jammer “For low-rate inftral production ‘Prototypes have been built or planned for these systems. However, the servrces are not ptannrng to use the Non-Lrne-cf-Srght Missile. Short-Range Attack Missile-Il. and Advanced Tactrcal Ftghter protc types for OTLE. “An OTLE, such as the one descrbed In footnote 3 on page 2. was successfully conducted P8ge 12 GAO/‘NSlADB@1O7 OT’&E Appendix II Examplesof SystemsNoted in Our Recent ReportsThat Did Not Have oT&E Planned Before Production StarkUp Abrams Tank: Block II Modifications Not Ready for Production (NSIADOO-67, Feb. 28, 1990) . Army will commit %166.4 million to the MlA2 tank program before development prototype testing is begun. . OSD had not approved the Army’s test and evaluation master plan as of August 1989 because it was concerned that the Army had not suffi- ciently planned live-fire testing and or&~, which needs to be completed and evaluated before the production decision is made in August 1991. Strategic Bombers: B-2 Program Status and Current Issues (NSWQO- 120, Feb. 22,199O) l B-2 aircraft acquisition strategy includes ordering a large number of planes before the necessary testing to demonstrate that the ES-2can per- form its mission is completed. ICBM Modernization: Rail Garrison Production Decision and Launch Car Acquisition Should Be Delayed (NSLAD 90-19, Dec. 7, 1989) . No or&~ of the complete weapon system (missiles and rail cars) will have been conducted prior to the initial production decision. . We recommended that the Secretary of Defense delay the April 1990 initial Rail Garrison production decision until the Air Force has con- ducted some or&~ of the complete system. Navy Weapons Testing: Defense Policy on Early Operational Testing (NSL~D-~~-98, May 8, 1989) The following seven programs were authorized for low-rate initial pro duction with no m&E: . ALQ-165 Airborne Self-Protection Jammer. l SQS-53C Sonar. l E6A Aircraft. l MK-45 Capsule Launching System. . Ocean Surveillance Information System Baseline Upgrade. l TB-23 Accelerated Thinline Towed Array. l AN/B%‘-2 Submarine Combat System. P8ge 13 GAO,‘TUSIAD(IOlM CT&E Examplea of Syatema Noted In Chu Recent Reporta That Did Not Ibe OT&E Planned Before Reduction start-up Navy/Air Force Still Developing Separate, Costly Radar Warning Receiv- ers - (~~~~-87-167, July 1, 1987) 0 Five radar warning receivers (RWR) started production before CT&E was completed. . Starting production before adequately testing the systems has resulted in the purchase of equipment that cannot be used for its intended pur- pose, production of RWR that were placed in storage rather than installed, and deployment of RWRjudged operationally unsuitable to U.S. combat forces by testing officials. Page 14 GAO/‘NSIAD96107 OTBE Examplesof Reports Illustrating Where Production DecisionsWere Made Before OT&E Wa Started or Completed Operational Test and Evaluation Can Contribute More to Decisionmak- % (NSIAD-W-67, Dec. 23, 1986) Our analysis showed 41 cases where production was approved before or&~ was started or completed. Thirty-one of the 41 cases were identi- fied in our reports and summarized in this December 23,1986, report. The 31 reports are listed below. 1. Adverse Effects of Large-Scale Production of Major Weapons Before Completion of Development and Testing (B-163068, Nov. 19, 1970) 2. The Importance of Testing and Evaluation in the Acquisition Process for Major Weapon Systems (B-163068, Aug. 7,1972) 3. Review of the Adequacy of Department of Defense Test Resources (~~~~76-84,Apr. 30,1975) 4. Effectiveness of Testing of Selected Major Weapon Systems (FSALI-76-74,June 4,1975) 6. Need for Additional Test and Evaluation on the Major Caliber Light- weight Gun (PSAD~~, Nov. 6,1976) 6. Navy Operational Test and Evaluation - A Valuable Tool Not Fully Utilized (~~~~7877, Mar. 29,1978) 7. Operational Testing of Air Force Systems Requires Several Improve- ments (FSAD7B102, June 2,1978) 8. Department of Defense’s Conduct of Operational Test and Evaluation of Foreign Built Weapon Systems (~~~79-131, July 25, 1978) 9. Practices and Procedures for FollowOn Operational Testing and EvaI- uation of Weapon Systems by the Military Services (~~~79-1, Oct. 19, 1978) 10. Army Operational Test and Evaluation Needs Improvements (PSAD-W~,Nov. 13, 1979) 11, Ml Tank’s Reliability is Still Uncertain ( ~SAD-M-20, Jan. 29, 1980) 12. F/A-18 Naval Strike Fighter: Its Effectiveness Is Uncertain (~~~~80-24, Feb. 14, 1980) P8ge 15 GAO/m3L4B~lo7 or&E Ehmplea of Reporta Ill~tiag when RoductIon xkcblon8 were Made Before ONE WM Started or Completed 13. Cruise Missiles: Status and Issues as They Near Production (I~AD-M-19,Feb. 28, 1980) 14. Future Procurements of Army’s Copperhead Projectile Should Be Contingent on Imnrovements in Performance and Reliabilitv (ps~~81-4. Nov. 13,198O) * 15. Review of the High Speed Anti-Radiation Missile Program (MASAD~~-7,Feb. 28, 1981) 16. Some Land Attack Cruise Missiles Acquisition Programs Need to Be Slowed Down (MA&D-s 1-9, Feb. 28, 1981) 17. Most Critical Testing Still Lies Ahead for Missiles in Theater Nuclear Modernization (MASADS1-16, Mar. 2,198l) 18. The Armv’s Advance Attack HehcoDter Is Not Readv for Production (MASAD-S~~, Dec. 1,198l) 19. Air Launched Cruise Missile Shows Promise but Problems Could Result in Operational Limitations (C- b1~sm-82-13, Feb. 26,1982) 20. Promzss of the Light Armored Vehicle Program Should Be Closelv Monitored ( ~~~~-82-41, Aug. 10,1982) 2 1. Results of Production Testing Should Be Considered Before Increas- ing Patriot’s Production ( MASADS~-7, Jan. 26,1983) 22. Acquisition of the Over-The-Horizon Backscatter Radar System Should Be Reevaluated (MATADOR- 14, Mar. 15,1983) 23. The B-l Bomber Program - A New Start ( ~~~-93-21, Apr. 13, 1983) 24. Better Planning and Management of Threat Simulators and Aerial Targets Is Crucial to Effective Weapon Systems Performance (M.kSADB-27, June 23,1983) 25. Air Force and Navy Trainer Aircraft Acquisition Programs (MASADS~-22, July 5, 1983) 26. Results of Forthcoming Critical Tests Are Needed to Confirm Army Remotely Piloted Vehicle’s Readiness for Production (NSL4D-84-72, Apr. 4, 1984) Page 16 GAO/‘NSL4B90-107 OI’&E Eumpla of ReportanlMlx8tlIlgwhere Prodoetioll Deciai0M were Made Before UME Wu Stand or Completed 27. Status of the Peacekeeper (MX) Weapon System (NSLAD8e112, May 9, 1984) 28. Army’s Decision to Begin Production of the High Mobility Multipur- pose Wheeled Vehicle Was Premature (NSIAD-SM~~, June 12,1984) 29. Army Has the Opportunity to Recompete DAS3 Purchases and Improve Automated Battlefield Support ( IIITEGW20, Sept. 28, 1984) 30. Production of Some Major Weapon Systems Began With Only Limited Operational Test and Evaluation Results (NS-, June 19, 1985) 31. Evaluation of Army’s Mobile Subscriber Equipment Program (NSIAD-86-117,July 16,1985) Page 17 GAO/NSLUM@lU7 UT%E Appendix IV Major Contributors to This &port Michael E. Motley, Associate Director National Security and Lester C. Farrington, Assistant Director International Affairs Maureen Machisak-Hemdon, Evaluator Division, Washington, D.C. Fred Harrison, Regional Management Representative Norfolk Regional Leslie Gregor, Evaluator-in-Charge Office Clifton Spruill, Site Senior Julie Chapman, Evaluator Jim Marshall, Advisor Ted Baird, Regional Management Representative Denver Regional Ernest Beran, Regional Assignment Manager Office P8ge 18 GAO/‘NS~~107 OTSE There is a 33”,, diwt)unt OII ordt~rs ft,r 100 or mow copies rnaibd to a \inglt* address. United States General Accounting Office Washington, DC 20548 1 Permit No. GlOO Official Business Penalty for Private Use $300
Weapons Testing: DOD Needs to Plan and Conduct More Timely Operational Tests and Evaluation
Published by the Government Accountability Office on 1990-05-17.
Below is a raw (and likely hideous) rendition of the original report. (PDF)