GAO United States General Accounting Office Washington, D.C. 20548 National Security and International Affairs Division B-229489 September 28, 1990 The Honorable Les Aspin Chairman, Committee on Armed Services House of Representatives Dear Mr. Chairman: As you requested, we have reviewed thestrategy for acquiring the Short-Range Unmanned Aerial Vehicle (UAV) system. Because of your Committee’s concern about the lack of success on past UAV programs and the need to assure that related acquisition weaknesses do not recur, we compared the Short-Range UAV acquisition strategy to that folloyed in acquiring the Pioneer UAV, a previously procured system which encoun- tered problems when deployed with the Navy’s operational forces. The Short-Range UAV program, expected to cost $1.5 billion, was initi- Background ated as a joint-service effort in response to congressional concern about the proliferation of UAV programs and the need to acquire UAV systems that could meet the requirements of more than one service. In this regard, Congress refused to authorize fiscal year 1988 funding for pro- curement of separate Army and Navy Short-Range UAVS and provided funds only for a joint program. Accordingly, the Department of Defense (DOD) is acquiring the Short-Range UAV to meet the needs of the Army, Navy, and Marine Corps. DOD plans to begin low rate production of the Short-Range system in fiscal year 1992 with the procurement of 8 of a total program quantity of 53 systems. Each system to be procured initially is to include 8 air vehicles, and the 64 vehicles are to be deployed in 1994. DOD'S acquisition strategy for the Short-Range UAV is to perform opera- Results in Brief tional testing before beginning low rate production. Despite the impor- tance of realistic operational testing, however, the Short-Range UAV acquisition strategy provides for testing in an environment not represen- tative of where the system is supposed to be deployed. This strategy places DOD at risk of becoming committed to the production of a system based on test results that may not be a valid indicator of the system’s capability. The strategy is similar to Pioneer’s strategy that resulted in spending $160 million for a system that still does not meet performance requirements. Page 1 GAO/NSIAlMO-234 U- ed Aerial Vehicles IS229489 DOD'S acquisition policies require that operational testing be held in a realistic environment representing combat conditions to the extent prac- tical. Nevertheless, DOD plans to limit the Short-Range system’s preproduction operational testing to areas that do not provide environ- ments typical of where current deployment plans indicate the system could be used in combat. Most of this preproduction testing will be lim- ited to a desert environment. We recognize that if the current Middle East situation continues until 1994, when the system is to be deployed, the Short-Range system deployment plans could be changed. However, the system’s currently required capability of locating targets under diverse conditions in multiple environments is not likely to be demon- strated before DOD becomes committed to its production. DOD also plans to begin full-rate production of the Short-Range system before verifying that it can be modified to meet Navy requirements. Although DOD plans to conduct operational testing before full-rate pro- duction, the system to be tested will not incorporate those features nec- essary to operate in a naval environment. If subsequent testing of the Navy variant were to show the system to be unsuitable for naval use, DOD would then be fully committed to a system not meeting congres- sional intent for a common-service system. DOD'S prior experience with the Pioneer system demonstrated the difficulties in adapting a UAV for naval use and the need for timely testing. Other shortcomings in the planned Short-Range system test program were identified by us and the Director, Defense Operational Test and Evaluation. When these shortcomings were brought to DOD'S attention, it agreed to correct them. DOD'S Short-Range UAV acquisition strategy includes preproduction oper- Recommendations ational testing, and DOD'S initially procured systems represent a signifi- cant portion of the total program quantity and are to be deployed to operational forces. We therefore recommend that the Secretary of Defense require that operational testing of the Short-Range UAV be con- ducted in diverse, realistic environments to provide reasonable assur- ance that it will meet requirements before permitting limited production of the land-based UAV system. We also recommend that the Secretary limit Short-Range UAV system production until satisfactory performance of the Navy variant is demonstrated and assure that actions are taken to correct the other shortcomings noted in the planned test program. Page 2 GAO/NSIAMlMS4 UNMNI ed Aerial Vehicles B-229489 Congress may wish to consider whether the congressional intent for a Matter for joint service system is jeopardized by the substantial risk that the Navy Congressional variant will be unsuitable, and, if so, Congress may wish to prohibit full- Consideration rat,e production of the Short-Range UAV until performance of the Navy variant is proven to be satisfactory. DOD agreed or partiallyagreed with most of the findings in this report Agency Comments and but disagreed with the recommendations and matter for congressional Our Evaluation consideration. DOD stated that an adequate evaluation of the system’s operational effectiveness and suitability can be accomplished without testing in all environments in which the system may be employed. DOD also stated that the system’s acquisition strategy and test program are consistent with the applicable DOD directive and that the system will be tested in representative operational environments. DOD indicated that the acquisition risks are inherently low because of the nondevelopmental nature of the program. DOD also stated that the risk would be further reduced by the test program for the Army and Marine Corps version, by engineering evaluations and design reviews of the Navy variant, and by preproduction testing of the Navy variant. Our basic concern is that the primary test site does not resemble the primary locations where the system is to be operationally deployed beginning in 1994 and therefore the test results could be very mis- leading. In fact, DOD'S restructured test plan recognizes that the testing environment constitutes a test limitation. Our past work has often demonstrated the importance of testing in a realistic operational environment, and DOD’S acquisition directive requires that when operational testing is conducted, it be conducted under realistic conditions. DOD’S comments present no evidence that negates the need to realistically test the system before committing to production. DOD also stated that it has no intention of delaying production of the system for the Army’s and Marine Corps’ use pending outcome of the Navy variant’s development. Finally, DOD pointed out that operational testing of the first Navy variant procured and other measures, such as design reviews, would sufficiently minimize the risk associated with the current acquisition strategy. Page 3 GAO/NsIAIH&294 U- ed Aerial Vehiclea B-229489 Under the current acquisition strategy the scheduled testing of the Navy variant will only reduce the risk of prematurely committing to produc- tion of that variant, rather than the basic system which will by then already be in full-rate production. If the Navy variant were to prove unsuccessful in the later testing, DOD would find itself fully committed to production of a system not meeting congressional intent for a common UAV. Experience with Pioneer shows that adapting a UAV for naval opera- tions is not a low-risk effort even though the Navy categorized it as such when justifying the program. We therefore continue to believe that DOD should retain the Short-Range system in low-rate production until opera- tional testing shows that the Navy variant’s performance is satisfactory. DOD refers to the Short-Range UAV as a nondevelopmental program. The Pioneer was also a so-called nondevelopmental program in its early phases but later required substantial development to solve problems associated with naval operations. In addition, testing of the Army and Marine Corps version of the Short-Range UAV will not reduce risk associ- ated with the Navy variant because the Army and Marine Corps version will not incorporate any of the modifications necessary for operating in a naval environment. Engineering evaluations, design reviews, and preproduction testing of the Navy variant should help preclude a pre- mature commitment to production of that variant. However, these efforts will not reduce the risk of becoming prematurely committed to the Short-Range program as a whole because the Army and Marine Corps land-based version will be in full-rate production by the time these efforts are completed. Appendix I discusses the Short-Range UAV acquisition strategy in more detail. Appendix II describes our objective, scope, and methodology. Appendix III sets forth DOD comments. Copies of this report are being sent to the Secretaries of Defense, the Army, and the Navy; the Director of the Office of Management and Budget; selected congressional committees; and other interested parties. Copies will be made available to others upon request. Page 4 GAO/T’BIAD9@234UNMNI ed Aerial Vehicles This report was prepared under the direction of Louis J. Rodrigues, Director, Command, Control, Communications, and Intelligence Issues, who may be reached on (202) 275-4841 if you or your staff have any questions concerning the report. Other major contributors to this report are listed in appendix IV. Sincerely yours, Frank C. Conahan Assistant Comptroller General Page 5 GAO/NsLU.MO-m U- Aerial Vehiclea Contents Appendix I 8 Evaluation of DOD’s Background Production Before Realistic Operational Testing 8 9 Acquisition Strategy Congressional Intent for a Joint-Service System Is at Risk 12 for the Short-Range Some Shortcomings of Original Test Plans Addressed 12 UAV Appendix II 14 Objective, Scope,and Methodology Appendix III 15 Comments From the GAO Comments 25 Department of Defense and Our Evaluation Appendix IV 29 Major Contributors to This Report Abbreviations DOD Department of Defense GAO General Accounting Office UAV Unmanned Aerial Vehicle P-6 GAO/-NSIABBO-234 Unmanued Aerial Vehlclea Page 7 GAO/NSIAD90-234U- ed Aerial Vehicles Evaluation of DOD’sAcquisition Strategy for the Short-RangeUAV are pilotless aircraft resembling small airplanes or helicopters that Background UAVS are remotely controlled or preprogrammed to be controlled by on-board equipment. DOD is acquiring UAvs to meet a VXi&y of military needs. A UAV system typically includes one or more air vehicles, a launch and recovery system, and a ground station for controlling the UAV’S flight and processing information from the UAV. The air vehicle carries the system’s payload, such as a television camera, with the specific type depending on the military mission to be accomplished. As part of its joint UAV program, DOD is acquiring the Short-Range system for use by the Army, Navy, and Marine Corps. The system is to provide the capability to perform reconnaissance, surveillance of enemy activi- ties, target acquisition, and other military missions. Initiation of the Short- The joint-service Short-Range program was initiated in response to con- Range Program gressional concern about the proliferation of UAV programs and the need to acquire UAVS that could meet the requirements of more than one ser- vice and eliminate duplicative programs. To achieve commonality among the services’ programs, Congress eliminated fiscal year 1988 funding within the services’ separate research, development, test, and evaluation accounts for individual UAV programs and consolidated the funding in a joint-service program. Also during the fiscal year 1988 budget process, Congress refused to authorize funding for the procurement of separate Navy and Army Short-Range UAVS. Congress provided that funds were available only for a joint program. Short-Range UAV Program DOD initiated acquisition of the Short-Range UAV in fiscal year 1989 as a Description common-service system. Contracts were awarded in September 1989 to two firms for nondevelopmentall candidate systems for a competitive fly-off in fiscal year 1991. DOD plans to evaluate the candidate systems and select the winning firm to begin low rate production in fiscal year 1992. The initial procurement is to include eight systems for deployment in 1994. ‘Nondevelopmental item means any item that is (1) commercially available, (2) in use by a US. agency or foreign government with which the United States has a mutual defense cooperation agree- ment, or (3) any of the items in (1) or (2) that require only minor modification. . Page 8 GAO/NSL4D9O-234U- ed Aerial Vehicles Appendix I Evalnation oPDOD’sAcquisition Strategy Por the Short-RangeUAV The initially produced systems, designated as Block 0, are planned for use by the Army and Marine Corps. However, DOD does not expect these systems to fully meet performance requirements and thus plans a research and development program to improve the Block 0 system’s capability and provide a variant of the system for naval use. The Navy variant is designated as Block I, while the improved Short-Range system is designated as Block II. Block II improvements will include use of a diesel or jet fuel engine to eliminate the use of more volatile gasoline. The improvements will also include growth in a number of mission payload and ground control station capabilities such as automatic tracking of targets and automated searching of designated areas. DOD plans to procure a total of 53 Short-Range systems: 27 systems for the Army, 18 for the Marine Corps, 5 for the Navy, and 3 for training purposes. The systems include 424 air vehicles or 8 vehicles per system. In addition to air vehicles, each system is to consist of multiple pay- loads, a launch and recovery station, and related equipment. The Short- Range UAV Decision Coordinating Paper, dated August 1989, projected total acquisition costs to be about $1.5 billion. However, according to DOD, that amount may need some revision, depending upon the winning contractor. Despite the importance of realistic operational testing, the preproduc- Production Before tion operational testing of the Short-Range system that DOD plans to con- Realistic Operational duct will not take place in environments representative of where it is Testing supposed to be deployed. We recognize that if the current Middle East situation continues until 1994, when the system is to be fielded, the Short-Range system deployment plans could be changed. Nonetheless, the test results, which are the basis for beginning production, may not be a valid indicator of the system’s currently required capability. The Navy’s experience with its Pioneer Unmanned Aerial Vehicle, first procured in 1986, illustrates the need for realistic operational testing of the Short-Range UAV system. Some of the lessons to be learned from pro- duction before testing in a realistic environment were vividly demon- strated during the acquisition of the Pioneer system. Experience With Pioneer The predecessors to Pioneer had been successfully used by Israeli forces in the Middle East. Thus, the Navy procured the Pioneer as a nondevelopmental system, without testing it, and deployed the system with its operational fQrces. Numerous problems ensued. Page 9 GAO/NSIAKMO-234Unmanned Aerial Vehicles Appendix I Evaluation of DOD’sAcquisition Strategy for the Short-RangeUAV After deployment at sea, engine failures caused crashes and led to tem- porary termination of Pioneer operations pending engine modifications and a change in the type of fuel used. However, the fuel change to avia- tion gasoline then forced the Navy to start a development program for a new Pioneer engine that would use a less volatile fuel considered accept- able for shipboard operations. Difficulty in landing Pioneer on the ship also caused crashes and required vehicle modifications and development of new flight control software. The air vehicle was also retrofitted with new foam-filled wings so that crashed vehicles could float until salvaged. Problems in controlling Pioneer at sea were compounded by electromag- netic interference from the ship’s high frequency radio, leading to at least one air vehicle loss. This required that the radio be shut down during Pioneer operations pending modifications to harden Pioneer against the interference. Placement of the air vehicle tracking unit behind the ship’s superstruc- ture created a blindspot where Pioneer contact was lost. This required that the ship be maneuvered extensively to keep the air vehicle out of the blindspot until another tracking unit could be acquired and placed in front of the ship’s superstructure. Pioneer’s cumbersome net recovery system interfered with helicopter operations and sometimes caused air vehicle and propeller damage when the air vehicle landed in the net. This required the Navy to develop a new net recovery system. Thus, the lack of realistic operational testing of the Pioneer UAV system left the Navy to costly and time-consuming trial and error as it tried to adapt a system used in a land-based desert environment to shipboard use. Four years after initial procurement, the Navy is still buying replacement hardware, such as completely modified air vehicles, to bring Pioneer systems up to a minimum essential level of performance. Planned Short-Range DOD plans to do operational testing of the Short-Range UAV system before Testing Is Not Realistic low rate production to minimize some of the risk that was inherent in the Pioneer UAV acquisition strategy. However, the planned testing, based on current deployment plans, is not realistic; significant risks remain. Thus, the test results, which are to be the basis for beginning production, may not be a valid indicator of the system’s capability. Operational testing is the primary means of predicting weapon system performance in a combat representative environment. Thus, operational testing should be held in a realistic environment representing combat Page 10 GAO/NSIAIHO-234 U- ed Aerial Vehicles I Appendix I Evaluation of DOD’sAcquisition Strategy for the Short-RangeUAV conditions to the extent practical. Our past work has also consistently illustrated the importance of conducting operational testing in a realistic environment. We reported in December 1986,* for example, that results of testing in an unrealistic environment can be misleading and of only limited usefulness in evaluating system performance. Preproduction Desert Testing Realistic operational testing of the Short-Range UAV would require, among other things, that the system be tested at locations resembling those where it might be used in combat. The Short-Range system is to be deployed in areas marked by hilly or mountainous terrain with dense forests and other vegetation and by varied climatic conditions such as cloudy weather, rain, snow, and other factors. Accordingly, the Short- Range system is supposed to be capable of performing reconnaissance, surveillance, and target acquisition under a variety of environmental conditions. DOD plans to conduct preproduction operational testing of the Block 0 system’s ability to perform such missions at Fort Huachuca, Arizona. The operational test facility at Fort Huachuca is typical desert terrain, generally flat and lacking trees and other vegetation. Our viewing of the planned test site showed few areas where potential targets could be situ- ated other than in open view. Thus, the planned testing to evaluate the Short-Range system’s capability to locate targets under various environ- mental conditions in which the UAV is to operate may prove misleading. Preproduction Maritime Testing DOD also plans to conduct preproduction maritime testing at a second location, the Pacific Missile Test Center. However, this testing will not include operations from Navy ships. Instead, testing will include opera- tions from runways on land and will therefore not demonstrate the system’s capability to operate in a naval environment. As discussed earlier, the Navy’s Pioneer experience demonstrated the difficulties in adapting a UAV system for use at sea. After crashing 14 of the air vehicles during maritime operations and 21 overall, and encoun- tering numerous other performance problems, the Navy redesigned and modified practically the entire Pioneer system. Despite these changes, made at a cost of about $50 million in Research & Development and replacement hardware, Pioneer does not meet its intended performance requirements. 2Weapon Performance: Operational Test and Evaluation Can Contribute More to Decisionmaking (GAOINSIAD87-61, Page 11 GAO/NSIAlMO-234 U- ed Aerial Vehicles Evahutlon oPDOD’sAcquisition Strategy Por the Short-RangeUAV DOD plans to commit to full-rate production of the Short-Range UAV Congressional Intent system before verifying that it can be modified to meet Navy require- for a Joint-Service ments. DOD’S acquisition policies require that operational testing be com- System Is at Risk pleted prior to full-rate production. Hence, DOD plans to conduct operational testing in 1992 before the system’s full-rate production begins. However, the system to be tested is the Block 0 version for use by the Army and Marine Corps and will not incorporate those features necessary to operate from Navy ships. Congress has already refused to authorize funding for separate Army and Navy short-range systems. If subsequent operational testing of the Navy variant were to show that a modified Short-Range system is unsuitable for Navy use, DOD would be faced with the alternatives of (1) cancelling the Short-Range production and starting a new program to acquire a system meeting both Army and Navy needs or (2) continuing the Short-Range production for the Army and Marine Corps and acquiring a separate system for the Navy. We and officials in the Office of the Secretary of Defense identified SomeShortcomings of other shortcomings in the Short-Range system’s original test program. Original Test Plans Subsequent to completion of our work, however, the UAV Joint Program Addressed Office agreed to address these concerns by significantly revising its test plans. Initially, we were concerned that the low-rate initial production decision for the Short-Range UAV was planned to follow an early operational assessment period rather than the currently planned operational testing. An official from the Office of the Secretary of Defense stated that an early operational assessment should not be considered a substitute for operational testing. Although DOD policy allows the initiation of low rate production based on an early operational assessment, DOD revised its acquisition strategy to provide for preproduction operational testing. Another shortcoming was that the test plan contained a provision that none of the criteria for judging the Short-Range system’s performance was to be considered as absolute pass or fail measures. Our concern with this provision was that it conflicted with the Secretary of Defense’s intent as recently stated in the July 1989 Defense Management Report to the President. The Secretary stated that DOD policy will be to discipline the acquisition process by defining minimum required accomplishments for advancing from one phase of the acquisition process to the next. This provision was removed from the test plan. Page 12 GAO/‘NSIAD90-234Unmanned Aerial Vehicles Evaluation of DOD’sAcquisition Strategy for the Short-RangeUAV We and Defense officials had concerns about the success threshold for locating targets accurately during Short-Range testing. The percentage of targets to be located accurately during testing had been tentatively set at one-fifth of the percentage specified in the Short-Range UAV per- formance requirements. In response to these concerns the Joint Program Office is revising its test plans to increase the goal for locational accu- racy so that it corresponds to the performance requirements. Page 13 GAO/NSIAJMlO-m Unmanned Aerial Vehicles Appendix II Objective,Scope,and Methodology In response to a request from the Chairman, House Armed Services Committee, we reviewed DOD'S acquisition strategy for the Short-Range UAV. The Chairman expressed concern about the lack of success on past UAV programs and the need to assure that weaknesses contributing to the lack of success do not recur. We obtained the information for this report by reviewing records and interviewing officials of the UAV Joint Program Office and Office of the Secretary of Defense in Washington, D.C.; Marine Corps officials in Quantico, Virginia; Short-Range UAV Project Office officials at Redstone Arsenal, Huntsville, Alabama; and Army Training and Doctrine Com- mand officials at Fort Huachuca, Arizona. We also visited the planned UAV test site at the c’i\v Joint Training Center at Fort Huachuca. We compared the acquisition strategy of the Short-Range UAV and the Pioneer UAV, a previously procured system that encountered problems when deployed with the Navy’s operational forces. After our initial efforts indicated weaknesses in the planned Short-Range system testing, we concentrated on evaluating the planned test program and comparing it to the Pioneer experience. We obtained information on Pioneer from officials of the Naval Air Systems Command, Washington, D.C. To determine the extent to which performance will be demonstrated, we compared the planned testing for the Short-Range system to its perform- ance requirements documents. We also analyzed planned testing docu- ments to determine whether they were consistent with DOD'S testing policies. Our review was performed from October 1989 through April 1990 in accordance with generally accepted government auditing standards. DOD provided written comments on a draft of this report. These comments are presented and evaluated in appendix III. Page 14 GAO/NSIAIM#234 Unmsnned Aerial Vehicles Appendix III CommentsFrom the Department of Defense and Our Evaluation Note: GAO comments supplementing those In the report text appear at the DIRECTOROFDEFENSERESEARCHANDENGINEERING end of this appendix. WASHINGTON, DC 20301-3010 August 20, 1990 Mr. Frank C. Conahan Assistant Comptroller General National Security and International Affairs Division General Accounting Office Washington, DC 20548 Dear Mr. Conahan: This is the Department of Defense (DOD) response to the General Accounting Office (GAO) Draft Report, "UWMAWWEDAERIAL VEHICLE: Realistic Testing Needed Before Production of Short- Range System," dated 10 July 1990,(GAO Code 395120/OSD Case 8410). The DOD concurs or partially concurs with most of the findings in the GAO report. However,. the DoD does not concur with either recommendation or the matter for congressional consideration. The Test and Evaluation Waster Plan for the Unmanned Aerial Vehicle-Short Range system, which describes the test program, is being restructured and will be reviewed by the Office of the Secretary of Defense (OSD). The responsible OSD officials wit I approve the test program only if it is adequate and consistent with applicable statutes and DOD directives. Therefore, the GACI See comment 1. assertion of inadequate testing cannot be addressed at this time. Detailed- comments on the report findings, recommendations, and matter for congressional considerations are provided in the enclosure. Thank you for this opportunity to review and to comment on the draft report. Sincerely, &,uffuharles M. Herzfeld Enclosure Page 16 GAO/‘NSIAD90-234Unmanned Aerial Vehicles Appendix Ill CknnmentaFrom the Department of Defense and Our Evaluation GAODRAE’T REPORT - DATED JULY 10, 1990 (GAO CODE 395120) OSD CASE 8410 YstwANNED ABRIAL VEHICLE: REALISTIC TESTING NEEDED BEFORE PRODUCTIONOF SHORT RANGE SYSTEW DEPARTMENTOF DRFWSE CDMMNTS * l * l l . FINDINGS FINDING A: Initiation of the Short-Ranae Unmanned Aerial Vehicle Prociram. The GAO reported that a joint service program was initiated by the Congress in response to concern about the proliferation of unmanned aerial vehicle programs and the need to acquire vehicles that meet the requirements of more than one Service. The GAO explained that, in order to achieve commonality, Congress eliminated the FY 1988 funding within the Services' separate research, development, test, and evaluation accounts for individual programs and consolidated the funding in a joint-service program. The GAO also reported that, during FY 1988, the Congress refused to authorize funding for the procurement of separate Navy and Army unmanned aerial vehicles-- providing funds, instead, for a joint program. The GAO reported that contracts were awarded in September 1989 to two firms for non-developmental candidate systems for a competitive fly-off in FY 1991. The GAO further reported that the DOD plans to procure a total of 53 Short-Range systems: 27 for the Army; 18 for the Marine Corps: 5 for the Navy: 3 for training purposes. The GAO found that the Short-Range Unmanned Aerial Vehicle Decision Coordinating Paper, dated August 1989, projected total acquisition costs of about $1.5 billion. The GAO noted that program officials now indicate that that amount may be revised. Now on pp. 8-9 (pp. T-lo/GAO Draft Report) DOD RESPONSE: Concur. Depending on the winning contractor, the actual program amount may need slight revision. FINDING B: Production Before Realistic -rational Testinq. The GAO reported that, contrary to its stated acquisition principles, the DOD does not plan to conduct preproduction operational testing of the Short-Range system in environments typical of where it may be used in combat. The GAO concluded, therefore, that the test results (from those tests that will be conducted) may not be a valid indicator of the system's capability. The GAO pointed out that the Navy's experience with the PIONEER Unmanned Aerial Vehicle, illustrated the need for realistic operational testing of the Short-Range Unmanned Aerial Vehicle system. The GAO noted that the lack of realistic operational testing of the PIONEER Unmanned Aerial Vehicle system led the Navy to costly and time consuming trial and Page 16 GAO/NSIAD4@234Unmanned Aerial Vehicles Appendix Ill CommentsFrom the Department of Defense and Our Evaluation error, as it tried to adapt, for shipboard uses, a system used in a land-based desert environment. The GAO reported that the Navy is still buying replacement hardware to bring the PIONEER systems up to a minimum essential level of performance. (pp. lo- Now on pp, 9-10 lZ/GAO Draft Report) DoD RESPONSE: Nonconcur. The Unmanned Aerial Vehicle-Short Range test program must be viewed in the context of its acquisition strategy. The acquisition strategy is non- developmental. Production funds are used to procure systems for a competitive fly-off and for follow-on systems. Full-scale development funds are not used to procure any systems. Production of two systems from each of two contractors was approved at the Milestone II/IIIA decision point. The Milestone IIIB decision point, which constitutes the low-rate production decision, will address the selection of one contractor to produce up to eight systems. The Milestone IIIC decision point, which constitutes the full production decision, will address the additional procurement of up to 43 additional Short Range systems. As part of selection criteria and prior to exercise of the first low rate production option, two tests will be performed. The first test, Technical Evaluation Test, will test each contractor system against the required system technical performance and includes survivability testing using the threat arrays at the See comment 2 Naval Weapons Center. The second test, Operational Test IIA, will assess each contractor system for operational effectiveness and suitability when employed by representative user personnel under realistic operational conditions. Operational Test IIA will also include a survivability evaluation using the results obtained from the Technical Evaluation Test, which will have used the threat arrays at the Naval Weapons Center. Thus, Operational Test IIA is a two-month test program that incorporates field exercises under simulated combat conditions, in which the system will be operated by soldiers and marines. Training will be provided to bring user personnel to operating proficiency prior to commencement of Operational Test IIA. The test results from the Technical Evaluation Test will be utilized to evaluate each system's performance, and results from Operational Test IIA will be utilized to evaluate each system's operational utility. That information will be provided for source selection evaluation. Prior to a Milestone IIIC decision to enter full rate production, Operational Test IIB will be performed on the winning contractor's system. Operational Test IIB will be conducted utilizing representative user personnel under realistic operational conditions to evaluate operational effectiveness and suitability to include reliability, maintainability, effectiveness-of-training, logistic supportability, and any correction of deficiencies previously noted. Operational Test IIB is also a two month test program in 2 Page 17 GAO/NSIAD90-234 Unmanned Aerial Vehicles Appendix III CommentsProm the Department of Defense and Our Evaluation which the system will be operated by Army and Marine Corps personnel in a field environment that incorporates threat simulators. See comment 3. Therefore, performance requirements and operational effectiveness and suitability will be assessed prior to the Milestone IIIB decision to enter low rate production, and prior to a Milestone IIIC decision to enter full scale production. It should be noted that OSD disapproved the initial UAV-SR Test and Evaluation Master Plan , which outlines the UAV-SR test program, in December 1989. The primary reasons for the disapproval concerned inadequacies in the operational test program. When a revised Master Plan is reviewed, the responsible OSD officials will approve the operational test program only if it is adequate and consistent with applicable statutes and DOD directives. The DoD recognizes significant problems were encountered in the effort to adapt the PIONEER for use by the Navy. Based on lessons learned from that process, the DOD is using the block upgrade concept to develop the Navy variant of the Short Range Unmanned Aerial Vehicle. See comment 4. Concerning statements on the PIONEER, the Navy's baseline concept of acquisition initially provided three PIONEER systems that were used during an extensive operational assessment period, during which time the requirements for United States Marine Corp (USMC) land and United States Navy (USN) maritime operations were further defined. Rather than a "time consuming trial and error" period, this assessment was conducted by both the operational users under actual operational conditions and by personnel at the Pacific Missile Test Center. The resulting assessment provided Navy planners with the necessary data to effectively make a baseline decision had the PIONEER program continued as planned. Additionally, the parts being bought are spares necessary to maintain the PIONEER at its currently acceptable operational level of performance. FINDING C: Planned Short-Ranae Testinq Is Not Realistic. The GAO concluded that, while the DoD plans to do some preproduction operational testing of the Short-Range Unmanned Aerial Vehicle system, the planned testing is not realistic and significant risks remain. The GAO further concluded that any test results, which are to be the basis for beginning production, may not be a valid indicator of the system's capability. The GAO reported that realistic operational testing of the Short-Range Unmanned Aerial Vehicle would require that the system be tested at locations resembling those where it might be used in combat. The GAO concluded that--because the system, which is to be deployed in diverse environments, including hilly or mountainous terrain, dense forests, rain, snow, and cloudy weather, is only going to be tested at Fort Auachuca, Arizona, in desert terrain-- the results may be misleading. 3 . Page 18 GAO/NSIAD!30-234Unmanned Aerial Vehicles Appendix m CommentaFrom the Department of Defense and Our Evaluation The GAO also found that the DOD plans to conduct preproduction maritime testing at the Pacific Missile Test Center. The GAO further found, however, that testing will not include operations from Navy ships. The GAO reported that the planned testing will only include operations from runways on land and, therefore, not demonstrate the system's capability to operate in a naval environment. The GAO referred to the difficulties in adapting the PIONEER to use at sea. According to the GAO, (1) 17 of the original 40 vehicles crashed, (2) the Navy redesigned and modified nearly the entire system at a cost of about $50 million in research and development and replacement hardware, and (3) the system still does not meet its intended performance Now on pp. 10-l 1. requirements. (pp. 12-lS/GAO Draft Report) DOD RESPONSE: Partially concur. The test program, composed of both technical and operational testing, will be conducted in the severe flight environments of Fort Buachuca (high altitude, high temperature, dust, etc.) and the Pacific Missile Test Center (moisture/water, fog and salt/fog, etc.). In order to increase the realism of operational tests, camouflage will be used to mask targets. The Pacific Missile Test Center early chamber technical testing will expand/supplement the natural conditions encountered to cover the remaining environmental specification requirements. The Naval Air Propulsion Center engine technical tests will determine the capability of the engines to meet all See comment 5. altitude and temperature specifications. An overriding factor in the identification of Fort Huachuca, Arizona, as the site for the OT-IIA was the extensive airspace and range scheduling limitations presented by other potential test sites. The environment at Fort Huachuca is extremely severe on system operation. The combination of altitude and temperature is equivalent to operating at an altitude of 10,000 feet. The terrain at Fort Huachuca includes mountains and some treed areas. The testing will take place during the Fort Huachuca rainy season. During this period, low-hanging clouds and periods of intense rain can be expected. The rainy season begins in the June timeframe. The Unmanned Aerial Vehicle Joint Project Office is in the process of identifying an additional site, with environmental conditions other than those of Fort Huachuca, for conduct of a portion of the Operational Test IIB. To correct the record, maritime operations training and testing See comment 6. of the PIONEER Unmanned Aerial Vehicle has resulted in fourteen air vehicles lost rather than the seventeen cited in this finding. As previously stated, the DOD recognizes significant problems were encountered in the effort to adapt the PIONEER for use by the Navy on board ships. The Department's block upgrade concept to develop the Navy variant of the Short Range Unmanned Aerial Vehicle is designed to take advantage of the lessons learned on PIONEER. Plans for the operational testing of the Navy Block I variant of the Unmanned Aerial Vehicle-Short Range during the follow-on 4 . Page 19 GAO/NSIAD-90-234UNMNI ed Aerial Vehicles - AppendixIII Comment8FYomtheDepartmentofDefense andOurEvaluation operational test and evaluation phase have not yet been developed. However, the DOD, particularly, operational test agencies, recognize the need to test weapon systems onboard ships, if the systems will be used in that environment. FINDING D: Conqressional Intent For A Joint-Service System Is At Risk. The GAO observed that DOD acquisition policies require that operational testing be completed prior to full-rate production. The GAO found, however, that the DOD plans to commit to full-rate production of the Short-Range Unmanned Aerial Vehicle before verifying that it can be modified to meet Navy requirements. The GAO reported that the DOD plans for operational testing in 1992--before full-rate production begins --involve the Block 0 version for use by the Army and Marine Corps, but will not incorporate those features necessary to operate from Navy ships. The GAO concluded, that if subsequent operational testing of the Navy variant were to show that a modified Short-Range system is unsuitable for Navy use, the DOD would be faced with the alternatives of (1) canceling the Short-Range production and starting a new program to acquire a system meeting both Army and Navy needs or (2) continuing the Short-Range production for the Army and Marine Corps and Now on p. 12. acquiring a separate system for the Navy. (pp. 15-16/GAO Draft Report) DOD RESPONSE: Partially concur. As provided for in the current contracts, as part of the downselect process, each contractor will submit for evaluation an engineering study, which will detail system configuration changes required for shipboard See comment 7 operations. Subsequent to the downselect, the Navy shipboard variant system will then complete technical and detailed design reviews. Full-rate producticn of Block 0 is planned to occur following design reviews of the Navy shipboard variant. Operational Testing of the Block I system will be conducted onboard ships prior to exercising the production option of the naval variant. During an Early Operational Assessment, the Navy will assess the See comment 8. Block 0 Unmanned Aerial Vehicle-Short Range system's capability to meet shipboard requirements. Of primary interest is the capability of the Short-Range Block 0 system to operate from ships while performing Reconnaissance, Surveillance, and Target Acquisition missions. It is important to realize that the preponderance of the current requirement for the Unmanned Aerial Vehicle-Short Range is for See comment 9 the Block 0 version. Only five of the 53 vehicles planned are the Navy variant, Block I. The approach recommended by the GAO would delay fielding of the 48 Block 0 systems by one to two years, while the Block I Navy variant completed its operational testing. The best way to operate an Unmanned Aerial Vehicle from ships, particularly launch and recovery operations, is still being 5 Page20 AppendixIII CommentaFromtheDepartmentofDefenae andOurEvaluation explored. It is possible that the Block 0 air vehicle may not be usable or may require extensive modifications for shipboard use. However, this would not preclude system interoperability or commonality in the use of other Unmanned Aerial Vehicle-Short Range system elements (mission planning stations, ground control stations, data terminals, etc,) in the Block I system. Testing of the Block 0 system will provide information applicable to the Navy variant. That information will be available before the full-rate production decision for the Block 0 system. FINDING E: Some shortcominqs of the The Orisinal Test Plans Addressed. The GAO reported that it (along with officials in the Office of the Secretary of Defense) identified other shortcomings in the Short-Range system's original test program that the Unmanned Aerial Vehicle Joint Program Office agreed to address by significantly revising the test plan. The GAO reported that, initially, the low-rate initial production decision for the Short-Range Unmanned Aerial Vehicle was planned to follow an early operational assessment period, rather than the currently planned operational testing. The GAO noted tha<, subsequently, the DOD revised its acquisition strategy to provide for preproduction operational testing. The GAO reported that another shortcoming was that the test plan contained a provision that none of the criteria for judging the Short-Range system's performance was to be considered as absolute pass or fail. The GAO found this provision in conflict with the intent of the Secretary of Defense to discipline the acquisition process by defining minimum required accomplishments for advancing from one phase to the next, as recently stated in July 1989 Defense Management Report to the President. The GAO reported that, subsequently, the conflicting provision was removed from the test plan. The GAO also reported that both GAO and DOD officials had concerns about the success threshold for locating targets accurately during Short-Range testing. The GAO found that the percentage of targets to be located accurately during testing had been tentatively set at one-fifth of the percentage specified in the Short-Range Unmanned Aerial Vehicle performance requirements. The GAO reported that, in response to those concerns, the Joint Program Office is revising its test plans to increase the goal for locational accuracy so that it corresponds Now on pp. 12-13. to the performance requirements. (pp. 16-la/GAO Draft Report) DoD RESPONSE: Concur. The revised Unmanned Aerial Vehicle- Short Range Test and Evaluation Master Plan has been revised to better reflect operational requirements and to provide for See comment 3. adequate operational testing prior to Milestone IIIB and IIC decisions. l l l l l 6 Page21 GAO/NSLAIMWB~UNMNI edAerialVehicles Appendix III CommentsFrom the Department of Defense and Our Evaluation RECONNFNDATIONS RRCONHRNDATION1: The GAO recommended that the Secretary of Defense require that the Short-Range Unmanned Aerial Vehicle system be operationally tested in diverse, realistic environments to provide reasonable assurance that it will meet requirements before permitting limited production of the Iand- based system. DoD RESPONSE Nonconcur. Department of Defense Directive 5000.3 states that demonstration of a system's technical capabilities and its operational effectiveness and suitability will be a key requirement for decisions to commit significant resources, to advance from one acquisition phase to another, and to field the system. An adequate evaluation of a system's operational effectiveness and suitability can be accomplished without testing in all of the operational environments in which a system may be employed. The Unmanned Aerial Vehicle Short Range acquisition strategy and, in particular, the test program, is See comment 10. consistent with the DOD directive for a non-developmental item program. The planned Technical Evaluation Test and Operational Test IIA will evaluate the system’s performance and operational effectiveness/suitability. The planned test program will test the system in meaningful representative operational environments and will provide data to support a Milestone IIIB decision to enter low rate production. RRCONNENDATION2: The GAO recommended that the Secretary of Defense limit Short-Range Unmanned Aerial Vehicle system production until satisfactor:r performance of the Navy variant is demonstrated and assure that actions are taken to correct the Now on p. 2. other shortcomings noted in the planned test program. (pp, 3- 4/GAO Draft Report) DOD RJZSl?ONSE: Nonconcur. The Unmanned Aerial Vehicle-Short Range acquisition strategy, with its provisions for production lot options, limits production of the Block I system until its performance is demonstrated. However, the Department of Defense does not intend to delay production of a successful Block 0 system, pending the outcome of the Block I development. As previously noted, 48 of the 53 systems planned are the Block 0 version. The Unmanned Aerial Vehicle - Short Range acquisition strategy includes a block upgrade approach, which allows for the evaluation and fielding of an initial baseline configuration, See comment 11 followed by block upgrades to meet the full operational requirements. The modular architecture of the system will facilitate upgrades. Block 0 is the initial, baseline system. Block I is the Navy shipboard variant. The described acquisition strategy has been fully reviewed and approved, and is in keeping with the non-developmental item nature of the system. The cognizant congressional committees have also been appraised of the acquisition strategy being approved. 7 Page 22 GAO/NSIADW234 U- ed Aerial Vehicles AppendixHI CommentaFromtheDepartmentofDefense andOurEvaluation AS provided in the current contracts, each contractor will submit for evaluation as part of the downselect process, an engineering study which will detail system configuration changes required for shipboard operations. These studies will be completed this fiscal year. Following this, a FY1992 contract option may be exercised for the integration, demonstration, and test of the Block I Navy shipboard variant. Not until FY1994 will the government exercise a contract option which will include the purchase of one system with Block I upgrades. See comment 7 Operational testing of this system in a shipboard environment will be conducted prior to exercising an option to procure additional Navy systems. This is the first of five Navy shipboard systems to be purchased. Therefore, the Block I engineering studies provided to each competing contractor as part of the downselect process: the technical design and the detailed design review subsequent to down select: and the operational testing of the initial Block I system prior to procuring additional systems, will sufficiently minimize the risk associated with the block upgrade approach. POR CONSIDERATION BY THE HOUSE COMMITTEE ON ARMHD SRRVICES MATTER FOR CONSIDERATION: To assure that congressional intent for a common-service system is achieved, the GAO suggested that the Committee may wish to prohibit full-rate production of the Short-Range Unmanned Aerial Vehicle until performance of the Now on p. 3 Navy variant is proven to be satisfactory. (p. 4/GAO Draft Report) DOD RESPONSE: Nonconcur. The non-developmental nature of the Unmanned Aerial Vehicle Short Range Program eliminates much of the risk inherent in a developmental program. The risk is further minimized by the Block 0 test program, the Block I engineering evaluations and design reviews, and the pre- production tests of the Block I Navy shipboard variant. The See comments 2, 4, 9, resultant risk is so small that the program delays and and 11. additional costs inherent in this suggestion exceed any additional risk avoidance. Operational testing of the Unmanned Aerial Vehicle-Short Range system before procurement of the Navy Block I variant will provide a basis for Block I development. The GAO cited problems with the performance of the PIONEER system onboard ships. Many of those problems were associated with the difficulties of adapting a land-based system for shipboard use. The DOD has learned from that experience. Many uncertainties and unknowns are still associated with the use of Unmanned Aerial Vehicle systems in worldwide military operations across a wide spectrum 8 Page23 GAO/NSIAlMO-234U- ed AerialVehicles - Appendix III CommentsProm the Department of Defense and Our Jhwluation of combat intensity. The Block upgrade concept will provide a measured approach to resolving those uncertainties and unknowns. 9 Page 24 GAO/NSIAD-90-234Unmanned Aerial Vehicles Appendix Ill CommentaProm the Department of Defense and Our Evaluation The following are GAO'S comments on the Department of Defense’s letter dated August 20, 1990, and its accompanying enclosure. 1. We believe that concerns about inadequate testing need to be GAO Comments addressed as part of the ongoing restructuring of the Test and Evalua- tion Master Plan. 2. The nondevelopmental nature of the program, does not negate the requirement for realistic testing. DOD directive 5000.3 on testing states that nondevelopmental items shall be guided by the same WD acquisi- tion principles as other systems. This directive requires testing of suita- bility for use in combat in an environment as operationally realistic as possible. DOD’S comments do not address our main concern that the Short-Range system’s preproduction operational testing will be limited primarily to the desert environment which does not resemble the primary locations where the system is to be deployed. These locations are marked by mountainous terrain with dense forests and adverse climatic conditions. From that standpoint, the planned testing of the Short-Range system is unrealistic and could yield misleading results on the system’s capability to locate targets. It is also important to point out that DOD'S comments address the planned testing prior to the full-rate production decision, while our report addresses the lack of realistic testing in DOD'S preproduction test plan. Moreover, the site referred to by DOD as Operational Test IIB has not yet been identified by DOD. Therefore, DOD’S response indicating that the operational tests leading up to a full-rate production decision will be conducted under realistic operational conditions is premature. Program officials indicated that discussions have occurred within DOD about selecting a site that would be more representative of a planned deploy- ment location than Fort Huachuca, but no decision has been made. 3. Our report recognizes that operational testing of the Short-Range system will be held before the low-rate and full-rate production deci- sions. Our concern with the testing before low-rate production (opera- tional test IIA) is that it is to be restricted primarily to an unrealistic environment. Our concern with the testing before full-rate production is that it will not evaluate the system’s capability to operate in a naval environment. Testing of the Navy variant is to occur only after DOD has committed to full-rate production. Page 26 GAO/NS~SO-234 Unman ed Aerial Vehicles Appendix Ill CommentsFrom the Department of Defense and Our Evaluation 4. We cited the Pioneer’s problems, in part, to illustrate the need for realistic operational testing before committing to production. In our view, DOD is not capitalizing on what should be a lesson learned from the Pioneer experience. Pioneer problems were discovered by deploying the system and waiting to see if it worked. When problems occurred, the Navy was forced to continually bring in experts from the Naval Air Propulsion Center, the Naval Air Test Center, the Pacific Missile Test Center, or send personnel to the subcontractors in Israel to find solutions. In 1987, the Navy pro- duced an Integrated Action Plan to address the numerous shortcomings of the system. In this plan the Pioneer acquisition strategy was entitled “Operate, Learn, Fix, Operate.” We see no difference between this and “trial and error.” The Navy recently (1990) described the Pioneer system’s level of per- formance as a “minimum essential capability,” (underscoring added). We would point out, that DOD and the Navy expended $28 million in research and development funds and an additional $22 million in pro- curement for replacement hardware over 4 years to reach this level of performance. 5. Although Fort Huachuca has a high altitude, the terrain is generally flat and treeless. Moreover, there is no indication in the revised draft test plan that DOD intends to overcome this drawback. To determine whether the rainy season at Fort Huachuca was signifi- cant and might indicate something about the ability of the Short-Range system’s ability to operate in general conditions of wetness, we acquired recent meteorological data for the closest metropolitan area to the Fort Huachuca test site. It showed that (1) less than 3 inches of rain fell per month during the 3-month rainy season (June, July, and August) and (2) rain fell in measurable amounts on only 5 days. This environment is not similar to that of the primary locations where the system is to be deployed. For example, at one planned deployment location the rainy season is 7 months, and measurable amounts of rain occur on average 16 days per month. Average monthly accumulation exceeds 8 inches per month, more than Fort Huachuca can expect in its entire 3-month rainy season. Range scheduling limitations presented by other potential test sites, as identified in DOD'S response, should not in our view be an overriding factor. The need to perform realistic testing should be the deciding Page 26 GAO/‘NSIAIMW-234Unmanned Aerial Vehicles Appendix Ill CommentsFrom the Department of Defense and Our Evaluation factor. Although DOD may be looking for a test site other than Fort Huachuca, the Operational Test IIB will not occur until after production of the Short-Range system has started. 6. In a draft of this report, we identified 17 Pioneer air vehicles as being lost during maritime operations. The final report has been modified to reflect DOD’S position that, of the 21 vehicles lost, 14 were lost during maritime operations. (Also see comment 4.) 7. We do not disagree with DOD’S comments; however, the fact remains that the Short-Range system will be in full-rate production before any operational testing of the Navy variant is done. 8. According to the Director, Defense Operational Test and Evaluation, an early operational assessment is no substitute for operational testing. 9. We did not recommend delaying the fielding of the Block 0 system. Our recommendation would require that the Block 0 system be retained in low-rate production pending verification that the Block I variant will meet Navy needs. This would reduce the risk of DOD becoming fully com- mitted to production of a system not meeting common-service require- ments. Further, DOD’S plan is to retrofit Block 0 systems to incorporate design features of the upgraded configurations; therefore, limiting Block 0 production would reduce the risks associated with the retrofit. Testing of the land-based system (Block 0) will not provide reasonable assurance that the various elements will meet the Navy’s requirements. Such assurances can only be obtained through operationally testing the system in a realistic naval environment. 10. DOD cannot conclusively determine whether the Short-Range UAV will be operationally effective or suitable based on its planned testing. The operational environment in which DOD intends to test is not at all similar to the diverse environments of the primary locations where M)D intends to deploy the system. In fact, DOD’S draft restructured test plan recog- nizes that the testing environment constitutes a test limitation. 11. Our recommendation does not affect low-rate production of the land- based system. Although specifically proscribed during congressional consideration of fiscal year 1988 UAV funding, DOD’s adherence to the current Short-Range UAV acquisition strategy could result in separate Army and Navy Short-Range UAV systems if the Navy variant proves a failure. This would not be consistent with congressional direction. Page 27 GAO/NSLAD-W234 Unmanned Aerial Vehicles AppemdixIII CommentsFrom the Department of Defense and Our Evaluation DOD’S acquisition strategy calls for entering full-rate production of the Short-Range system for use by the Army and Marine Corps before con- ducting any testing of the block upgraded Navy variant. Scheduled testing of the Navy variant will only reduce the risk of prematurely committing to production of that system, not the basic system, which will already be in full-rate production. If the Navy variant were to prove unsuccessful in the later testing, DODwould find itself fully committed to production of a system not meeting congressional intent for a common UAV. Further, as brought out in our report, experience with Pioneer shows that adapting a UAV for naval operations is not a low-risk effort even though the Navy categorized it as such when justifying the pro- gram. We therefore continue to believe that DOD should retain the Short- Range system in low-rate production until operational testing shows that the Navy variant’s performance is satisfactory. Page 28 GAO/Ni%UMM34U- ed Aerial Vehlclea Appendix IV Major Contributors to This Report Charles A. Ward, Evaluator-in-Charge National Security and International Affairs Division, Washington, D.C. Jackie B. Guin, Assistant Director Atlanta Regional Bobby D. Hall, Evaluator Office Terry D. Wyatt, Evaluator (396120) Page 29 GAO/‘NSIAD~~ UNMNI ed Aerial Vehicles
Unmanned Aerial Vehicles: Realistic Testing Needed Before Production of Short-Range System
Published by the Government Accountability Office on 1990-09-28.
Below is a raw (and likely hideous) rendition of the original report. (PDF)