United States General Accounting Office GAO Report to the Chairman, Subcommittee on Defense, Committee on Appropriations, House of Representatives i i. August 1990 .-, TEST AND EVALUATION A Proposed Framework for Measuring the Use oft Test Facilities GAO United States General Accounting Office Washington, D.C. 20548 National Security and International Affairs Division B-210919 August 8,199O The Honorable John P. Murtha Chairman, Subcommittee on Defense Committee on Appropriations House of Representatives Dear Mr. Chairman: The military services operate and maintain 21 major test ranges, collec- tively known as the Major Range and Test Facility Base (MRTFB), which have investments over 525 billion and an annual operating budget of about $3.5 billion. The Office of the Secretary of Defense (OSD) is respon- sible for centralized management of the ranges and for making budg- etary decisions on range expansion, consolidation, and improvements. However, the ranges do not collect and report standard information to assess overall range capacity and use. In response to the former Chairman’s request, we have developed a framework for measuring the use of MRTFB test ranges. Our proposed framework, which the ranges could adopt with minor data collection modifications, should be useful to both defense and congressional decisionmakers. MRTFISranges comprise large land, sea, and air masses that are divided Background into various test sites, Generic equipment and instrumentation, such as tracking radars, can be found throughout the ranges, while individual test sites contain equipment and instrumentation required for specific types of tests. The ranges are used for such purposes as testing aircraft, bombs, and missiles; tanks and other tracked vehicles; ordnance; and environmental effects on weapon systems and underwater tests of munitions. Although DOD has long recognized the need for a common measurement Problems in Assessing of its test ranges’ capacity and use, the ranges do not collect comparable Test Capabilities data. As a result, DOD cannot readily identify either excess testing capacity or the need for additional capacity. Page 1 GAO/NSIAB90-91 Test Facilities H-210919 DOD-Proposed Range Use In 1988, DOD'S Range Commanders Council established a working group to develop a Range Vtilization Measurement System that would accu- Measurement System Was rately convey the degree to which each range’s test capacity was being Not Adopted used. The system was developed in response to an informal request from the Deputy Director, Defense Research and Engineering (Test and Eval- uation), the OSDofficial responsible for MKTFH management issues. But according to the Deputy Director’s office, OSDofficials did not adopt the system because they believed it would be labor intensive and t,oo costly. Implementing the system would have entailed additional data collection efforts, such as detaikd information on personnel and equipment use. In addition to collecting data on personnel and equipment use, the Range I Ttilization Measurement System would have recorded a general activity indicator (to bc determined by each range) and land/sea/airspace reported by (1) total missions using a particular land/sea/airspace, (2) the number of mission schedules denied because of land/sea/air- space constraints, and (3) the total number of requested schedules involving land/sea/airspace. The system provided for recording unavail- ability of a test site for testing owing to constraining factors, such as time needed to maintain the site and prepare it for testing or delays caused by weather conditions. Further discussion of the DOD proposal versus our proposal is contained on pages 5 and 6. - Recognizing that test ranges that offer similar capabilities carry out sim- Our Proposed ilar test functions, we developed a proposal to measure use and capacity Framework for of MRTFB air testing sites based on a functional approach. We selected air Measuring the Use of testing-that is, air-to-air or air-to-ground testing and aircraft flight performance testing-because it represents the majority of testing done Test Facilities within the MIITFR. The three main components of our proposed framework are to . identify similar test functions, . establish common capacity levels, and l identify what types of constraints prevent testing at a test site. By defining functional categories within air testing, DOD could identify test sites that provide similar air and ground space, equipment, and instrumentation to perform various air test functions and could then measure their use. By assuming that test equipment and instrumenta- tion on a site are necessary, although not all items may be used for each test, our approach focuses on airspace and real estate as the key factors Page3 GAO/NSIA@SO-91 Test F’acilities &210919 proposal then could potentially be applied to other types of tests, such as testing of ordnance and vehicles. Emphasis on Measures of We propose addressing the use of test facilities from a functional per- spective, considering the capacity and use of the facilities. To determine Functional U.ses available test capacity, all test functions need to be identified and tracked. Examples of functional categories include (1) testing of aircraft flight performance and (2) testing of aircraft weapon systems, such as test and evaluation of fire-control and bombing systems against fixed and moving ground targets. Before our proposed framework can be applied, test sites and testing functions will need to be defined using standard terms of reference. For example, the key factors for defining air test sites to be used for a test function should be required airspace and real estate. Range officials agreed that standard definitions would be critical to drawing conclu- sions on comparable testing capabilities. Establishing Capacity We propose that “capacity” be defined as available daylight hours, with potential maximum capacity as 12 hours a day. In air testing, daylight is Levels generally required to record flight performance or bomb impact. How- ever, even when 12 hours of daylight are available, current salary and personnel ceilings restrict personnel resources to 8 hours a day and/or 40 hours a week. Identifying Constraints Several key factors that affect range use are generally outside the That Prevent Testing ranges’ control. Test sites are often not used because of safety restric- tions resulting from adjacent testing. For example, a piece of real estate may be restricted from use because another test is using the airspace overhead. Furthermore, weather sometimes prevents testing, which results in the need to reschedule. Finally, range users often cancel planned tests. On the other hand, test sites that are available for testing, but have not been scheduled for use by customers, represent nonuse of available capacity. Our proposal is similar to the DOD-proposed Range Utilization Measure- ment System in that we recognize the need to categorize and account for constraining factors that prevent test sites from being used. However, we distinguish between (1) ranges not available for testing because of constraining factors that prevent use (see fig. 1, for example) and Page 5 GAO/NSlALHO-91Test Facilities 8210919 nonuse of test sites was due to constraints on site use, such as airspace restricted for other tests, or to no requests to use existing capacity. To capture data on use and nonuse, as well as reasons for nonuse, the automated data now used for billing purposes could be combined with the nonautomated data used to schedule tests. Although the four ranges we visited did not automatically identify the reasons for nonuse, they considered these reasons when scheduling tests. Thus, data on nonuse were either already available or could be collected easily by establishing standard categories, such as cancellations of planned tests, for the periods when testing is not done. Data in these categories could then be formally entered into existing data collection systems. As officials at the sites visited pointed out, data collection guidelines would need to be implemented uniformly to obtain comparable statistics on all sites. Comparable statistics would be useful in managing existing test resources as well as in justifying requests for increased resources. Such statistics could identify (1) unused capacity or (2) sites at which use could be increased by transferring personnel or equipment. Overtime or shift assignments could be considered, when feasible, to extend use of a test site beyond the 40.hour work week if use measurements showed that additional capacity was needed. In developing common measures of use and capacity for sites having Objectives, Scope, and similar test functions, WCvisited four major test ranges: Methodology - the Armament Division, 3246th Test Wing, Eglin Air Force Base, Florida; - the Naval Air Test Center, Patuxent River, Maryland; - the Air Force Flight Test Center, Edwards Air Force Base, California; and l the Naval Weapons Center, China Lake, California. We analyzed how the ranges scheduled tests, billed customers, and accounted for use of test facilities. We held discussions with OSD and range officials on the feasibility of developing a common measurement method and other topics. Our review was performed from January to <July 1989 according to generally accepted government auditing standards. Page 7 GAO/NSIAD-99.91Test Facilities Page 9 GAO/NSIAWO-91 Test Facilities Page 11 GAO/NSIAtWO-91Test Facilities Appendix I Synopsesof Test Functions Performed At The Test RangesWeVisited The Naval Weapons Center, China Lake, California, is a large and com- Naval Weapons Center plex range. Test capabilities include supersonic test tracks, an explosive test site, a propulsion test site, a live ordnance environmental test site, and an electronic warfare threat environment site. Extensive instrumen- tation is available, including radars, theodolites, cameras, telemetry sys- tems, and communications. The center conducts testing and evaluation of air- and surface-launched weapons, electronic warfare systems, mis- siles, life-support systems, and parachute systems. Page 13 GAO/NSIADM-91 Test Facilities . ,, -4 . -. -,-.---,a - - I ,. ,,. . --- g .‘. %cp~?~ts for copies of GAO reports should be sent to: * si i. U.S. General Accounting Office Post Office Box 6015 ; Gaithersburg, Maryland 20877 I Telephone 202-275-5241 The fiit five copies of each report are free. Additional copies are $2.00 each. There is a 25% discount on orders for 100 or more copies mailed to a single address. Orders must be prepaid by cash or T n i ‘iT ‘:I:“! Fn~F=rllm *IT warl Appendix II Major Contributors to This Report Michael E. Motley, Associate Director National Security and Lester C. Farrington. Assistant Director International Affairs Patricia L. Martin, Evaluator-in-Charge Division, Washington, Anne W. Howe, Evaluator Marilyn Mauch, Social Science Analyst D.C. Alfred Lilliendahl, Operations Research Analyst Page 14 GAO/NSIAD-99.91Test Facilities Appendix I Synopses of Test F’unetions Performed At The Test Ranges We Visited The Armament Division, 3246th Test Wing, Eglin Air Force Base, Eglin Air Force Base Florida, manages a large and complex range encompassing 724 square miles of land and 98.000 square miles of test area in the Gulf of Mexico. The test wing is responsible for the development, test, and evaluation of all Air Force nonnuclear air armaments, electronic combat systems, target acquisition and weapon delivery systems, base intrusion and detection systems, electronic systems, and aerial targets. It is also responsible for climatic simulation test and evaluation and determina- tion of electromagnetic and electro-optical signatures. Typically, Eglin tests ordnance and munitions, emphasizing warhead performance, fuzing, ter- minal effects, aerodynamics, ballistics, and aircraft compatibility; air-to-air and air-to-ground operations that focus on target acquisition and weapon delivery systems; electronic system capabilities; and sensors, emphasizing electro-optical, laser, infrared, and millimeter wave operations. The Naval Air Test Center, Patuxent River, Maryland, maintains flight Naval Air Test Center test facilities that provide actual and simulated conditions for all in-service and planned naval aircraft weapon system programs. Flight testing is conducted in 50,000 square miles of restricted airspace over the Chesapeake Bay and offshore operating areas in the Atlantic Ocean. The Chesapeake Test Range, with its computer-linked video, theodolite, radar, and laser tracking equipment and telemetry capability, is used for (1) flight testing of aircraft and airborne systems and (2) testing of weapon and aircraft compatibility, including weapon carriage release, separation, and accuracy. The Air Force Flight Test Center, Edwards Air Force Base, California, Edwards Air Force conducts and supports tests of manned and unmanned aerospace vehi- Base cles and recovery of research vehicles. The center uses two precision impact ranges, four spin test ranges, two high-altitude supersonic corri- dors, and two low-level routes. Mission control consists of nine separate control rooms using telemetry, radar, cinetheodolite, real-time processing, and related instrumentation to conduct the testing. The center conducts performance and flying quality evaluations, precision bombing, rocket firing, photo and infrared resolution, and radar fidelity testing. Page 12 GAO/NSIAMKl-91 Test Facilities Contents Letter Appendix I Synopses of Test Functions Performed At The Test Ranges We Visited Appendix II 14 Major Contributors to This Report Table 1: Hypothetical Test Site Use for 1 Work Week 6 Figure Figure 1: Eglin Test Environment 4 Abbreviations DOD Department of Defense MKTFH Major Range and Test Facility Base OSD Office of the Secretary of Defense Page 10 GAO/NSIADSOSl Test Facilities B-210919 As requested, we did not obtain written agency comments on this report. However, we discussed the issues with officials from the Office of the Deputy Director, Defense Research and Engineering (Test and Evalua- tion), and with officials from the four ranges we visited. They generally agreed that our approach had potential and could easily be imple- mented, and we incorporated their comments as appropriate. We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Deputy Director, Defense Research and Engineering (Test and Evaluation). Please contact me at (202) 275-4587 if you or your staff have any ques- tions concerning this report. Other major contributors to this report are listed in appendix II. Sincerely yours, Paul F. Math Director, Research, Development, Acquisition, and Procurement Issues Page S GAO/NSIAMIO-91 Test Facilities B-210919 (2) nonuse of available capacity because no tests were requested. We define test site “use” as the time needed not only for testing but also for the activities necessary to support tests, such as test preparation and site maintenance. Using Our Proposed Table 1 shows how use and nonuse of hypothetical test sites on a range might be tracked and reported, including consideration of legitimate Framework in a constraints on testing. This example is provided to show how use and Hypothetical Situation nonuse categories could be established and data recorded. Available capacity is based on a work schedule of 40 hours a week. Table 1: Hypothetical Test Site Use for 1 Work Week Site A Site B Hours Percent Hours Percent Site m use for Maintenance 10~~ 25.0 ..~~.. 9 22.5 Test preparation 9 22.5 7 17.5 Testing 6 15.0 4 10.0 Total 25 62.5 20 50.0 Site not in use due to Weather 5 12.5 0 0.0 - Cancellations 0 00 ~____~. 10 25.0 Safety conflict 0 0.0 IO 25 0 No scheduled tests 10 250 -0 00 Total 15 37.5 20 50.0 Total 40 100.0 40 100.0 In the example above, site A had 10 hours of unused available capacity when no tests were scheduled. However, site B had nonuse that resulted from legitimate constraints: cancellations and safety conflicts. Minimal Modifications to We anticipate that only minor modification to the ranges’ data collection Data Collection Systems systems will be necessary to use our proposed framework. Unlike the Range Utilization Measurement System, our approach does not include Needed details of actual personnel and equipment use, because we believe such details are not necessary to obtain comparable use statistics. Instead, we focus on the reasons for nonuse of test sites. Therefore, current data collection systems would need to be modified to account for whether Page 6 GAO/NSIAlHO-91 Test Facilities B-210919 for defining availability of a test site for air testing. Boundaries would be established for air and ground space necessary to perform specific tests. (See fig. 1 for a diagram of the test environment at Eglin, a major range where air testing is conducted. Appendix I gives a synopsis of test functions at Eglin and other ranges we visited while developing our proposal.) Figure 1: Eglin Test Environment Test Sites I Range I Restricted Real Estate Restricted Airspace Under our approach, WD would be able to identify, categorize, measure, and compare the use and nonuse of similar air testing facilities, thus accounting for total capacity. With minimal added effort in data collec- tion, MRTFB ranges could adopt our approach on a trial basis to obtain an overview of capacity and use at air testing sites. We believe that our Page 4 GAO/NSlAtMO-91 Test Facilities B210919 - We previously expressed concerns about problems with the ranges’ information in 1987,’ and the House Committee on Appropriations Surveys and Investigations Staff also found similar problems in 1987.-’ As a result of its review, the Committee directed OSDto submit itn MRTFB management plan to the Committee in 1988 and to take a stronger role in managing the MKTFH. Current Information 1s Not The ranges currently maintain automated records of staff and equip- Comparable or Consistent ment for billing purposes and nonautomated records for scheduling tests requested by potential users. While these records are useful for sched- uling and billing, they cannot be used to measure test site use among ranges on a common basis or to identify potential excess capacity. The ranges’ test-scheduling systems identify personnel and instrumenta- tion availability while stressing safe operations. As part of the systems, the ranges produce daily schedules, and at the completion of each day, they note cancellations and tests done, including the reasons test sites were not used. However, the four ranges we visited tracked different units of measurement, even for similar test functions. For example, one range tracked individual equipment used while others tracked staff- hours used in testing or the number of missions flown. Common defini- tions of capacity, or measures of the availability of sites to perform tests or test functions, would be required to measure use of the sites on a common basis. Also, the ranges did not consistently track and report the reasons that test facilities were not used. As a result, WD could not determine poten- tial capacity or unused capacity where additional test work load could be accommodated. Further, the data available did not identify con- straints to testing, such as weather conditions. The lack of such data resulted in the appearance of low use and nonuse of existing capacity, even though the ranges may have been prevented from testing for rea- sons beyond their control. Achieving a common measurement for nonuse of test sites that recognizes constraints on site use is particularly important. ‘Letters to the Director,OperationalTest and Evaluation, and the Acting Deputy UnderSecretaryof Defense(DevelopmentTest and Evaluation). datedApnl 16, 1987 ‘Managementof the Ma~urRangeand Test Faclhty Baseof the Departmentof Defense(HouseCom- mittee on Appropriations,Surveysand InvestigationsStaff, June 9, 1987). Page 2 GAO/NSIAD-99.91Test Facilities
Test and Evaluation: A Proposed Framework for Measuring the Use of Test Facilities
Published by the Government Accountability Office on 1990-08-08.
Below is a raw (and likely hideous) rendition of the original report. (PDF)