oversight

Test and Evaluation: A Proposed Framework for Measuring the Use of Test Facilities

Published by the Government Accountability Office on 1990-08-08.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United   States   General   Accounting   Office

    GAO          Report to the Chairman, Subcommittee
                 on Defense, Committee on
                 Appropriations, House of
                 Representatives
i
i. August 1990
.-,              TEST AND
                 EVALUATION
                 A Proposed
                 Framework for
                 Measuring the Use oft
                 Test Facilities
GAO
                        United States
                        General Accounting Office
                        Washington, D.C. 20548

                        National Security and
                        International Affairs Division

                        B-210919

                        August 8,199O

                        The Honorable John P. Murtha
                        Chairman, Subcommittee on Defense
                        Committee on Appropriations
                        House of Representatives

                        Dear Mr. Chairman:

                        The military services operate and maintain 21 major test ranges, collec-
                        tively known as the Major Range and Test Facility Base (MRTFB), which
                        have investments over 525 billion and an annual operating budget of
                        about $3.5 billion. The Office of the Secretary of Defense (OSD) is respon-
                        sible for centralized management of the ranges and for making budg-
                        etary decisions on range expansion, consolidation, and improvements.
                        However, the ranges do not collect and report standard information to
                        assess overall range capacity and use.

                        In response to the former Chairman’s request, we have developed a
                        framework for measuring the use of MRTFB test ranges. Our proposed
                        framework, which the ranges could adopt with minor data collection
                        modifications, should be useful to both defense and congressional
                        decisionmakers.


                        MRTFISranges comprise large land, sea, and air masses that are divided
Background              into various test sites, Generic equipment and instrumentation, such as
                        tracking radars, can be found throughout the ranges, while individual
                        test sites contain equipment and instrumentation required for specific
                        types of tests. The ranges are used for such purposes as testing aircraft,
                        bombs, and missiles; tanks and other tracked vehicles; ordnance; and
                        environmental effects on weapon systems and underwater tests of
                        munitions.


                         Although DOD has long recognized the need for a common measurement
Problems in Assessing    of its test ranges’ capacity and use, the ranges do not collect comparable
Test Capabilities        data. As a result, DOD cannot readily identify either excess testing
                         capacity or the need for additional capacity.




                         Page 1                                           GAO/NSIAB90-91 Test Facilities
                             H-210919




DOD-Proposed Range Use       In 1988, DOD'S Range Commanders Council established a working group
                             to develop a Range Vtilization Measurement System that would accu-
Measurement System Was       rately convey the degree to which each range’s test capacity was being
Not Adopted                  used. The system was developed in response to an informal request from
                             the Deputy Director, Defense Research and Engineering (Test and Eval-
                             uation), the OSDofficial responsible for MKTFH management issues. But
                             according to the Deputy Director’s office, OSDofficials did not adopt the
                             system because they believed it would be labor intensive and t,oo costly.
                             Implementing the system would have entailed additional data collection
                             efforts, such as detaikd information on personnel and equipment use.

                             In addition to collecting data on personnel and equipment use, the Range
                             I Ttilization Measurement System would have recorded a general activity
                             indicator (to bc determined by each range) and land/sea/airspace
                             reported by (1) total missions using a particular land/sea/airspace,
                             (2) the number of mission schedules denied because of land/sea/air-
                             space constraints, and (3) the total number of requested schedules
                             involving land/sea/airspace. The system provided for recording unavail-
                             ability of a test site for testing owing to constraining factors, such as
                             time needed to maintain the site and prepare it for testing or delays
                             caused by weather conditions. Further discussion of the DOD proposal
                             versus our proposal is contained on pages 5 and 6.

                                                    -
                             Recognizing that test ranges that offer similar capabilities carry out sim-
Our Proposed                 ilar test functions, we developed a proposal to measure use and capacity
Framework for                of MRTFB air testing sites based on a functional approach. We selected air
Measuring the Use of         testing-that     is, air-to-air or air-to-ground testing and aircraft flight
                             performance testing-because it represents the majority of testing done
Test Facilities              within the MIITFR.

                             The three main components of our proposed framework are to

                         . identify similar test functions,
                         . establish common capacity levels, and
                         l identify what types of constraints prevent testing at a test site.

                             By defining functional categories within air testing, DOD could identify
                             test sites that provide similar air and ground space, equipment, and
                             instrumentation to perform various air test functions and could then
                             measure their use. By assuming that test equipment and instrumenta-
                             tion on a site are necessary, although not all items may be used for each
                             test, our approach focuses on airspace and real estate as the key factors


                             Page3                                             GAO/NSIA@SO-91
                                                                                            Test F’acilities
                          &210919




                          proposal then could potentially be applied to other types of tests, such
                          as testing of ordnance and vehicles.


Emphasis on Measures of   We propose addressing the use of test facilities from a functional per-
                          spective, considering the capacity and use of the facilities. To determine
Functional U.ses          available test capacity, all test functions need to be identified and
                          tracked. Examples of functional categories include (1) testing of aircraft
                          flight performance and (2) testing of aircraft weapon systems, such as
                          test and evaluation of fire-control and bombing systems against fixed
                          and moving ground targets.

                          Before our proposed framework can be applied, test sites and testing
                          functions will need to be defined using standard terms of reference. For
                          example, the key factors for defining air test sites to be used for a test
                          function should be required airspace and real estate. Range officials
                          agreed that standard definitions would be critical to drawing conclu-
                          sions on comparable testing capabilities.


Establishing Capacity     We propose that “capacity” be defined as available daylight hours, with
                          potential maximum capacity as 12 hours a day. In air testing, daylight is
Levels                    generally required to record flight performance or bomb impact. How-
                          ever, even when 12 hours of daylight are available, current salary and
                          personnel ceilings restrict personnel resources to 8 hours a day and/or
                          40 hours a week.


Identifying Constraints   Several key factors that affect range use are generally outside the
That Prevent Testing      ranges’ control. Test sites are often not used because of safety restric-
                          tions resulting from adjacent testing. For example, a piece of real estate
                          may be restricted from use because another test is using the airspace
                          overhead. Furthermore, weather sometimes prevents testing, which
                          results in the need to reschedule. Finally, range users often cancel
                          planned tests. On the other hand, test sites that are available for testing,
                          but have not been scheduled for use by customers, represent nonuse of
                          available capacity.

                          Our proposal is similar to the DOD-proposed Range Utilization Measure-
                          ment System in that we recognize the need to categorize and account for
                          constraining factors that prevent test sites from being used. However,
                          we distinguish between (1) ranges not available for testing because of
                          constraining factors that prevent use (see fig. 1, for example) and


                           Page 5                                           GAO/NSlALHO-91Test Facilities
                             8210919




                             nonuse of test sites was due to constraints on site use, such as airspace
                             restricted for other tests, or to no requests to use existing capacity.

                             To capture data on use and nonuse, as well as reasons for nonuse, the
                             automated data now used for billing purposes could be combined with
                             the nonautomated data used to schedule tests. Although the four ranges
                             we visited did not automatically identify the reasons for nonuse, they
                             considered these reasons when scheduling tests. Thus, data on nonuse
                             were either already available or could be collected easily by establishing
                             standard categories, such as cancellations of planned tests, for the
                             periods when testing is not done. Data in these categories could then be
                             formally entered into existing data collection systems. As officials at the
                             sites visited pointed out, data collection guidelines would need to be
                             implemented uniformly to obtain comparable statistics on all sites.

                             Comparable statistics would be useful in managing existing test
                             resources as well as in justifying requests for increased resources. Such
                             statistics could identify (1) unused capacity or (2) sites at which use
                             could be increased by transferring personnel or equipment. Overtime or
                             shift assignments could be considered, when feasible, to extend use of a
                             test site beyond the 40.hour work week if use measurements showed
                             that additional capacity was needed.


                             In developing common measures of use and capacity for sites having
Objectives, Scope, and       similar test functions, WCvisited four major test ranges:
Methodology
                         - the Armament Division, 3246th Test Wing, Eglin Air Force Base,
                           Florida;
                         - the Naval Air Test Center, Patuxent River, Maryland;
                         - the Air Force Flight Test Center, Edwards Air Force Base, California;
                           and
                         l the Naval Weapons Center, China Lake, California.

                             We analyzed how the ranges scheduled tests, billed customers, and
                             accounted for use of test facilities. We held discussions with OSD and
                             range officials on the feasibility of developing a common measurement
                             method and other topics. Our review was performed from January to
                             <July 1989 according to generally accepted government auditing
                             standards.




                              Page 7                                           GAO/NSIAD-99.91Test Facilities
Page 9   GAO/NSIAWO-91 Test Facilities
Page 11   GAO/NSIAtWO-91Test Facilities
                       Appendix I
                       Synopsesof Test Functions Performed At
                       The Test RangesWeVisited




                       The Naval Weapons Center, China Lake, California, is a large and com-
Naval Weapons Center   plex range. Test capabilities include supersonic test tracks, an explosive
                       test site, a propulsion test site, a live ordnance environmental test site,
                       and an electronic warfare threat environment site. Extensive instrumen-
                       tation is available, including radars, theodolites, cameras, telemetry sys-
                       tems, and communications. The center conducts testing and evaluation
                       of air- and surface-launched weapons, electronic warfare systems, mis-
                       siles, life-support systems, and parachute systems.




                        Page 13                                         GAO/NSIADM-91 Test Facilities
                   . ,,



              -4




          .
-.
                                -,-.---,a               -   -    I   ,.   ,,.                     . ---




     g                    .‘.
                                            %cp~?~ts for copies of GAO reports                should be sent to:
     *
     si
     i.
                                            U.S. General Accounting             Office
                                            Post Office Box 6015
     ;                                      Gaithersburg, Maryland              20877
     I

                                            Telephone           202-275-5241

                                            The fiit  five copies of each report              are free. Additional   copies are
                                            $2.00 each.

                                            There is a 25% discount             on orders for 100 or more copies mailed to a
                                            single address.

                                            Orders must be prepaid              by cash or
                                            T n i ‘iT ‘:I:“! Fn~F=rllm             *IT warl
Appendix II

Major Contributors to This Report


                            Michael E. Motley, Associate Director
National Security and       Lester C. Farrington. Assistant Director
International Affairs       Patricia L. Martin, Evaluator-in-Charge
Division,     Washington,   Anne W. Howe, Evaluator
                            Marilyn Mauch, Social Science Analyst
D.C.                        Alfred Lilliendahl, Operations Research Analyst




                            Page 14                                           GAO/NSIAD-99.91Test Facilities
Appendix I

Synopses of Test F’unetions Performed At
The Test Ranges We Visited

                        The Armament Division, 3246th Test Wing, Eglin Air Force Base,
Eglin Air Force Base    Florida, manages a large and complex range encompassing 724 square
                        miles of land and 98.000 square miles of test area in the Gulf of Mexico.
                        The test wing is responsible for the development, test, and evaluation of
                        all Air Force nonnuclear air armaments, electronic combat systems,
                        target acquisition and weapon delivery systems, base intrusion and
                        detection systems, electronic systems, and aerial targets. It is also
                        responsible for climatic simulation test and evaluation and determina-
                        tion of electromagnetic and electro-optical signatures. Typically, Eglin
                        tests

                        ordnance and munitions, emphasizing warhead performance, fuzing, ter-
                        minal effects, aerodynamics, ballistics, and aircraft compatibility;
                        air-to-air and air-to-ground operations that focus on target acquisition
                        and weapon delivery systems;
                        electronic system capabilities; and
                        sensors, emphasizing electro-optical, laser, infrared, and millimeter
                        wave operations.


                        The Naval Air Test Center, Patuxent River, Maryland, maintains flight
Naval Air Test Center   test facilities that provide actual and simulated conditions for all
                        in-service and planned naval aircraft weapon system programs. Flight
                        testing is conducted in 50,000 square miles of restricted airspace over
                        the Chesapeake Bay and offshore operating areas in the Atlantic Ocean.
                        The Chesapeake Test Range, with its computer-linked video, theodolite,
                        radar, and laser tracking equipment and telemetry capability, is used for
                        (1) flight testing of aircraft and airborne systems and (2) testing of
                        weapon and aircraft compatibility, including weapon carriage release,
                        separation, and accuracy.


                        The Air Force Flight Test Center, Edwards Air Force Base, California,
Edwards Air Force       conducts and supports tests of manned and unmanned aerospace vehi-
Base                    cles and recovery of research vehicles. The center uses two precision
                        impact ranges, four spin test ranges, two high-altitude supersonic corri-
                        dors, and two low-level routes. Mission control consists of nine separate
                        control rooms using telemetry, radar, cinetheodolite, real-time
                        processing, and related instrumentation to conduct the testing. The
                        center conducts performance and flying quality evaluations, precision
                        bombing, rocket firing, photo and infrared resolution, and radar fidelity
                        testing.



                        Page 12                                         GAO/NSIAMKl-91 Test Facilities
Contents


Letter
Appendix I
Synopses of Test
Functions Performed
At The Test Ranges
We Visited
Appendix II                                                                                      14
Major Contributors to
This Report
                        Table 1: Hypothetical Test Site Use for 1 Work Week                       6

Figure                  Figure 1: Eglin Test Environment                                          4




                        Abbreviations

                        DOD       Department of Defense
                        MKTFH     Major Range and Test Facility Base
                        OSD       Office of the Secretary of Defense


                        Page 10                                        GAO/NSIADSOSl Test Facilities
B-210919




As requested, we did not obtain written agency comments on this report.
However, we discussed the issues with officials from the Office of the
Deputy Director, Defense Research and Engineering (Test and Evalua-
tion), and with officials from the four ranges we visited. They generally
agreed that our approach had potential and could easily be imple-
mented, and we incorporated their comments as appropriate.

We are sending copies of this report to the Secretary of Defense; the
Secretaries of the Air Force, Army, and Navy; and the Deputy Director,
Defense Research and Engineering (Test and Evaluation).

Please contact me at (202) 275-4587 if you or your staff have any ques-
tions concerning this report. Other major contributors to this report are
listed in appendix II.

Sincerely yours,




Paul F. Math
Director, Research, Development,
  Acquisition, and Procurement Issues




Page S                                          GAO/NSIAMIO-91 Test Facilities
                                      B-210919




                                      (2) nonuse of available capacity because no tests were requested. We
                                      define test site “use” as the time needed not only for testing but also for
                                      the activities necessary to support tests, such as test preparation and
                                      site maintenance.


Using Our Proposed                    Table 1 shows how use and nonuse of hypothetical test sites on a range
                                      might be tracked and reported, including consideration of legitimate
Framework in a                        constraints on testing. This example is provided to show how use and
Hypothetical Situation                nonuse categories could be established and data recorded. Available
                                      capacity is based on a work schedule of 40 hours a week.

Table 1: Hypothetical Test Site Use
for 1 Work Week                                                                        Site A               Site B
                                                                                   Hours Percent        Hours Percent
                                      Site m use for
                                         Maintenance                                   10~~     25.0
                                                                                              ..~~..        9     22.5
                                         Test preparation                               9       22.5        7     17.5
                                         Testing                                        6       15.0        4     10.0
                                      Total                                            25       62.5       20     50.0

                                      Site not in use due to
                                         Weather                                        5       12.5        0      0.0
                                      - Cancellations                                   0        00
                                                                                              ~____~.      10     25.0
                                         Safety conflict                                0        0.0       IO     25 0
                                         No scheduled tests                            10       250       -0       00
                                      Total                                            15      37.5        20     50.0

                                      Total                                            40     100.0        40    100.0


                                      In the example above, site A had 10 hours of unused available capacity
                                      when no tests were scheduled. However, site B had nonuse that resulted
                                      from legitimate constraints: cancellations and safety conflicts.


Minimal Modifications to              We anticipate that only minor modification to the ranges’ data collection
Data Collection Systems               systems will be necessary to use our proposed framework. Unlike the
                                      Range Utilization Measurement System, our approach does not include
Needed                                details of actual personnel and equipment use, because we believe such
                                      details are not necessary to obtain comparable use statistics. Instead, we
                                      focus on the reasons for nonuse of test sites. Therefore, current data
                                      collection systems would need to be modified to account for whether




                                       Page 6                                           GAO/NSIAlHO-91 Test Facilities
                                                     B-210919




                                                     for defining availability of a test site for air testing. Boundaries would
                                                     be established for air and ground space necessary to perform specific
                                                     tests. (See fig. 1 for a diagram of the test environment at Eglin, a major
                                                     range where air testing is conducted. Appendix I gives a synopsis of test
                                                     functions at Eglin and other ranges we visited while developing our
                                                     proposal.)

Figure 1: Eglin Test Environment




                       Test Sites                I
                                                                   Range                    I
                                    Restricted       Real Estate
                                                                               Restricted       Airspace




                                                     Under our approach, WD would be able to identify, categorize, measure,
                                                     and compare the use and nonuse of similar air testing facilities, thus
                                                     accounting for total capacity. With minimal added effort in data collec-
                                                     tion, MRTFB ranges could adopt our approach on a trial basis to obtain an
                                                     overview of capacity and use at air testing sites. We believe that our


                                                     Page 4                                                GAO/NSlAtMO-91 Test Facilities
                                 B210919




                                                       -
                                 We previously expressed concerns about problems with the ranges’
                                 information in 1987,’ and the House Committee on Appropriations
                                 Surveys and Investigations Staff also found similar problems in 1987.-’
                                 As a result of its review, the Committee directed OSDto submit itn MRTFB
                                 management plan to the Committee in 1988 and to take a stronger role
                                 in managing the MKTFH.


Current   Information   1s Not   The ranges currently maintain automated records of staff and equip-
Comparable or Consistent         ment for billing purposes and nonautomated records for scheduling tests
                                 requested by potential users. While these records are useful for sched-
                                 uling and billing, they cannot be used to measure test site use among
                                 ranges on a common basis or to identify potential excess capacity.

                                 The ranges’ test-scheduling systems identify personnel and instrumenta-
                                 tion availability while stressing safe operations. As part of the systems,
                                 the ranges produce daily schedules, and at the completion of each day,
                                 they note cancellations and tests done, including the reasons test sites
                                 were not used. However, the four ranges we visited tracked different
                                 units of measurement, even for similar test functions. For example, one
                                 range tracked individual equipment used while others tracked staff-
                                 hours used in testing or the number of missions flown. Common defini-
                                 tions of capacity, or measures of the availability of sites to perform tests
                                  or test functions, would be required to measure use of the sites on a
                                  common basis.

                                 Also, the ranges did not consistently track and report the reasons that
                                 test facilities were not used. As a result, WD could not determine poten-
                                 tial capacity or unused capacity where additional test work load could
                                 be accommodated. Further, the data available did not identify con-
                                 straints to testing, such as weather conditions. The lack of such data
                                 resulted in the appearance of low use and nonuse of existing capacity,
                                 even though the ranges may have been prevented from testing for rea-
                                 sons beyond their control. Achieving a common measurement for nonuse
                                 of test sites that recognizes constraints on site use is particularly
                                 important.



                                  ‘Letters to the Director,OperationalTest and Evaluation, and the Acting Deputy UnderSecretaryof
                                  Defense(DevelopmentTest and Evaluation). datedApnl 16, 1987
                                  ‘Managementof the Ma~urRangeand Test Faclhty Baseof the Departmentof Defense(HouseCom-
                                  mittee on Appropriations,Surveysand InvestigationsStaff, June 9, 1987).



                                  Page 2                                                        GAO/NSIAD-99.91Test Facilities