oversight

Electronic Combat: Consolidation Master Plan Does Not Appear to Be Cost-Effective

Published by the Government Accountability Office on 1997-07-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                  United States General Accounting Office

GAO               Report to Congressional Requesters




July 1997
                  ELECTRONIC
                  COMBAT
                  Consolidation Master
                  Plan Does Not Appear
                  to Be Cost-Effective




GAO/NSIAD-97-10
             United States
GAO          General Accounting Office
             Washington, D.C. 20548

             National Security and
             International Affairs Division

             B-272629

             July 10, 1997

             The Honorable Connie Mack
             The Honorable Bob Graham
             United States Senate

             The Honorable Joe Scarborough
             House of Representatives

             In response to your request, we have reviewed the Department of
             Defense’s (DOD) Electronic Combat Consolidation Master Plan. As agreed
             with your office, our objective was to assess the costs and benefits of
             DOD’s consolidation plans for open air ranges, hardware-in-the-loop
             facilities, and installed system test facilities used in electronic combat
             testing.


             In its report on the Fiscal Year 1996 National Defense Authorization Act,
Background   the Senate Armed Services Committee criticized DOD for not having a clear
             approach to consolidating test infrastructure and recommended
             reductions in DOD’s Test and Evaluation support accounts. The Senate
             Appropriations Committee agreed with the authorizing committee,
             recommended reductions to the fiscal year 1996 Test and Evaluation
             support accounts, and acknowledged the need to constrain spending in
             this area. Subsequently, in the Fiscal Year 1996 National Defense
             Appropriations Act, the Congress limited the obligation of specified funds
             until DOD provided the defense authorizing and appropriating committees
             with an Electronic Combat Consolidation Master Plan to establish a
             DOD-wide infrastructure for electronic combat testing. In March 1996, DOD
             published its Master Plan.

             In transmitting the Master Plan to the Congress, the Under Secretary of
             Defense for Acquisition and Technology stated that DOD would revisit the
             Plan in the broader context of section 277 of the National Defense
             Authorization Act for Fiscal Year 1996, and adjust the Plan as appropriate.
             Section 277 directs DOD to develop a consolidation and restructure plan for
             its laboratories and test and evaluation centers for the 21st century. This
             effort is not yet complete.

             According to the Master Plan, DOD considered 17 of the services’ electronic
             combat test facilities for consolidation. The Army controls 4 of the
             17 facilities, the Navy controls 6, and the Air Force controls 7. The




             Page 1                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                       B-272629




                       conclusion of the Master Plan is that the assets of three of the seven
                       facilities managed by the Air Force will be moved to other Air Force
                       locations. No interservice consolidations and no intraservice consolidation
                       of the four Army or six Navy facilities are proposed in the Plan. The three
                       facilities to be relocated are

                   •   the Air Force Electronic Warfare Evaluation Simulator (AFEWES) in Fort
                       Worth, Texas;
                   •   the Real-time Electronic Digitally Controlled Analyzer Processor (REDCAP)
                       in Buffalo, New York; and
                   •   the Electro-Magnetic Test Environment (EMTE) at Eglin Air Force Base,
                       Florida.

                       AFEWES   is a specialized hardware-in-the-loop facility that simulates
                       individual radar and missile threats to aircraft and electronic combat
                       hardware. REDCAP is a specialized hardware-in-the-loop facility that
                       simulates an integrated air defense system with command, control, and
                       communications networks. EMTE is an open air range providing radar and
                       simulated missile threats to aircraft in flight; it is collocated at Eglin Air
                       Force Base with the Air Force’s development and test and evaluation
                       activities for armaments. Installed system test facility consolidation was
                       not proposed in the Master Plan. For purposes of this review, we focused
                       on three open air ranges, two hardware-in-the-loop facilities, and two
                       installed system test facilities. The remaining 10 are other kinds of
                       electronic combat test facilities, such as research laboratories or radar
                       cross-section measurement facilities or are service unique capabilities.
                       DOD’s electronic combat test process and the role the various kinds of
                       facilities play in that process are explained briefly in appendix I.


                       Implementation of the Electronic Combat Consolidation Master Plan will
Results in Brief       result in less effective electronic combat testing capabilities.

                   •   The planned relocation of EMTE will eliminate DOD’s current capability to
                       test electronic combat systems in conditions that typify many potential
                       threat locations. DOD will be left with two open air ranges with very similar
                       environmental characteristics and will no longer have the ability to test in
                       diverse conditions needed to understand environmental effects on
                       electronic combat systems.
                   •   The planned REDCAP relocation will mean replacing existing hardware
                       simulation capability with digital computer models, thus reducing DOD’s




                       Page 2                        GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                           B-272629




                           current capability to simulate realistic aircraft strike scenarios with high
                           confidence and fidelity.

                           The Master Plan did not contain any cost analysis and did not identify any
                           savings expected from the consolidations. Estimates used to support 1995
                           Base Closure and Realignment Commission (BRAC) deliberations, as well
                           as data provided by users indicate that the consolidation may increase
                           DOD’s electronic combat testing costs. In addition, the Master Plan does
                           not contain any analysis or recommendations regarding consolidation of
                           installed system test facility workloads across the services although the
                           Navy and the Air Force are spending $512 million for construction of
                           another anechoic chamber to provide a controlled electromagnetic
                           environment at Patuxent River, Maryland, and other upgrades to their
                           current primary installed system test facilities at Patuxent River and
                           Edwards Air Force Base, California.

                           Consequently, the Master Plan, if implemented, may not achieve the most
                           cost-effective DOD-wide infrastructure. The root cause of this was DOD
                           officials’ inability to overcome service parochialism during the Master
                           Plan’s development. This parochialism resulted in a “gentlemen’s
                           agreement” between the Air Force and the Navy to focus on intraservice
                           rather than interservice consolidations. Prior joint service studies
                           performed on an interservice basis had identified alternatives for more
                           cost-effective consolidations. However, the recommendations of these
                           studies were never implemented. If this continues, service rivalry could
                           adversely affect DOD’s ongoing, congressionally mandated
                           section 277/vision 21 consolidation effort, which is considering the
                           broader issue of DOD’s testing and laboratory facilities.



Principal Findings

Planned Consolidation of   The proposal in the Master Plan to relocate EMTE would eliminate a test
Open Air Ranges Will       facility that provides unique advantages and keep two testing facilities
Reduce Effectiveness       with overlapping capabilities. DOD’s acquisition regulations require systems
                           to be evaluated in operationally realistic environments, including the
                           expected range of natural environmental conditions. Currently, its
                           electronic combat open air ranges replicate diverse threat environments
                           where the services must be prepared to conduct operations.




                           Page 3                        GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                             B-272629




Testing Equipment in         DOD’s 5000.2R acquisition regulations require testing in natural
Diverse Environments Is      environmental conditions representative of intended areas of operations
Critical                     (e.g. temperature, pressure, humidity, fog, precipitation, clouds, blowing
                             dust and sand, steep terrain, storm surge and tides, etc.). Testing in diverse
                             conditions provides performance data needed to understand
                             environmental effects on electronic combat systems. This information is
                             critical to making informed acquisition and mission planning decisions,
                             thereby reducing the risk of buying ineffective equipment and the potential
                             for casualties during wartime.

                             DOD  studies also document the importance of testing electronic combat
                             equipment in diverse environments. For example, a 1994 joint service
                             study of electronic combat open air ranges expressed the need for
                             electronic combat testing in the correct natural environment. Test results
                             for electronic combat systems demonstrate that performance can differ
                             significantly in differing environments.

                             Testing in diverse environments is also important for collecting data to
                             support development of realistic computer models. DOD believes modeling
                             and simulation can be used to reduce the cost of live tests, but to improve
                             levels of confidence in models they must be built on high fidelity data
                             collected from diverse environments.


Plan Would Eliminate         DOD’s proposed open air range consolidation as described in the Master
Diversity Found in Current   Plan would eliminate diversity by keeping only desert ranges and thereby
Open Air Ranges              reduce electronic combat open air range testing effectiveness. The Air
                             Force and the Navy control three primary open air ranges for testing
                             electronic combat systems. These include two western ranges, one at
                             China Lake, California, and one managed by Edwards Air Force Base,
                             California. Both feature dry, desert climates with steep, rocky terrain. The
                             third range, EMTE, at Eglin Air Force Base on the Florida panhandle,
                             features a land/sea interface, high humidity, and a subtropical, forested
                             environment, and an over water test range.

                             The Master Plan states that preservation of militarily unique electronic
                             combat test facilities was an important criterion for deciding which
                             facilities to close. However, EMTE is unique among DOD’s open air ranges,
                             and the 1994 joint service study noted that one of the primary
                             disadvantages of closing EMTE would be the loss of terrain and
                             geographical diversity, since both remaining ranges would be located in
                             the desert.



                             Page 4                        GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                                       B-272629




Current Open Air Ranges                Both western ranges provide a capability for conducting essential
Represent Potential Threat             electronic combat testing over terrain representative of projected middle
Environments                           eastern threat environments. Conversely, EMTE provides DOD with an
                                       environment more typical of most of the other projected U.S. threat
                                       locations, including North Korea and the Balkans. Table 1 identifies the
                                       terrain of countries that are representative of possible locations for future
                                       conflicts that are of concern to the United States. In comparison, table 2
                                       demonstrates that the unique environmental characteristics of EMTE—over
                                       water, land/sea interface, and foliage—are prevalent in most of the
                                       potential threat locations identified in table 1.

Table 1: Potential Threat Locations
and Terrain Correlation                                            Sea/land
                                       Location       Over water   interface      Desert        Foliage       Mountain
                                       Iraq                                       X                           X
                                       Iran           X            X              X             X             X
                                       N. Korea       X            X                            X             X
                                       China          X            X              X             X             X
                                       Libya          X            X                            X             X
                                       Cuba           X            X                            X             X
                                       Balkans        X            X                            X             X

Table 2: Open Air Ranges and Terrain
Correlation                                                        Sea/land
                                       Location       Over water   interface      Desert        Foliage       Mountain
                                       EMTE           X            X                            X
                                       China Lake                                 X                           X
                                       Air Force                                  X                           X
                                       Western Test
                                       Range



                                       The Master Plan proposal to move the REDCAP facility from Buffalo and
REDCAP at New                          colocate it with the Air Force’s installed system test facility at Edwards Air
Location Will Be Less                  Force Base will reduce electronic combat testing effectiveness. The intent
Capable                                is to reestablish what the Air Force calls a “core” REDCAP capability at the
                                       new location by developing a computer model to simulate REDCAP
                                       hardware. However, the model will not simulate all of the current REDCAP
                                       testing features.

                                       Establishing a core REDCAP capability means not utilizing much of the
                                       REDCAP hardware, and its associated functions, even though the Air Force




                                       Page 5                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                          B-272629




                          completed upgrading this hardware in 1996 at a cost of $75 million over
                          the past 8 years. The core REDCAP at the proposed new location will be less
                          capable than the complete REDCAP at its current location.

                          Some of the REDCAP hardware functions that the Air Force does not plan to
                          make available in core REDCAP do not exist anywhere else in DOD.
                          According to DOD and Air Force officials, the REDCAP facility in Buffalo is
                          unique. For instance, REDCAP can currently simulate a realistic scenario of
                          a strike package of multiple aircraft approaching targets protected by
                          multiple threat radars and threat aircraft incorporated into an integrated
                          air defense system. The proposed core REDCAP will not be able to simulate
                          this scenario. Simulating many aircraft versus many threat systems is
                          important because integrated air defense systems exist in a number of
                          potential threat locations and integrated defenses are projected by DOD to
                          be a growth area among potential threat nations.


                          The Master Plan did not contain any cost analysis or identify the savings
Planned                   expected from the consolidations. Our analysis of prior estimates used to
Consolidations May        support the 1995 BRAC deliberations and other data provided by users
Increase Costs            indicates the consolidations may increase DOD’s testing costs. More
                          specifically (1) BRAC-related data indicates that a complete EMTE relocation
                          would not be cost-effective, (2) cost estimates provided to BRAC regarding
                          the relocation of REDCAP and AFEWES were understated, and (3) increased
                          costs that will be incurred by user organizations were not considered in
                          Air Force cost estimates.


Master Plan Includes No   Senior Air Force test officials told us that the Air Force selected EMTE,
Evidence of Savings       REDCAP, and AFEWES for consolidation because they believed they would
                          ultimately save money by relocating them. The Electronic Combat
                          Consolidation Master Plan, however, includes no evidence that any
                          savings will result and, in fact, contains no cost data at all.

                          The Secretary of Defense recommended the relocation of REDCAP and
                          AFEWES and the partial relocation of EMTE to the 1995 BRAC. BRAC approved
                          the REDCAP relocation, rejected the AFEWES proposal, and significantly
                          scaled back the partial relocation of EMTE. The Master Plan, however,
                          incorrectly states that selecting EMTE for relocation reflects decisions of
                          the 1995 BRAC.




                          Page 6                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                           B-272629




BRAC Found No Savings in   The 1995 BRAC scaled back the Secretary’s recommendation to realign the
Relocating EMTE in Total   EMTE open air range at Eglin Air Force Base. DOD proposed transferring
                           17 systems designed to simulate various threat radars and missiles, but
                           BRAC determined that was too costly and would “never net a return on
                           investment.” Ultimately, however, BRAC did approve the movement of
                           10 systems (for which the BRAC account was eventually charged
                           $6.1 million), but required DOD to leave limited capability systems at Eglin
                           to support the Air Force’s Special Operations Forces, Armaments Division,
                           and Air Warfare Center, which are also at Eglin. Nevertheless, the 1996
                           Master Plan says the Air Force plans to “relocate” EMTE, not move just 10
                           systems.

                           According to Air Force officials, “relocate” means 17 systems will be
                           moved. Ten will be operated at the new location and 7 will be cannibalized
                           for parts. Air Force test officials maintain that the Special Operations
                           Forces, Air Warfare Center and Armaments Division do not need these
                           17 systems at Eglin, and they will leave behind some systems to meet the
                           customers’ needs. EMTE users, such as the Special Operations Forces and
                           the 53rd Test Wing and the Army Aviation Test Directorate, told us that the
                           systems the Air Force plans to leave will not meet their needs for
                           accomplishing realistic testing because they do not have the capability to
                           receive and process testing data for subsequent analysis. Air Force test
                           officials told us users can travel to the Air Force’s western test range to
                           meet their test requirements.


REDCAP Relocation Costs    To mitigate the impact of the reduction in REDCAP effectiveness described
Not Fully Disclosed        earlier in this report, the Air Force has awarded a $6.2-million contract to
                           design and build a digital computer model of REDCAP that it intends to use
                           instead of the REDCAP hardware that will be stored. This additional cost,
                           however, was not included in the Air Force cost estimate that BRAC used in
                           deciding to relocate REDCAP.

                           The Air Force had recommended to the 1995 BRAC that the REDCAP facility
                           be relocated to Edwards Air Force Base. The 1995 BRAC found that Air
                           Force cost estimates to relocate were understated, but decided to accept
                           the recommendation as they believed it would still result in overall
                           savings. As a result, the BRAC account makes available to the Air Force
                           $3.7 million to relocate REDCAP. Using Air Force cost figures, BRAC
                           projected the operating cost to the government of REDCAP at the new
                           location will be $100,000 compared to $1 million annually at the current




                           Page 7                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                           B-272629




                           location, BRAC anticipated a 4-year return on investment (4 x $0.9 million).
                           (The remainder of REDCAP’s operations are funded by customer receipts.)

                           Since the cost of the new computer model was not taken into account, the
                           Air Force will not achieve a relatively quick return on investment. The
                           additional $6.2 million means it will take an additional 7 years to recoup
                           costs based on Air Force projected savings of $0.9 million per year. This
                           11-year (4 + 7) return is well beyond the 1995 BRAC norm of seeking a
                           6-year or less return on investment.


AFEWES Move Delayed        The Air Force recommended to the 1995 BRAC that the AFEWES facility in
                           Fort Worth be relocated to Edwards Air Force Base. The Air Force had
                           estimated a cost of $8.9 million to close AFEWES and move it. BRAC did not
                           accept the recommendation though because BRAC estimated it would cost
                           $34.9 million to close the facility and would be over 100 years before a
                           return on investment was realized. Nevertheless, the Air Force included
                           the AFEWES relocation in the 1996 Master Plan. Air Force officials told us
                           they are now attempting to modify their outyear budgets so they can move
                           the AFEWES facility sometime in the year 2000 time frame.


User Costs Will Increase   Special Operations Forces based at Hurlburt Field, Florida, adjacent to
With EMTE Closure          Eglin Air Force Base, are users of EMTE. After the EMTE relocation,
                           however, Special Operations Forces’ electronic combat testing will be
                           conducted at the Air Force’s western test range. As a result, Special
                           Operations Forces officials estimate that their electronic combat testing
                           will cost $23 million over the next 5 years, whereas they have spent only
                           $4 million for electronic combat testing over the last 4 years.

                           We reviewed the analysis supporting this estimate and found it to be
                           realistic. The $19 million in additional cost results from sending aircraft,
                           their crews, and support personnel temporarily to the western test range
                           more often than in the past. In contrast, there are no temporary duty costs
                           associated with testing Special Operations Forces aircraft at EMTE.

                           In addition to the Special Operations Forces, another user organization
                           based at Eglin, the 53rd Test Wing, estimates that the proposed EMTE
                           relocation may cost them as much as an additional $1 million per year.
                           This additional cost would provide for an estimated 20 additional trips to
                           the Air Force’s western test range to perform electronic combat testing
                           that in the past has been performed at Eglin Air Force Base.



                           Page 8                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                         B-272629




                         DOD’s Master Plan does not contain any analysis or recommendations
Installed System Test    regarding consolidation of installed system test facility workloads across
Facility Consolidation   the services. The Navy and the Air Force are spending $512 million for
Not Practical            construction of a new anechoic chamber to provide a controlled
                         electromagnetic environment at Patuxent River, Maryland, and other
                         upgrades to their current primary installed system test facilities at
                         Patuxent River and Edwards Air Force Base, California. These projects
                         have progressed too far to make any interservice consolidation practical at
                         this time, however.

                         The Navy has a fighter-sized anechoic chamber, has already spent
                         $227 million, and has plans to spend an additional $101 million, to (1) add
                         a new, medium-sized anechoic chamber and (2) upgrade the electronic
                         combat test laboratory shared by both the fighter and medium-sized
                         chambers. The Navy is planning to have the medium-sized chamber
                         completed in fiscal year 1999.

                         Completion of this work is timed to conduct testing on the Navy’s E-6 and
                         P-3 aircraft. (These specialized aircraft are too large to fit into the fighter
                         sized facility.) Meanwhile, the Air Force has plans to spend over
                         $184 million through fiscal year 2002 to make the same electronic combat
                         test upgrades to its Edwards Air Force Base installed system test facility
                         as the Navy is making at Patuxent River.

                         The Edwards Air Force Base facility is large enough to accommodate any
                         military aircraft except a C-5 transport. Navy officials agreed that the
                         Edwards facility is large enough to accommodate their medium-sized E-6
                         and P-3 aircraft; however, they maintain that the Edwards facility is not
                         advanced enough right now to conduct the testing on these aircraft. Navy
                         officials also insist they cannot postpone their testing until fiscal year 2002
                         when the Edwards facility upgrade is scheduled to be completed.
                         Furthermore, they say, the Air Force has blocked out most of the available
                         test time at the Edwards facility for its future F-22 fighter, an aircraft that
                         would fit in the Patuxent River chamber.




                         Page 9                         GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                           B-272629




                           In the past 3 years, DOD has conducted two joint service studies of possible
More Cost-Effective        consolidation of electronic combat test facilities. One study done in 1994 is
Alternatives to            referred to as the Board of Directors study and is cited as justification for
Planned Relocations        the conclusions in the Master Plan.1 The other study is known as the 1995
                           Joint Cross Service Group study, which was done in support of the 1995
Ignored                    BRAC process.2 These studies identified a more cost-effective interservice
                           electronic combat consolidation as compared to the intraservice approach
                           reflected in the Master Plan. However, the lack of interservice cooperation
                           undermined the more cost-effective efforts.


Open Air Range             To reduce excess capacity, the Master Plan recommends relocating test
Consolidation Does Not     assets from EMTE to the western test range managed by Edwards Air Force
Reflect a More Effective   Base and cites the 1994 Board of Directors Study as justification.
                           According to the study, DOD’s open air range workload capacity is 6,000
Alternative                test hours per year, while actual workload in fiscal year 1993 was 4,867
                           test hours, and actual workload is projected to decline to 4,000 hours per
                           year. Based on this workload data, DOD determined it will only need two of
                           the three current open air range facilities in the future.

                           However, that 1994 study, as well as the 1995 Joint Cross Service Group
                           study done in support of the BRAC process, ranked EMTE as a more valuable
                           electronic combat test capability than the Navy’s China Lake open air
                           range. The 1994 study also projected that relocating test assets from China
                           Lake to EMTE and the Air Force’s western test range would produce about
                           $47 million more in savings over 5 years than relocating EMTE.

                           DOD and Air Force officials with knowledge of the studies told us that the
                           Navy participated fully in both studies, but once it became apparent that
                           EMTE would rank higher than China Lake, the Navy would not cooperate in
                           implementing the study’s conclusions.


Electronic Linking of      In addition to comparing the EMTE and China Lake open air ranges, the
REDCAP and AFEWES a        1994 Board of Directors Study considered the possibility of achieving
More Cost-Effective        “synergy” between hardware-in-the-loop facilities, like AFEWES or REDCAP,
                           by colocating them with installed system test facilities, like those
Alternative

                           1
                           The Board of Directors is made up of the Service Vice Chiefs in their role as the Test and Evaluation
                           Executive Agent. Board of Directors study team members were drawn from each of the services.
                           2
                            The Joint Cross Service Group was led by representatives of the Office of the Secretary of Defense
                           and included team members from each of the services. The group examined potential consolidations
                           for airframe and armaments testing, as well as electronic combat testing.



                           Page 10                                GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                          B-272629




                          maintained by the Air Force at Edwards, or the Navy at Patuxent River,
                          Maryland. However, the Board of Directors study concluded that
                          relocation would require 200 years to net a return on investment. Instead,
                          according to a 1995 study conducted for the Air Force, electronic linking
                          of REDCAP and AFEWES to an installed system test facility was far more
                          cost-effective than relocating them.

                          Despite the findings of these studies, the Air Force plans to relocate
                          AFEWES and REDCAP. At the same time, the Office of the Secretary of
                          Defense and the Navy are undertaking the High Level Architecture Project
                          to electronically link REDCAP and AFEWES’ hardware with the Navy’s
                          installed system test facility at Patuxent River. This link will allow DOD to
                          test electronic combat systems on an aircraft in an installed system test
                          facility and do hardware-in-the-loop testing without having to physically
                          move the systems to REDCAP or AFEWES. This approach is consistent with
                          the 1995 study commissioned by the Air Force.


                          The failure of the Master Plan effort to achieve any DOD-wide electronic
Master Plan Process       combat testing consolidations despite direction from the Congress to do
Stifled by Intraservice   so is due to service parochialism. This resulted in focusing on intraservice
Focus                     rather than interservice consolidations.

“Gentlemen’s Agreement”   According to officials involved in the development of the Master Plan,
Prevented Interservice    because no DOD-wide consolidations could be agreed upon, Air Force and
Open Air Range            Navy representatives responsible for writing the Master Plan reached a
                          “gentlemen’s agreement.” The agreement was that there would be no
Consolidation Effort      interservice consolidation until all intraservice consolidations were
                          complete. The impact of this agreement was that the Master Plan
                          consolidation effort for open air ranges focused only on whether to
                          relocate EMTE or the western test range since they are both Air Force
                          facilities, instead of focusing on all three open air ranges to ensure that the
                          two kept would represent what was in the best interest of all of DOD.


                          In a memorandum transmitting the Master Plan to the Congress in
Intraservice Focus        March 1996, the Under Secretary of Defense for Acquisition and
Could Interfere With      Technology stated that DOD would revisit the Master Plan in the broader
Broader                   context of section 277 of the National Defense Authorization Act for Fiscal
                          Year 1996, and adjust the Plan as appropriate. Section 277 directs DOD to
Consolidation Effort      develop a consolidation and restructure plan for its laboratories and test
                          and evaluation centers for the 21st century.



                          Page 11                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                     B-272629




                     This plan, which DOD calls vision 21, will be based on the requirements to
                     support the test and evaluation of future weapon systems and identify the
                     critical test facilities needed to support them. DOD maintains that vision 21
                     will include both intraservice and interservice restructuring. However,
                     based on the inability of DOD to implement proposed interservice
                     consolidations originating from its prior studies of electronic combat test
                     consolidation, we are concerned that the intraservice focus that interfered
                     with development of a DOD-wide Electronic Combat Master Plan will
                     undermine the vision 21 effort.


                     Because (1) the loss of electronic combat effectiveness was not given
Recommendation       adequate consideration in the development of DOD’s Electronic Combat
                     Consolidation Master Plan, (2) the Master Plan contained no costs or
                     evidence of savings, and (3) service parochialism was allowed to interfere
                     with development of the Master Plan, we recommend that the Secretary of
                     Defense take steps to make sure that the methodology for the ongoing
                     section 277/vision 21 effort include the following criteria: (1) accurate,
                     comparable, and reliable data on the true cost of operating the services’
                     test and evaluation infrastructure; (2) the needs of and costs to test facility
                     customers; (3) the maintenance of geographical and topographical
                     diversity in the test facility base; (4) the requirement that proposed
                     consolidations be cost-effective for DOD as a whole; and (5) measures to
                     ensure that implementation of cost-effective decisions cannot be
                     constrained or avoided.


                     Because DOD’s Electronic Combat Consolidation Master Plan may not
Matter for           provide for the most cost-effective DOD-wide infrastructure for electronic
Congressional        combat testing as directed by the Congress, the Congress may wish to
Consideration        consider directing the Secretary of Defense to defer the transferring of
                     electronic combat test assets until DOD completes its vision 21 plan for
                     restructuring its laboratories and test and evaluation centers.


                     In commenting on a draft of this report, DOD indicated that it did not agree
Agency Comments      with our findings, recommendation, or matter for congressional
and Our Evaluation   consideration. According to DOD’s response, the consolidations proposed
                     in the Electronic Combat Consolidation Master Plan and addressed in our
                     report are in keeping with the intent of the Congress to reduce the test
                     infrastructure. We disagree. The Congress directed DOD to develop a
                     DOD-wide infrastructure for electronic combat testing. DOD’s Master Plan




                     Page 12                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
              B-272629




              did not consider any of the Army and the Navy electronic combat test
              facilities as possibilities for consolidation and merely transfers Air Force
              test functions to other Air Force locations.

              DOD’s response indicated that the services made decisions to consolidate
              in areas that would have the least impact on DOD’s ability to perform
              effective test and evaluation. This response is not supported by the facts.
              For instance, the plan to close the EMTE electronic combat open air range
              at Eglin Air Force Base will leave DOD with no non-desert electronic
              combat test range for tactical fighters and two desert test ranges—one
              each for the Navy and the Air Force. This is not consistent with DOD’s
              testing policy that calls for testing to be conducted in a range of natural
              environments.

              DOD  commented that its planned consolidations reflect the 1995 BRAC
              legislation and the services’ plans to implement congressional direction.
              Our review showed that the planned actions will go beyond, not “reflect,”
              the 1995 BRAC legislation as the Air Force intends to relocate the entire
              EMTE function from Eglin Air Force Base, not limit itself to the
              BRAC-directed realignment of 10 systems (8 threat and 2 podded systems.)
              The Air Force intends to move AFEWES, as well. This planned move is
              inconsistent with direction from the 1995 BRAC.

              DOD believes diversity in the testing environments is desirable, but
              inconsequential, so long as DOD maintains the capability to replicate
              geographical and topographical characteristics through modeling and
              simulation and other work arounds. Our review indicated that DOD does
              not need to rely in large measure on computer models and work arounds.
              Instead, DOD could have considered keeping its non-desert range at Eglin,
              and could have considered consolidating the Air Force’s and the Navy’s
              desert ranges into one to keep the diverse test environments required by
              its regulations and still reduce from three ranges to two. We have modified
              the language from our draft report concerning our matter for
              congressional consideration to ensure that it is not misconstrued and to
              help focus attention on the desirability of considering a more
              cost-effective alternative. DOD’s comments are reprinted as appendix II,
              along with our detailed evaluation of them.


              To accomplish our objective, we examined DOD’s March 1996 Electronic
Scope and     Combat Consolidation Master Plan and DOD studies of potential electronic
Methodology   combat test facility consolidations. Because the Electronic Combat



              Page 13                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
B-272629




Consolidation Master Plan did not include any cost data, we gathered cost
data from affected sites, as well as the Air Force Materiel Command, and
other DOD studies of electronic combat test consolidation. We interviewed
officials from the Office of the Secretary of Defense, the Army, the Navy,
and the Air Force responsible for or involved in the electronic combat test
process. We also interviewed contractor personnel involved in the
electronic combat test process. We visited open air ranges,
hardware-in-the-loop facilities, installed system test facilities, and
observed electronic combat tests in progress. We reviewed DOD policy and
guidance on testing and evaluation, as well.

We performed our work at the Offices of the Secretaries of Defense, the
Navy, and the Air Force; the Offices of the Chief of Naval Operations and
the Air Force Chief of Staff; the Air Force Materiel Command,
Wright-Patterson Air Force Base, Ohio; Edwards Air Force Base,
California; Nellis Air Force Base, Nevada; Eglin Air Force Base, Florida;
Hurlburt Field, Florida; Army Aviation and Technical Test Center, Fort
Rucker, Alabama; Army Missile Command, Redstone Arsenal, Huntsville,
Alabama; Naval Air Warfare Centers at Patuxent River, Maryland, China
Lake, California, and Point Mugu, California; and REDCAP at Buffalo, New
York.

We performed our review from March 1996 to March 1997 in accordance
with generally accepted government auditing standards.


We are sending copies of this report to interested congressional
committees; the Secretaries of Defense, the Army, the Navy, and the Air
Force; the Director, Office of Management and Budget; and other
interested parties. We will make copies available to others upon request.

If you have any questions about this report, I may be reached at
(202) 512-4841. Major contributors to this report are listed in appendix III




Louis J. Rodrigues
Director, Defense Acquisitions Issues


Page 14                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Page 15   GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Contents



Letter                                                                                                  1


Appendix I                                                                                             18
                          Predict-Test-Compare Replaces Fly-Fix-Fly                                    18
The Department of         Hardware-in-the-Loop Facilities Provide Controlled Conditions                19
Defense’s Electronic        for Test
                          Effects of Electronic Combat System on Platform Determined in                20
Combat Test Process         Installed System Test Facility
                          Open Air Range Provides Real-World Test Scenarios                            20

Appendix II                                                                                            22

Comments From the
Office of the Secretary
of Defense
Appendix III                                                                                           48

Major Contributors to
This Report
Tables                    Table 1: Potential Threat Locations and Terrain Correlation                   5
                          Table 2: Open Air Ranges and Terrain Correlation                              5




                          Abbreviations

                          AFEWES     Air Force Electronic Warfare Evaluation Simulator
                          BRAC       Base Closure and Realignment Commission
                          DOD        Department of Defense
                          EMTE       Electro-Magnetic Test Environment
                          REDCAP     Real-time Electronic Digitally Controlled Analyzer Processor


                          Page 16                     GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Page 17   GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix I

The Department of Defense’s Electronic
Combat Test Process

                       Electronic combat systems, such as radar jammers and warning receivers,
Predict-Test-Compare   are most often associated with tactical fighter aircraft because of the
Replaces Fly-Fix-Fly   threat posed to them by modern surface-to-air missiles. However,
                       electronic combat systems are found today on all types of platforms.
                       These include ground vehicles, surface and subsurface naval vessels,
                       missiles, helicopters, and other fixed-wing aircraft besides tactical
                       fighters. Hence, wherever the services and their contractors develop or
                       test platforms and major subsystems for those platforms, electronic
                       combat test facilities have been established as necessary support
                       functions.

                       In the past 10 years, the Department of Defense (DOD) has spent more than
                       $300 million to build and upgrade electronic combat test capabilities. The
                       vast majority of this new investment has gone into hardware-in-the-loop
                       and installed system test facilities, which are highly scientific, laboratory
                       type facilities, and open air ranges that try to replicate real world
                       environments. These new and upgraded facilities were designed and built
                       to accommodate DOD’s revised electronic combat test process.

                       DOD’s revised electronic combat test process utilizing
                       hardware-in-the-loop, installed system test facilities, and finally, open air
                       ranges fits into a broader test philosophy referred to as
                       “Predict-Test-Compare.” According to a former test official,
                       Predict-Test-Compare was implemented to ensure more rigorous testing
                       was done before fielding because of a general belief in DOD that its
                       electronic combat systems did not work very well. According to the Air
                       Force, past electronic warfare programs have displayed a pattern of latent
                       deficiencies manifesting themselves in operational test and evalution,
                       necessitating expensive fixes and retesting. Predict-Test-Compare
                       replaced DOD’s “fly-fix-fly” model that emphasized open air range testing as
                       the primary test method.

                       Fly-fix-fly relied too much on trial and error at open air ranges to find and
                       correct problems. Often the systems were concurrently built and tested
                       and already fielded before successful fixes were identified. Typical
                       outcomes of a fly-fix-fly philosophy are the costly, repeated, and
                       continuing attempts to fix the ALQ-161 electronic warfare suite on the Air
                       Force’s B-1 Bombers, and the SLQ-32 electronic warfare suite on the
                       Navy’s surface combatants.

                       In contrast to trial and error, Predict-Test-Compare is based on the
                       scientific method of interplay between inductive and deductive reasoning.



                       Page 18                       GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                        Appendix I
                        The Department of Defense’s Electronic
                        Combat Test Process




                        After subjecting systems to testing on the ground under tightly controlled
                        conditions, testers compare the test outcomes to their predictions to
                        induce hypotheses that explain the outcomes. The inductive hypotheses,
                        in turn, are analyzed by developers and testers to deduce what
                        hypothetical fixes are necessary to produce more desirable outcomes in
                        subsequent tests. Thus, Predict-Test-Compare is an iterative process in
                        which understanding why a system behaves as it does is essential to
                        successfully predicting how the system will behave when it is modified.


                        Controlling for the conditions of a test is the number one requirement for
Hardware-in-the-Loop    ensuring that test outcomes are explainable. Hardware-in-the-loop
Facilities Provide      facilities provide this capability in the electronic combat test process. In
Controlled Conditions   their laboratory type environments, testers can control for external
                        variables found in realistic environments such as terrain effects and
for Test                background noise that might influence test outcomes.
                        Hardware-in-the-loop testing provides the capability to provide repeatable
                        measurements and verification of protection techniques and system
                        effectiveness.

                        The hardware-in-the-loop facility is the first place a new or modified piece
                        of electronic combat equipment faces an actual or simulated threat radar.
                        Prior to hardware-in-the-loop testing, a developer begins with a concept
                        for electronic combat equipment to fill a requirement, say an ability to
                        deceive a new threat radar. The developer typically will design a computer
                        model representative of the concept. The electronic combat tester will
                        then subject the conceptual model to an increasingly rigorous test against
                        validated computer models of threat radars. Once a computer model that
                        works against the threat models is developed, real electronic combat
                        hardware that tries to replicate the model’s behaviour is built. The
                        electronic combat hardware is then subjected to the hardware-in-the-loop
                        testing, that is, it is tested against actual or simulated threat radar
                        hardware.

                        If testers cannot demonstrate that the hardware will work as predicted
                        within the controlled conditions of the hardware-in-the-loop facility, a
                        system should not proceed to the next phases of the test process. Success
                        at installed system test facilities or open air ranges after failure in the
                        hardware-in-the-loop facility might be evidence of a positive effect from
                        environmental influences, for example, electronic signals bouncing
                        uncontrollably off of terrain features to confuse a threat radar, a factor
                        that will not always be present in every wartime environment.



                        Page 19                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                        Appendix I
                        The Department of Defense’s Electronic
                        Combat Test Process




                        In addition, systems that have failed in the real world can be brought back
                        to the hardware-in-the-loop facility to evaluate and improve their
                        performance. According to test officials, serious problems with the ALQ-99
                        system used on the EA-6B and EF-111 stand-off jamming aircraft were
                        unraveled and solutions identified in the Real-time Electronic Digitally
                        Controlled Analyzer Processor (REDCAP) hardware-in-the-loop facility
                        before the ALQ-99 went on to successful testing at the open air range. In a
                        more recent example, the Air Force Electronic Warfare Evaluation
                        Simulator (AFEWES) hardware-in-the-loop facility was able to recreate and
                        simulate the conditions that led to the shootdown of Captain Scott
                        O’Grady’s F-16 over Bosnia in 1995. The AFEWES results were subsequently
                        proven in real aircraft testing at the Electro-Magnetic Test Environment
                        (EMTE) Open Air Range at Eglin Air Force Base.


                        After the hardware is tested in the hardware-in-the-loop facility, it is then
Effects of Electronic   placed on the platform intended to eventually carry the hardware for
Combat System on        installed system testing. Installed system test facilities consist of anechoic
Platform Determined     chambers in which simultaneous operation of electronic warfare systems
                        and host platform avionics and munitions can be conducted. It is in the
in Installed System     installed system test facility that systems and subsystems are tested
Test Facility           together for electromagnetic interference and electromagnetic
                        compatibility, both of which have been major problems in the past. For
                        instance, a number of U.S. aircraft have had radar jammers, radars, and
                        radar warning receivers in the past that conflicted with each other. By
                        identifying the conflicts before flying at the open air range, testers can
                        more quickly isolate and solve problems. Once the Air Force and the Navy
                        complete their ongoing upgrades to their installed system test facilities,
                        they will be able to test systems for effectiveness under a wide range of
                        realistic threat and operational conditions while still on the ground.


                        Finally, when the hardware has been proven successful in each of the
Open Air Range          earlier steps, the electronic combat test process ends with open air testing
Provides Real-World     against actual or simulated threat radars in real-world environments.
Test Scenarios          Real-world phenomena encountered during open air testing can include
                        terrain effects, multi-path propagation, electromagnetic interference from
                        commercial systems, and other conditions that affect the atmospheric
                        propagation of electronic signals. While often thought of as the place for a
                        “final exam,” probably because of the association open air ranges have
                        with operational testing, open air ranges also can have a developmental
                        role. According to DOD officials, a properly managed and operated open air



                        Page 20                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix I
The Department of Defense’s Electronic
Combat Test Process




range can provide the proper mix of scientific accuracy and real-world
effects to allow electronic combat system developers to know if what they
have observed in the hardware-in-the-loop facility and installed system test
facility will hold true in the real world. The example cited above, in which
the AFEWES hardware-in-the-loop and EMTE open air range facilities together
unraveled, recreated, and demonstrated how the F-16 was shot down in
1995 over Bosnia provides evidence of this.




Page 21                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II

Comments From the Office of the Secretary
of Defense




See comment 1.




See comment 2.




See comment 3.




See comment 4.




                 Page 22   GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                 Appendix II
                 Comments From the Office of the Secretary
                 of Defense




See comment 5.

See comment 6.




See comment 7.




                 Page 23                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                        Appendix II
                        Comments From the Office of the Secretary
                        of Defense




See comments 1, 2, 3,
and 4.




See comment 8.




See comment 9.


See comment 10.




                        Page 24                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 11.




See comment 12.




See comment 13.




                  Page 25                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 14.




See comment 15.




See comment 16.




See comment 17.




                  Page 26                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 18.




See comment 19.




                  Page 27                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 19.




See comment 20.




See comment 21.




See comment 22.




                  Page 28                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 23.




See comment 24.




See comment 25.




                  Page 29                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 26.




See comment 27.




See comment 28.




                  Page 30                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 29.




See comment 31.




                  Page 31                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 30.




See comment 31.




See comment 32.


See comment 33.




                  Page 32                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 34.




See comment 35.




See comment 36.




                  Page 33                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 37.




                  Page 34                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 38.




See comment 3.




See comment 39.




                  Page 35                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
                  Appendix II
                  Comments From the Office of the Secretary
                  of Defense




See comment 39.




                  Page 36                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
               Appendix II
               Comments From the Office of the Secretary
               of Defense




               The following are GAO’s comments on DOD’s letter dated March 10, 1997.


               1. The Congress directed DOD to develop a plan “to establish a DOD-wide
GAO Comments   infrastructure for electronic combat testing.” DOD’s proposed plan fails to
               establish a DOD-wide infrastructure. Instead, DOD’s plan did not consider
               any of the 10 Army and Navy electronic combat test facilities as
               possibilities for consolidation or the results of DOD studies that identified
               consolidations that would result in a more cost-effective DOD-wide
               infrastructure.

               Our report does not conflict with the report entitled Defense
               Infrastructure (GAO/HR-97-7, Feb. 1997). In fact, this report substantiates its
               conclusions. The prior report stated that:

               “. . . breaking down cultural resistance to change, overcoming service parochialism, and
               setting forth a clear framework for a reduced defense infrastructure are key to avoiding
               waste and inefficiency. To do this, the Secretary of Defense and the Service Secretaries
               need to give greater structure to their efforts by developing an overall strategic plan.”


               In this report, we point out that the process used by the services in
               developing the Electronic Combat Consolidation Master Plan did not
               overcome parochialism, as evidenced by the lack of effort to consolidate
               across service lines. The Master Plan does not reflect a DOD-wide strategic
               plan, but rather merely an Air Force plan to move Air Force functions to
               other Air Force locations.

               DOD’s comment that “. . . the Services made decisions to consolidate in
               areas where they would have the least impact on the Department to
               perform effective T&E” is not supported by the facts. For instance, the
               plan to close the EMTE electronic combat open air range at Eglin Air Force
               Base will leave DOD with no non-desert electronic combat test range for
               tactical fighters, and two desert test ranges—one for the Navy and one for
               the Air Force. This is contrary to DOD’s testing policy that requires testing
               to be conducted in a range of natural environments. As an alternative, DOD
               could have considered, but decided to forego, the option of consolidating
               the test assets of the two desert ranges into one, and keep its only
               non-desert electronic combat open air range.

               As our report shows, the Air Force intends to “relocate” the EMTE function
               from Eglin Air Force Base, not limit itself to the Base Closure and
               Realignment Commission (BRAC) directed realignment. If the Air Force




               Page 37                            GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




transfers more than eight threat systems and two podded threat systems
out of Eglin, its actions will go beyond, not “reflect,” the 1995 BRAC
recommendation. The 1995 BRAC recommendation involves the movement
of only 8 threat systems and 2 podded threat systems, but DOD’s Master
Plan states that EMTE consists of 65 highly instrumented threat systems and
high fidelity validated simulators.

2. How funding for upgrades was authorized and appropriated is not
relevant to the issue of whether a facility should have been considered for
consolidation or whether more cost-effective consolidation alternatives
exist.

3. According to Air Force test policy, modeling and simulation is not an
adequate replacement for actual hardware testing because it cannot
predict absolute performance and effectiveness with high confidence or
achieve the same degree of fidelity for complex functions as testing of the
hardware itself.

The ongoing vision 21 consolidation effort gives DOD the opportunity to
consider how it will maintain geographical and topographical diversity,
among other things, and still achieve “as few [facilities] as is practicable
and possible.” For instance, DOD could consider keeping its non-desert
range at Eglin, and consolidate the Air Force’s and the Navy’s desert
ranges into one to keep the diverse test environments required by its
regulations and still reduce from three ranges to two.

4. There may be no significant loss of capabilities if the Air Force limits the
movement from EMTE to the eight systems and two pods that are described
in the BRAC decision and keeps the other residual test assets available for
testing at Eglin. However, if the Air Force carries out the Master Plan
proposal to “relocate” the EMTE function to accomplish a reduction from
three to two electronic combat ranges, there will be a loss in DOD’s current
ability to test with high fidelity and confidence. Testing only in dry, desert
air over rocky, mountainous terrain will limit DOD’s real-world testing to
one environment and one set of operating conditions. Moreover, the desert
ranges are not representative of most places in which DOD must be
prepared to fight.

5. Although the Air Force maintains there is no question of affordability in
the proposed move of AFEWES, the 1995 BRAC found that such a move would
cost $34.9 million and take over 100 years to achieve a return on that
investment. The Air Force’s refusal to consider electronic linking, despite



Page 38                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




an independent Air Force contractor’s conclusion that linking would be far
more cost-effective, demonstrates that the Air Force is not in step with the
rest of DOD, which is demonstrating electronic linking of AFEWES, REDCAP,
and the Navy’s anechoic facilities at Patuxent River, Maryland.

6. We agree that the Air Force should keep REDCAP and AFEWES test
capabilities for which there are test requirements. These test requirements
are outlined in Air Force Manual 99-112, Electronic Warfare Test and
Evaluation Process—Direction and Methodology for EW Testing.
According to the manual, hardware-in-the-loop facilities (such as AFEWES
and REDCAP) are an important test category because they represent the first
opportunity to test components against simulations of hostile weapon
system hardware or actual hostile weapon system hardware. That is why
we question the Air Force’s plan to put REDCAP hardware in storage in
favor of an unproven digital computer model.

7. We continue to believe that the transfer of test assets should be deferred
until the ongoing vision 21 consolidation effort is complete because this
would provide DOD with an opportunity to create a plan for a future
DOD-wide infrastructure for its testing, instead of an infrastructure that
preserves each service’s ability to maintain its own set of separate
facilities across the test spectrum. The 1995 BRAC decisions have a 6-year
implementation period. The planned transfers do not have to be made
immediately to satisfy BRAC. We have modified the words in the matter for
congressional consideration to more clearly articulate our position. Also
see comments 1, 2, and 3.

8. Open air ranges are used to evaluate electronic combat systems in
background, clutter, noise, and dynamic environments. Dynamic
environments contain numerous important variables besides those
mentioned in DOD’s comments. According to the Air Force’s electronic
combat test manual, an operationally realistic open air test environment
includes real-world phenomena such as terrain effects, multi-path
propagation, electromagnetic interference from commercial sources, and
effects caused by atmospheric propagation factors (i.e., the tendency of
atmospheric conditions to enhance or inhibit signal transmission).

Providing realistic and diverse representations of threat radar systems in
the numbers (“density”) and dispersion (“laydown”) that the system under
test would be expected to defeat in actual electronic combat does not
negate the requirement to test in operationally realistic environments. Also
see comment 3.



Page 39                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




9. The disadvantage of climatic predictability at the desert test ranges is
that the effects of various meteorological conditions cannot be observed.

10. The cost of testing at the western test range, the specifics of which the
Air Force has classified, far exceed those at EMTE at Eglin. In fact,
eliminating EMTE eliminates the Air Force’s lower cost range. In addition,
allowing foreign customers to utilize the Eglin range generates revenue,
but for classified reasons most foreign customers are precluded from
using the western range.

11. While it seems clear that moving EMTE’s threat systems to the Air
Force’s western test range could improve the western test range’s
technical capability, it does not automatically follow that this is the most
cost-effective solution for DOD as a whole to pursue.

12. If environmental effects were as well understood and accounted for in
electronic combat testing as DOD’s response claims, real-world testing at
open air ranges would not be required; testing indoors at
hardware-in-the-loop and installed system test facilities would be an
adequate substitute. Environmental effects on electronic combat system
performance can be more accurately determined on open air ranges where
the system is exposed to the complexities of different real-world
environments.

Furthermore, without the ability to test in at least two distinct
representative environments (e.g. wet and flat versus dry and
mountainous), DOD will be unable to predict with significant assurance
how an electronic combat system will perform in any environment other
than the one in which it was tested. Hence, because the electronic combat
test environment provided by the Eglin range provides DOD with its only
alternative to the desert test environment, DOD’s response that “the specific
electronic combat environment offered at EMTE is not critical to RF [radio
frequency] testing . . .” is not supportable. In addition, DOD regulations and
the Air Force electronic combat test process require testing under
real-world representative environment and operating conditions whether
or not DOD believes that a given specific test environment is not critical for
a given type of testing.

13. We did not assert that testing conducted in the environment at EMTE is
scientifically “of a higher value” than testing done in a desert environment.
What we stated was that DOD must prepare to fight in diverse




Page 40                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




environments; testing conducted in diverse environments is of a higher
value than testing limited to a single environment.

An operationally realistic test environment allows testers to gain insight
into understanding how a system will perform in that environment. Testers
cannot assume that the system will perform the same way in different
environments. If DOD reduces its testing capability to only a desert
environment, it will not be able to prove its systems work in anything
other than a desert environment. This is contrary to DOD testing policy that
requires testing to be conducted in a range of natural environments. In
addition, testing indoors in a contractor’s laboratory is not considered an
acceptable substitute for real-world testing on the aircraft according to the
Air Force’s electronic combat test process guide.

Neither Point Mugu nor Vandenberg Air Force Base have the necessary
threat system test assets to create realistic threat environments for
electronic combat testing for tactical aircraft systems. To utilize Point
Mugu or Vandenburg for this purpose, DOD would essentially be recreating
EMTE on the west coast. Moreover, no naval battle group currently has the
capability to create a realistic open air threat density and laydown of
hostile land-based surface-to-air missile and anti-aircraft artillery systems.
Also see comments 2 and 10.

14. If these sites mentioned by DOD “easily support” electronic combat
testing of tactical aircraft, they should have been considered for
consolidation along with EMTE in the Master Plan process. However, the
reality is that none of the places mentioned by DOD has the test assets to
create the realistic threat densities and laydowns that DOD earlier in its
response said were the most important factors in developing “operational
realism and diversity.”

DOD’s statement that once an electronic combat system is operating at
airspeeds and altitudes normal for tactical aircraft the environmental
conditions at the surface have little or no effect on performance
unrealistically assumes no aircraft will ever be called upon to fly at low
altitudes (such as flying low to avoid radar detection). Moreover, DOD’s
statement is counter to its policy statement on the need to operationally
test in different environments.

15. In addition to electronic combat testing, Eglin conducts other kinds of
testing, including bombing and live missile firings. Moreover, the main
civilian air corridor between Los Angeles and destinations further east,



Page 41                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




including Las Vegas, one of the nation’s fastest growing cities, buttresses
against the restricted air space available to the Air Force and the Navy at
their desert test ranges. Also see comment 1.

16. DOD’s comment seems to assume that keeping EMTE would mean that
the strengths of the Air Force’s western test range, which it delineates
here, would have to be sacrificed. We do not suggest that the western test
range be closed instead of EMTE. Also see comment 1.

17. Operational testers have been using and continue to use operationally
relevant scenarios at EMTE. Test aircraft at EMTE can also fly with live
ordnance through simulated hostile airspace and live ordnance can be
delivered on a real target. Also see comment 2.

18. Our point is that the body of potential hostile nations contains a variety
of environments, not just desert. Testing at EMTE and in the desert allows
the operational tester to gain insight into electronic combat system
performance in multiple environments.

The threat dispersion at EMTE can be changed if necessary, as it has been in
the past. In fact, the threat dispersion at all of the ranges should be
changed regularly to ensure that testing includes operationally relevant
scenarios since many modern threat systems are designed to be mobile.

China Lake is a facility with naval surface-to-air missiles located deep in a
desert ringed by mountains. Placing naval surface-to-air missiles at EMTE
with its flat terrain, humid environment and littoral location could provide
a more realistic and operationally relevant scenario for naval aircraft.

Despite DOD’s assertion that severe operational limitations exist at EMTE,
EMTE’s annual workload historically has been significantly greater than the
two desert test ranges. The Air Force and the Navy both use EMTE for
testing despite the presence of the desert ranges. Thus, it appears their
past testing behavior indicates they believe the benefits of EMTE outweigh
any such limitations.

19. REDCAP at Edwards will be less capable as a hardware-in-the-loop
facility because the Air Force intends to put the hardware in storage,
replacing it with a digital computer model to simulate actual hardware
testing.




Page 42                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




According to Air Force Manual 99-112, Electronic Warfare Test and
Evaluation Process—Direction and Methodology for EW Testing, the Air
Force’s electronic combat testing policy requires hardware-in-the-loop
testing. Also, REDCAP currently has paying customers who do want to use
it. Furthermore, hardware-in-the-loop facilities such as REDCAP and AFEWES
use “real equipment.” It is in digital modeling, such as DOD’s comment
proposes as a substitute for REDCAP, where actual electronic combat
systems are replaced by software representations instead of real
equipment.

The software-based computer model of REDCAP being developed may cost
less to operate than the actual REDCAP hardware-in-the-loop facility, just as
flight simulators cost less to operate than actual aircraft. However,
modeling and simulation is not hardware-in-the-loop testing. Because they
are different kinds of testing with different purposes, they are not directly
comparable for purposes of determining which is more cost-effective.

DOD’s statement that “Currently, REDCAP goes practically unused” is not
supported by recent usage data. Reimbursable costs from test customers
are up significantly over the past 3 years. Recent customers include a
major U.S. Air Force aircraft program that used the REDCAP Mission Level
Assessment Tool for several months, as well as a foreign customer having
some of its electronic combat hardware tested. See also comment 5.

20. DOD’s Master Plan included no cost estimates. We reported (1) the cost
estimates that were independently arrived at by BRAC, which do not
support relocating AFEWES or all of EMTE; (2) known additional costs that
the Air Force will incur by replacing REDCAP with a digital model, which
will in turn allow the Air Force to keep down the cost-estimate for the
REDCAP move; and (3) additional costs that current EMTE customers report
they will incur as a result of the EMTE closing. DOD’s comments provide no
evidence to suggest that these are wrong.

21. According to the BRAC language, some EMTE assets were specifically
directed to be left at Eglin “to support” several customers, including the
Special Operations Forces, as well as the Air Force Materiel Command
Armaments/Weapons Test and Evaluation activities, and other users. DOD’s
position that the BRAC legislation prohibits testing and limits customer
support to providing training capability is not adressed in the BRAC
direction.




Page 43                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




22. We agree the cost analysis to support any test facility closure should
include additional costs to users associated with the relocation.

23. According to the BRAC recommendation regarding Eglin, BRAC expected
DOD to use the Master Plan process to come up with the “optimal”
consolidation plan. Closing EMTE (not just relocating those 10 test assets
recommended to be moved by BRAC), relocating AFEWES despite BRAC’s
determination that this would not be cost-effective, and ignoring Army and
Navy test facilities completely as possibilities for consolidation, does not
support DOD’s claim that the Master Plan is “the result of BRAC decisions.”
Moreover, previous DOD cost-effectiveness studies concluded that the
three relocations planned to be relocated by the Air Force will not be
cost-effective.

24. The Navy and the Air Force authors of the Master Plan told us they did
not consider costs in the Master Plan because there was no requirement to
do so.

25. It is not clear why DOD raises the issue of the REDCAP contractor’s
estimate of the cost of moving REDCAP. We do not use that figure in our
report. Our report shows that the Air Force intends to replace REDCAP
hardware being moved from its current location with a digital computer
model that will simulate REDCAP. The Air Force’s contracted cost for the
model is $6.2 million. If the Air Force was not replacing the REDCAP
hardware with the digital model, it would have to reestablish the REDCAP
hardware at some unknown additional cost. Hence, the cost to make
REDCAP operational at the new location is either (1) the cost of the move
plus the digital model (with current hardware going into storage) or
(2) the cost of the move plus set-up costs for the current hardware (with
no digital model). Since DOD has selected option number (1), $6.2 million
should be added to the cost of the REDCAP move. See also comments
2 and 5.

26. REDCAP does have some outdated systems. But as our report shows,
REDCAP also just completed a $75 million upgrade. Also, customer usage
and receipts over the past 3 years have increased.

27. We have changed the title of this finding.

28. According to Special Operations Forces test officials, EMTE provides a
more cost-effective test capability to meet their needs compared to
traveling to the western test range. Also see comment 21.



Page 44                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




29. Air Force officials reported to BRAC that they anticipated saving
$3.7 million per year after spending $6.1 million to move the threat systems
out of EMTE. Even if this savings materializes, it will not offset the
additional costs anticipated by the current users of EMTE. Special
Operations Forces officials told us they must use operational aircraft from
Hurlburt Field, Florida, adjacent to Eglin, to accomplish their testing
because they have no dedicated test aircraft at either Edwards or Eglin Air
Force Base.

30. DOD has no studies to show that the relocations delineated in the
Master Plan are cost-effective, and now claims that its 1994 and 1995 joint
studies, which do not support the Master Plan moves, were incomplete
and flawed. We spoke with Air Force, Army, and DOD Inspector General
officials involved in preparation or oversight of the 1994 and 1995 studies
and they do not agree the studies were flawed. They told us what made the
recommendations of these studies “unrealistic” was not the content, but
the refusal of the Navy to consider closing China Lake while the Air Force
retained two open air ranges. Navy officials associated with China Lake do
maintain the studies were incomplete and flawed.

The specific examples provided to us on 15 August 1996 represent the
dissenting position that China Lake’s open air range was not given
adequate consideration in studies that compared it to the Eglin open air
range. This data does not support the alternative position that the Master
Plan proposal to relocate EMTE is cost-effective.

31. We agree that the Air Force’s cuts in funding for investment at EMTE
over the past several years, coupled with the Navy’s increased investment
funding at the China Lake range, could affect the outcome of a comparison
of the two if the 1994 study was redone today. DOD, however, has not done
such a study to demonstrate that the outcome would be different.

32. The claim that Eglin has a capacity disadvantage does not appear to be
accurate. During the run up to the 1990-91 Gulf War, the Eglin Range
conducted the largest share of electronic combat testing of the three open
air ranges. During fiscal year 1993, 2,133 hours of testing were conducted
at EMTE, while China Lake and the western test range conducted 1,649 and
1,085 hours, respectively.

33. The referenced DOD Inspector General’s report compared EMTE with the
western test range. We do not assert that EMTE should be kept in lieu of the




Page 45                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




western test range. The Inspector General’s report did not consider the
Navy’s open air range at China Lake compared to EMTE.

34. The 1995 study conducted for the Air Force by Georgia Tech Research
Institute concluded that electronic linking would be far more
cost-effective than relocating AFEWES and REDCAP. In addition, a 1994 Board
of Directors synergy study concluded that moving the
hardware-in-the-loop facilities would not be cost-effective. We know of no
study that concludes it is less expensive to relocate and reassemble
AFEWES or REDCAP hardware at a new location.


DOD’s position that successful electronic linking will be impossible due to
the laws of physics has not yet materialized. DOD’s project to link REDCAP
and AFEWES with Patuxent is well underway, and as DOD states, the REDCAP
link “shows potential.” Additional support for the linking project comes
from the Georgia Tech study concluding that linking will be more
cost-effective, and the 1994 DOD synergy study concluding that moving the
hardware-in-the-loop facilities is less cost-effective. Hence, DOD could have
advocated pursuing electronic linking instead of relocation of REDCAP and
AFEWES in the Master Plan.


35. The 1994 synergy study conducted for DOD’s Test and Evaluation Board
of Directors concluded that it would take 200 years to recover the
investment to relocate and reassemble the hardware-in-the-loop facilities
at the Edwards Air Force Base installed system test facility for “one stop
shopping.” As a result, the Navy shows no inclination to relocate its
hardware-in-the-loop facility from Point Mugu, California, to its installed
system test facility at Patuxent River, Maryland.

36. Even taking into account the continued operations and maintenance
costs at AFEWES and REDCAP, the 1994 DOD synergy study and the 1995
Georgia Tech Research Institute study concluded that these moves would
not be cost-effective.

37. The memoranda cited by DOD were all written in 1992 and referred to
another study that concluded that keeping China Lake’s open air range
was less cost-effective than EMTE. The DOD joint service studies cited in our
report were conducted in 1994 and 1995. Although DOD asserts that it is not
service parochialism that prevents interservice consolidation from
occurring, we note that DOD has now produced three studies with a
conclusion that China Lake is less cost-effective to keep, yet the Master
Plan calls for assets to be relocated from one Air Force location to another



Page 46                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix II
Comments From the Office of the Secretary
of Defense




Air Force location. The Director of Air Force Test and Evaluation told us
that this is because the Navy would not consider relocating China Lake’s
test assets.

38. As with the Electronic Combat Consolidation Master Plan, we believe
that service parochialism may interfere with the ongoing vision 21 effort.
There have been no DOD-wide electronic combat test consolidations in the
Major Range Test Facility Base despite a number of studies that have
recommended such consolidations.

39. We have modified the language from our draft matter for congressional
consideration to ensure that our focus is not misconstrued by others.




Page 47                          GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Appendix III

Major Contributors to This Report


                        Delores Cohen
National Security and   Charles Ward
International Affairs
Division, Washington,
D.C.
                        Mark Lambert
Atlanta Regional
Office
                        Don Springman
Chicago Regional
Office




(707149)                Page 48         GAO/NSIAD-97-10 Electronic Combat Test Consolidation
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested