oversight

Unmanned Aerial Vehicles: Realistic Testing Needed Before Production of Short-Range System

Published by the Government Accountability Office on 1990-09-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

GAO
                   United States
                   General Accounting  Office
                   Washington, D.C. 20548

                   National Security and
                   International Affairs Division

                   B-229489

                   September 28, 1990

                   The Honorable Les Aspin
                   Chairman, Committee on Armed Services
                   House of Representatives

                   Dear Mr. Chairman:

                   As you requested, we have reviewed thestrategy for acquiring the
                   Short-Range Unmanned Aerial Vehicle (UAV) system. Because of your
                   Committee’s concern about the lack of success on past UAV programs and
                   the need to assure that related acquisition weaknesses do not recur, we
                   compared the Short-Range UAV acquisition strategy to that folloyed in
                   acquiring the Pioneer UAV, a previously procured system which encoun-
                   tered problems when deployed with the Navy’s operational forces.


                   The Short-Range UAV program, expected to cost $1.5 billion, was initi-
Background         ated as a joint-service effort in response to congressional concern about
                   the proliferation of UAV programs and the need to acquire UAV systems
                   that could meet the requirements of more than one service. In this
                   regard, Congress refused to authorize fiscal year 1988 funding for pro-
                   curement of separate Army and Navy Short-Range UAVS and provided
                   funds only for a joint program. Accordingly, the Department of Defense
                   (DOD) is acquiring the Short-Range UAV to meet the needs of the Army,
                   Navy, and Marine Corps.

                   DOD plans to begin low rate production of the Short-Range system in
                   fiscal year 1992 with the procurement of 8 of a total program quantity
                   of 53 systems. Each system to be procured initially is to include 8 air
                   vehicles, and the 64 vehicles are to be deployed in 1994.


                   DOD'S acquisition strategy for the Short-Range UAV is to perform opera-
Results in Brief   tional testing before beginning low rate production. Despite the impor-
                   tance of realistic operational testing, however, the Short-Range UAV
                   acquisition strategy provides for testing in an environment not represen-
                   tative of where the system is supposed to be deployed. This strategy
                   places DOD at risk of becoming committed to the production of a system
                   based on test results that may not be a valid indicator of the system’s
                   capability. The strategy is similar to Pioneer’s strategy that resulted in
                   spending $160 million for a system that still does not meet performance
                   requirements.


                    Page 1                               GAO/NSIAlMO-234 U-    ed Aerial Vehicles
                  IS229489




                  DOD'S acquisition policies require that operational testing be held in a
                  realistic environment representing combat conditions to the extent prac-
                  tical. Nevertheless, DOD plans to limit the Short-Range system’s
                  preproduction operational testing to areas that do not provide environ-
                  ments typical of where current deployment plans indicate the system
                  could be used in combat. Most of this preproduction testing will be lim-
                  ited to a desert environment. We recognize that if the current Middle
                  East situation continues until 1994, when the system is to be deployed,
                  the Short-Range system deployment plans could be changed. However,
                  the system’s currently required capability of locating targets under
                  diverse conditions in multiple environments is not likely to be demon-
                  strated before DOD becomes committed to its production.

                  DOD also plans to begin full-rate production of the Short-Range system
                  before verifying that it can be modified to meet Navy requirements.
                  Although DOD plans to conduct operational testing before full-rate pro-
                  duction, the system to be tested will not incorporate those features nec-
                  essary to operate in a naval environment. If subsequent testing of the
                  Navy variant were to show the system to be unsuitable for naval use,
                  DOD would then be fully committed to a system not meeting congres-
                  sional intent for a common-service system. DOD'S prior experience with
                  the Pioneer system demonstrated the difficulties in adapting a UAV for
                  naval use and the need for timely testing.

                  Other shortcomings in the planned Short-Range system test program
                  were identified by us and the Director, Defense Operational Test and
                  Evaluation. When these shortcomings were brought to DOD'S attention, it
                  agreed to correct them.


                  DOD'S Short-Range UAV acquisition strategy includes preproduction oper-
Recommendations   ational testing, and DOD'S initially procured systems represent a signifi-
                  cant portion of the total program quantity and are to be deployed to
                  operational forces. We therefore recommend that the Secretary of
                  Defense require that operational testing of the Short-Range UAV be con-
                  ducted in diverse, realistic environments to provide reasonable assur-
                  ance that it will meet requirements before permitting limited production
                  of the land-based UAV system. We also recommend that the Secretary
                  limit Short-Range UAV system production until satisfactory performance
                  of the Navy variant is demonstrated and assure that actions are taken to
                  correct the other shortcomings noted in the planned test program.




                  Page 2                                 GAO/NSIAMlMS4 UNMNI ed Aerial Vehicles
                      B-229489




                      Congress may wish to consider whether the congressional intent for a
Matter for            joint service system is jeopardized by the substantial risk that the Navy
Congressional          variant will be unsuitable, and, if so, Congress may wish to prohibit full-
Consideration          rat,e production of the Short-Range UAV until performance of the Navy
                       variant is proven to be satisfactory.


                      DOD agreed or partiallyagreed with most of the findings in this report
Agency Comments and   but disagreed with the recommendations and matter for congressional
Our Evaluation        consideration.

                      DOD stated that an adequate evaluation of the system’s operational
                      effectiveness and suitability can be accomplished without testing in all
                      environments in which the system may be employed. DOD also stated
                      that the system’s acquisition strategy and test program are consistent
                      with the applicable DOD directive and that the system will be tested in
                      representative operational environments.

                      DOD indicated that the acquisition risks are inherently low because of the
                      nondevelopmental nature of the program. DOD also stated that the risk
                      would be further reduced by the test program for the Army and Marine
                      Corps version, by engineering evaluations and design reviews of the
                      Navy variant, and by preproduction testing of the Navy variant.

                      Our basic concern is that the primary test site does not resemble the
                      primary locations where the system is to be operationally deployed
                      beginning in 1994 and therefore the test results could be very mis-
                      leading. In fact, DOD'S restructured test plan recognizes that the testing
                      environment constitutes a test limitation.

                      Our past work has often demonstrated the importance of testing in a
                      realistic operational environment, and DOD’S acquisition directive
                      requires that when operational testing is conducted, it be conducted
                      under realistic conditions. DOD’S comments present no evidence that
                      negates the need to realistically test the system before committing to
                      production.

                      DOD also stated that it has no intention of delaying production of the
                      system for the Army’s and Marine Corps’ use pending outcome of the
                      Navy variant’s development. Finally, DOD pointed out that operational
                      testing of the first Navy variant procured and other measures, such as
                      design reviews, would sufficiently minimize the risk associated with the
                      current acquisition strategy.


                       Page 3                                 GAO/NsIAIH&294   U-    ed Aerial Vehiclea
B-229489




Under the current acquisition strategy the scheduled testing of the Navy
variant will only reduce the risk of prematurely committing to produc-
tion of that variant, rather than the basic system which will by then
already be in full-rate production. If the Navy variant were to prove
unsuccessful in the later testing, DOD would find itself fully committed to
production of a system not meeting congressional intent for a common
UAV. Experience with Pioneer shows that adapting a UAV for naval opera-
tions is not a low-risk effort even though the Navy categorized it as such
when justifying the program. We therefore continue to believe that DOD
should retain the Short-Range system in low-rate production until opera-
tional testing shows that the Navy variant’s performance is satisfactory.

DOD refers to the Short-Range UAV as a nondevelopmental     program. The
Pioneer was also a so-called nondevelopmental program in its early
phases but later required substantial development to solve problems
associated with naval operations. In addition, testing of the Army and
Marine Corps version of the Short-Range UAV will not reduce risk associ-
ated with the Navy variant because the Army and Marine Corps version
will not incorporate any of the modifications necessary for operating in
a naval environment. Engineering evaluations, design reviews, and
preproduction testing of the Navy variant should help preclude a pre-
mature commitment to production of that variant. However, these
efforts will not reduce the risk of becoming prematurely committed to
the Short-Range program as a whole because the Army and Marine
Corps land-based version will be in full-rate production by the time
these efforts are completed.


Appendix I discusses the Short-Range UAV acquisition strategy in more
detail. Appendix II describes our objective, scope, and methodology.
Appendix III sets forth DOD comments.

Copies of this report are being sent to the Secretaries of Defense, the
Army, and the Navy; the Director of the Office of Management and
Budget; selected congressional committees; and other interested parties.
Copies will be made available to others upon request.




 Page 4                                GAO/T’BIAD9@234UNMNI ed Aerial Vehicles
This report was prepared under the direction of Louis J. Rodrigues,
Director, Command, Control, Communications, and Intelligence Issues,
who may be reached on (202) 275-4841 if you or your staff have any
questions concerning the report. Other major contributors to this report
are listed in appendix IV.

Sincerely yours,




Frank C. Conahan
Assistant Comptroller General




Page 5                                GAO/NsLU.MO-m U-         Aerial Vehiclea
Contents



Appendix I                                                                                        8
Evaluation of DOD’s     Background
                        Production Before Realistic Operational Testing
                                                                                                  8
                                                                                                  9
Acquisition Strategy    Congressional Intent for a Joint-Service System Is at Risk               12
for the Short-Range     Some Shortcomings of Original Test Plans Addressed                       12
UAV
Appendix II                                                                                      14
Objective, Scope,and
Methodology
Appendix III                                                                                     15
Comments From the       GAO Comments                                                             25

Department of
Defense and Our
Evaluation
Appendix IV                                                                                      29
Major Contributors to
This Report




                        Abbreviations

                        DOD       Department of Defense
                        GAO       General Accounting Office
                        UAV       Unmanned Aerial Vehicle


                        P-6                                   GAO/-NSIABBO-234
                                                                             Unmanued Aerial Vehlclea
Page 7   GAO/NSIAD90-234U-   ed Aerial Vehicles
Evaluation of DOD’sAcquisition Strategy for
the Short-RangeUAV

                                are pilotless aircraft resembling small airplanes or helicopters that
Background                 UAVS
                           are remotely controlled or preprogrammed to be controlled by on-board
                           equipment. DOD is acquiring UAvs to meet a VXi&y of military needs.

                           A UAV system typically includes one or more air vehicles, a launch and
                           recovery system, and a ground station for controlling the UAV’S flight
                           and processing information from the UAV. The air vehicle carries the
                           system’s payload, such as a television camera, with the specific type
                           depending on the military mission to be accomplished.

                           As part of its joint UAV program, DOD is acquiring the Short-Range system
                           for use by the Army, Navy, and Marine Corps. The system is to provide
                           the capability to perform reconnaissance, surveillance of enemy activi-
                           ties, target acquisition, and other military missions.


Initiation of the Short-   The joint-service Short-Range program was initiated in response to con-
Range Program              gressional concern about the proliferation of UAV programs and the need
                           to acquire UAVS that could meet the requirements of more than one ser-
                           vice and eliminate duplicative programs. To achieve commonality
                           among the services’ programs, Congress eliminated fiscal year 1988
                           funding within the services’ separate research, development, test, and
                           evaluation accounts for individual UAV programs and consolidated the
                           funding in a joint-service program.

                           Also during the fiscal year 1988 budget process, Congress refused to
                           authorize funding for the procurement of separate Navy and Army
                           Short-Range UAVS. Congress provided that funds were available only for
                           a joint program.


Short-Range UAV Program    DOD  initiated acquisition of the Short-Range UAV in fiscal year 1989 as a
Description                common-service system. Contracts were awarded in September 1989 to
                           two firms for nondevelopmentall candidate systems for a competitive
                           fly-off in fiscal year 1991. DOD plans to evaluate the candidate systems
                           and select the winning firm to begin low rate production in fiscal year
                           1992. The initial procurement is to include eight systems for deployment
                           in 1994.



                           ‘Nondevelopmental item means any item that is (1) commercially available, (2) in use by a US.
                           agency or foreign government with which the United States has a mutual defense cooperation agree-
                           ment, or (3) any of the items in (1) or (2) that require only minor modification.
                                                              .


                           Page 8                                            GAO/NSL4D9O-234U-            ed Aerial Vehicles
                          Appendix I
                          Evalnation oPDOD’sAcquisition Strategy Por
                          the Short-RangeUAV




                          The initially produced systems, designated as Block 0, are planned for
                          use by the Army and Marine Corps. However, DOD does not expect these
                          systems to fully meet performance requirements and thus plans a
                          research and development program to improve the Block 0 system’s
                          capability and provide a variant of the system for naval use. The Navy
                          variant is designated as Block I, while the improved Short-Range system
                          is designated as Block II. Block II improvements will include use of a
                          diesel or jet fuel engine to eliminate the use of more volatile gasoline.
                          The improvements will also include growth in a number of mission
                          payload and ground control station capabilities such as automatic
                          tracking of targets and automated searching of designated areas.

                          DOD plans to procure a total of 53 Short-Range systems: 27 systems for
                          the Army, 18 for the Marine Corps, 5 for the Navy, and 3 for training
                          purposes. The systems include 424 air vehicles or 8 vehicles per system.
                          In addition to air vehicles, each system is to consist of multiple pay-
                          loads, a launch and recovery station, and related equipment. The Short-
                          Range UAV Decision Coordinating Paper, dated August 1989, projected
                          total acquisition costs to be about $1.5 billion. However, according to
                          DOD, that amount may need some revision, depending upon the winning
                          contractor.


                          Despite the importance of realistic operational testing, the preproduc-
Production Before         tion operational testing of the Short-Range system that DOD plans to con-
Realistic Operational     duct will not take place in environments representative of where it is
Testing                   supposed to be deployed. We recognize that if the current Middle East
                          situation continues until 1994, when the system is to be fielded, the
                          Short-Range system deployment plans could be changed. Nonetheless,
                          the test results, which are the basis for beginning production, may not
                          be a valid indicator of the system’s currently required capability.

                          The Navy’s experience with its Pioneer Unmanned Aerial Vehicle, first
                          procured in 1986, illustrates the need for realistic operational testing of
                          the Short-Range UAV system. Some of the lessons to be learned from pro-
                          duction before testing in a realistic environment were vividly demon-
                          strated during the acquisition of the Pioneer system.


Experience With Pioneer    The predecessors to Pioneer had been successfully used by Israeli forces
                           in the Middle East. Thus, the Navy procured the Pioneer as a
                           nondevelopmental system, without testing it, and deployed the system
                           with its operational fQrces. Numerous problems ensued.


                           Page 9                                      GAO/NSIAKMO-234Unmanned Aerial Vehicles
                           Appendix I
                           Evaluation of DOD’sAcquisition Strategy for
                           the Short-RangeUAV




                           After deployment at sea, engine failures caused crashes and led to tem-
                           porary termination of Pioneer operations pending engine modifications
                           and a change in the type of fuel used. However, the fuel change to avia-
                           tion gasoline then forced the Navy to start a development program for a
                           new Pioneer engine that would use a less volatile fuel considered accept-
                           able for shipboard operations.
                           Difficulty in landing Pioneer on the ship also caused crashes and
                           required vehicle modifications and development of new flight control
                           software. The air vehicle was also retrofitted with new foam-filled
                           wings so that crashed vehicles could float until salvaged.
                           Problems in controlling Pioneer at sea were compounded by electromag-
                           netic interference from the ship’s high frequency radio, leading to at
                           least one air vehicle loss. This required that the radio be shut down
                           during Pioneer operations pending modifications to harden Pioneer
                           against the interference.
                           Placement of the air vehicle tracking unit behind the ship’s superstruc-
                           ture created a blindspot where Pioneer contact was lost. This required
                           that the ship be maneuvered extensively to keep the air vehicle out of
                           the blindspot until another tracking unit could be acquired and placed in
                            front of the ship’s superstructure.
                            Pioneer’s cumbersome net recovery system interfered with helicopter
                           operations and sometimes caused air vehicle and propeller damage when
                           the air vehicle landed in the net. This required the Navy to develop a
                            new net recovery system.

                           Thus, the lack of realistic operational testing of the Pioneer UAV system
                           left the Navy to costly and time-consuming trial and error as it tried to
                           adapt a system used in a land-based desert environment to shipboard
                           use. Four years after initial procurement, the Navy is still buying
                           replacement hardware, such as completely modified air vehicles, to
                           bring Pioneer systems up to a minimum essential level of performance.


Planned Short-Range         DOD plans to do operational testing of the Short-Range UAV system before
Testing Is Not Realistic    low rate production to minimize some of the risk that was inherent in
                            the Pioneer UAV acquisition strategy. However, the planned testing,
                            based on current deployment plans, is not realistic; significant risks
                            remain. Thus, the test results, which are to be the basis for beginning
                            production, may not be a valid indicator of the system’s capability.

                            Operational testing is the primary means of predicting weapon system
                            performance in a combat representative environment. Thus, operational
                            testing should be held in a realistic environment representing combat


                            Page 10                                      GAO/NSIAIHO-234 U-   ed Aerial Vehicles
                                                                                                                                      I




                                   Appendix I
                                   Evaluation of DOD’sAcquisition Strategy for
                                   the Short-RangeUAV




                                   conditions to the extent practical. Our past work has also consistently
                                   illustrated the importance of conducting operational testing in a realistic
                                   environment. We reported in December 1986,* for example, that results
                                   of testing in an unrealistic environment can be misleading and of only
                                   limited usefulness in evaluating system performance.

Preproduction   Desert Testing     Realistic operational testing of the Short-Range UAV would require,
                                   among other things, that the system be tested at locations resembling
                                   those where it might be used in combat. The Short-Range system is to be
                                   deployed in areas marked by hilly or mountainous terrain with dense
                                   forests and other vegetation and by varied climatic conditions such as
                                   cloudy weather, rain, snow, and other factors. Accordingly, the Short-
                                   Range system is supposed to be capable of performing reconnaissance,
                                   surveillance, and target acquisition under a variety of environmental
                                   conditions.

                                   DOD plans to conduct preproduction operational testing of the Block 0
                                   system’s ability to perform such missions at Fort Huachuca, Arizona.
                                   The operational test facility at Fort Huachuca is typical desert terrain,
                                   generally flat and lacking trees and other vegetation. Our viewing of the
                                   planned test site showed few areas where potential targets could be situ-
                                   ated other than in open view. Thus, the planned testing to evaluate the
                                   Short-Range system’s capability to locate targets under various environ-
                                   mental conditions in which the UAV is to operate may prove misleading.

Preproduction   Maritime Testing   DOD  also plans to conduct preproduction maritime testing at a second
                                   location, the Pacific Missile Test Center. However, this testing will not
                                   include operations from Navy ships. Instead, testing will include opera-
                                   tions from runways on land and will therefore not demonstrate the
                                   system’s capability to operate in a naval environment.

                                   As discussed earlier, the Navy’s Pioneer experience demonstrated the
                                   difficulties in adapting a UAV system for use at sea. After crashing 14 of
                                   the air vehicles during maritime operations and 21 overall, and encoun-
                                   tering numerous other performance problems, the Navy redesigned and
                                   modified practically the entire Pioneer system. Despite these changes,
                                   made at a cost of about $50 million in Research & Development and
                                   replacement hardware, Pioneer does not meet its intended performance
                                   requirements.


                                    2Weapon Performance: Operational Test and Evaluation Can Contribute More to Decisionmaking
                                    (GAOINSIAD87-61,



                                    Page 11                                         GAO/NSIAlMO-234 U-           ed Aerial Vehicles
                       Evahutlon oPDOD’sAcquisition Strategy Por
                       the Short-RangeUAV




                       DOD plans to commit to full-rate production of the Short-Range UAV
Congressional Intent   system before verifying that it can be modified to meet Navy require-
for a Joint-Service    ments. DOD’S acquisition policies require that operational testing be com-
System Is at Risk      pleted prior to full-rate production. Hence, DOD plans to conduct
                       operational testing in 1992 before the system’s full-rate production
                       begins. However, the system to be tested is the Block 0 version for use
                       by the Army and Marine Corps and will not incorporate those features
                       necessary to operate from Navy ships.

                       Congress has already refused to authorize funding for separate Army
                       and Navy short-range systems. If subsequent operational testing of the
                       Navy variant were to show that a modified Short-Range system is
                       unsuitable for Navy use, DOD would be faced with the alternatives of (1)
                       cancelling the Short-Range production and starting a new program to
                       acquire a system meeting both Army and Navy needs or (2) continuing
                       the Short-Range production for the Army and Marine Corps and
                       acquiring a separate system for the Navy.


                       We and officials in the Office of the Secretary of Defense identified
SomeShortcomings of    other shortcomings in the Short-Range system’s original test program.
Original Test Plans    Subsequent to completion of our work, however, the UAV Joint Program
Addressed              Office agreed to address these concerns by significantly revising its test
                       plans.

                       Initially, we were concerned that the low-rate initial production decision
                       for the Short-Range UAV was planned to follow an early operational
                       assessment period rather than the currently planned operational testing.
                       An official from the Office of the Secretary of Defense stated that an
                       early operational assessment should not be considered a substitute for
                       operational testing. Although DOD policy allows the initiation of low rate
                       production based on an early operational assessment, DOD revised its
                       acquisition strategy to provide for preproduction operational testing.

                        Another shortcoming was that the test plan contained a provision that
                        none of the criteria for judging the Short-Range system’s performance
                        was to be considered as absolute pass or fail measures. Our concern with
                        this provision was that it conflicted with the Secretary of Defense’s
                        intent as recently stated in the July 1989 Defense Management Report to
                        the President. The Secretary stated that DOD policy will be to discipline
                        the acquisition process by defining minimum required accomplishments
                        for advancing from one phase of the acquisition process to the next.
                        This provision was removed from the test plan.


                        Page 12                                    GAO/‘NSIAD90-234Unmanned Aerial Vehicles
Evaluation of DOD’sAcquisition Strategy for
the Short-RangeUAV




We and Defense officials had concerns about the success threshold for
locating targets accurately during Short-Range testing. The percentage
of targets to be located accurately during testing had been tentatively
set at one-fifth of the percentage specified in the Short-Range UAV per-
formance requirements. In response to these concerns the Joint Program
Office is revising its test plans to increase the goal for locational accu-
racy so that it corresponds to the performance requirements.




Page 13                                       GAO/NSIAJMlO-m Unmanned Aerial Vehicles
Appendix II

Objective,Scope,and Methodology


              In response to a request from the Chairman, House Armed Services
              Committee, we reviewed DOD'S acquisition strategy for the Short-Range
              UAV. The Chairman expressed concern about the lack of success on past
              UAV programs and the need to assure that weaknesses contributing to
              the lack of success do not recur.

              We obtained the information for this report by reviewing records and
              interviewing officials of the UAV Joint Program Office and Office of the
              Secretary of Defense in Washington, D.C.; Marine Corps officials in
              Quantico, Virginia; Short-Range UAV Project Office officials at Redstone
              Arsenal, Huntsville, Alabama; and Army Training and Doctrine Com-
              mand officials at Fort Huachuca, Arizona. We also visited the planned
              UAV test site at the c’i\v Joint Training Center at Fort Huachuca.

              We compared the acquisition strategy of the Short-Range UAV and the
              Pioneer UAV, a previously procured system that encountered problems
              when deployed with the Navy’s operational forces. After our initial
              efforts indicated weaknesses in the planned Short-Range system testing,
              we concentrated on evaluating the planned test program and comparing
              it to the Pioneer experience. We obtained information on Pioneer from
              officials of the Naval Air Systems Command, Washington, D.C.

              To determine the extent to which performance will be demonstrated, we
              compared the planned testing for the Short-Range system to its perform-
              ance requirements documents. We also analyzed planned testing docu-
              ments to determine whether they were consistent with DOD'S testing
              policies.

              Our review was performed from October 1989 through April 1990 in
              accordance with generally accepted government auditing standards. DOD
              provided written comments on a draft of this report. These comments
              are presented and evaluated in appendix III.




              Page 14                               GAO/NSIAIM#234 Unmsnned Aerial Vehicles
Appendix III

CommentsFrom the Department of Defense
and Our Evaluation

Note: GAO comments
supplementing those In the
report text appear at the                      DIRECTOROFDEFENSERESEARCHANDENGINEERING
end of this appendix.
                                                         WASHINGTON,    DC   20301-3010




                                                                                                  August   20,   1990




                             Mr. Frank C. Conahan
                             Assistant    Comptroller      General
                             National    Security   and International
                                Affairs   Division
                             General Accounting       Office
                             Washington,     DC 20548
                             Dear   Mr.   Conahan:
                                  This is the Department of Defense (DOD) response to the
                             General Accounting  Office  (GAO) Draft Report, "UWMAWWEDAERIAL
                             VEHICLE: Realistic   Testing Needed Before Production of Short-
                             Range System," dated 10 July 1990,(GAO Code 395120/OSD Case
                             8410).
                                   The DOD concurs or partially     concurs with most of the
                             findings    in the GAO report.     However,. the DoD does not concur
                             with either    recommendation or the matter for congressional
                             consideration.
                                   The Test and Evaluation        Waster Plan for the Unmanned Aerial
                             Vehicle-Short     Range system, which describes           the test program, is
                             being restructured        and will   be reviewed     by the Office      of the
                             Secretary     of Defense (OSD). The responsible             OSD officials     wit I
                             approve the test program only if it is adequate and consistent
                             with applicable      statutes     and DOD directives.        Therefore,    the GACI
See comment 1.               assertion     of inadequate    testing   cannot be addressed at this
                             time.
                                  Detailed- comments on the report findings,     recommendations,
                             and matter for congressional     considerations  are provided   in the
                             enclosure.     Thank you for this opportunity   to review and to
                             comment on the draft    report.
                                                                             Sincerely,


                                                                         &,uffuharles     M. Herzfeld
                             Enclosure




                                     Page 16                                          GAO/‘NSIAD90-234Unmanned Aerial Vehicles
                        Appendix Ill
                        CknnmentaFrom the Department of Defense
                        and Our Evaluation




                                    GAODRAE’T REPORT - DATED JULY 10, 1990
                                       (GAO CODE 395120) OSD CASE 8410
                      YstwANNED ABRIAL VEHICLE: REALISTIC TESTING NEEDED BEFORE
                                  PRODUCTIONOF SHORT RANGE SYSTEW
                                          DEPARTMENTOF DRFWSE CDMMNTS
                                                       *    l   *    l   l


      .                                                    FINDINGS

                 FINDING A: Initiation             of the Short-Ranae            Unmanned Aerial
                 Vehicle      Prociram.     The GAO reported          that a joint        service program
                 was initiated         by the Congress in response               to concern about the
                 proliferation         of unmanned aerial         vehicle      programs and the need
                 to acquire       vehicles     that meet the requirements                of more than one
                 Service.       The GAO explained         that,     in order to achieve
                 commonality,        Congress eliminated          the FY 1988 funding within               the
                 Services'       separate research,         development,         test,     and evaluation
                 accounts for individual            programs and consolidated                 the funding     in
                 a joint-service          program.     The GAO also reported               that, during FY
                 1988, the Congress refused to authorize                      funding      for the
                 procurement        of separate Navy and Army unmanned aerial                      vehicles--
                 providing       funds, instead,       for a joint         program.        The GAO reported
                 that contracts          were awarded in September 1989 to two firms for
                 non-developmental           candidate     systems for a competitive               fly-off    in
                 FY 1991.        The GAO further       reported        that the DOD plans to procure
                 a total      of 53 Short-Range        systems:          27 for the Army; 18 for the
                 Marine Corps: 5 for the Navy: 3 for training                          purposes.       The GAO
                  found that the Short-Range Unmanned Aerial                       Vehicle Decision
                 Coordinating         Paper, dated August 1989, projected                    total
                  acquisition       costs of about $1.5 billion.                  The GAO noted that
                  program officials          now indicate       that that amount may be revised.
Now on pp. 8-9     (pp. T-lo/GAO Draft Report)
                 DOD RESPONSE: Concur.   Depending on the winning                         contractor,       the
                 actual program amount may need slight  revision.

                  FINDING B: Production              Before Realistic        -rational       Testinq.
                  The GAO reported           that, contrary      to its stated acquisition
                  principles,       the DOD does not plan to conduct preproduction
                  operational       testing      of the Short-Range         system in environments
                  typical     of where it may be used in combat.                    The GAO concluded,
                  therefore,      that the test results            (from those tests that will          be
                  conducted)      may not be a valid          indicator      of the system's
                  capability.         The GAO pointed out that the Navy's experience                   with
                  the PIONEER Unmanned Aerial               Vehicle,    illustrated      the need for
                  realistic     operational         testing   of the Short-Range         Unmanned Aerial
                  Vehicle     system.        The GAO noted that the lack of realistic
                  operational        testing      of the PIONEER Unmanned Aerial            Vehicle
                  system led the Navy to costly               and time consuming trial           and




                         Page 16                                             GAO/NSIAD4@234Unmanned Aerial Vehicles
                         Appendix Ill
                         CommentsFrom the Department of Defense
                         and Our Evaluation




                  error,  as it tried   to adapt, for shipboard     uses, a system used
                  in a land-based    desert environment.      The GAO reported  that the
                  Navy is still   buying replacement     hardware to bring the PIONEER
                  systems up to a minimum essential       level of performance.    (pp. lo-
Now on pp, 9-10   lZ/GAO Draft Report)
                  DoD RESPONSE: Nonconcur.               The Unmanned Aerial           Vehicle-Short
                  Range test program must be viewed in the context                        of its
                  acquisition      strategy.       The acquisition        strategy      is non-
                  developmental.          Production     funds are used to procure systems for
                  a competitive       fly-off     and for follow-on         systems.        Full-scale
                  development      funds are not used to procure any systems.
                  Production     of two systems        from each of two contractors                was
                  approved at the Milestone            II/IIIA     decision     point.       The Milestone
                  IIIB decision       point,     which constitutes        the low-rate        production
                  decision,    will     address the selection          of one contractor           to
                  produce up to eight systems.                 The Milestone      IIIC decision        point,
                  which constitutes           the full   production      decision,      will     address the
                  additional     procurement        of up to 43 additional            Short Range
                  systems.
                  As part of selection         criteria      and prior   to exercise     of the first
                  low rate production         option,      two tests will    be performed.      The
                  first    test,    Technical    Evaluation       Test, will  test each contractor
                  system against        the required       system technical     performance    and
                  includes      survivability      testing     using the threat     arrays at the
See comment 2     Naval Weapons Center.
                  The second test, Operational             Test IIA, will       assess each
                  contractor       system for operational         effectiveness          and suitability
                  when employed by representative               user personnel         under realistic
                  operational       conditions.       Operational       Test IIA will        also include
                  a survivability         evaluation    using the results         obtained           from the
                  Technical      Evaluation      Test, which will        have used the threat
                  arrays at the Naval Weapons Center.                   Thus, Operational              Test IIA
                  is a two-month test program that incorporates                      field       exercises
                  under simulated         combat conditions,         in which the system will               be
                  operated by soldiers           and marines.        Training    will      be provided        to
                  bring user personnel           to operating     proficiency        prior       to
                  commencement of Operational             Test IIA.        The test results             from the
                  Technical       Evaluation     Test will     be utilized     to evaluate             each
                  system's performance,            and results     from Operational           Test IIA will
                  be utilized        to evaluate     each system's operational              utility.        That
                   information      will    be provided    for source selection             evaluation.
                   Prior to a Milestone           IIIC    decision     to enter full         rate
                   production,       Operational       Test IIB will       be performed         on the
                   winning contractor's           system.       Operational     Test IIB will          be
                   conducted utilizing          representative         user personnel         under
                   realistic       operational       conditions     to evaluate       operational
                   effectiveness        and suitability         to include     reliability,
                   maintainability,         effectiveness-of-training,              logistic
                   supportability,         and any correction          of deficiencies          previously
                   noted.      Operational      Test IIB is also a two month test program                     in


                                                                  2




                          Page 17                                           GAO/NSIAD90-234 Unmanned Aerial Vehicles
                       Appendix III
                       CommentsProm the Department of Defense
                       and Our Evaluation




                 which the system will         be operated by Army and Marine Corps
                 personnel   in a field       environment  that incorporates threat
                 simulators.
See comment 3.   Therefore,      performance      requirements      and operational
                 effectiveness       and suitability       will    be assessed prior     to the
                 Milestone     IIIB decision        to enter low rate production,         and prior
                 to a Milestone        IIIC decision      to enter full       scale production.     It
                 should be noted that OSD disapproved                 the initial   UAV-SR Test and
                 Evaluation      Master Plan , which outlines            the UAV-SR test program,
                 in December 1989.          The primary reasons for the disapproval
                 concerned inadequacies           in the operational        test program.      When a
                 revised Master Plan is reviewed,               the responsible     OSD officials
                 will    approve the operational          test program only if it is adequate
                 and consistent        with applicable       statutes    and DOD directives.
                 The DoD recognizes     significant problems were encountered    in the
                 effort  to adapt the PIONEER for use by the Navy.        Based on
                 lessons learned from that process,       the DOD is using the block
                 upgrade concept to develop the Navy variant        of the Short Range
                 Unmanned Aerial    Vehicle.
See comment 4.   Concerning      statements      on the PIONEER, the Navy's baseline
                 concept of acquisition           initially        provided     three PIONEER systems
                 that were used during an extensive                   operational     assessment
                 period,     during which time the requirements                   for United States
                 Marine Corp (USMC) land and United States Navy (USN) maritime
                 operations      were further       defined.        Rather than a "time consuming
                 trial    and error"      period,     this assessment was conducted by both
                 the operational         users under actual operational               conditions    and by
                 personnel      at the Pacific        Missile      Test Center.       The resulting
                 assessment provided          Navy planners with the necessary data to
                 effectively       make a baseline          decision     had the PIONEER program
                 continued      as planned.       Additionally,          the parts being bought are
                 spares necessary to maintain                the PIONEER at its currently
                 acceptable       operational     level of performance.
                 FINDING C: Planned            Short-Ranae   Testinq    Is Not Realistic.       The
                 GAO concluded that,          while the DoD plans to do some preproduction
                 operational      testing     of the Short-Range Unmanned Aerial           Vehicle
                 system, the planned           testing   is not realistic     and significant
                 risks remain.         The   GAO further    concluded that any test results,
                 which are to be the           basis for beginning      production,    may not be a
                 valid   indicator      of   the system's capability.
                 The GAO reported     that realistic      operational      testing      of the
                 Short-Range Unmanned Aerial         Vehicle would require           that the
                 system be tested at locations         resembling      those where it might be
                 used in combat.      The GAO concluded that--because              the system,
                 which is to be deployed in diverse           environments,        including   hilly
                 or mountainous    terrain,   dense forests,        rain,    snow, and cloudy
                 weather,   is only going to be tested at Fort Auachuca, Arizona,
                 in desert terrain--     the results     may be misleading.



                                                              3


                                                       .


                        Page 18                                        GAO/NSIAD!30-234Unmanned Aerial Vehicles
                            Appendix m
                            CommentaFrom the Department of Defense
                            and Our Evaluation




                     The GAO also found that the DOD plans to conduct preproduction
                     maritime    testing    at the Pacific      Missile       Test Center.     The GAO
                     further   found, however, that testing              will   not include    operations
                     from Navy ships.         The GAO reported        that the planned testing         will
                     only include      operations     from runways on land and, therefore,              not
                     demonstrate      the system's capability           to operate in a naval
                     environment.        The GAO referred      to the difficulties          in adapting
                     the PIONEER to use at sea.            According      to the GAO, (1) 17 of the
                     original    40 vehicles      crashed,    (2) the Navy redesigned          and
                     modified    nearly the entire        system at a cost of about $50 million
                     in research and development and replacement                   hardware, and (3)
                     the system still       does not meet its intended             performance
Now on pp. 10-l 1.   requirements.        (pp. 12-lS/GAO Draft Report)
                     DOD RESPONSE: Partially              concur.       The test       program,     composed of
                     both technical        and operational        testing,     will      be conducted        in the
                     severe flight       environments      of Fort Buachuca (high altitude,                    high
                     temperature,       dust, etc.)      and the Pacific         Missile       Test Center
                     (moisture/water,        fog and salt/fog,          etc.).      In order to increase
                     the realism of operational             tests,     camouflage will          be used to
                     mask targets.         The Pacific     Missile      Test Center early           chamber
                     technical     testing     will    expand/supplement         the natural        conditions
                     encountered      to cover the remaining            environmental          specification
                     requirements.         The Naval Air Propulsion            Center engine technical
                     tests will      determine      the capability        of the engines to meet all
See comment 5.       altitude    and temperature         specifications.
                     An overriding      factor   in the identification          of Fort Huachuca,
                     Arizona,     as the site for the OT-IIA was the extensive                  airspace
                     and range scheduling        limitations        presented   by other potential
                     test sites.       The environment         at Fort Huachuca is extremely
                     severe on system operation.               The combination      of altitude      and
                     temperature     is equivalent        to operating      at an altitude      of 10,000
                     feet.     The terrain     at Fort Huachuca includes            mountains and some
                     treed areas.       The testing       will   take place during the Fort
                     Huachuca rainy season.           During this period,         low-hanging       clouds
                     and periods     of intense     rain can be expected.            The rainy season
                     begins in the June timeframe.               The Unmanned Aerial         Vehicle Joint
                     Project Office       is in the process of identifying              an additional
                     site,    with environmental        conditions      other than those of Fort
                     Huachuca, for conduct of a portion                of the Operational        Test IIB.
                     To correct    the record, maritime operations         training    and testing
See comment 6.       of the PIONEER Unmanned Aerial         Vehicle   has resulted      in fourteen
                     air vehicles     lost rather than the seventeen cited in this
                     finding.     As previously    stated,   the DOD recognizes       significant
                     problems were encountered        in the effort     to adapt the PIONEER for
                     use by the Navy on board ships.           The Department's     block     upgrade
                     concept to develop the Navy variant           of the Short Range Unmanned
                     Aerial   Vehicle    is designed to take advantage of the lessons
                     learned on PIONEER.
                     Plans for the operational     testing               of the Navy Block  I variant            of
                     the Unmanned Aerial   Vehicle-Short                 Range during the follow-on


                                                                     4



                                                          .

                             Page 19                                            GAO/NSIAD-90-234UNMNI ed Aerial Vehicles
                                                                                                               -
                       AppendixIII
                       Comment8FYomtheDepartmentofDefense
                       andOurEvaluation




                 operational       test and evaluation    phase have not yet been
                 developed.        However, the DOD, particularly,    operational test
                 agencies,      recognize   the need to test weapon systems onboard
                 ships,    if   the systems will     be used in that environment.
                 FINDING D: Conqressional               Intent   For A Joint-Service            System Is
                 At Risk.     The GAO observed that DOD acquisition                     policies     require
                 that operational         testing     be completed prior          to full-rate
                 production.        The GAO found, however, that the DOD plans to
                 commit to full-rate           production      of the Short-Range          Unmanned
                 Aerial    Vehicle     before verifying         that it can be modified             to meet
                 Navy requirements.            The GAO reported         that the DOD plans for
                 operational      testing      in 1992--before        full-rate      production
                 begins --involve       the Block 0 version           for use by the Army and
                 Marine Corps, but will            not incorporate          those features       necessary
                 to operate from Navy ships.                The GAO concluded,          that if
                 subsequent operational            testing     of the Navy variant           were to show
                 that a modified        Short-Range        system is unsuitable           for Navy use,
                 the DOD would be faced with the alternatives                       of (1) canceling
                 the Short-Range production               and starting        a new program to acquire
                 a system meeting both Army and Navy needs or (2) continuing                             the
                 Short-Range production            for the Army and Marine Corps and
Now on p. 12.    acquiring     a separate system for the Navy.                   (pp. 15-16/GAO Draft
                 Report)
                 DOD RESPONSE:          Partially     concur.     As provided      for in the current
                 contracts,      as part of the downselect            process,    each contractor
                 will    submit for evaluation            an engineering    study, which will
                 detail    system configuration            changes required      for shipboard
See comment 7    operations.         Subsequent to the downselect,            the Navy shipboard
                 variant     system will        then complete technical         and detailed    design
                 reviews.       Full-rate       producticn     of Block 0 is planned to occur
                 following      design reviews of the Navy shipboard                 variant.
                 Operational      Testing of the Block I system will                 be conducted
                 onboard ships prior             to exercising    the production        option of the
                 naval variant.
                 During an Early Operational       Assessment, the Navy will        assess the
See comment 8.   Block 0 Unmanned Aerial      Vehicle-Short     Range system's capability
                 to meet shipboard    requirements.       Of primary     interest is the
                 capability  of the Short-Range       Block 0 system to operate from
                 ships while performing      Reconnaissance,     Surveillance,    and Target
                 Acquisition  missions.
                  It is important    to realize   that the preponderance    of the current
                  requirement   for the Unmanned Aerial      Vehicle-Short  Range is for
See comment 9     the Block 0 version.       Only five of the 53 vehicles    planned are
                  the Navy variant,    Block I.     The approach recommended by the GAO
                  would delay fielding     of the 48 Block 0 systems by one to two
                  years, while the Block I Navy variant        completed its operational
                  testing.
                  The best way to operate an Unmanned Aerial                    Vehicle    from ships,
                  particularly launch and recovery operations,                    is still     being


                                                                5




                         Page20
                         AppendixIII
                         CommentaFromtheDepartmentofDefenae
                         andOurEvaluation




                    explored.      It is possible      that the Block 0 air vehicle              may not
                    be usable or may require          extensive      modifications      for shipboard
                    use.      However, this would not preclude             system interoperability
                    or commonality      in the use of other Unmanned Aerial                 Vehicle-Short
                    Range system elements (mission planning                 stations,     ground control
                    stations,     data terminals,      etc,)    in the Block I system.             Testing
                    of the Block 0 system        will    provide information          applicable     to the
                    Navy variant.       That information        will   be available       before the
                    full-rate     production   decision      for the Block 0 system.
                    FINDING E: Some shortcominqs           of the The Orisinal            Test Plans
                    Addressed.      The GAO reported      that it (along with officials              in
                    the Office     of the Secretary      of Defense) identified            other
                    shortcomings     in the Short-Range        system's original          test program
                    that the Unmanned Aerial        Vehicle Joint Program Office               agreed to
                    address by significantly        revising      the test plan.          The GAO
                    reported   that,     initially,   the low-rate        initial    production
                    decision   for the Short-Range        Unmanned Aerial          Vehicle    was planned
                    to follow     an early operational        assessment period,          rather than
                    the currently      planned operational        testing.        The GAO noted tha<,
                    subsequently,      the DOD revised      its acquisition         strategy     to
                    provide for preproduction        operational        testing.
                    The GAO reported       that another shortcoming      was that the test plan
                    contained    a provision     that none of the criteria      for judging the
                    Short-Range     system's performance     was to be considered       as
                    absolute    pass or fail.       The GAO found this provision      in conflict
                    with the intent       of the Secretary   of Defense to discipline        the
                    acquisition     process by defining     minimum required      accomplishments
                    for advancing      from one phase to the next, as recently          stated in
                    July 1989 Defense Management Report to the President.                The GAO
                    reported    that,    subsequently,   the conflicting    provision    was
                    removed from the test plan.
                    The GAO also reported         that both GAO and DOD officials              had
                    concerns about the success threshold                for locating      targets
                    accurately      during Short-Range       testing.       The GAO found that the
                    percentage      of targets     to be located accurately          during testing
                    had been tentatively         set at one-fifth         of the percentage
                    specified     in the Short-Range        Unmanned Aerial       Vehicle     performance
                    requirements.         The GAO reported      that,     in response to those
                    concerns,     the Joint Program Office           is revising     its test plans to
                    increase   the goal for locational            accuracy so that it corresponds
Now on pp. 12-13.   to the performance         requirements.        (pp. 16-la/GAO Draft Report)
                     DoD RESPONSE: Concur.          The revised Unmanned Aerial      Vehicle-
                     Short Range Test and Evaluation          Master Plan has been revised              to
                     better    reflect operational     requirements    and to provide for
See comment 3.       adequate operational      testing    prior  to Milestone  IIIB and IIC
                     decisions.

                                                         l   l   l   l   l




                                                                 6




                           Page21                                            GAO/NSLAIMWB~UNMNI edAerialVehicles
                       Appendix III
                       CommentsFrom the Department of Defense
                       and Our Evaluation




                                                    RECONNFNDATIONS
                  RRCONHRNDATION1: The GAO recommended that the Secretary                  of
                  Defense require     that the Short-Range Unmanned Aerial          Vehicle
                  system be operationally     tested in diverse,       realistic
                  environments    to provide reasonable      assurance that it will        meet
                  requirements    before permitting    limited    production     of the Iand-
                  based system.
                  DoD RESPONSE Nonconcur.              Department of Defense Directive              5000.3
                  states that demonstration            of a system's    technical      capabilities
                  and its operational         effectiveness     and suitability        will   be a key
                  requirement      for decisions       to commit significant        resources,        to
                  advance from one acquisition            phase to another,       and to field          the
                  system.       An adequate evaluation        of a system's operational
                  effectiveness       and suitability       can be accomplished        without
                  testing     in all of the operational         environments      in which a system
                  may be employed.          The Unmanned Aerial       Vehicle    Short Range
                  acquisition      strategy    and, in particular,        the test program, is
See comment 10.   consistent      with the DOD directive        for a non-developmental             item
                  program.
                  The planned Technical         Evaluation    Test and Operational     Test IIA
                  will    evaluate    the system’s    performance   and operational
                  effectiveness/suitability.            The planned test program will       test
                  the system in meaningful          representative   operational    environments
                  and will     provide data to support a Milestone          IIIB decision     to
                  enter low rate production.
                  RRCONNENDATION2: The GAO recommended that the Secretary               of
                  Defense limit       Short-Range Unmanned Aerial     Vehicle system
                  production    until    satisfactor:r  performance   of the Navy variant   is
                  demonstrated     and assure that actions       are taken to correct   the
Now on p. 2.      other shortcomings         noted in the planned test program. (pp, 3-
                  4/GAO Draft Report)
                  DOD RJZSl?ONSE: Nonconcur.             The Unmanned Aerial         Vehicle-Short
                  Range acquisition         strategy,     with its provisions         for production
                  lot options,       limits    production      of the Block I system until             its
                  performance      is demonstrated.          However, the Department of Defense
                  does not intend to delay production                 of a successful        Block 0
                  system, pending the outcome of the Block I development.                           As
                  previously      noted, 48 of the 53 systems planned are the Block 0
                  version.      The Unmanned Aerial          Vehicle - Short Range acquisition
                  strategy    includes       a block upgrade approach,           which allows for the
                  evaluation      and fielding       of an initial      baseline     configuration,
See comment 11    followed    by block upgrades to meet the full                 operational
                   requirements.        The modular architecture           of the system will
                   facilitate     upgrades.       Block 0 is the initial,          baseline      system.
                  Block I is the Navy shipboard              variant.      The described
                  acquisition       strategy     has been fully       reviewed and approved, and
                   is in keeping with the non-developmental                  item nature of the
                   system.       The cognizant        congressional      committees have also been
                   appraised     of the acquisition         strategy     being approved.


                                                                7




                         Page 22                                         GAO/NSIADW234 U-            ed Aerial Vehicles
                              AppendixHI
                              CommentaFromtheDepartmentofDefense
                              andOurEvaluation




                        AS   provided     in the current       contracts,      each contractor        will
                        submit for evaluation           as part of the downselect             process,      an
                        engineering       study which will        detail    system configuration            changes
                        required      for shipboard      operations.        These studies        will   be
                        completed this fiscal           year.     Following      this,    a FY1992 contract
                        option may be exercised            for the integration,           demonstration,         and
                        test of the Block I Navy shipboard                variant.        Not until     FY1994
                        will    the government exercise           a contract       option which will
                        include     the purchase of one system with Block I upgrades.
See comment 7           Operational        testing   of this system in a shipboard               environment
                        will    be conducted prior         to exercising       an option      to procure
                        additional       Navy systems.        This is the first          of five Navy
                        shipboard       systems to be purchased.            Therefore,      the Block I
                        engineering        studies   provided     to each competing contractor                as
                        part of the downselect           process:      the technical       design and the
                        detailed      design review subsequent to down select:                   and the
                        operational        testing   of the initial       Block I system prior             to
                        procuring       additional     systems, will      sufficiently        minimize the
                         risk associated         with the block upgrade approach.



                                                  POR CONSIDERATION BY THE HOUSE
                                                   COMMITTEE ON ARMHD SRRVICES

                        MATTER FOR CONSIDERATION: To assure that congressional              intent
                        for a common-service   system is achieved,       the GAO suggested that
                        the Committee may wish to prohibit     full-rate       production   of the
                        Short-Range  Unmanned Aerial  Vehicle until        performance    of the
Now on p. 3             Navy variant  is proven to be satisfactory.           (p. 4/GAO Draft
                        Report)
                        DOD RESPONSE:         Nonconcur.     The non-developmental      nature of the
                        Unmanned Aerial        Vehicle   Short Range Program eliminates        much of
                        the risk inherent          in a developmental   program.     The risk is
                        further    minimized       by the Block 0 test program, the Block I
                        engineering       evaluations    and design reviews,     and the pre-
                        production      tests of the Block I Navy shipboard          variant.    The
See comments 2, 4, 9,   resultant     risk is so small that the program delays and
and 11.                 additional      costs inherent      in this suggestion     exceed any
                        additional      risk avoidance.
                         Operational     testing  of the Unmanned Aerial        Vehicle-Short       Range
                         system before procurement         of the Navy Block I variant         will
                         provide a basis for Block I development.             The GAO cited problems
                         with the performance      of the PIONEER system onboard ships.               Many
                         of those problems were associated           with the difficulties         of
                         adapting    a land-based   system for shipboard        use.     The DOD has
                         learned from that experience.           Many uncertainties        and unknowns
                         are still    associated   with the use of Unmanned Aerial            Vehicle
                         systems in worldwide      military     operations   across a wide spectrum


                                                                        8




                                Page23                                            GAO/NSIAlMO-234U-            ed AerialVehicles
                                                                                  -
     Appendix III
     CommentsProm the Department of Defense
     and Our Jhwluation




of combat intensity.  The Block upgrade concept        will     provide a
measured approach to resolving  those uncertainties           and unknowns.




                                      9




      Page 24                                 GAO/NSIAD-90-234Unmanned Aerial Vehicles
               Appendix Ill
               CommentaProm the Department of Defense
               and Our Evaluation




               The following are GAO'S comments on the Department of Defense’s letter
               dated August 20, 1990, and its accompanying enclosure.


               1. We believe that concerns about inadequate testing need to be
GAO Comments   addressed as part of the ongoing restructuring of the Test and Evalua-
               tion Master Plan.

               2. The nondevelopmental nature of the program, does not negate the
               requirement for realistic testing. DOD directive 5000.3 on testing states
               that nondevelopmental items shall be guided by the same WD acquisi-
               tion principles as other systems. This directive requires testing of suita-
               bility for use in combat in an environment as operationally realistic as
               possible.

               DOD’S comments do not address our main concern that the Short-Range
               system’s preproduction operational testing will be limited primarily to
               the desert environment which does not resemble the primary locations
               where the system is to be deployed. These locations are marked by
               mountainous terrain with dense forests and adverse climatic conditions.
               From that standpoint, the planned testing of the Short-Range system is
               unrealistic and could yield misleading results on the system’s capability
               to locate targets.

               It is also important to point out that DOD'S comments address the
               planned testing prior to the full-rate production decision, while our
               report addresses the lack of realistic testing in DOD'S preproduction test
               plan. Moreover, the site referred to by DOD as Operational Test IIB has
               not yet been identified by DOD. Therefore, DOD’S response indicating that
               the operational tests leading up to a full-rate production decision will be
               conducted under realistic operational conditions is premature. Program
               officials indicated that discussions have occurred within DOD about
               selecting a site that would be more representative of a planned deploy-
               ment location than Fort Huachuca, but no decision has been made.

               3. Our report recognizes that operational testing of the Short-Range
               system will be held before the low-rate and full-rate production deci-
               sions. Our concern with the testing before low-rate production (opera-
               tional test IIA) is that it is to be restricted primarily to an unrealistic
               environment. Our concern with the testing before full-rate production is
               that it will not evaluate the system’s capability to operate in a naval
               environment. Testing of the Navy variant is to occur only after DOD has
               committed to full-rate production.


                Page 26                                 GAO/NS~SO-234 Unman ed Aerial Vehicles
Appendix Ill
CommentsFrom the Department of Defense
and Our Evaluation




4. We cited the Pioneer’s problems, in part, to illustrate the need for
realistic operational testing before committing to production. In our
view, DOD is not capitalizing on what should be a lesson learned from the
Pioneer experience.

Pioneer problems were discovered by deploying the system and waiting
to see if it worked. When problems occurred, the Navy was forced to
continually bring in experts from the Naval Air Propulsion Center, the
Naval Air Test Center, the Pacific Missile Test Center, or send personnel
to the subcontractors in Israel to find solutions. In 1987, the Navy pro-
duced an Integrated Action Plan to address the numerous shortcomings
of the system. In this plan the Pioneer acquisition strategy was entitled
“Operate, Learn, Fix, Operate.” We see no difference between this and
“trial and error.”

The Navy recently (1990) described the Pioneer system’s level of per-
formance as a “minimum essential capability,” (underscoring added).
We would point out, that DOD and the Navy expended $28 million in
research and development funds and an additional $22 million in pro-
curement for replacement hardware over 4 years to reach this level of
performance.

5. Although Fort Huachuca has a high altitude, the terrain is generally
flat and treeless. Moreover, there is no indication in the revised draft
test plan that DOD intends to overcome this drawback.

To determine whether the rainy season at Fort Huachuca was signifi-
cant and might indicate something about the ability of the Short-Range
system’s ability to operate in general conditions of wetness, we acquired
recent meteorological data for the closest metropolitan area to the Fort
Huachuca test site. It showed that (1) less than 3 inches of rain fell per
month during the 3-month rainy season (June, July, and August) and
(2) rain fell in measurable amounts on only 5 days. This environment is
not similar to that of the primary locations where the system is to be
deployed. For example, at one planned deployment location the rainy
season is 7 months, and measurable amounts of rain occur on average 16
days per month. Average monthly accumulation exceeds 8 inches per
month, more than Fort Huachuca can expect in its entire 3-month rainy
season.

 Range scheduling limitations presented by other potential test sites, as
 identified in DOD'S response, should not in our view be an overriding
 factor. The need to perform realistic testing should be the deciding


 Page 26                                 GAO/‘NSIAIMW-234Unmanned Aerial Vehicles
Appendix Ill
CommentsFrom the Department of Defense
and Our Evaluation




factor. Although DOD may be looking for a test site other than Fort
Huachuca, the Operational Test IIB will not occur until after production
of the Short-Range system has started.

6. In a draft of this report, we identified 17 Pioneer air vehicles as being
lost during maritime operations. The final report has been modified to
reflect DOD’S position that, of the 21 vehicles lost, 14 were lost during
maritime operations. (Also see comment 4.)

7. We do not disagree with DOD’S comments; however, the fact remains
that the Short-Range system will be in full-rate production before any
operational testing of the Navy variant is done.

8. According to the Director, Defense Operational Test and Evaluation,
an early operational assessment is no substitute for operational testing.

9. We did not recommend delaying the fielding of the Block 0 system.
Our recommendation would require that the Block 0 system be retained
in low-rate production pending verification that the Block I variant will
meet Navy needs. This would reduce the risk of DOD becoming fully com-
mitted to production of a system not meeting common-service require-
ments. Further, DOD’S plan is to retrofit Block 0 systems to incorporate
design features of the upgraded configurations; therefore, limiting Block
0 production would reduce the risks associated with the retrofit.

Testing of the land-based system (Block 0) will not provide reasonable
assurance that the various elements will meet the Navy’s requirements.
Such assurances can only be obtained through operationally testing the
system in a realistic naval environment.

10. DOD cannot conclusively determine whether the Short-Range UAV will
be operationally effective or suitable based on its planned testing. The
operational environment in which DOD intends to test is not at all similar
to the diverse environments of the primary locations where M)D intends
to deploy the system. In fact, DOD’S draft restructured test plan recog-
nizes that the testing environment constitutes a test limitation.

11. Our recommendation does not affect low-rate production of the land-
based system. Although specifically proscribed during congressional
consideration of fiscal year 1988 UAV funding, DOD’s adherence to the
current Short-Range UAV acquisition strategy could result in separate
Army and Navy Short-Range UAV systems if the Navy variant proves a
failure. This would not be consistent with congressional direction.


 Page 27                                 GAO/NSLAD-W234   Unmanned Aerial Vehicles
AppemdixIII
CommentsFrom the Department of Defense
and Our Evaluation




DOD’S acquisition strategy calls for entering full-rate production of the
Short-Range system for use by the Army and Marine Corps before con-
ducting any testing of the block upgraded Navy variant. Scheduled
testing of the Navy variant will only reduce the risk of prematurely
committing to production of that system, not the basic system, which
will already be in full-rate production. If the Navy variant were to prove
unsuccessful in the later testing, DODwould find itself fully committed to
production of a system not meeting congressional intent for a common
UAV. Further, as brought out in our report, experience with Pioneer
shows that adapting a UAV for naval operations is not a low-risk effort
even though the Navy categorized it as such when justifying the pro-
gram. We therefore continue to believe that DOD should retain the Short-
 Range system in low-rate production until operational testing shows
that the Navy variant’s performance is satisfactory.




 Page 28                                 GAO/Ni%UMM34U-       ed Aerial Vehlclea
Appendix IV

Major Contributors to This Report


                        Charles A. Ward, Evaluator-in-Charge
National Security and
International Affairs
Division, Washington,
D.C.
                        Jackie B. Guin, Assistant Director
Atlanta Regional        Bobby D. Hall, Evaluator
Office                  Terry D. Wyatt, Evaluator




 (396120)                Page 29                               GAO/‘NSIAD~~   UNMNI ed Aerial Vehicles