Review of the Operational Test and Evaluation of the Fast Automatic Shuttle Transfer System and Other Systems

Published by the Government Accountability Office on 1971-09-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                           ,(-.   ! .


                The Ronorable                             ._-
                The Secretary           of Defense       ',

                      Attention:           Assistant     Secretary   of Defense

                Dear Mr.     Secretary:

                        The General Accounting  Office has made a ~~e+e$ewof the operational
                test and evaluation     of the Fast @tomatic     Shuttle Transfer (FAST) system
                audother     systems (GAO Code 77104).    The review was performed at the
                Navy's Operational    Test and Evaluation    Force (OFTEVFOR), Norfolk,

                        In conjunction      with this review,       we noted that production       commit-
                ments had been authorized            by the Navy before adequate testing         of about
                one-third    of the systems assigned to OFTEXFOR as of September 30, 1970,
                for operational       testing     and evaluation.       Consequently,   OFTEVFOR is un-
                able, in these cases, to perform sufficient                 operational  evaluation     of
                new equipment to determine its suitability                for service use. As pointed
                out in previous       reports,     concurrent     development and production       fre-
                quently results       in additional       expenditures    of time and money to iden-
                tify    and correct     deficiencies      and may cause delays in deployment of
                effective    systems.

                       Our work indicates     a need for management improvements which, we
                believe,    would supplement the current measures being taken by the De-                      --
                partment of Defense and the Navy to improve operational              test and evalu-         .
                ation procedures.      Although we did not assess the urgency of the pro-
                curements and other factors        which the Navy deeisionmakers        may have
                considered    in these cases, we are reporting       our fLndings at OFTEZFOR
                and our suggestions      for earlier    operational  evaluation    to permit timely
                consideration    of any risks related      to urgent requirements.

                      OP!i!EVFOR's operational    evaluation   of newly developed equipment nor-
                mally follows   the developing      command's tecbnBzK%valuation           ai?d&termi-
                nation that the equipment meets the technical           requirements.       The Chief
                of Naval Operations     (CNO) then directs     OFTEWOR to perform independent
                test and evaluation     in an operational    environment.       OFTEWOR determines
                whether the equipment meets the operational          requirements      and makes recom-
                mendations 'concerning     its suitability   for service use.         This includes
                making a determination       that the equipment can be operated,          maintained,
                and supported logistically

                                             50fH    A

       CNO instructions        provide that new equipment till           not be committed
to large-Scale       production     until  its suitability        for service use has been
established     through operational        testing.       When new items are submitted
for testing     in accordance with this policy,              OPTEVFOR's test and evalua-
tions are usually        performed using prototype           models of the equipment.
In cases where urgency or important              operational      considerations      indicate
a need to deviate from this policy,              procurement     prior to completion          of
operational     testing    may be authorized.         In instances      where procurement
is authorized      before equipment has undergone operational                 tests and
evaluations,      service use and operational           testing    normally     occur con-
currently,    as in the case of the FAST system.                In these instances,
OPTEXFOR usually utilizes           equipment already delivered          to the fleet       to
make its tests.

       To meet its responsibilities,        OPTEVFOR is organized into three test
and evaluation    squadrons located at Key West, Florida,          and Point Mngu
and China Lake, California,         and two test and evaluation     detachments
located at New London, Connecticut,          and Key West, Florida.      It has a
total   military  strength    of about 1,400 officers     and enlisted     personnel
and about 24 civilian      personnel.      OFTEVFOR has about 40 assigned mili-
tary aircraft    and uses other shore facilities        on both the east and west
coasts as well as ships of both the Atlantic           and Paciffc   Fleets.

        The results of OPTEXFOR's tests of new equipment and its recommenda-
tions are submitted     directly to the CNO. Decisions  as to the accept-
ability    of new equipment for operational use are made by the CNO.


        Our review of the FAST system, which is used for underway replenish-
ment of ammunition and stores between supply and combatant ships, iden-
tified    problems that cau result   when new equipment is procured without
completing      operational testing.  We found that the FAST system was in-
stalled    on about 50 ships before operational    testing and service use
showed that the system was not reliable       and could not be effectively

       Procurement       of the FAST system started       in 1960. CR0 did not assign
this equipment to OPTEVFOR for operational               evaluation  until October 1964;
operational      testing    started    in April   1966 and was completed in April 1968.
OFTEVFOR recommended that the system not he accepted for operational                use
until   correction       of a number of deficiencies.         The Navy has since removed
or modified most of the equipment developed for this system in order to
provide a more reliable           andssimpler   transfer    system.

      In view of our findings on the FAST system, we broadened our retiew
to cover allwstems   assigned to OPTEXFORfor operational   testing as of
September 30, 1970. As of this date, the CNO bad assigned to OPTEVFOR



41 items of equipment         for   operational    testing   to determine   suitability
for service use.

       We    requested the Navy to complete a questionnaire       on each of these
projects       to facilitate   our evaluation  of whether OFTEVFOR was being uti-
lized in       accordance with its assigned mission.      The Navy returned   ques-
tionnaires        for 39 of the 41 projects.    While the questionnaires    contained
certain      inconsistencies,    we determined the following:

                                                             No. of
                 Contracts      for production     of
                 items for service use entered
                    Before or about the date
                    operational     testing    was

                   After operational     testing
                   was requested     but before
                   tests were started

                   After  operational tests
                   were requested but before
                   the tests were completed                       k

Thus, production     commitments had been authorized        before the completion
of operational    test and evaluations     for about one-third       of these systems.
In these cases, OPTEVFORwas determining         suitability      for service use after
procurement    commitments had been made. Pertinent          data regarding   each of
the 13. oases summarized'sbove     is included   in the appendix to this report.

       These 13 cases involve    conditions    similar   to those discussed in our
report to the Congress entitled        "Admrse Effects     of Large-Scale     Produc-
tion of Major Weapons Before Completion of Development and Testing"
(~-163058 dated November 191 1970).         In that report,     we pointed out that
most of the Navy's major weapons systems were approved for large-scale
production    before development and testing       were completed.     The report
also commented that (1) when concurrent         development and production        oc-
curred,    weapons frequently   did not perform as intended resulting          in ex-
penditures    of large sums of money and time to identify          and correct    defi-
ciencies    and (2) the deployment of effective        weapons may not have been
accelerated    and, in fact,  may have been delayed.
      We made a limited          examination     into the 13 cases where production
commitments were authorized            prior   to completion   of operational evaluations
and found      adverse   effects    similar    to those described in the above report.


      In one case, we found that contracts for production of 83 gun
pods costing $18.6 million were awarded in December 1964 and June 1965.
After units of this equipment were placed in service, various Navy
operational commandsreported that the equipmeut was not satisfactory.
During OFTEVFOR'sevaluation-- requested by CNOin August 1965 and com-
pleted in March 1968--it was also determined that the gun pods were
unsatisfactory.   In October 1966 CNO directed that no additional  units
be procured because tbfs equipment could not be considered reliable.
        With respect to timely utilization  of OFTEVFORwe found that de-
lays have occurred in the assignment of projects to OPTEVFOR. In addi-
tion, we found that delays have occurred in the commencementof tests
by OPTEVFORafter projects have been assigned by the         0. We noted,
for example, that CPTEVFOR(1) was not requested by the CNOto perform
operational tests on one item until about two years after the date of
initial    procurement action, and (2) did not start actual testing on
another item until over four years bad elapsed from the date testing
was requested.      The appendix to this report illustrates  similar delays
for other items.
         In March 1970, in commenting on the draft of our earlier report
(B-163058), the Navy stated it would revise its instructions           regarding
 concurrent development and testing of weapons systems. Subsequently,
in an attachment to a memollandumdated December 21, 1970, to the Secre-
tary of Defense, the Secretary of the Navy cited certain weaknesses in
the conduct of operational tests and evaluatfons.           To improve the effec-
tiveness of operational tests and evaluations,       the Secretary stated that
he planned to (1) centralize the test and evaluation forces to achieve
a greater depth and variety of analytical       capability,    data processing
facilities,     and instrumentation;   (2) give OFTEVFORa role earlier in the
development process; and (3) strengthen the present system of monttoring
the correction of deficiencies       revealed during operational evaluations.
       In February 1971the Office of tbe Secretary of Defense established
a position&    Deputy Mrector for Test and Evaluation with across-tbe-
board responsibilities   for the Office of the Secretary of Defense in test
and evaluation matters.     Also, in March 1971, the Navy established a Di-
rector, Research, Development, Test and Evaluation (RD!t%E) within CEO.
In May 1971, the Navy established an Assistant Mrector for oT&E and a
Test and Evaluation Ditision within the Navy's Office of the Mrector,
        We were fnformed by an OFTEVFORofficial in Juue 1971 that no
specific changes had occurred in the conduct of 0P8E at the operating
level.   At a meeting with officials of the Navy's Office of the Mrector
for RlX&E in August 1971, we were informed that certain guidance con-
cerning the conduct of operational test and evaluation had recently been

received     from the Secretary        of Defense, but in view of the newness of
their    organization,  actions        had not yet been taken at the Service head-
quarters     level to implement        these instructions.

                                                                            MEN?= AVAILABLE
        The delays in requesting          OPECVFORto perform operational         tests and
in the start of testing          indicate     a need in cases of urgency (1) to assign
higher priorities        to operational       test and evaluation    effort   and (2) to
emphasize earlier        operational      evaluation   by the independent     test agency
to permit consideration          by the decisionmaker       of any rLsks related      to
large-scale      production     of urgent requirements       before completion      of oper-
ational    testing.      We believe that a decision to commence production               be-
fore completion       of testing      because of urgent need should be accompanied
by an equally urgent effort            to complete the required      operational     evalua-

        We also believe that,      in cases where urgency may be a factor,           there
is a need for earlier       coordination     of the technical      and operational     eval-
uations.      In this way, optimum fnformation        --concerning    system operational
effectiveness      and the associated     risks--may     be made available     to the de-
cisiomnaker     earlier   in the acquisition      cycle.

       Our major concern in such instances                is whether the decisionmaker
has sufficient      data available        to assure and to document that the risks
connected with the decision to proceed to full-scale                        production    because
of urgent requirements         are reduced to acceptable              levels so that,      in his
opinion,    the decision will not jeopardize                effective     operational     use of
the equipment.       In instances        wbere the absence of military              urgency per-
mits the use of initial          or ptlot     production       units to complete realistic
operational     evaluations,      we are also concerned that the initial                  procure-
ment is approved only for the limited                 quantities      needed for a determina-
tion of operational        suitability.         From our experience,           it appears that
when the approval for large-scale               production       of an item is made before
completion     of operational        evaluation     testing,      harmful cost and perform-
ance consequences usually            occur.

       Reassignments    of responsibility      and organizational   changes made or
now under consideration       at the close of our review may result        in
strengthening    controls    over the matters noted during our review.          How-
ever, as of June 1971, some 15 months after the date of the Navy's reply
to our earlier     draft report    (B-163058),    we noted no significant     changes
at the Navy's test and evaluation         operating    levels.

      Accordingly,    we are recommending that,       together    with the organi-
zational   changes, specific    steps be taken to assure that (1) initial
contracts    for production  units are awarded only for the minimum quanti-
ties needed for operational       testing and for any urgent requirements
which must be met while operational       suitability       is being established;

.       -



            .           ’


                            (2) in cases of essential urgent need, the decisionmaker has s&ficient
                            operational test data to reasonably establish and document that risks
                            resulting from the urgent requfrements are acceptable; and (3) OPTEVFOR
                            be directed to proceed with the required tests on a priority    basis.   In
                            this regard, we further recommend that OPTEVFOR   be required to monitor
                            the development of systems and components requiring operational evalua-
                            tions in order to better plan ORTEVFOR'sfuture workload and to arrange
                            for testing at the earliest possible date, parbicularly    when urgency is
                            a compelling factor.
                                 We would appreciate your comments and advice of any specific ac-
                            tions planned or being t&ten to improve the utilization    of operational
                            test and evaluations within the Navy. If you or your representatives
                            wish to discuss these matters or require additional information,     please
                            contact Mr. Harold E. Rubin, Associate Director,    code 129, extension
                            4515   l

                                  Since this report contains recommendations for your consideration, \_ _
                            copfes are being sent to the Appropriations   and Government Operations     '
                            Comm$ttees of both Houses of the Congress under the provisions of Set- :j '
                            tfon 236 of the Legislative   Reorganization Act of 1970. We will appre-
                            ciate receiving copies of the statements you furnish the specified
                            Committees in accordance with these provisions.
                                  Copies of this letter are also being sent to the Mrector of De-
                            fense Research aad Engineering and the Secretary of the Navy for their
                                                                    Sincerely   yours,


                                                                                                  - 6-


                             OPERATIONAL TEST AND EVALUATION FORCE

                        Schedule      of Pertinent   Data for Projects  Sbowlng
                                   Procurements    Before Completion of
                                    Operational    Test and Evaluation
                                          as of March 31, 1971

             Date of develop-
              ing agency's             Date assigned     Testing    by OFTETJFOR
 Case         request for               to OPTEXFOR       Date          Date       pate procured
number        evaluation                   by CNO        started      completed  for service use

                  6,‘28/6                   8/26/65      g/10/6        3/12/aa          '$$!i~e
                10/30/66                  11/23/66       3/29/7l           -
                 3/9+/67                   5/z/67           x0/67          -
                11/13/67                   3/2w3         3/12/f@           -             l;;:gd
                 12/7/67                   e/19/68           5/a           -           7/25/ad
                 7/15/a                                      2/70          3/70b       g/13/@
                   8/5/a                    '$if$           81433          7/70c       9/27/Bd
                 5/22/69                    7/28/69      B/31/70          w70              FY 69
                 6/24/e                   lOl3a9             5/70          2/71       11/19/68d
                 l/15/70                    4/10/70                                    10/7/70
                   2/5/70                   5/25/70                                    9/29/70
                 3/10/70                   '+/22/70          2/71         -                FY 71e
                 6/x2/70                   9/16/w         3/5/71          -            8/22/@

aOpen end project       to provide for testing     of additional  components       as needed.
bAdditional      component of system to he tested.
'Items to be tested on supplemental         delivery   vehicles.
dMultiple    procurement     contracts awarded commencing with thekitial
     procurement    date shown.
eExact dates not shown in questionnaires           completed by the Navy.