oversight

Test and Evaluation: The Director, Operational Test and Evaluation's Role in Test Resources

Published by the Government Accountability Office on 1990-08-27.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                               TEST AND
                               EVALUATION
                               The Director,
                               Operational Test and
                               Evaluation’s Role in
                               Test Resources


                                                        3b
                                               IIlllllll
                                                142081




                            __-.--
GAO/NSIAI)-90-I   28   --
                                                                                      .-   -_--1--
                ..“-...   -.   _--....-.._   l __-   __.___   --_-   __-______   --
.-__.,,   I._

                                                                                                     -   ;;
                  United States
GAO               General Accounting Office
                  Washington, D.C. 20648

                  National Security and
                  Intxwnational Affairs Division

                  B-238677
                  August 27,199O

                  The Honorable Les Aspin
                  Chairman, Committee on Armed Services
                  Houseof Representatives
                  Dear Mr. Chairman:

                  As requested,we evaluated whether certain tasks were within the mis-
                  sion of the Office of the Director, Operational Test and Evaluation
                  (DUI’&E).’ Specifically, we examined the Director’s (1) role as Chairman
                  of the Test and Evaluation Committee and (2) use of early operational
                  assessments.You expressedparticular concern about whether opera-
                  tional test and evaluation (OT&E) is independent of development testing.

                  Until recently, the Director had major responsibilities chairing a com-
Resultsin Brief   mittee that planned, programmed, and budgeted for development test
                  resources.As a result, the legislatively mandated independenceand sep-
                  aration that must exist between development and operational testing
                  was at risk. While this committee was recently disestablished,the
                  Department of Defense(DOD) is considering a new structure. We believe
                  the Director should function as an adviser-not an officer or member-
                  for any future committee or council that plans, programs, and budgets
                  for development test resources.

                  Using early operational assessments,which help to determine whether
                  weapon systems are ready for operational testing, is within DOl%E’S mis-
                  sion becausethe assessmentsoffer advice to the DOD acquisition deci-
                  sionmakers. Although these assessmentsrely primarily on development
                  and not operational test data, they can be useful in filling a void when
                  actual operational test results are not available. However, these assess-
                  ments are not substitutes for actual OT&E and the basis for developing
                  the assessmentsshould be fully disclosedto avoid misunderstanding as
                  to their nature and use.


                  The Congress,concernedthat OI’&E was not receiving sufficient emphasis
Background        and independent oversight, directed the establishment of MJT&E
                  (P.L. 98-94, Sept. 24, 1983). The statute designatedthe Director as the
            Y
                  ‘In practice, DOME is used to mean both the Director and the office under the Director’s responsi-
                  bility. To avoid confusion, we refer to the Director as the Director and to the office as DOT&E.



                  Page 1                                                            GAO/NSIAD9O-128      DOD Testing
                      0288677




                      principal CT&E official within senior DOD managementto ensure that sys-
                      tems being acquired are ready for production. To ensure that DUI’&E is
                      independent, the Congresslegislated on several occasions,the organiza-
                      tional separation between development and operational testing-the
                      two primary types of DOD testing during the acquisition process.In addi-
                      tion, the Director may not be assignedany responsibility for develop-
                      ment test and evaluation except to provide advice to officials
                      responsible for such testing.

                      The roles and responsibilities of the Director are described in 10 U.S.C.
                      section 138. According to the statute, the Director is responsible for
                      prescribing policies and procedures for W&E,
                      providing guidance to and consulting with defenseofficials both gener-
                      ally and with respect to specific programs,
                      monitoring and reviewing all W&E,
                      coordinating joint (JT&E,
                      analyzing and reporting on both the adequacy of the testing and the
                      results, and
                      reviewing and making recommendationsto the Secretary of Defenseon
                      all budgetary and financial matters relating to or&~.


Development and       Development test and evaluation is done throughout the acquisition pro-
Operational Testing   cessto ensure the attainment of technical performance specifications,
                      program objectives, and weapon system supportability. Development
                      testing is normally done by the agency responsible for developing the
                      system. It usessuch techniques as modeling, simulation, prototypes, or
                      full-scale engineering development system models to determine the
                      extent that a system meets technical specifications.
                      W&E  is the field test, under realistic combat conditions, of major weapon
                      systems, equipment, components,or munitions for the purpose of deter-
                      mining the operational effectiveness and suitability2 of the weapons,
                      equipment, or munitions used in combat by typical military users. Each
                      service has an operational test agency that is responsible for CJIBE,
                      reporting test results, and providing an evaluation of the tested system’s
                      operational effectiveness and suitability.

                      20perational effectiveness means the ability of a system to accomplish its mission when placed in use
                      in the planned operational environment. Operational suitability is the degree to which a system can
                      be placed satisfactorily in field use considering, among other factors, availability, maintainability,
                      and logistica support.



                      Psge 2                                                             GAO/NEXADM-128       DOD Ted&j
                         Bma077




                         In November 1989, the Congressadded provisions concerningOJ%Ein
                         connection with the decision to go beyond low-rate initial production.3
                         The Congressstated that m&E doesnot include an operational assess-
                         ment basedexclusively on (1) computer modeling, (2) simulation, or
                         (3) an analysis of system requirements, engineering proposals, design
                         specifications, or any other information contained in program
                         documents.

                         The Deputy Secretary of Defenseappointed the Director as Chairman of
The Director’s Role in   the Test and Evaluation Committee. As Committee Chairman, the
DevelopmentTest          Director presided over the operations of the Committee, which was
Resources                responsible for planning, programming, and budgeting for development
                         test resources.As a result, the Director’s independencewas jeopardized
                         becausethe Director had influence over the types of development test
                         assetsused by the services.Responsibility for development test
                         resourcesrests with the services.We believe the Director should func-
                         tion as an adviser on any future committee or council that is given
                         responsibility for development test resources.
                         In 1987, the Congressinserted languagein nor&~‘sstatute through the
                         National DefenseAuthorization Act for Fiscal Years 1988-89,which
                         states that “the Director may not be assignedany responsibility for
                         developmental test and evaluation, other than the provision of advice to
                         officials responsible for such testing.” According to the conference
                         report on this act, the languagewas inserted to ensure that responsibili-
                         ties for operational testing are separate from functions associatedwith
                         development testing. The conferencereport further stated that the Sec-
                         retary of Defenseshould refrain from any realignment or new arrange-
                         ment of test and evaluation activities, oversight responsibilities, or
                         functions.
                         We recently issued an unclassified version of a classified report4 that
                         criticized DOT&E'Smanagementof the OT&ECapability Improvement Pro-
                         gram. In that report, we stated that m&E was performing management
                         functions by acquiring operational test resourcesused in testing major
                         weapon systems.We recommendedthat the Secretary of Defensedirect

                         3Low-rate initial production is the production of a system in the minimum quantity needed to conduct
                         UI’&E, to establish an initial production base for the system, and to permit sn orderly increase in the
                         production rate for the system sufficient to lead to full-rate production upon the successful comple
                         tion of operational testing.
                         4(GAO/NSIAD-QO-141,Mar.
                                            30,lQQO).


                         Page 8                                                             GAO/NSIAIMO-1ZS       DOD Terthg
                                                                                                    ,
                          B238877




                          and assurethat the servicesplan, program, and budget for adequatetest
                          resourcesneededto conduct operational testing of weapon systems
                          effectively. We explained that the Director should continue to perform
                          oversight and policy functions by reviewing and making recommenda-
                          tions to the Secretary of Defenseto assurethat adequate test resources
                          are acquired.


The Director Influenced   The Test and Evaluation Committee was established to support the
the Committee in          Under Secretary of Defensefor Acquisition, who, by law, is responsible
                          for establishing policies for acquiring DOD’S weapon systems and the
Development Test          development testing of such systems. (See10 U.S.C.133(b)(2).) The
ResourceMatters           Committee was to provide a forum for key representatives from the
                          Office of the Secretary of Defense,the services,and other agenciesto
                          identify and resolve issuesand to develop recommendationson the DOD
                          acquisition system in the test and evaluation area.

                          The Committee’s charter specified that the Chairman had major roles
                          and responsibilities regarding the operations of the Committee. The
                          Committee Chair was to, among other things, direct and supervise oper-
                          ations of the Committee, schedule and preside at meetings, select items
                          submitted by the Committee membersto be addressedby the Com-
                          mittee, and establish panels to carry out Committee assignedprojects.
                          The Committee attempted to obtain a consensusamong all members
                          regarding matters brought before the Committee. As such, the Director
                          participated in the decision-making processby concurring or not concur-
                          ring on these matters, thus having influence over the types of test assets
                          used by the services.
                          As Committee Chairman, the Director initially provided advice and
                          presented recommendationson various test and evaluation issuesto the
                          Deputy Secretary of Defensefor a decision. In 1986, the issuesincluded
                          such things as a proposed recommendation to improve spacesystem test
                          capabilities. In 1988, the Director recommendedestablishing the Central
                          Test and Evaluation Investment Program, which received $83 million in
                          fiscal year 1990 for the purpose of acquiring primarily development test
                          resourcesfor new weapon systems. The Director believed that the lack
                          of development and operational test investments by the serviceswas
                          reaching crisis proportions.
                          Instead of continuing to serve in an advisory capacity by presenting rec-
                          ommendations on test resourcesto the Deputy Secretary of Defense,the



                          Page 4                                          GAO/NSL4D90-128   DOD Testing
                       B-288877




                       Director becamea manager of these resourcesby taking on responsi-
                       bility for the program. In November 1988, the Deputy Secretary of
                       Defensecreated the program and directed that it be managedby the
                       Committee. On November 30,1988, managementof the program was
                       established and the Committee Chairman assumedresponsibility and
                       accountability for the managementand effective allocation of the pro-
                       gram funds.6Managementof test resourcesmay dilute DOT&E’S oversight
                       function becauseit gives DW&E a direct influence over the types of test
                       assetsacquired and used by the services.For example, MJT&E may be
                       reluctant to criticize a test plan that contains inadequate testing
                       resourcesif that office was responsible for managing or played a part in
                       the acquisition of such resources.
                       Oncethe Committee approved the projects to be funded under the pro-
                       gram, the Committee’s resourcepanel established the parameters for the
                       projects that were to be executed by the lead servicesor agency. In turn,
                       the lead service or agency developed a project managementplan that
                       specifically outlined how the project would be executed. Thesetwo
                       efforts established the “contract” for the project between the lead agent
                       and the Committee.

                       The projects were then executed by the lead service or agency.The
                       Committee established program corporate investment priorities with the
                       goal of preventing unnecessaryduplication, encouraging multiservice
                       use of equipment, and providing critically neededtest capabilities. Pri-
                       orities were to be determined in all phasesof program management
                       (planning, programming, budgeting, and budget execution) by the Com-
                       mittee. As a r&%ilt; the Direstgr, by Msuming maJorrespansibilitiee fsr
                       the Committee that manageddevelopment test resource funding, placed
                       his independenceat risk.


The Committee May Be   DOD’s  responseto a draft of this report stated that the Test and Evalua-
Replacedby a Council   tion Committee under the DefenseAcquisition Board has been disestab-
                       lished. However, we were told that a DOD Test and Evaluation Resource
                       Council may be created and may be co-chaired by the Director. At pre-
                       sent, it is unclear how such a council will operate. If the council is estab-
                       lished to plan, program, and budget for development test resourcesand
                       the Director servesas a co-chairman or even as a council member, we
                       “The funding for the program is found in the appropriation for the Deputy Director, Defense
                       Research and Engineering (Test and Evaluation), Office of the Secretary of Defense, who is respon-
                       sible for development testing. Fiscal oversight, distinct from the Committee’s responsibility for man-
                       agement and corporate priority setting, is the job of the Deputy Director.



                       Page 6                                                             GAO/NSIAD-M-128       DOD Testing
                            B-238677




                            believe the Director would continue to exceedthe role of providing
                            advice to officials responsible for development test matters.
                            In the past, the Test and Evaluation Committee membership consistedof
                            the Committee chair, vice chair, members, and invited participants/
                            advisers. The Committee Chair and memberswere to reach a consensus
                            on various issuesbrought before the Committee. The invited partici-
                            pants and advisers were to provide information on specific matters. In
                            keeping with m&E’s responsibilities for UI%E oversight as addressedin
                            its statute, it would be more appropriate for the Director to be an
                            adviser on any future test resourcescouncil that is created. As such, the
                            Director should continue to perform oversight and policy functions by
                            reviewing test plans and results and making recommendationsto the
                            Secretary of Defenseso that test resource needsare given appropriate
                            consideration.

                            Our review showed that m&~‘s use of early operational assessments,
Useof Early                 which DGD began using in 1988, are not prohibited by m&E’s statute.
Operational                 They allow DOT&E to offer information to DOD acquisition decisionmakers
AssessmentsIs Within        as to whether weapon systems are ready for m&E and whether opera-
                            tional shortfalls are being identified and corrected early in the acquisi-
m&E’s Mission               tion process.These assessments,which are basedprimarily on
                            development rather than OR&E data, are being done during the early
                            phasesof the acquisition processwhen actual (F&E results are not avail-
                            able. However, as noted in a previous report,Bthey are not substitutes
                            for m&E and the basis for the assessmentsshould be fully disclosed
                            when they are reported to congressionaland DOD decisionmakers.
                            According to the Director, an independent evaluation is made of the
                            operational assessmentsdevelopedby the operational test agencies.This
                            evaluation is provided as advice to the Under Secretary of Defensefor
                            Acquisition through the DefenseAcquisition Board process,which is
                            used to overseemajor system acquisitions.


The Director’s Past         In the past, the Director has attempted to redefine the way test and
Attempts to Redefine Test   evaluation is viewed within DOD. Until the Director began advocating the
                            use of early operational assessments,initial m&E was that portion of
and Evaluationu             actual OUCE done throughout the acquisition processbefore the decision

                            (‘Navy Weapons Testing: Defense Policy on Early Operational Testing (GAO/NSIAD89-98,
                            May 8,19f.W.



                            Page 6                                                        GAO/NSLW4tO-128      DOD Testing
to proceed to production. It was accomplishedusing a prototype,
preproduction article, or a low-rate initial production article as the test
item. The “final exam,” or the latter phase of initial ONE usually
entailed dedicated operational testing of production representative test
articles using typical operational personnel in as realistic a combat envi-
ronment as possible.

In a report to the HouseArmed ServicesCommittee dated September 25,
1987, the Secretary of Defenseproposed to redefine initial UI%E as that
component of testing initiated at program inception, to forecast opera-
tional effectiveness and suitability. It was to be a tool to provide insights
about the potential operational worth of a system throughout its acqui-
sition life. As opposedto doing actual UME, initial W&E would take
advantage of any test results and could use modeling, simulation, and
paper analysesto make assessments7
DOD’s current policy requires early and progressive assessmentsof oper-
ational capability, including realistic operational testing before full-scale
production starts. It is the current DOD perspective that an operational
test agency should evaluate all pertinent information as it becomes
available, regardless of the source,for input into early periodic opera-
tional assessments.

On January 26, 1990, DOD issued guidelines that prescribe uniform pro-
ceduresfor preparing and approving Test and Evaluation Master Plans.
The plans describe, among other things, the development and opera-
tional testing to be performed on a weapon system throughout the
acquisition process.Theseguidelines state that the plans will show how
operational assessmentsand testing are structured at each acquisition
decision point. Further, they show how operational testing will, or has,
evaluated the system in an environment as operationally realistic as
possible. If W&E cannot be done or completed early in the development
process,then this is to be clearly stated and the reason(s)explained in
the plans.

The guidelines state that operational assessmentsare done before or in
support of the full-scale engineering development phase. They are used
to identify significant trends noted in development efforts, program-
matic voids, areas of risk, adequacy of requirements, and the ability of
the program to support adequate operational testing. The assessments

‘According to a JWI’&E official, this new definition for initial UlXkE was synonymous with an early
operational assessment.



Page 7                                                            GAO/NSIAD-90-128      DOD Testing
                           B!ma677




                           are to be basedon all information relevant to the program, and can be
                           made at anytime. However, they are not to be consideredsubstitutes for
                           Ul%E in support of full-production decisions.



Early Operational          Performing early operational assessmentswhen production representa-
AssessmentsCan Be          tive test articles are not available is a step forward in filling a void in
                           the availability of actual (T&E results. Furthermore, the assessmentsare
Useful                     consideredadvisory in nature and can be useful to decisionmakers.For
                           example, the C-17A aircraft was the first major acquisition program
                           that relied on an early operational assessmentas a decision-making
                           tools In December1987, the Deputy Secretary of Defenserequired the
                           Air Force to submit an assessmentof the C-17A regarding (1) meeting
                           low-rate initial production criteria and (2) progresstowards initial oper-
                           ational capability.
                           In September 1988, the Air Force Operational Test and Evaluation
                           Center completed the C-17A assessmentto support a low-rate initial
                           production decision consisting of 40 C-17A aircraft. Sinceno production
                           representative systems/subsystemswere available, no operational tests
                           were done. The center’s assessmentwas basedon all available informa-
                           tion sourcesto addresssuch areas as attainment of the production cri-
                           teria listed in the C-17A’s Test and Evaluation Master Plan. These
                           sourcesincluded, but were not limited to, results of critical design
                           reviews, development test results, demonstrations, the center’s partici-
                           pation in planning committees, program documents, and interviews.

                           In January 1989, the Deputy Secretary of Defenseapproved low-rate
                           initial production of the C-17A aircraft, which reduced the requested
                           procurement of 40 aircraft to 10 aircraft. BecauseC-17A aircraft were
                           not available for UI%E, the early operational assessmentwas beneficial
                           in that it provided timely insight into potential problems for the C-17A
                           aircraft basedon essentially development test data.


Our Earlier Report Dealt   In our 1989 report to the Chairman, Subcommitteeon Seapowerand
With Disclosure of Early   Strategic and Critical Materials, HouseCommittee on Armed Services,
                           we evaluated the Navy’s o-r&~ done before decisionswere made on the
Operational Assessments    full-scale development or low-rate initial production of weapon systems.
             ”             During fiscal years 1986 to 1988, the Navy typically approved weapon

                           sDOD is just beginning to use assessmentsin the acquisition process. As a result, there are very few
                           examples of early operational assessments.



                           Page 8
                     B-288677




                     systems for full-scale development and, in many cases,for low-rate ini-
                     tial production before any (JT&E was completed. However, the Navy did
                     do a limited number of operational assessmentsthat supported the early
                     milestone decisions.
                     Basedon our evaluation of DOD’S testing policy and the Navy’s use of
                     operational assessments,we recommendedin our 1989 report that the
                     Secretary of Defenseclarify when it is appropriate for decisionmakers
                     to rely on operational assessmentsthat do not include the operational
                     testing of any hardware and when actual W&E must occur. In addressing
                     this issue, the Secretary was to assurethe independenceof the services’
                     m&E agencieswas not compromised and that the basis of the weapon
                     system assessmentswere fully disclosedwhen the assessmentswere
                     reported to congressionaland DOD decisionmakers.
                     In responseto our report, DOD stated that it agreed with our recommen-
                     dations. DOD further stated that the policy as to when it is appropriate
                     for decisionmakers to rely on operational assessmentswould be included
                     in the publication of DOD Manual 5000.3-M-5,volume 3. Due to changes
                     in DOD’S testing policy being considered as part of DOD’S DefenseManage-
                     ment Review, many directives and regulations are being consolidated,
                     revised, or terminated. For example, DOD plans to publish a new Direc-
                     tive 6000.1~addressingthe acquisition processas well as a manual
                     addressing acquisition documentation and reporting procedures.As a
                     result, test and evaluation policies and procedures will not be published
                     until sometime in the future.

                     We recommendthat the Secretary of Defensekeep development and
Recommendation       operational testing independent and separate by ensuring that the
                     Director provides advice only to those committees or councils that are
                     responsible for the planning, programming, and budgeting of develop-
                     ment test resources.The Director should perform oversight and policy
                     functions by reviewing test plans and results and making recommenda-
                     tions to the Secretary of Defenseso that test resourceneedsare given
                     appropriate consideration.

                         agreed that development and operational testing should be kept
Agency Commentsand   DOD
                     independent and separate. However, DOD disagreedthat DCJI%E had man-
Our Evaluation       aged development test resources.DOD stated that neither the Test and
                     ‘In the past, DOD Directive 6000.1 addressed the subject of major weapon system acquisitions.



                     Page 9                                                           GAO/NSIAD99.128       DOD Testing
B-238677




Evaluation Committee nor the chairman ever manageddevelopment test
resourcesor any program such as the Central Test and Evaluation
Investment Program that involves such resources.As such, DOD has
stated it was already in compliance with our recommendation and that
the Director does not now function as chairman of any committee or any
other organization responsible for managing test resourcesnor is there
any intent to assign such responsibilities to the Director.
Although we recognizethat test and evaluation resource management
structures are presently undergoing changes,the Director had major
responsibilities chairing a committee that planned, programmed, and
budgeted for development test resources.As Committee Chairman, the
Director exercised major responsibilities over the operations of the Com-
mittee. Further, the Committee was responsible for planning, program-
ming, and budgeting for the Central Test and Evaluation Investment
Program. Although we did not identify examples where the Director had
unduly influenced the services,we believe his responsibilities created
the perception that the Director’s independencefrom development test
matters was jeopardized by allowing for a direct influence over the test
assetsused by the services.

DOD told us that a corporate mechanism has been established to focus
attention on the need for test resources.DOD believes that this attention
has forced the servicesto work together to identify requirements, elimi-
nate duplication, and obtain the necessaryfunds to support the
requirements.
Basedon DOD’S comments,we have modified our recommendation to
better emphasizethe actions we believe are needed.(DOD’S detailed com-
ments and our evaluation of these comments are presented in app. II.)

We are sending copies of this report to the Secretariesof Defense,the
Navy, Army, and Air Force and to interested parties. Copieswill also be
made available to others on request.




Page 10                                         GAO/NSIAD-90-128   DOD Testing
Pleasecontact me at (202) 2’758400 if you or your staff have any ques-
tions concerning this report. Major contributors to this report are listed
in appendix III.

Sincerely yours,




Paul F. Math
Director, Research,Development,
  Acquisition, and Procurement Issues




Page 11                                         GAO/NSIAD-S@128   DOD Testing
Contents


Letter
Appendix I
Scopeand
Methodology
Appendix II
CommentsFrom the        GAO Comments
Department of
Defense
Appendix III                                                                                       24
Major Contributors to
This Report




                        Abbreviations

                        DOD       Department of Defense
                        DOT&E     Office of the Director, Operational Test And Evaluation
                        W&E       operational test and evaluation


                        Page 12                                        GAO/NSIAD-90-128   DOD Testing
Page 13   GAO/NSIAD-90-128   DOD Testing
Appendix I

Scopeand Methodology


             To assessthe Director’s role as Chairman of the Test and Evaluation
             Committee, we reviewed the scopeof DUNE'S legislation as it pertained
             to development test matters. In particular, we focused on a legislative
             provision that stated that the Director was not to be assignedany
             responsibility for development test and evaluation, other than the provi-
             sion of advice to officials responsible for such testing.
             We interviewed Office of the Secretary of Defenseofficials and reviewed
             documents pertaining to the Test and Evaluation Committee. We
             reviewed the Director’s involvement in test resourcesponsorship as the
             Committee Chairman and in the normal role as DCW.&E. In particular, we
             interviewed Office of the Secretary of Defenseofficials and reviewed
             documents pertaining to the Central Test and Evaluation Investment
             Program becausethe program is used to acquire primarily development
             test resources.
             To assessthe Director’s use of early operational assessments,we evalu-
             ated the scopeof DtYIXE'S legislation as it pertained to development test
             matters and whether that office was responsible for development test
             and evaluation, other than the provision of advice to officials respon-
             sible for such testing.

             Sincethe early operational assessmentis an evolving concept, only lim-
             ited information is available on it. Primarily, we reviewed documents
             that set out DCYWE'S policy on early operational assessments.We also
             interviewed service officials from the Navy Commander,Operational
             Test and Evaluation Force, Norfolk, Virginia; Air Force Operational Test
             and Evaluation Center, Albuquerque, New Mexico; and the Army Opera-
             tional Test and Evaluation Agency, Alexandria, Virginia, to obtain their
             views on early operational assessments.
             Our review was performed in accordancewith generally acceptedgov-
             ernment auditing standards between October 1988 and January 1990.




             Page 14                                         GAO/NSIAD-90-128   DOD Testing
A$&ndix      II

Comments From the Department of Defense


Note: GAO comments
supplementing those in the
report text appear at the
end of this appendix.                                     OFFICE OF THE SECRETARY OF DEFENSE
                                                                 WASHINGTON, DC 20301-1700

                                                                            6 JUN I:%

                             OPERATlONAL
                               lE8’l AND
                              EVALUATION


                                   Mr. Frank C. Conahan
                                   Assistant    Comptroller    General
                                   National    Security    and International
                                     Affairs    Division
                                   U.S. General Accounting        Office
                                   Washington,     DC 20540
                                   Dear Mr.         Conahan:
                                          This    is the Department             of Defense     (DOD) response      to the
                                   General      Accounting       Office       (GAO) draft     report,       "DOD TESTING:
                                   Questionable       Tasks Performed by the Director,             Operational   Test and
                                   Evaluation,"       dated April       13, 1990 (GAO Code 396224/OSD
                                   Case 8303).         The Department         agrees with the recommendation,            but
                                   does not        agree    with      the     GAO assertion       that     the  Director,
                                   Operational      Test and Evaluation          managed development test resources.
See comment 1.                     It is the DOD position                that     neither  the Test and Evaluat.ion
                                   Committee nor the Chairman has ever managed the Central                       Test and
                                   Evaluation      Investment     Program, as asserted          in the report.
                                          The Department    concurs and is already      in compliance    with the
                                   recommendation     that "development     and operational      testing  be kept
                                   independent    and separate."       The Director,      Operational    Test and
See comment 2.                     Evaluation   is not now assigned as chairman of any committee or any
                                   other organization      responsible   for management of test resources,
                                   nor is there intent      to assign such responsibilities.
                                         The detailed      DOD comments on the       report    findings  and
                                   recommendation     are provided   in the enclosure.       The Department
                                   appreciates    the opportunity  to comment on this       ft report.
                                                                                        Y-7


                                                                                             C. Duncan

                                   Enclosure




                                       Page    16                                                 GAO/NSIAD-90-128   DOD Testing
                        Appendix   II
                        CommentaFkomtheDepartmentofDefense




                                   GAO DRAFT REPORT - DATED APRIL 13, 1990
                                       GAO CODE 396224 - OSD Case 8303
                    "DOD TESTING:        QUESTIONABLE TASKS PERFORMEDBY THE DIRECTOR,
                                         OPERATIONAL TEST AND EVALUATION"
                                          DEPARTMENT OF DEFENSE COMMJ5NTS
                                                           * * * * *
                                                            FINDINGS
                0      FINDING A: v:                            mce        of #J&Q&Director.         ODmtiona
                       3-t       and ~=mia,um          .     The GAO observed            that the Congress,
                       concerned        that      operational          test      and evaluation           was not
                       receiving        sufficient          emphasis        and independent            oversight,
                       established         the Director,            Operational        Test and Evaluation
                        (P.L. 98-94, September 24, 1983).                       According      to the GAO, the
                       statute      designated       the Director          as the principal          operational
                       test and evaluation             official       within      senior DOD management to
                       ensure that systems being acquired                       are ready for production.
                       The GAO pointed           out that,       to ensure the independence                 of the
                       Director,         Operational           Test      and Evaluation,             on several
                       occasions,          the     Congress          legislated         the      organizational
                       separation       between development               and operational         testing--
                       the two primary           types of DOD testing               during     the acquisition
                       process.       The GAO observed that the Director,                     Operational       Test
                       and Evaluation,           may not be assigned                any responsibility            for
                       development        test and evaluation--              except to provide          advice to
Nowon pp. l-2          officials       responsible         for such testing.               (pp. 2-4/GAO Draft
                       Report)
                       goD   RESPOND:         Concur.

                0      FIINDING:       These                                      Test  Resource*    .
                       The GAO found that the Director,          as Chairman of the Test and
                       Evaluation    Committee,     had recently       become involved       in the
                       management of development       test resources.         The GAO explained
                       that managing development        test resources       goes beyond merely
                       providing   advice to officials      responsible    for   development    test
                       matters and puts at risk the independence           of the Director      from
                       development   test matters because it gives the Director             control
                       and influence     over the types of development          test assets used
                       by the Services.
                       The GAO pointed       out that   in 1987, the Congress       inserted
                       language in the National     Defense Authorization    Act for Fiscal
                       years    1988-89,   relating   to the Office       of the Director,
                       Operational    Test and Evaluation,   which states     "The Director
                                                                                                      Enclosure




                        Page16                                                              GAO/NSIAD-!bO-128DODTesting
                           Appendix II
                           CemmentsFre~the Department        0fDefenre




                     may    not be assigned any responsibility                   for   developmental       test
                     and evaluation,             other      than      the provision          of    advice     to
                     officials       responsible         for such testing."              According      to the
                     GAO,      the conference          report      on this       Act had the language
                     inserted       to ensure        that       responsibilities          for operational
                     testing        are      separate         from      functiona        associated        with
                     development       testing.       The GAO noted that the conference                 report
                     further      stated     that the Secretary             of Defense should refrain
                     from any realignment           or new arrangement            of test and evaluation
                     activities,        oversight       responsibilities,            or functions,
                     The GAO further        referred   to a recently      issued GAO report,          w
                                                               t of DOD'S Owerational            Test a
                                                 tv mnt               Prm            dated March 20,
                     1990 (OSD Case 8137), which criticized                  the'management         of the
                     Operational        Test     and Evaluation          Capability         Improvement
                     Program.       The GAO concluded          that the Director,          Operational
                     Teat and Evaluation,           was performing       management functions            by
                     acquiring     test resources       used in testing       major weapon systems.
                     The GAO asserted        that management of test resources              dilutes    the
                     Director's      oversight     function     because it gives the Director,
                     Operational      Test and Evaluation,          a direct     influence       over the
                     types of teat assets acquired              and wed by the Services.               The
                     GAO concluded that the Director              may be reluctant        to criticize
                     a test plan that contains            inadequate    testing      resources      if his
                     office     was responsible       for managing or played a part in the
Nowon pp.34          acquisition      of such resources.           (PP. l-2,      PP. d-S/GAO Draft
                     Report)

                     -RESPONSE:             Nonconcur.       The Department          acknowledges       that
                     there      were some statements              in the early             correspondence
                     creating       the Central      Test and Evaluation             Investment      Program
                     that      assigned      management responsibility                to the Test and
                     Evaluation       Committee (hereafter          referred       to as the Committee)
                     and the Chairman of the Committee                    (hereafter      referred      to as
See comment 1.       the Chairman).            In practice,      however,        the Chairman did not
                     manage. As specified            in the Committee charter,               the Committee
                     provided        a forum and catalyst            for the review            of DOD test
                     resource       matters     and recommended alternatives.                    It did not
                     have management authority              under the charter.              Moreover,      the
                     Chairman did not have decision                   authority;        consequently,        he
                     could not manage teat re8ources.                    The Chairman's         role was to
                     facilitate        the review process          and cause agreement to occur
                     where that was possible.               In that capacity,            for all members
                     of the Committee,             whether    they were judged to be of the
                     development         or operational       test      communities,        the Director,
                     Operational        Test and Evaluation        or his representative            provided
                     "advice      to officials      responsible        for such testing."




                 Y




                           Page17
                                                                                                 i

     Appendix II
     Commenti From the Department      of Defense




                                                 .
0   EIINDING:          TheisMsnaainaDevelorrraent
    vResourcea.           The GAO observed that the Test and Evaluation
    Committee was established           to support    the Under Secretary         of
    Defense     for Acquisition        who, by law,       is responsible        for
    acquiring     DOD weapon systems and the development             testing      of
    such systems.       The GAO noted that the Committee was to provide
    a forum      for   key representatives         from the Office         of the
    Secretary      of Defense,      the Services,      and other   agencies       to
    identify    and resolve     issues and to develop recommemdations on
    the DOD acquisition         system in the test and evaluation            area.
    They learned      that the Deputy Secretary          of Defense appointed
    the Director,      Operational     Test and Evaluation     as the Chairman
    of the Committee.
    The GAO explained               that,      as the Chairman,            the Director
    initially       provided       advice     and presented        recommendations         on
    various     test and evaluation            issues to the Deputy Secretary              of
    Defense for a decision.                   The GAO found,          however,     that    in
    1986,      the     issues       included        such   things        as a proposed
    recommendation          to improve space system test capabilities.                    The
    GAO further         found that,        in    1988, the Director           recommended
    establishing          the Central          Test    and Evaluation           Investment
    Program, which received              $83 million     in FY 1990 for the purpose
    of acquiring        primarily       development      test resources         for new
    weapon systems.               The GAO acknowledged             that     the Director
    believed      that the lack of development                  and operational         test
    investments       by the Services           was reaching      crisis     proportions.
    The GAO concluded     that,     instead    of continuing      to serve in a
    advisory    capacity    by presenting          recommendations         on test
    resources   to the Deputy Secretary           of Defense,     the Director,
    Operational    Test and Evaluation,          became a manager of those
    resources   by taking on the responsibility           for the program. The
    GAO emphasized that,        in November 1988, the Deputy Secretary
    of Defense created the program and directed              that it be managed
    by the Committee.        The GAO concluded,          therefore,       that the
    Committee     Chairman       became     ultimately        responsible       and
    accountable   for the management and funding of development test
    resources.
    While      agreeing      the program          is executed            by the Military
    Services,       the GAO further         concluded        that the Committee sets
    program       corporate      investment      'priorities            with  the goal of
    preventing       unnecessary      duplication,         encouragingmulti-Service
    use of equipment,              and providing             critically        needed test
    capabilities.         The GAO observed that the priorities                     are to be
    determined         in all      phases      of program             management by the
    Committee;        therefore,       the Director,              Operational      Test and
    Evaluation,       by controlling        development          test resource funding,




     Page 18                                                         GAO/NSIAMO-128       DOD Testing
                      Appendix II
                      Comments From the Department     of Defense




                     ha8     exceeded       the       mission         of   providing   advice          to
Now on pp, 4-5       decisionmakers.         (pp.    l-2, pp.       6-7/GAO Draft Report)

See comment 3.       PoD:                  Partially       concur.        The majority      of the teat
                     resources     that will        be acquired       under the Central          Test and
                     Evaluation       Investment        Program will        directly     support      both
                     operational       and developmental         testing.       Those teat resources
                     that do not directly           support operational          testing  will    provide
                     data that will        be used to support            operational     effectiveness
                     and suitability           assessments         or they         will   be used as
                     evaluation/analysis           tools.
See comment 1.       The Department acknowledges         that a Deputy Secretary       Memorandum
                     did direct     the Committee        to manage the Central            Teat and
                     Evaluation   Investment      Program and that a subsequent Committee
                     Memorandum stated       that    the "...Chairman         . . . is ultimately
                     responsible    and accountable        for the management and effective
                     allocation   of [Central      Teat and Evaluation      Investment     Program]
                     funds."     However,      in recognition        of the concerns         of the
                     Congress, in actual practice,           the Director   nevel: exercised      his
                     Department mandate to "manage" the Central Test and Evaluation
                     Investment   Program.       As defined     above, the Director      has m
                     manacred the Central       Test and Evaluation       Investment    Program.
See comment 4.       Rather     than managing,          as Chairman     of the Committee,        the
                     Director     ensured a systematic         process for review of all teat
                     resource     matters.        In fact,    in recommending the management
                     approach      for   the Central        Teat and Evaluation         Investment
                     Program, the Director           proposed that the funding       be placed in
                     the Deputy Director        Defense Research and Engineering          (Teat and
                     Evaluation]       appropriation,       recognizing    that  office      as the
                     Department test resource            manager.

                 0                                            ,* F t          ia Uncertain .      The GAO
                     FINDING:           The
                     explained      that the continued            need for the Committee is being
                     questioned       as a part of the DOD Defense Management Review.
                     According      to the GAO, the Committee functions                     may be placed
                     under      a council       that      could      possibly      be chaired      by the
                     Director,      Operational        Test and Evaluation.            The GAO observed
                     that     it is currently          unclear      whether      such a council      would
                     continue      to manage development                test     resources.       The GAO
                     concluded,        however,      that      if the council         does manage such
                     resources      and the Director            is the council       Chairman, then the
                     Director,      Operational        Test and Evaluation,           would continue     to
                     be      in    the      position          of     managing        development      test
                     resources--thus          still       going     beyond      the mission      of only
                     providing     advice to officials            responsible      for development test
Now on pp. 5-6       matters.        (pp. 1-2, p. S/GAO Draft Report)




                      Page 19                                                      GAO/NSIAD-99-128    DOD Testing
                    Appendlrn
                    C4munent8RomtheDepnrtmentofDefenoe




                   -RESPONSE:             Partially       Concur.      The Test and Evaluation
                   Committee       under     the Defense        Acquisition        Board has been
                   disestablished.         However, the DOD Test and Evaluation             Activity
                   Consolidation        Study,      conducted       as a part      of the Defense
Seecomment2.       Management Review,           does propose the establishment              of a DOD
                   Test and Evaluation          Resource Council,         independently    chartered
                   under     the Deputy         Secretary     of Defense.            If the Deputy
                   Secretary      approves the Council,           it will     be charteqed    in such
                   a way that         there      is neither       the perception        or   fact    of
                   organizational        bias.

               0   EUlR+LG:         Pne of lSaUv         OPerational                        1s Wi&inAhsi
                             aa            .        The GAO found that                the use of early
                   operational         assessments,          which the DOD began using in 1988,
                   are not prohibited               by the Director,             Operational        Test and
                   Evaluation        statute.         The GAO explained            that the assessments
                   allow the Director              to offer        information       to DOD acquisition
                   decisionmakers            as to whether           weapon systems          are ready for
                   operational          teat      and evaluation            and whether          operational
                   shortfalls       are being identified               and corrected         early in
                   the acquisition             process.          The GAO pointed           out that      these
                   aaseaamenta,          which are baaed primarily               on development         rather
                   than operational             teat      and evaluation         data,      are being done
                   during     the early          phases of the acquisition                    process,     when
                   actual      operational           teat       and evaluation           results     are not
                   available.           The GAO emphasized,              however,      that they are not
                   substitutes         for operational           test and evaluation           and the basis
                   for the assessments               should be fully          disclosed       when they are
                   reported     to congressional              and DOD decisionmakers,             as noted in
                   its prior       report,        NAw                   TESTINC .        efanse Pow
                                                            , '* datedMay     8, 1989 (OSD Case 7800).
                           According       to the GAO, the Director          stated     that  an
                   independent       evaluation    is made of the operational       asaesaments
                   developed      by the operational       teat agencies.       The GAO noted
                   that     this    evaluation     is provided     as advice    to the Under
                   Secretary       of Defense      for Acquisition     through    the Defense
                   Acquisition       Board process that is used to oversee        major system
Nowon p-6          acquisitions.          (pp. l-2, pp. 0-14/GAO Draft Report)

                   I)oD:                  Concur.




                    Page20                                                            GAO/NSIAD@O.128      DODTesting
                                 Appendix Jl
                                 Commenta From the Department of Defense




                                                                 *****

                                                              RECOMMJZNDATION

                            0   -:                   The GAO recommended that the Secretary
                                of Defense assure that development        and operational     testing    be
                                kept independent     and separate.     (The GAO emphasized that the
                                Secretary    should ensure that the Director        does not chair or
                                manage committees      or programs that involve       development     test
                                resourcea   that can compromise the independence          of development
Now on p. 9                     and operational    testing.)     (p. lS/GAO Draft Report)

See comments 1 and 2.           -RESPONSE!             Concur.   The Department    is already     complying
                                with this      recommendation      because   (1) the Director      does not
                                manage any test resources         and (2) there is no intent      to assign
                                any such responsibilities         to him. The Secretary       has found the
                                Director’s      counsel on Operational      Test and Evaluation-related
                                budgetary     and financial     matters to be invaluable        and intends
                                to     ensure     that    such   advice    remains    available      through
                                established      Department    corporate   and functional      mechanisms.




                        Y




                                 Page 21                                               GAO/NSIAB9O-128   DOD Tasting
               Appendix II
               Canments Prom the Department   of Defeme




               The following are GAO'S comments on DOD'S letter dated June 6, 1990.


               1. As explained in the report, the Director had major responsibilities
GAO Comments   chairing a Committee that planned, programmed, and budgeted for
               development test resources.Instead of continuing to serve in an advi-
               sory capacity by presenting recommendationson test resources,the
               Director took on responsibilities for managing these resources.This type
               of participation placed at risk the Director’s independencefrom develop-
               ment test matters becauseit gave the Director influence over the types
               of test assetsacquired and used by the services.
               We modified our report to discussin greater detail how we believe the
               Director influenced the Committee in development test resources.We
               explain that the Chairman had major roles and responsibilities
               regarding the operations of the Committee. (Seep. 4.) In addition, the
               Committee’s resource panel served as the working arm for the Com-
               mittee. For example, the resourcepanel developed a “contract” between
               the Committee and lead service or agency on how specific projects
               would be executed. (Seep. 6.)

               2. While we recognizethat the Committee has been disestablished, we
               have included information in the report about it becausethe Committee
               offers a historical perspective on how such a council could function. For
               example, the Committee’s charter provides a framework as to how the
               DOD Test and Evaluation ResourceCouncil’s charter could be structured.
               By reviewing the Committee’s charter, we found that the Chairman had
               major responsibilities dealing with the operations of the Committee that
               would not be appropriate. As such, the Director had influence over the
               types of test assetsused by the services.(Seep. 4.)
               DOD states that the Council will be chartered in such a way that there is
               neither the perception or fact of organizational bias. In DOD'S official oral
               comments,we were told that the Council could possibly be co-chaired by
               the Director. We have modified the report to show the past membership
               of the Test and Evaluation Committee. (Seep. 6.) By reviewing the
               charter of the Committee, we found that the membership consistedof
               the Committee chair, vice chair, members,and invited participants/
               advisers. The invited participants and advisers were to participate
               baaedon specific matters to be addressed.In keeping with the Director’s
               responsibilities for m&E oversight as addressedin its statute, we believe
               the Director should more appropriately be an adviser on any future
               council that may be created.


               Page 22                                           GAO/Nf3IAD9O-128   DODTesting
Appendix II
Chmunenta From the Department   of Defense




3. We believe that the Director should not have provided more than
advice to officials responsible for development testing and continue to
believe that the Director should not exceedproviding advice by
becoming involved with development test resourcesor any combination
of development and or&~ test resources.In addition, we recently issued a
report that criticized D0ME’S managementof VII&E resources.We recom-
mended that the Secretary of Defensedirect and assurethat the services
plan, program, and budget for adequate test resourcesneededto con-
duct operational testing of weapon systems effectively. In April 1990,
the Director emphasizedto the servicesthe importance of their plan-
ning, budgeting, and programming for adequatetest resources.
4. We agreethat the Director proposed that the funding for the program
be placed in the Deputy Director, DefenseResearchand Engineering
(Test and Evaluation) appropriation. However, the Deputy Director was
only to be the fiscal agent for the funds. The Committee, on the other
hand, was responsible for managementand corporate priority setting.
Priorities were to be determined in all phasesof program management
(planning, programming, budgeting, and budget execution). In effect, we
believe this constituted control over development test resource funds.




Page 23                                       GAO/NSIAD-30428   DOD Testing
Appendix III

Major Contributors to This &port


                        Michael E. Motley, Associate Director
National Security and   Lester C. Farrington, Assistant Director
International Affairs   Charles D. Groves,Evaluator-in-Charge
Division, Washington
DC.

Office of General
Counsel




(296224)                Page 24                                    GAO/NSIAD-90-128   DOD Testing
‘I’hth first five copitbs of that-h GAO rc”port art& frt*e. Additional copies
tire $2 tech. Orders should be sent to 01th following address, accom-
pauitAt1 by a check or mont*y order made oul. t,o t,he Superiuteudeut
of I)ocumtmts, when ntxessary. Orders for 100 or more copies tm be
mailed I,0 a single address are discounted 25 ptmeut.

IJ.S. <&neral Acrcouut.ing Office
I’.(). Hox 6015
Gait.htmburg,   MD 20877

Ordrrs   may also be placed by calling    (202) 275-6241.
I
    -.___. _ _.__   _I” _- ._.__   ..__ “..-... ___” -.-.... -I _-.. I.. .-_. I. - _.._-.._. --   .-._ - -...- -..-I.-   -. ..- ._ - ._




                                                                                                                                          ;
    --1----