TEST AND EVALUATION The Director, Operational Test and Evaluation’s Role in Test Resources 3b IIlllllll 142081 __-.-- GAO/NSIAI)-90-I 28 -- .- -_--1-- ..“-... -. _--....-.._ l __- __.___ --_- __-______ -- .-__.,, I._ - ;; United States GAO General Accounting Office Washington, D.C. 20648 National Security and Intxwnational Affairs Division B-238677 August 27,199O The Honorable Les Aspin Chairman, Committee on Armed Services Houseof Representatives Dear Mr. Chairman: As requested,we evaluated whether certain tasks were within the mis- sion of the Office of the Director, Operational Test and Evaluation (DUI’&E).’ Specifically, we examined the Director’s (1) role as Chairman of the Test and Evaluation Committee and (2) use of early operational assessments.You expressedparticular concern about whether opera- tional test and evaluation (OT&E) is independent of development testing. Until recently, the Director had major responsibilities chairing a com- Resultsin Brief mittee that planned, programmed, and budgeted for development test resources.As a result, the legislatively mandated independenceand sep- aration that must exist between development and operational testing was at risk. While this committee was recently disestablished,the Department of Defense(DOD) is considering a new structure. We believe the Director should function as an adviser-not an officer or member- for any future committee or council that plans, programs, and budgets for development test resources. Using early operational assessments,which help to determine whether weapon systems are ready for operational testing, is within DOl%E’S mis- sion becausethe assessmentsoffer advice to the DOD acquisition deci- sionmakers. Although these assessmentsrely primarily on development and not operational test data, they can be useful in filling a void when actual operational test results are not available. However, these assess- ments are not substitutes for actual OT&E and the basis for developing the assessmentsshould be fully disclosedto avoid misunderstanding as to their nature and use. The Congress,concernedthat OI’&E was not receiving sufficient emphasis Background and independent oversight, directed the establishment of MJT&E (P.L. 98-94, Sept. 24, 1983). The statute designatedthe Director as the Y ‘In practice, DOME is used to mean both the Director and the office under the Director’s responsi- bility. To avoid confusion, we refer to the Director as the Director and to the office as DOT&E. Page 1 GAO/NSIAD9O-128 DOD Testing 0288677 principal CT&E official within senior DOD managementto ensure that sys- tems being acquired are ready for production. To ensure that DUI’&E is independent, the Congresslegislated on several occasions,the organiza- tional separation between development and operational testing-the two primary types of DOD testing during the acquisition process.In addi- tion, the Director may not be assignedany responsibility for develop- ment test and evaluation except to provide advice to officials responsible for such testing. The roles and responsibilities of the Director are described in 10 U.S.C. section 138. According to the statute, the Director is responsible for prescribing policies and procedures for W&E, providing guidance to and consulting with defenseofficials both gener- ally and with respect to specific programs, monitoring and reviewing all W&E, coordinating joint (JT&E, analyzing and reporting on both the adequacy of the testing and the results, and reviewing and making recommendationsto the Secretary of Defenseon all budgetary and financial matters relating to or&~. Development and Development test and evaluation is done throughout the acquisition pro- Operational Testing cessto ensure the attainment of technical performance specifications, program objectives, and weapon system supportability. Development testing is normally done by the agency responsible for developing the system. It usessuch techniques as modeling, simulation, prototypes, or full-scale engineering development system models to determine the extent that a system meets technical specifications. W&E is the field test, under realistic combat conditions, of major weapon systems, equipment, components,or munitions for the purpose of deter- mining the operational effectiveness and suitability2 of the weapons, equipment, or munitions used in combat by typical military users. Each service has an operational test agency that is responsible for CJIBE, reporting test results, and providing an evaluation of the tested system’s operational effectiveness and suitability. 20perational effectiveness means the ability of a system to accomplish its mission when placed in use in the planned operational environment. Operational suitability is the degree to which a system can be placed satisfactorily in field use considering, among other factors, availability, maintainability, and logistica support. Psge 2 GAO/NEXADM-128 DOD Ted&j Bma077 In November 1989, the Congressadded provisions concerningOJ%Ein connection with the decision to go beyond low-rate initial production.3 The Congressstated that m&E doesnot include an operational assess- ment basedexclusively on (1) computer modeling, (2) simulation, or (3) an analysis of system requirements, engineering proposals, design specifications, or any other information contained in program documents. The Deputy Secretary of Defenseappointed the Director as Chairman of The Director’s Role in the Test and Evaluation Committee. As Committee Chairman, the DevelopmentTest Director presided over the operations of the Committee, which was Resources responsible for planning, programming, and budgeting for development test resources.As a result, the Director’s independencewas jeopardized becausethe Director had influence over the types of development test assetsused by the services.Responsibility for development test resourcesrests with the services.We believe the Director should func- tion as an adviser on any future committee or council that is given responsibility for development test resources. In 1987, the Congressinserted languagein nor&~‘sstatute through the National DefenseAuthorization Act for Fiscal Years 1988-89,which states that “the Director may not be assignedany responsibility for developmental test and evaluation, other than the provision of advice to officials responsible for such testing.” According to the conference report on this act, the languagewas inserted to ensure that responsibili- ties for operational testing are separate from functions associatedwith development testing. The conferencereport further stated that the Sec- retary of Defenseshould refrain from any realignment or new arrange- ment of test and evaluation activities, oversight responsibilities, or functions. We recently issued an unclassified version of a classified report4 that criticized DOT&E'Smanagementof the OT&ECapability Improvement Pro- gram. In that report, we stated that m&E was performing management functions by acquiring operational test resourcesused in testing major weapon systems.We recommendedthat the Secretary of Defensedirect 3Low-rate initial production is the production of a system in the minimum quantity needed to conduct UI’&E, to establish an initial production base for the system, and to permit sn orderly increase in the production rate for the system sufficient to lead to full-rate production upon the successful comple tion of operational testing. 4(GAO/NSIAD-QO-141,Mar. 30,lQQO). Page 8 GAO/NSIAIMO-1ZS DOD Terthg , B238877 and assurethat the servicesplan, program, and budget for adequatetest resourcesneededto conduct operational testing of weapon systems effectively. We explained that the Director should continue to perform oversight and policy functions by reviewing and making recommenda- tions to the Secretary of Defenseto assurethat adequate test resources are acquired. The Director Influenced The Test and Evaluation Committee was established to support the the Committee in Under Secretary of Defensefor Acquisition, who, by law, is responsible for establishing policies for acquiring DOD’S weapon systems and the Development Test development testing of such systems. (See10 U.S.C.133(b)(2).) The ResourceMatters Committee was to provide a forum for key representatives from the Office of the Secretary of Defense,the services,and other agenciesto identify and resolve issuesand to develop recommendationson the DOD acquisition system in the test and evaluation area. The Committee’s charter specified that the Chairman had major roles and responsibilities regarding the operations of the Committee. The Committee Chair was to, among other things, direct and supervise oper- ations of the Committee, schedule and preside at meetings, select items submitted by the Committee membersto be addressedby the Com- mittee, and establish panels to carry out Committee assignedprojects. The Committee attempted to obtain a consensusamong all members regarding matters brought before the Committee. As such, the Director participated in the decision-making processby concurring or not concur- ring on these matters, thus having influence over the types of test assets used by the services. As Committee Chairman, the Director initially provided advice and presented recommendationson various test and evaluation issuesto the Deputy Secretary of Defensefor a decision. In 1986, the issuesincluded such things as a proposed recommendation to improve spacesystem test capabilities. In 1988, the Director recommendedestablishing the Central Test and Evaluation Investment Program, which received $83 million in fiscal year 1990 for the purpose of acquiring primarily development test resourcesfor new weapon systems. The Director believed that the lack of development and operational test investments by the serviceswas reaching crisis proportions. Instead of continuing to serve in an advisory capacity by presenting rec- ommendations on test resourcesto the Deputy Secretary of Defense,the Page 4 GAO/NSL4D90-128 DOD Testing B-288877 Director becamea manager of these resourcesby taking on responsi- bility for the program. In November 1988, the Deputy Secretary of Defensecreated the program and directed that it be managedby the Committee. On November 30,1988, managementof the program was established and the Committee Chairman assumedresponsibility and accountability for the managementand effective allocation of the pro- gram funds.6Managementof test resourcesmay dilute DOT&E’S oversight function becauseit gives DW&E a direct influence over the types of test assetsacquired and used by the services.For example, MJT&E may be reluctant to criticize a test plan that contains inadequate testing resourcesif that office was responsible for managing or played a part in the acquisition of such resources. Oncethe Committee approved the projects to be funded under the pro- gram, the Committee’s resourcepanel established the parameters for the projects that were to be executed by the lead servicesor agency. In turn, the lead service or agency developed a project managementplan that specifically outlined how the project would be executed. Thesetwo efforts established the “contract” for the project between the lead agent and the Committee. The projects were then executed by the lead service or agency.The Committee established program corporate investment priorities with the goal of preventing unnecessaryduplication, encouraging multiservice use of equipment, and providing critically neededtest capabilities. Pri- orities were to be determined in all phasesof program management (planning, programming, budgeting, and budget execution) by the Com- mittee. As a r&%ilt; the Direstgr, by Msuming maJorrespansibilitiee fsr the Committee that manageddevelopment test resource funding, placed his independenceat risk. The Committee May Be DOD’s responseto a draft of this report stated that the Test and Evalua- Replacedby a Council tion Committee under the DefenseAcquisition Board has been disestab- lished. However, we were told that a DOD Test and Evaluation Resource Council may be created and may be co-chaired by the Director. At pre- sent, it is unclear how such a council will operate. If the council is estab- lished to plan, program, and budget for development test resourcesand the Director servesas a co-chairman or even as a council member, we “The funding for the program is found in the appropriation for the Deputy Director, Defense Research and Engineering (Test and Evaluation), Office of the Secretary of Defense, who is respon- sible for development testing. Fiscal oversight, distinct from the Committee’s responsibility for man- agement and corporate priority setting, is the job of the Deputy Director. Page 6 GAO/NSIAD-M-128 DOD Testing B-238677 believe the Director would continue to exceedthe role of providing advice to officials responsible for development test matters. In the past, the Test and Evaluation Committee membership consistedof the Committee chair, vice chair, members, and invited participants/ advisers. The Committee Chair and memberswere to reach a consensus on various issuesbrought before the Committee. The invited partici- pants and advisers were to provide information on specific matters. In keeping with m&E’s responsibilities for UI%E oversight as addressedin its statute, it would be more appropriate for the Director to be an adviser on any future test resourcescouncil that is created. As such, the Director should continue to perform oversight and policy functions by reviewing test plans and results and making recommendationsto the Secretary of Defenseso that test resource needsare given appropriate consideration. Our review showed that m&~‘s use of early operational assessments, Useof Early which DGD began using in 1988, are not prohibited by m&E’s statute. Operational They allow DOT&E to offer information to DOD acquisition decisionmakers AssessmentsIs Within as to whether weapon systems are ready for m&E and whether opera- tional shortfalls are being identified and corrected early in the acquisi- m&E’s Mission tion process.These assessments,which are basedprimarily on development rather than OR&E data, are being done during the early phasesof the acquisition processwhen actual (F&E results are not avail- able. However, as noted in a previous report,Bthey are not substitutes for m&E and the basis for the assessmentsshould be fully disclosed when they are reported to congressionaland DOD decisionmakers. According to the Director, an independent evaluation is made of the operational assessmentsdevelopedby the operational test agencies.This evaluation is provided as advice to the Under Secretary of Defensefor Acquisition through the DefenseAcquisition Board process,which is used to overseemajor system acquisitions. The Director’s Past In the past, the Director has attempted to redefine the way test and Attempts to Redefine Test evaluation is viewed within DOD. Until the Director began advocating the use of early operational assessments,initial m&E was that portion of and Evaluationu actual OUCE done throughout the acquisition processbefore the decision (‘Navy Weapons Testing: Defense Policy on Early Operational Testing (GAO/NSIAD89-98, May 8,19f.W. Page 6 GAO/NSLW4tO-128 DOD Testing to proceed to production. It was accomplishedusing a prototype, preproduction article, or a low-rate initial production article as the test item. The “final exam,” or the latter phase of initial ONE usually entailed dedicated operational testing of production representative test articles using typical operational personnel in as realistic a combat envi- ronment as possible. In a report to the HouseArmed ServicesCommittee dated September 25, 1987, the Secretary of Defenseproposed to redefine initial UI%E as that component of testing initiated at program inception, to forecast opera- tional effectiveness and suitability. It was to be a tool to provide insights about the potential operational worth of a system throughout its acqui- sition life. As opposedto doing actual UME, initial W&E would take advantage of any test results and could use modeling, simulation, and paper analysesto make assessments7 DOD’s current policy requires early and progressive assessmentsof oper- ational capability, including realistic operational testing before full-scale production starts. It is the current DOD perspective that an operational test agency should evaluate all pertinent information as it becomes available, regardless of the source,for input into early periodic opera- tional assessments. On January 26, 1990, DOD issued guidelines that prescribe uniform pro- ceduresfor preparing and approving Test and Evaluation Master Plans. The plans describe, among other things, the development and opera- tional testing to be performed on a weapon system throughout the acquisition process.Theseguidelines state that the plans will show how operational assessmentsand testing are structured at each acquisition decision point. Further, they show how operational testing will, or has, evaluated the system in an environment as operationally realistic as possible. If W&E cannot be done or completed early in the development process,then this is to be clearly stated and the reason(s)explained in the plans. The guidelines state that operational assessmentsare done before or in support of the full-scale engineering development phase. They are used to identify significant trends noted in development efforts, program- matic voids, areas of risk, adequacy of requirements, and the ability of the program to support adequate operational testing. The assessments ‘According to a JWI’&E official, this new definition for initial UlXkE was synonymous with an early operational assessment. Page 7 GAO/NSIAD-90-128 DOD Testing B!ma677 are to be basedon all information relevant to the program, and can be made at anytime. However, they are not to be consideredsubstitutes for Ul%E in support of full-production decisions. Early Operational Performing early operational assessmentswhen production representa- AssessmentsCan Be tive test articles are not available is a step forward in filling a void in the availability of actual (T&E results. Furthermore, the assessmentsare Useful consideredadvisory in nature and can be useful to decisionmakers.For example, the C-17A aircraft was the first major acquisition program that relied on an early operational assessmentas a decision-making tools In December1987, the Deputy Secretary of Defenserequired the Air Force to submit an assessmentof the C-17A regarding (1) meeting low-rate initial production criteria and (2) progresstowards initial oper- ational capability. In September 1988, the Air Force Operational Test and Evaluation Center completed the C-17A assessmentto support a low-rate initial production decision consisting of 40 C-17A aircraft. Sinceno production representative systems/subsystemswere available, no operational tests were done. The center’s assessmentwas basedon all available informa- tion sourcesto addresssuch areas as attainment of the production cri- teria listed in the C-17A’s Test and Evaluation Master Plan. These sourcesincluded, but were not limited to, results of critical design reviews, development test results, demonstrations, the center’s partici- pation in planning committees, program documents, and interviews. In January 1989, the Deputy Secretary of Defenseapproved low-rate initial production of the C-17A aircraft, which reduced the requested procurement of 40 aircraft to 10 aircraft. BecauseC-17A aircraft were not available for UI%E, the early operational assessmentwas beneficial in that it provided timely insight into potential problems for the C-17A aircraft basedon essentially development test data. Our Earlier Report Dealt In our 1989 report to the Chairman, Subcommitteeon Seapowerand With Disclosure of Early Strategic and Critical Materials, HouseCommittee on Armed Services, we evaluated the Navy’s o-r&~ done before decisionswere made on the Operational Assessments full-scale development or low-rate initial production of weapon systems. ” During fiscal years 1986 to 1988, the Navy typically approved weapon sDOD is just beginning to use assessmentsin the acquisition process. As a result, there are very few examples of early operational assessments. Page 8 B-288677 systems for full-scale development and, in many cases,for low-rate ini- tial production before any (JT&E was completed. However, the Navy did do a limited number of operational assessmentsthat supported the early milestone decisions. Basedon our evaluation of DOD’S testing policy and the Navy’s use of operational assessments,we recommendedin our 1989 report that the Secretary of Defenseclarify when it is appropriate for decisionmakers to rely on operational assessmentsthat do not include the operational testing of any hardware and when actual W&E must occur. In addressing this issue, the Secretary was to assurethe independenceof the services’ m&E agencieswas not compromised and that the basis of the weapon system assessmentswere fully disclosedwhen the assessmentswere reported to congressionaland DOD decisionmakers. In responseto our report, DOD stated that it agreed with our recommen- dations. DOD further stated that the policy as to when it is appropriate for decisionmakers to rely on operational assessmentswould be included in the publication of DOD Manual 5000.3-M-5,volume 3. Due to changes in DOD’S testing policy being considered as part of DOD’S DefenseManage- ment Review, many directives and regulations are being consolidated, revised, or terminated. For example, DOD plans to publish a new Direc- tive 6000.1~addressingthe acquisition processas well as a manual addressing acquisition documentation and reporting procedures.As a result, test and evaluation policies and procedures will not be published until sometime in the future. We recommendthat the Secretary of Defensekeep development and Recommendation operational testing independent and separate by ensuring that the Director provides advice only to those committees or councils that are responsible for the planning, programming, and budgeting of develop- ment test resources.The Director should perform oversight and policy functions by reviewing test plans and results and making recommenda- tions to the Secretary of Defenseso that test resourceneedsare given appropriate consideration. agreed that development and operational testing should be kept Agency Commentsand DOD independent and separate. However, DOD disagreedthat DCJI%E had man- Our Evaluation aged development test resources.DOD stated that neither the Test and ‘In the past, DOD Directive 6000.1 addressed the subject of major weapon system acquisitions. Page 9 GAO/NSIAD99.128 DOD Testing B-238677 Evaluation Committee nor the chairman ever manageddevelopment test resourcesor any program such as the Central Test and Evaluation Investment Program that involves such resources.As such, DOD has stated it was already in compliance with our recommendation and that the Director does not now function as chairman of any committee or any other organization responsible for managing test resourcesnor is there any intent to assign such responsibilities to the Director. Although we recognizethat test and evaluation resource management structures are presently undergoing changes,the Director had major responsibilities chairing a committee that planned, programmed, and budgeted for development test resources.As Committee Chairman, the Director exercised major responsibilities over the operations of the Com- mittee. Further, the Committee was responsible for planning, program- ming, and budgeting for the Central Test and Evaluation Investment Program. Although we did not identify examples where the Director had unduly influenced the services,we believe his responsibilities created the perception that the Director’s independencefrom development test matters was jeopardized by allowing for a direct influence over the test assetsused by the services. DOD told us that a corporate mechanism has been established to focus attention on the need for test resources.DOD believes that this attention has forced the servicesto work together to identify requirements, elimi- nate duplication, and obtain the necessaryfunds to support the requirements. Basedon DOD’S comments,we have modified our recommendation to better emphasizethe actions we believe are needed.(DOD’S detailed com- ments and our evaluation of these comments are presented in app. II.) We are sending copies of this report to the Secretariesof Defense,the Navy, Army, and Air Force and to interested parties. Copieswill also be made available to others on request. Page 10 GAO/NSIAD-90-128 DOD Testing Pleasecontact me at (202) 2’758400 if you or your staff have any ques- tions concerning this report. Major contributors to this report are listed in appendix III. Sincerely yours, Paul F. Math Director, Research,Development, Acquisition, and Procurement Issues Page 11 GAO/NSIAD-S@128 DOD Testing Contents Letter Appendix I Scopeand Methodology Appendix II CommentsFrom the GAO Comments Department of Defense Appendix III 24 Major Contributors to This Report Abbreviations DOD Department of Defense DOT&E Office of the Director, Operational Test And Evaluation W&E operational test and evaluation Page 12 GAO/NSIAD-90-128 DOD Testing Page 13 GAO/NSIAD-90-128 DOD Testing Appendix I Scopeand Methodology To assessthe Director’s role as Chairman of the Test and Evaluation Committee, we reviewed the scopeof DUNE'S legislation as it pertained to development test matters. In particular, we focused on a legislative provision that stated that the Director was not to be assignedany responsibility for development test and evaluation, other than the provi- sion of advice to officials responsible for such testing. We interviewed Office of the Secretary of Defenseofficials and reviewed documents pertaining to the Test and Evaluation Committee. We reviewed the Director’s involvement in test resourcesponsorship as the Committee Chairman and in the normal role as DCW.&E. In particular, we interviewed Office of the Secretary of Defenseofficials and reviewed documents pertaining to the Central Test and Evaluation Investment Program becausethe program is used to acquire primarily development test resources. To assessthe Director’s use of early operational assessments,we evalu- ated the scopeof DtYIXE'S legislation as it pertained to development test matters and whether that office was responsible for development test and evaluation, other than the provision of advice to officials respon- sible for such testing. Sincethe early operational assessmentis an evolving concept, only lim- ited information is available on it. Primarily, we reviewed documents that set out DCYWE'S policy on early operational assessments.We also interviewed service officials from the Navy Commander,Operational Test and Evaluation Force, Norfolk, Virginia; Air Force Operational Test and Evaluation Center, Albuquerque, New Mexico; and the Army Opera- tional Test and Evaluation Agency, Alexandria, Virginia, to obtain their views on early operational assessments. Our review was performed in accordancewith generally acceptedgov- ernment auditing standards between October 1988 and January 1990. Page 14 GAO/NSIAD-90-128 DOD Testing A$&ndix II Comments From the Department of Defense Note: GAO comments supplementing those in the report text appear at the end of this appendix. OFFICE OF THE SECRETARY OF DEFENSE WASHINGTON, DC 20301-1700 6 JUN I:% OPERATlONAL lE8’l AND EVALUATION Mr. Frank C. Conahan Assistant Comptroller General National Security and International Affairs Division U.S. General Accounting Office Washington, DC 20540 Dear Mr. Conahan: This is the Department of Defense (DOD) response to the General Accounting Office (GAO) draft report, "DOD TESTING: Questionable Tasks Performed by the Director, Operational Test and Evaluation," dated April 13, 1990 (GAO Code 396224/OSD Case 8303). The Department agrees with the recommendation, but does not agree with the GAO assertion that the Director, Operational Test and Evaluation managed development test resources. See comment 1. It is the DOD position that neither the Test and Evaluat.ion Committee nor the Chairman has ever managed the Central Test and Evaluation Investment Program, as asserted in the report. The Department concurs and is already in compliance with the recommendation that "development and operational testing be kept independent and separate." The Director, Operational Test and See comment 2. Evaluation is not now assigned as chairman of any committee or any other organization responsible for management of test resources, nor is there intent to assign such responsibilities. The detailed DOD comments on the report findings and recommendation are provided in the enclosure. The Department appreciates the opportunity to comment on this ft report. Y-7 C. Duncan Enclosure Page 16 GAO/NSIAD-90-128 DOD Testing Appendix II CommentaFkomtheDepartmentofDefense GAO DRAFT REPORT - DATED APRIL 13, 1990 GAO CODE 396224 - OSD Case 8303 "DOD TESTING: QUESTIONABLE TASKS PERFORMEDBY THE DIRECTOR, OPERATIONAL TEST AND EVALUATION" DEPARTMENT OF DEFENSE COMMJ5NTS * * * * * FINDINGS 0 FINDING A: v: mce of #J&Q&Director. ODmtiona 3-t and ~=mia,um . The GAO observed that the Congress, concerned that operational test and evaluation was not receiving sufficient emphasis and independent oversight, established the Director, Operational Test and Evaluation (P.L. 98-94, September 24, 1983). According to the GAO, the statute designated the Director as the principal operational test and evaluation official within senior DOD management to ensure that systems being acquired are ready for production. The GAO pointed out that, to ensure the independence of the Director, Operational Test and Evaluation, on several occasions, the Congress legislated the organizational separation between development and operational testing-- the two primary types of DOD testing during the acquisition process. The GAO observed that the Director, Operational Test and Evaluation, may not be assigned any responsibility for development test and evaluation-- except to provide advice to Nowon pp. l-2 officials responsible for such testing. (pp. 2-4/GAO Draft Report) goD RESPOND: Concur. 0 FIINDING: These Test Resource* . The GAO found that the Director, as Chairman of the Test and Evaluation Committee, had recently become involved in the management of development test resources. The GAO explained that managing development test resources goes beyond merely providing advice to officials responsible for development test matters and puts at risk the independence of the Director from development test matters because it gives the Director control and influence over the types of development test assets used by the Services. The GAO pointed out that in 1987, the Congress inserted language in the National Defense Authorization Act for Fiscal years 1988-89, relating to the Office of the Director, Operational Test and Evaluation, which states "The Director Enclosure Page16 GAO/NSIAD-!bO-128DODTesting Appendix II CemmentsFre~the Department 0fDefenre may not be assigned any responsibility for developmental test and evaluation, other than the provision of advice to officials responsible for such testing." According to the GAO, the conference report on this Act had the language inserted to ensure that responsibilities for operational testing are separate from functiona associated with development testing. The GAO noted that the conference report further stated that the Secretary of Defense should refrain from any realignment or new arrangement of test and evaluation activities, oversight responsibilities, or functions, The GAO further referred to a recently issued GAO report, w t of DOD'S Owerational Test a tv mnt Prm dated March 20, 1990 (OSD Case 8137), which criticized the'management of the Operational Test and Evaluation Capability Improvement Program. The GAO concluded that the Director, Operational Teat and Evaluation, was performing management functions by acquiring test resources used in testing major weapon systems. The GAO asserted that management of test resources dilutes the Director's oversight function because it gives the Director, Operational Test and Evaluation, a direct influence over the types of teat assets acquired and wed by the Services. The GAO concluded that the Director may be reluctant to criticize a test plan that contains inadequate testing resources if his office was responsible for managing or played a part in the Nowon pp.34 acquisition of such resources. (PP. l-2, PP. d-S/GAO Draft Report) -RESPONSE: Nonconcur. The Department acknowledges that there were some statements in the early correspondence creating the Central Test and Evaluation Investment Program that assigned management responsibility to the Test and Evaluation Committee (hereafter referred to as the Committee) and the Chairman of the Committee (hereafter referred to as See comment 1. the Chairman). In practice, however, the Chairman did not manage. As specified in the Committee charter, the Committee provided a forum and catalyst for the review of DOD test resource matters and recommended alternatives. It did not have management authority under the charter. Moreover, the Chairman did not have decision authority; consequently, he could not manage teat re8ources. The Chairman's role was to facilitate the review process and cause agreement to occur where that was possible. In that capacity, for all members of the Committee, whether they were judged to be of the development or operational test communities, the Director, Operational Test and Evaluation or his representative provided "advice to officials responsible for such testing." Y Page17 i Appendix II Commenti From the Department of Defense . 0 EIINDING: TheisMsnaainaDevelorrraent vResourcea. The GAO observed that the Test and Evaluation Committee was established to support the Under Secretary of Defense for Acquisition who, by law, is responsible for acquiring DOD weapon systems and the development testing of such systems. The GAO noted that the Committee was to provide a forum for key representatives from the Office of the Secretary of Defense, the Services, and other agencies to identify and resolve issues and to develop recommemdations on the DOD acquisition system in the test and evaluation area. They learned that the Deputy Secretary of Defense appointed the Director, Operational Test and Evaluation as the Chairman of the Committee. The GAO explained that, as the Chairman, the Director initially provided advice and presented recommendations on various test and evaluation issues to the Deputy Secretary of Defense for a decision. The GAO found, however, that in 1986, the issues included such things as a proposed recommendation to improve space system test capabilities. The GAO further found that, in 1988, the Director recommended establishing the Central Test and Evaluation Investment Program, which received $83 million in FY 1990 for the purpose of acquiring primarily development test resources for new weapon systems. The GAO acknowledged that the Director believed that the lack of development and operational test investments by the Services was reaching crisis proportions. The GAO concluded that, instead of continuing to serve in a advisory capacity by presenting recommendations on test resources to the Deputy Secretary of Defense, the Director, Operational Test and Evaluation, became a manager of those resources by taking on the responsibility for the program. The GAO emphasized that, in November 1988, the Deputy Secretary of Defense created the program and directed that it be managed by the Committee. The GAO concluded, therefore, that the Committee Chairman became ultimately responsible and accountable for the management and funding of development test resources. While agreeing the program is executed by the Military Services, the GAO further concluded that the Committee sets program corporate investment 'priorities with the goal of preventing unnecessary duplication, encouragingmulti-Service use of equipment, and providing critically needed test capabilities. The GAO observed that the priorities are to be determined in all phases of program management by the Committee; therefore, the Director, Operational Test and Evaluation, by controlling development test resource funding, Page 18 GAO/NSIAMO-128 DOD Testing Appendix II Comments From the Department of Defense ha8 exceeded the mission of providing advice to Now on pp, 4-5 decisionmakers. (pp. l-2, pp. 6-7/GAO Draft Report) See comment 3. PoD: Partially concur. The majority of the teat resources that will be acquired under the Central Test and Evaluation Investment Program will directly support both operational and developmental testing. Those teat resources that do not directly support operational testing will provide data that will be used to support operational effectiveness and suitability assessments or they will be used as evaluation/analysis tools. See comment 1. The Department acknowledges that a Deputy Secretary Memorandum did direct the Committee to manage the Central Teat and Evaluation Investment Program and that a subsequent Committee Memorandum stated that the "...Chairman . . . is ultimately responsible and accountable for the management and effective allocation of [Central Teat and Evaluation Investment Program] funds." However, in recognition of the concerns of the Congress, in actual practice, the Director nevel: exercised his Department mandate to "manage" the Central Test and Evaluation Investment Program. As defined above, the Director has m manacred the Central Test and Evaluation Investment Program. See comment 4. Rather than managing, as Chairman of the Committee, the Director ensured a systematic process for review of all teat resource matters. In fact, in recommending the management approach for the Central Teat and Evaluation Investment Program, the Director proposed that the funding be placed in the Deputy Director Defense Research and Engineering (Teat and Evaluation] appropriation, recognizing that office as the Department test resource manager. 0 ,* F t ia Uncertain . The GAO FINDING: The explained that the continued need for the Committee is being questioned as a part of the DOD Defense Management Review. According to the GAO, the Committee functions may be placed under a council that could possibly be chaired by the Director, Operational Test and Evaluation. The GAO observed that it is currently unclear whether such a council would continue to manage development test resources. The GAO concluded, however, that if the council does manage such resources and the Director is the council Chairman, then the Director, Operational Test and Evaluation, would continue to be in the position of managing development test resources--thus still going beyond the mission of only providing advice to officials responsible for development test Now on pp. 5-6 matters. (pp. 1-2, p. S/GAO Draft Report) Page 19 GAO/NSIAD-99-128 DOD Testing Appendlrn C4munent8RomtheDepnrtmentofDefenoe -RESPONSE: Partially Concur. The Test and Evaluation Committee under the Defense Acquisition Board has been disestablished. However, the DOD Test and Evaluation Activity Consolidation Study, conducted as a part of the Defense Seecomment2. Management Review, does propose the establishment of a DOD Test and Evaluation Resource Council, independently chartered under the Deputy Secretary of Defense. If the Deputy Secretary approves the Council, it will be charteqed in such a way that there is neither the perception or fact of organizational bias. 0 EUlR+LG: Pne of lSaUv OPerational 1s Wi&inAhsi aa . The GAO found that the use of early operational assessments, which the DOD began using in 1988, are not prohibited by the Director, Operational Test and Evaluation statute. The GAO explained that the assessments allow the Director to offer information to DOD acquisition decisionmakers as to whether weapon systems are ready for operational teat and evaluation and whether operational shortfalls are being identified and corrected early in the acquisition process. The GAO pointed out that these aaseaamenta, which are baaed primarily on development rather than operational teat and evaluation data, are being done during the early phases of the acquisition process, when actual operational teat and evaluation results are not available. The GAO emphasized, however, that they are not substitutes for operational test and evaluation and the basis for the assessments should be fully disclosed when they are reported to congressional and DOD decisionmakers, as noted in its prior report, NAw TESTINC . efanse Pow , '* datedMay 8, 1989 (OSD Case 7800). According to the GAO, the Director stated that an independent evaluation is made of the operational asaesaments developed by the operational teat agencies. The GAO noted that this evaluation is provided as advice to the Under Secretary of Defense for Acquisition through the Defense Acquisition Board process that is used to oversee major system Nowon p-6 acquisitions. (pp. l-2, pp. 0-14/GAO Draft Report) I)oD: Concur. Page20 GAO/NSIAD@O.128 DODTesting Appendix Jl Commenta From the Department of Defense ***** RECOMMJZNDATION 0 -: The GAO recommended that the Secretary of Defense assure that development and operational testing be kept independent and separate. (The GAO emphasized that the Secretary should ensure that the Director does not chair or manage committees or programs that involve development test resourcea that can compromise the independence of development Now on p. 9 and operational testing.) (p. lS/GAO Draft Report) See comments 1 and 2. -RESPONSE! Concur. The Department is already complying with this recommendation because (1) the Director does not manage any test resources and (2) there is no intent to assign any such responsibilities to him. The Secretary has found the Director’s counsel on Operational Test and Evaluation-related budgetary and financial matters to be invaluable and intends to ensure that such advice remains available through established Department corporate and functional mechanisms. Y Page 21 GAO/NSIAB9O-128 DOD Tasting Appendix II Canments Prom the Department of Defeme The following are GAO'S comments on DOD'S letter dated June 6, 1990. 1. As explained in the report, the Director had major responsibilities GAO Comments chairing a Committee that planned, programmed, and budgeted for development test resources.Instead of continuing to serve in an advi- sory capacity by presenting recommendationson test resources,the Director took on responsibilities for managing these resources.This type of participation placed at risk the Director’s independencefrom develop- ment test matters becauseit gave the Director influence over the types of test assetsacquired and used by the services. We modified our report to discussin greater detail how we believe the Director influenced the Committee in development test resources.We explain that the Chairman had major roles and responsibilities regarding the operations of the Committee. (Seep. 4.) In addition, the Committee’s resource panel served as the working arm for the Com- mittee. For example, the resourcepanel developed a “contract” between the Committee and lead service or agency on how specific projects would be executed. (Seep. 6.) 2. While we recognizethat the Committee has been disestablished, we have included information in the report about it becausethe Committee offers a historical perspective on how such a council could function. For example, the Committee’s charter provides a framework as to how the DOD Test and Evaluation ResourceCouncil’s charter could be structured. By reviewing the Committee’s charter, we found that the Chairman had major responsibilities dealing with the operations of the Committee that would not be appropriate. As such, the Director had influence over the types of test assetsused by the services.(Seep. 4.) DOD states that the Council will be chartered in such a way that there is neither the perception or fact of organizational bias. In DOD'S official oral comments,we were told that the Council could possibly be co-chaired by the Director. We have modified the report to show the past membership of the Test and Evaluation Committee. (Seep. 6.) By reviewing the charter of the Committee, we found that the membership consistedof the Committee chair, vice chair, members,and invited participants/ advisers. The invited participants and advisers were to participate baaedon specific matters to be addressed.In keeping with the Director’s responsibilities for m&E oversight as addressedin its statute, we believe the Director should more appropriately be an adviser on any future council that may be created. Page 22 GAO/Nf3IAD9O-128 DODTesting Appendix II Chmunenta From the Department of Defense 3. We believe that the Director should not have provided more than advice to officials responsible for development testing and continue to believe that the Director should not exceedproviding advice by becoming involved with development test resourcesor any combination of development and or&~ test resources.In addition, we recently issued a report that criticized D0ME’S managementof VII&E resources.We recom- mended that the Secretary of Defensedirect and assurethat the services plan, program, and budget for adequate test resourcesneededto con- duct operational testing of weapon systems effectively. In April 1990, the Director emphasizedto the servicesthe importance of their plan- ning, budgeting, and programming for adequatetest resources. 4. We agreethat the Director proposed that the funding for the program be placed in the Deputy Director, DefenseResearchand Engineering (Test and Evaluation) appropriation. However, the Deputy Director was only to be the fiscal agent for the funds. The Committee, on the other hand, was responsible for managementand corporate priority setting. Priorities were to be determined in all phasesof program management (planning, programming, budgeting, and budget execution). In effect, we believe this constituted control over development test resource funds. Page 23 GAO/NSIAD-30428 DOD Testing Appendix III Major Contributors to This &port Michael E. Motley, Associate Director National Security and Lester C. Farrington, Assistant Director International Affairs Charles D. Groves,Evaluator-in-Charge Division, Washington DC. Office of General Counsel (296224) Page 24 GAO/NSIAD-90-128 DOD Testing ‘I’hth first five copitbs of that-h GAO rc”port art& frt*e. Additional copies tire $2 tech. Orders should be sent to 01th following address, accom- pauitAt1 by a check or mont*y order made oul. t,o t,he Superiuteudeut of I)ocumtmts, when ntxessary. Orders for 100 or more copies tm be mailed I,0 a single address are discounted 25 ptmeut. IJ.S. <&neral Acrcouut.ing Office I’.(). Hox 6015 Gait.htmburg, MD 20877 Ordrrs may also be placed by calling (202) 275-6241. I -.___. _ _.__ _I” _- ._.__ ..__ “..-... ___” -.-.... -I _-.. I.. .-_. I. - _.._-.._. -- .-._ - -...- -..-I.- -. ..- ._ - ._ ; --1----
Test and Evaluation: The Director, Operational Test and Evaluation's Role in Test Resources
Published by the Government Accountability Office on 1990-08-27.
Below is a raw (and likely hideous) rendition of the original report. (PDF)