,(-. ! . DEFENSE DIVISION ~-163058 The Ronorable ._- The Secretary of Defense ', Attention: Assistant Secretary of Defense (Comptroller) Dear Mr. Secretary: The General Accounting Office has made a ~~e+e$ewof the operational test and evaluation of the Fast @tomatic Shuttle Transfer (FAST) system audother systems (GAO Code 77104). The review was performed at the Navy's Operational Test and Evaluation Force (OFTEVFOR), Norfolk, Virginia. In conjunction with this review, we noted that production commit- ments had been authorized by the Navy before adequate testing of about one-third of the systems assigned to OFTEXFOR as of September 30, 1970, for operational testing and evaluation. Consequently, OFTEVFOR is un- able, in these cases, to perform sufficient operational evaluation of new equipment to determine its suitability for service use. As pointed out in previous reports, concurrent development and production fre- quently results in additional expenditures of time and money to iden- tify and correct deficiencies and may cause delays in deployment of effective systems. Our work indicates a need for management improvements which, we believe, would supplement the current measures being taken by the De- -- partment of Defense and the Navy to improve operational test and evalu- . ation procedures. Although we did not assess the urgency of the pro- curements and other factors which the Navy deeisionmakers may have considered in these cases, we are reporting our fLndings at OFTEZFOR and our suggestions for earlier operational evaluation to permit timely consideration of any risks related to urgent requirements. O??TEXFOR's MISSION ARD FUNCTIONS OP!i!EVFOR's operational evaluation of newly developed equipment nor- mally follows the developing command's tecbnBzK%valuation ai?d&termi- nation that the equipment meets the technical requirements. The Chief of Naval Operations (CNO) then directs OFTEWOR to perform independent test and evaluation in an operational environment. OFTEWOR determines whether the equipment meets the operational requirements and makes recom- mendations 'concerning its suitability for service use. This includes making a determination that the equipment can be operated, maintained, and supported logistically 50fH A ~-163058 CNO instructions provide that new equipment till not be committed to large-Scale production until its suitability for service use has been established through operational testing. When new items are submitted for testing in accordance with this policy, OPTEVFOR's test and evalua- tions are usually performed using prototype models of the equipment. In cases where urgency or important operational considerations indicate a need to deviate from this policy, procurement prior to completion of operational testing may be authorized. In instances where procurement is authorized before equipment has undergone operational tests and evaluations, service use and operational testing normally occur con- currently, as in the case of the FAST system. In these instances, OPTEXFOR usually utilizes equipment already delivered to the fleet to make its tests. To meet its responsibilities, OPTEVFOR is organized into three test and evaluation squadrons located at Key West, Florida, and Point Mngu and China Lake, California, and two test and evaluation detachments located at New London, Connecticut, and Key West, Florida. It has a total military strength of about 1,400 officers and enlisted personnel and about 24 civilian personnel. OFTEVFOR has about 40 assigned mili- tary aircraft and uses other shore facilities on both the east and west coasts as well as ships of both the Atlantic and Paciffc Fleets. The results of OPTEXFOR's tests of new equipment and its recommenda- tions are submitted directly to the CNO. Decisions as to the accept- ability of new equipment for operational use are made by the CNO. PROCUREMENTCOMMlTMEiYTSMA~ REl?OR'E COMPLJEION OF OPERATIONAL TESTING Our review of the FAST system, which is used for underway replenish- ment of ammunition and stores between supply and combatant ships, iden- tified problems that cau result when new equipment is procured without completing operational testing. We found that the FAST system was in- stalled on about 50 ships before operational testing and service use showed that the system was not reliable and could not be effectively maintained. Procurement of the FAST system started in 1960. CR0 did not assign this equipment to OPTEVFOR for operational evaluation until October 1964; operational testing started in April 1966 and was completed in April 1968. OFTEVFOR recommended that the system not he accepted for operational use until correction of a number of deficiencies. The Navy has since removed or modified most of the equipment developed for this system in order to provide a more reliable andssimpler transfer system. In view of our findings on the FAST system, we broadened our retiew to cover allwstems assigned to OPTEXFORfor operational testing as of September 30, 1970. As of this date, the CNO bad assigned to OPTEVFOR -2. . B-163058 41 items of equipment for operational testing to determine suitability for service use. We requested the Navy to complete a questionnaire on each of these projects to facilitate our evaluation of whether OFTEVFOR was being uti- lized in accordance with its assigned mission. The Navy returned ques- tionnaires for 39 of the 41 projects. While the questionnaires contained certain inconsistencies, we determined the following: No. of projects Contracts for production of items for service use entered into: Before or about the date operational testing was requested After operational testing was requested but before tests were started After operational tests were requested but before the tests were completed k Thus, production commitments had been authorized before the completion of operational test and evaluations for about one-third of these systems. In these cases, OPTEVFORwas determining suitability for service use after procurement commitments had been made. Pertinent data regarding each of the 13. oases summarized'sbove is included in the appendix to this report. These 13 cases involve conditions similar to those discussed in our report to the Congress entitled "Admrse Effects of Large-Scale Produc- tion of Major Weapons Before Completion of Development and Testing" (~-163058 dated November 191 1970). In that report, we pointed out that most of the Navy's major weapons systems were approved for large-scale production before development and testing were completed. The report also commented that (1) when concurrent development and production oc- curred, weapons frequently did not perform as intended resulting in ex- penditures of large sums of money and time to identify and correct defi- ciencies and (2) the deployment of effective weapons may not have been accelerated and, in fact, may have been delayed. We made a limited examination into the 13 cases where production commitments were authorized prior to completion of operational evaluations and found adverse effects similar to those described in the above report. -3- B-163058 In one case, we found that contracts for production of 83 gun pods costing $18.6 million were awarded in December 1964 and June 1965. After units of this equipment were placed in service, various Navy operational commandsreported that the equipmeut was not satisfactory. During OFTEVFOR'sevaluation-- requested by CNOin August 1965 and com- pleted in March 1968--it was also determined that the gun pods were unsatisfactory. In October 1966 CNO directed that no additional units be procured because tbfs equipment could not be considered reliable. With respect to timely utilization of OFTEVFORwe found that de- lays have occurred in the assignment of projects to OPTEVFOR. In addi- tion, we found that delays have occurred in the commencementof tests by OPTEVFORafter projects have been assigned by the 0. We noted, for example, that CPTEVFOR(1) was not requested by the CNOto perform operational tests on one item until about two years after the date of initial procurement action, and (2) did not start actual testing on another item until over four years bad elapsed from the date testing was requested. The appendix to this report illustrates similar delays for other items. AGENCYACTIONS In March 1970, in commenting on the draft of our earlier report (B-163058), the Navy stated it would revise its instructions regarding concurrent development and testing of weapons systems. Subsequently, in an attachment to a memollandumdated December 21, 1970, to the Secre- tary of Defense, the Secretary of the Navy cited certain weaknesses in the conduct of operational tests and evaluatfons. To improve the effec- tiveness of operational tests and evaluations, the Secretary stated that he planned to (1) centralize the test and evaluation forces to achieve a greater depth and variety of analytical capability, data processing facilities, and instrumentation; (2) give OFTEVFORa role earlier in the development process; and (3) strengthen the present system of monttoring the correction of deficiencies revealed during operational evaluations. In February 1971the Office of tbe Secretary of Defense established a position& Deputy Mrector for Test and Evaluation with across-tbe- board responsibilities for the Office of the Secretary of Defense in test and evaluation matters. Also, in March 1971, the Navy established a Di- rector, Research, Development, Test and Evaluation (RD!t%E) within CEO. In May 1971, the Navy established an Assistant Mrector for oT&E and a Test and Evaluation Ditision within the Navy's Office of the Mrector, RlYIi?E. We were fnformed by an OFTEVFORofficial in Juue 1971 that no specific changes had occurred in the conduct of 0P8E at the operating level. At a meeting with officials of the Navy's Office of the Mrector for RlX&E in August 1971, we were informed that certain guidance con- cerning the conduct of operational test and evaluation had recently been -41 B-163058 received from the Secretary of Defense, but in view of the newness of their organization, actions had not yet been taken at the Service head- quarters level to implement these instructions. CONCWTSIONSAND RECOMMENDATIONS MEN?= AVAILABLE The delays in requesting OPECVFORto perform operational tests and in the start of testing indicate a need in cases of urgency (1) to assign higher priorities to operational test and evaluation effort and (2) to emphasize earlier operational evaluation by the independent test agency to permit consideration by the decisionmaker of any rLsks related to large-scale production of urgent requirements before completion of oper- ational testing. We believe that a decision to commence production be- fore completion of testing because of urgent need should be accompanied by an equally urgent effort to complete the required operational evalua- tioas. We also believe that, in cases where urgency may be a factor, there is a need for earlier coordination of the technical and operational eval- uations. In this way, optimum fnformation --concerning system operational effectiveness and the associated risks--may be made available to the de- cisiomnaker earlier in the acquisition cycle. Our major concern in such instances is whether the decisionmaker has sufficient data available to assure and to document that the risks connected with the decision to proceed to full-scale production because of urgent requirements are reduced to acceptable levels so that, in his opinion, the decision will not jeopardize effective operational use of the equipment. In instances wbere the absence of military urgency per- mits the use of initial or ptlot production units to complete realistic operational evaluations, we are also concerned that the initial procure- ment is approved only for the limited quantities needed for a determina- tion of operational suitability. From our experience, it appears that when the approval for large-scale production of an item is made before completion of operational evaluation testing, harmful cost and perform- ance consequences usually occur. Reassignments of responsibility and organizational changes made or now under consideration at the close of our review may result in strengthening controls over the matters noted during our review. How- ever, as of June 1971, some 15 months after the date of the Navy's reply to our earlier draft report (B-163058), we noted no significant changes at the Navy's test and evaluation operating levels. Accordingly, we are recommending that, together with the organi- zational changes, specific steps be taken to assure that (1) initial contracts for production units are awarded only for the minimum quanti- ties needed for operational testing and for any urgent requirements which must be met while operational suitability is being established; -5- . - I . . . ’ B-163058 (2) in cases of essential urgent need, the decisionmaker has s&ficient operational test data to reasonably establish and document that risks resulting from the urgent requfrements are acceptable; and (3) OPTEVFOR be directed to proceed with the required tests on a priority basis. In this regard, we further recommend that OPTEVFOR be required to monitor the development of systems and components requiring operational evalua- tions in order to better plan ORTEVFOR'sfuture workload and to arrange for testing at the earliest possible date, parbicularly when urgency is a compelling factor. We would appreciate your comments and advice of any specific ac- tions planned or being t&ten to improve the utilization of operational test and evaluations within the Navy. If you or your representatives wish to discuss these matters or require additional information, please contact Mr. Harold E. Rubin, Associate Director, code 129, extension 4515 l Since this report contains recommendations for your consideration, \_ _ copfes are being sent to the Appropriations and Government Operations ' Comm$ttees of both Houses of the Congress under the provisions of Set- :j ' tfon 236 of the Legislative Reorganization Act of 1970. We will appre- ciate receiving copies of the statements you furnish the specified Committees in accordance with these provisions. Copies of this letter are also being sent to the Mrector of De- fense Research aad Engineering and the Secretary of the Navy for their information. Sincerely yours, Mrector - 6- -- APPENDIX1 ~ARTMEXQWTREMAW OPERATIONAL TEST AND EVALUATION FORCE Schedule of Pertinent Data for Projects Sbowlng Procurements Before Completion of Operational Test and Evaluation as of March 31, 1971 Date of develop- ing agency's Date assigned Testing by OFTETJFOR Case request for to OPTEXFOR Date Date pate procured number evaluation by CNO started completed for service use 6,‘28/6 8/26/65 g/10/6 3/12/aa '$$!i~e 10/30/66 11/23/66 3/29/7l - 3/9+/67 5/z/67 x0/67 - 11/13/67 3/2w3 3/12/f@ - l;;:gd 12/7/67 e/19/68 5/a - 7/25/ad 7/15/a 2/70 3/70b g/13/@ 8/5/a '$if$ 81433 7/70c 9/27/Bd 5/22/69 7/28/69 B/31/70 w70 FY 69 6/24/e lOl3a9 5/70 2/71 11/19/68d l/15/70 4/10/70 10/7/70 2/5/70 5/25/70 9/29/70 3/10/70 '+/22/70 2/71 - FY 71e 6/x2/70 9/16/w 3/5/71 - 8/22/@ aOpen end project to provide for testing of additional components as needed. bAdditional component of system to he tested. 'Items to be tested on supplemental delivery vehicles. dMultiple procurement contracts awarded commencing with thekitial procurement date shown. eExact dates not shown in questionnaires completed by the Navy.
Review of the Operational Test and Evaluation of the Fast Automatic Shuttle Transfer System and Other Systems
Published by the Government Accountability Office on 1971-09-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)