Test and Evaluation: Improvements Are Being Made in the Department of Defense's Test Planning

Published by the Government Accountability Office on 1990-09-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

q                                                                                                                                                    ‘F
I .-..
              _..-_..-.. __.....__
           _..__.                .I._....                  Urlitrvi
                                           ---.. - -..-...,__.
                                     - .._-..                       Stirks
                                                            _._____-.._       (;c~rrttrul
                                                                    --_--_-____-                          Aworlnt~itlg   Of’fiw     I_-_--   --..-

1     (-$t!&-)
                                                                                 IZt:port to the Ranking Minority
31                                                                               Medxr, Chnmittee on Chvemnrtntal
I                                                                                AfTairs, U.S. Senate

      ,.““”    .,   .   ,.,   .*_I   .   I   .I-_   II   .   “I_.   I-----.--

      St~ptcwrlM~r lW0
                                                                                 TEST AND

                                                                                 Improvements Are
                                                                                 Being Made in the
                                                                                 Department of
                                                                                 Defense’s Test

                                                                                RESTRICTED--      Not to be released outside the
                                                                                General Accounting Offlce unless speck&ally
                                                                                approved by the Of’fke of Congressional

_.__ .._. -..   ..I-_   .._ _.-_._   ----.-   .   - .._._._ l_..“l ..-..---..-._   “.. ..--.-.-   - l...ll--.-.   ---_-----   -ll--“-.-
                                                                                                                                          I--   ,,
                       United States
GAO                    General Accounting Office
                       Washington, D.C. 20648

                       National Security and
                       International Affairs Division


                       September 28,1QQ0

                       The Honorable William V. Roth, Jr.
                       Ranking Minority Member
                       Committee on Governmental Affairs
                       United States Senate

                       Dear Senator Roth:

                       At your request, we reviewed the Department of Defense’s (DOD) test
                       planning processes focusing on the Test and Evaluation Master Plan
                       (TEMP) process. Our objectives were to review (1) those processes that
                       the services have instituted for creating, evaluating, and approving
                       TEMPS; (2) the strengths and weaknesses of the services’ test planning
                       and TEMP processes; and (3) the extent to which recent and ongoing DOD
                       actions address TEMP weaknesses that have been of particular concern
                       (e.g., lack of timeliness, inadequate test schedules, insufficient test
                       resources, etc.).

                           has focused a great deal of attention on improving test planning
Results in Brief       DOD
                       processes and has made several improvements in TEMP guidance. More
                       specifically, our review indicated the following:

                   . The military services have each instituted similar TEMP and test planning
                     processes in accordance with Office of the Secretary of Defense (OSD)
                   l DOD has adequately identified the key weaknesses and strengths of the
                     TEMP and test planning processes.
                   . DOD actions address historical weaknesses and, if properly implemented,
                     should significantly improve the timeliness and quality of TEMPS and the
                     test planning processes.

                       In the process of acquiring weapons systems, it is axiomatic that test
Background             planning should start early, usually at the initiation of a major defense
                       program. Moreover, this planning should provide the basis for identi-
                       fying required test resources and test schedules.

                       Test and evaluation is an integral part of the systems acquisition pro-
                       cess. Within DOD, the basis for all test and evaluation requirements
                       related to a particular system acquisition program is the TEMP. This key

                       Page1                                    GAO/NSIAD-90.303DefenseTestPlanning

                        management control document is required to be used by OSD and all mili-
                        tary service components in planning, reviewing, and approving weapon
                        system test and evaluation.

                        The TEMP is the primary means by which OSD and the military services
                        determine the adequacy of test planning. The TEMP also defines perform-
                        ance criteria that a program is to meet before it can advance to the next
                        acquisition milestone. Department of Defense Directive 5000.3, Test and
                        Evaluation, requires that each major system acquisition program have
                        an approved TEMP before the start of any testing.

GAO’s Analysis

Military Service TEMP   Each service structures its acquisition process based on guidance in DOD
ProcessesAre Similar    directives and instructions. There are few, if any, significant differences
                        in the way the services have implemented OSD guidance concerning
                        TEMPS. For example, in each service the TEMP is prepared by the program
                        management office responsible for developing acquisition and develop-
                        ment testing plans. Operational test information contained in the TEMP is
                        provided by the military services’ operational test agencies.

                        As needed, each service uses similar test coordination and working
                        groups to execute the TEMP and test planning processes. Also, the OSD
                        and service TEMP guidance require that certain potential problem areas
                        (test resource sufficiency, test schedules, threat realism, etc.) be

DOD Has Enumerated      We found that, as part of self-initiated efforts to improve their TEMP and
TEMP Weaknesses         test planning processes, OSD and the services have identified the key
                        weaknesses. A major concern is a lack of TEMP timeliness throughout
                        DOD. For example, in January 1990, the Under Secretary of Defense for
                        Acquisition, commenting on disapproved TEMPS for 22 major systems
                        acquisition programs, stated that “ . ..TEMPS seem to be chronically out of
                        date, awaiting revisions, or involved in lengthy review and approval
                        cycles....” Other TEMP weaknesses identified by DOD include insufficient
                        resources planned for tests, inadequate threat realism, and poorly
                        written critical operational issues.

                        Page 2                                   GAO/NSIAIMO-303 Defense Test Planning

                            Although we found no criteria concerning how long each phase of the
                            TEMP coordination and review cycle should be, several DOD officials told
                            us that a lack of timeliness was the predominant weakness in the TEMP
                            process. They indicated that OSD TEMP review and service TEMP coordina-
                            tion cycles were too long and, as a result, the TEIMP was almost always
                            out of date and lagged behind detailed test planning. There has also been
                            a lack of consensus concerning the appropriate role, definition, and level
                            of detail for the TEMP. For example, the TEMP'S utility as a management
                            oversight tool or working-level document and the value added by the
                            various review and approval levels are still issues of concern to some
                            DOD officials. Resolving this lack of consensus remains a joint OSD and
                            service goal.

Strength of the TEMP        The TEMP process in each of the services assures that the expertise and
ProcessIs the Involvement   perspectives of relevant organizations are utilized in test planning, pri-
                            marily through test coordination and working groups. A key strength of
of Key Organizations        the process is the formality and discipline of this involvement, which is
                            accepted as valid and useful throughout DOD. Some DOD officials believe
                            that the true value of the TEMP process is the planning involved -not
                            the detailed, approved TEMP document. Also, in advance, the TEMP and
                            test planning processes establish expectations regarding responsibilities,
                            required test resources, and organizational involvement.

DOD Has Taken Action        DOD  has taken actions that should address identified weaknesses and
Improve Test Planning       improve TEMPS and related test planning. In particular, DOD has focused
                            attention on improving the timeliness of the TEMP process as a key goal.
                            Specific actions include: (1) revision of guidelines and (2) establishment
                            of a review of the TEMP process. As a result, 0s~'~ TEMP review cycle has
                            improved significantly during the past year. For example, DOD reported
                            that in the first quarter of fiscal year 1990, the OSD TEMP review cycle
                            was 90 days. In the second quarter, OSDtook an average of 45 days to
                            process the TEMPS.

Recently Issued TEMP        In 1988, OSD initiated a field survey to determine what needed to be done
Guidelines Are a Good       to improve the usefulness and timeliness of TEMPS. Based primarily on
                            the results of the survey, new guidelines were issued in January 1990.
Start                       The revised guidelines refer to existing DOD guidance for details
             u              regarding TEMP creation and evaluation.

                            Page 3                                   GAO/NSLAD-90-303 Defense Test Planning
                                These revised guidelines target many of the issues of concern to the
                                Congress and DOD, including timeliness, test resource sufficiency, and
                                adequacy of test schedules. For example, to improve the timeliness of
                                TJZMP creation, evaluation, and approval, the guidance (1) limits the basic
                                TEMP document to 30 pages of text and (2) requires a new integrated test
                                schedule supporting the acquisition strategy that relates resources to
                                test schedules.

                                A significant area in the revised guidelines requires a discussion of the
                                effects of test resource shortfalls on the assessment of test results. We
                                believe this should provide decisionmakers with a clear idea of the risks
                                and impacts associated with failure to provide adequate test resources.

Current Improvement             In January 1990, the Under Secretary of Defense for Acquisition, con-
Actions Should Increase         cerned about continuing TEMP timeliness and content problems, initiated
                                another osu-level review of the TEMP cycle. The goal of this review was
the Effectiveness of Test       to determine how the TEMP and test and evaluation processes could be
Planning                        improved. The Director, Operational Test and Evaluation, was directed
                                to coordinate this review.

                                Interim and final reports were issued in March and May 1990, respec-
                                tively. In the process, many of the weaknesses identified by OSD'S 1988
                                field survey were revalidated. The interim report concluded that the
                                revised TEMP guidance issued in January 1990 was needed and that the
                                TEMP needs to be tied to other key program decision documents, such as
                                the Mission Need Statement, Baseline Document, and Cost and Opera-
                                tional Effectiveness Analysis (see glossary).

                                The study concluded that this action could reduce the primary reasons
                                OSD rejects TEMPS,namely (1) insufficient definition of the mission at the
                                systems level and (2) inadequate specification or representation of the
                                operational environment in the test plan.

                                The final report highlighted that each of the military services has an
                                ongoing TEMP process improvement initiative and discussed overall plans
                                to improve TEMP and test planning processes. The report called for

                            .   TEMPupdates based on test events rather than on other acquisition
                            . early osD-approved TEMPS and a Test Concept Brief before a system
                              acquisition request for proposal is made,

                                Page4                                             GAO/NSL4IMW-303Defe~eTestPlannhg

                                                                    ..‘?‘;   ,.    ,,,
    1   .


              l use of the TEMP as a contract (e.g., an agreement between OSD and the
                services, as well as among organizations within the services), and
              . definition of the level of detail appropriate for the TEMP.

                  These key initiatives cited in the interim and final test and evaluation
                  improvement reports have not yet been implemented. Recently revised
                  acquisition directives have not yet been approved, and a policy state-
                  ment describing how to better integrate the TEMP with other key pro-
                  gram decision documents has not yet been issued.

                  To review the TEMP and test planning process, we looked at records and
Scopeand          interviewed officials at test and program oversight offices within OSD,
Methodology       service headquarters, test and evaluation agencies, and program execu-
                  tive and management offices. We analyzed OSD'S and the services’ TEMP
                  guidance and related documents, searched the relevant literature, and
                  synthesized the results of past and ongoing work by DOD.

                  We compared and contrasted those processes that the services have
                  instituted for creating, evaluating, and approving TEMPS; examined the
                  strengths and weaknesses of the services’ TEMP and test planning
                  processes; and examined the extent to which recent and ongoing WD
                  actions address TEMP and test planning weaknesses.

                  We conducted our work between January and September 1990 in accor-
                  dance with generally accepted government auditing standards.

                  As requested, we did not obtain written agency comments on this report.
                  However, we discussed the information in this report with officials from
                  the Office of the Deputy Director, Defense Research and Engineering
                  (Test and Evaluation), and the Office of the Director, Operational Test
                  and Evaluation. They supported our findings, and we incorporated their
                  specific comments in the report where appropriate.

                  As agreed with your office, unless you publicly announce its contents
                  earlier, we plan no further distribution of this report until 30 days from
                  the date of this letter. At that time, we will send copies of this report to
                  the Secretary of Defense and the Secretaries of the Army, the Navy, and
                  the Air Force; the Deputy Director, Defense Research and Engineering
                  (Test and Evaluation); the Director, Operational Test and Evaluation;
                  the Director, Office of Management and Budget; and other interested

                  Page 5                                    GAO/NSlAD-M3    Defense Test Plaming
Please contact me at (202) 276-8400 if you or your staff have any ques-
tions. Major contributors to this report are listed in appendix I.

Sincerely yours,

Paul F. Math
Director, Research, Development,
  Acquisition, and Procurement Issues

Page 6                                  GAO/NSIAtMO-303 Defense Test Planning
Page 7   GAO/NSIAD-90-303 Defense Test Planning
Appendix I                                                                                            4
Major Contributors to This Report

                                           ‘y, Associate Director
National Security and   ~~~~$~~~o’     ---,_------_----
                                      @t.nn             - ----__
                                            Assist,ant, Director
International Affairs   Anne W.’Howe, Evaluator
Division, Washington,
                        Fred Harrison,     Regional Management Representative
Norfolk Regional        Clifton Spruill,   Evaluator-in-Charge
Office                  Robert Floren,     Evaluator
                        Dawn Roberts,      Evaluator

                        Ted Baird, Regional Management Representative
Denver Regional         Ernest Beran, Regional Assignment Manager

                        Page 8                                      GA0/N?3IAD90-303 Defense Test Plamhg
Baseline Document             For full-scale development programs, description of the system require-
                              ments, the unit cost, the total development cost, and the milestone
                              schedule. For production programs, description of the system require-
                              ments, the total program cost, the cost profile, and the production

Cost and Operational          Assessment of program operational effectiveness against mission areas.
Effective Analysis

Critical Operational Issues   Aspects of a system’s operational capability that must be questioned
                              before its operational effectiveness can be properly evaluated.

Mission Need Statement        Description of the need for a new major weapon system and its use to
                              support the services’ initial request for funds.

(2962se)                      Page 9                                  GAO/NSIAD-90303 Defense Test Planning
..I ..._.-._.._
              .:. .. .” - - -..__-...
                                 I- __^.
                                      - ..-._   -1 __.__- - -_--___..                             ------_..         -_-~--

                                                           Ordering     Information

                                                           The first five copies of each GAO report are free. Additional   copies
                                                           are $2 each. Orders should be sent to the following ;tddrthss, xcom-
                                                           panied by a check or money order made out to the Sup~riIlt,end~rIt.
                                                           of Docnments, when necessary. Orders for 100 or more copies to be
                                                           mailed to 21single address are discounted 25 percen 1,.

                                                           IIS. General Accounting Office
                                                           I’.<). Hox 60 16
                                                           Gaitht~rsburg, MD 20877

                                                           Orders may also be placed by calling    (202) 2756241.

I” “1”1   .“, I   .“.__..   .” I   I ..lllll_   “.ll.. ...”   “. .I..   . . -.., .l...l.ll” -“-” -.-... “..ll_“_~““--._“.l--.“-~-----~~   ~-   ‘+-