oversight

Chemical and Biological Defense: Program Planning and Evaluation Should Follow Results Act Framework

Published by the Government Accountability Office on 1999-08-16.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                   United States General Accounting Office

GAO                Report to Congressional Requesters




August 1999
                   CHEMICAL AND
                   BIOLOGICAL
                   DEFENSE

                   Program Planning and
                   Evaluation Should
                   Follow Results Act
                   Framework




GAO/NSIAD-99-159
United States General Accounting Office                                                              National Security and
Washington, D.C. 20548                                                                        International Affairs Division



                                    B-282699                                                                                    Letter

                                    August 16, 1999

                                    The Honorable Robert C. Byrd
                                    Ranking Minority Member
                                    Committee on Appropriations
                                    United States Senate

                                    The Honorable Carl Levin
                                    Ranking Minority Member
                                    Committee on Armed Services
                                    United States Senate

                                    Since the Persian Gulf War, Members of Congress have raised concerns
                                    regarding the adequacy of technology used by the Department of Defense
                                    (DOD) to detect, identify, prepare for, and protect troops against chemical
                                    and biological (CB) weapons.1 In 1993, the National Defense Authorization
                                    Act for Fiscal Year 1994 (P.L. 103-160) directed the Secretary of Defense to
                                    take actions designed to improve the Department’s CB defense capabilities,
                                    including coordination and integration of all CB defense programs into
                                    what is now the CB Defense Program. More recently, concerns that
                                    terrorists might move beyond using conventional weapons to chemical or
                                    biological devices led Congress to authorize the federal government to
                                    improve domestic capabilities to respond to such incidents. With the
                                    initiation of these domestic preparedness programs in fiscal year 1997,
                                    federal research and development efforts to develop nonmedical
                                    CB defense technology expanded considerably, and they continue to grow.2
                                    According to the White House, the President’s fiscal year 2000 budget
                                    request includes over $10 billion to combat terrorism. Almost $1.4 billion is
                                    for programs specifically aimed at terrorist threats from chemical,
                                    biological, radiological, or nuclear weapons, an amount which exceeds the




                                    1
                                     See Chemical and Biological Defense: Emphasis Remains Insufficient to Resolve Continuing Problems
                                    (GAO/NSIAD-96-103, Mar. 29, 1996) and Chemical Weapons: DOD Does Not Have a Strategy to Address
                                    Low-Level Exposures (GAO/NSIAD-98-228, Sept. 23, 1998).
                                    2
                                     Nonmedical technologies refer to technologies for detecting, identifying, protecting against, or
                                    decontaminating personnel and equipment of chemical and biological agents. By contrast, examples of
                                    medical research and development include the development of prophylactics such as vaccines, medical
                                    diagnostics for determining exposure to chemical or biological agents, and therapeutic drugs or
                                    procedures for countering the effects of exposure.




                   Leter            Page 1                                     GAO/NSIAD-99-159 Chemical and Biological Defense
                      B-282699




                      funding of less than $1 billion for military programs to counter chemical
                      and biological threats.

                      In 1993 Congress enacted the Government Performance and Results Act
                      (commonly referred to as the Results Act). The legislation was designed to
                      have agencies focus on the performance and results of their programs
                      rather than on program activities and resources, as they had traditionally
                      done. Congress sought to shift federal management and oversight from its
                      preoccupation with program staffing, activity levels, and tasks completed
                      to program results—that is, to the real difference that federal programs
                      make in people’s lives. Congressional reports and administrative guidance
                      indicate that programs such as the CB Defense Program should follow the
                      Results Act’s outcome-oriented principles, including the establishment of
                      general goals as well as quantifiable, measurable, outcome-oriented
                      performance goals and related measures.

                      As you requested, we examined the extent to which DOD has applied the
                      Results Act’s outcome-oriented principles to the CB Defense Program,
                      focusing in particular on research, development, testing, and evaluation
                      (RDT&E) activities that lead to new technologies and defensive
                      capabilities. Specifically, we assessed whether (1) CB Defense Program
                      goals are explicit and measurable, (2) the CB Defense Program has
                      performance measures that assess outcomes and impacts rather than
                      outputs and activities, and (3) organizations executing the CB Defense
                      RDT&E activities have incorporated Results Act principles in their program
                      planning and evaluation. A companion report Chemical and Biological
                      Defense: Coordination of Nonmedical Chemical and Biological Research
                      and Development Programs (GAO/NSIAD-99-160, Aug. 16, 1999) examines
                      coordination on nonmedical CB defense research and development
                      programs.



Results in Brief      DOD’s CB Defense Program in general, and its RDT&E activities in
                      particular, have not incorporated key Results Act principles, as evidenced
                      by the fact that the goals of the program are vague and unmeasurable and
                      do not articulate specific desired impacts. Program planners do not
                      explain, for example, the meaning of goals such as denying military
                      advantage or allowing U.S. forces to operate largely unimpeded by
                      chemical and biological attacks. In the absence of explicit and measurable
                      goals, it is difficult to assess whether the program has been successful in
                      achieving its goals.




              Leter   Page 2                          GAO/NSIAD-99-159 Chemical and Biological Defense
             B-282699




             The performance measures of CB Defense Program RDT&E emphasize
             activities rather than impacts. The program is not being evaluated
             according to its impact on the defensive or operational capabilities of
             U.S. forces, either individually or collectively. CB Defense Program
             planners use roadmaps to track program progress toward meeting
             chemical and biological defense goals. These goals frequently take the
             form of advanced concept technology demonstrations.3 However, the
             demonstration of a new defensive technology or capability is not a measure
             of the program’s impact or contribution to the military’s ability to survive,
             fight, and win in chemical and biological environments. For example, these
             technology demonstrations may still need additional engineering and
             manufacturing development or product and concept development, as well
             as successful operational testing, before production begins and warfighters
             are equipped.

             CB Defense Program research and development organizations have
             incorporated Results Act principles inconsistently. Only one organization
             has adopted the Results Act planning and evaluation tools. The remaining
             research and development organizations cited either the utilization of
             equivalent planning tools or the unique challenges of evaluating research
             and development activities as reasons why they had not or could not adopt
             the Results Act processes.

             We are recommending that the Secretary of Defense take actions to
             develop a performance plan for the CB Defense Program based on the
             outcome-oriented management principles embodied in the Results Act.



Background   Consistent with the National Defense Authorization Act for Fiscal Year
             1994,4 the Secretary of Defense assigned responsibility for the overall
             coordination and integration of the CB Defense Program to a single office,
             the Deputy Assistant to the Secretary of Defense for Counterproliferation
             and Chemical/Biological Defense. The office is responsible for approving
             all planning, programming, and budgeting documents, ensuring

             3
              Advanced concept technology demonstrations assess the military utility of mature technologies and
             their capabilities in realistic operational scenarios. CB defense capabilities that have been explored
             through these technology demonstrations include the capability to (1) provide early warning of remote
             biological warfare agents; (2) detect, warn, dewarn, identify, protect, and decontaminate air bases and
             seaports against biological attack; and (3) integrate biological and chemical detection and early
             warning capability at an air base or seaport.
             4
                 P.L. 103-160, sec. 1701.




             Page 3                                      GAO/NSIAD-99-159 Chemical and Biological Defense
B-282699




coordination between the medical and nonmedical CB defense efforts, and
overseeing management oversight in accordance with the law. Several
organizations within DOD conduct RDT&E activities on behalf of the CB
Defense Program. These include the Defense Advanced Research Projects
Agency (DARPA), the Soldier and Biological Chemical Command, and the
Joint Program Office for Biological Defense. In addition, CB defense
research and development is conducted at the national laboratories of the
Department of Energy (DOE), funded both by DOD and DOE.

The program addresses five defensive capabilities, three of which are
nonmedical: contamination avoidance, protection, and decontamination,5
as well as medical chemical defense and medical biological defense. These
areas comprise the framework that DOD uses to formulate nonmedical CB
Defense Program requirements. When doctrinal, training, or organizational
solutions cannot satisfy warfighters’ needs in these areas, DOD seeks new
equipment through the research, development, and acquisition cycle. 6 CB
defense funding is divided between the program’s two primary activities:
RDT&E and procurement. Of the CB Defense Program budget of
$717 million proposed for fiscal year 2000, $340 million (47 percent) would
be for RDT&E and the remaining $377 million (53 percent) for
procurement.

The Results Act is the primary legislative framework through which
agencies, at all levels, are required to set strategic goals, measure
performance, and report on the degree to which goals are met. The
outcome-oriented principles of the Results Act, which Congress
anticipated would be institutionalized and practiced at all organizational
levels in federal agencies, include (1) establishing general goals and
quantifiable, measurable, outcome-oriented performance goals and related
measures; (2) developing strategies for achieving the goals, including
strategies for overcoming or mitigating major impediments to goal
achievement; (3) ensuring that goals at lower organizational levels align
with and support general goals; and (4) identifying the resources that will
be required to achieve the goals.


5
 Contamination avoidance includes detecting, avoiding, and bypassing contaminated areas; protection
consists of individual and collective protection; decontamination is the restoration of combat power
after a CB attack.
6
 DOD categorizes RDT&E into five budget activities: basic research (6.1 account), applied research
(6.2), advanced technology development (6.3), demonstration/validation (6.4), and engineering and
manufacturing development (6.5). DOD refers to “activities 6.1 to 6.3” as Defense Science and
Technology.




Page 4                                     GAO/NSIAD-99-159 Chemical and Biological Defense
B-282699




In its guidance on Results Act implementation, the Chief Financial Officers
Council advised agencies that to comply with the spirit and intent of the
act, the goals and measures used at lower organizational levels should be
linked with the agency’s strategic goals.

The Quadrennial Defense Review (QDR) serves as DOD’s overall strategic
planning document. The QDR directs DOD organizations at all levels to
review their strategic and mission objectives in order to ensure that they
link to the goals and objectives of the QDR and that Results Act
performance plans indicate progress toward meeting QDR goals. DOD
implementing guidance states that the goals, objectives, measures of
success, quantifiable performance measures, and program outcome
evaluations of subordinate organizations should be linked to the DOD
corporate goals articulated in the QDR and made operational in the DOD
performance plan. DOD does not routinely link its performance measures
to specific organizational units or individuals, which have sufficient
flexibility, discretion, and authority to accomplish desired results. DOD’s
performance plan for fiscal year 2000 does not specifically discuss the CB
Defense Program. (CB Defense Program RDT&E activities are aggregated
with those of other modernization activities to support DOD’s second
corporate goal to “prepare now for an uncertain future by pursuing a
focused modernization effort that maintains U.S. qualitative superiority in
key warfighting capabilities.”)

Congressional reports and administrative guidance indicate that programs
such as the CB Defense Program should follow the outcome-oriented
principles of the Results Act. In our assessment of the adherence of the
program to the act, we only evaluate of the first of four principles—
establishing general goals and quantifiable, measurable, outcome-oriented
performance goals, and related measures.7




7
 Tasks required for implementing the first principle include (1) identifying the organization’s mission
and long-term strategic goals, (2) describing how the organization’s annual performance goals are
related to its long-term goals, (3) specifying annual performance goals for each program activity,
(4) identifying the performance measures the organization will use to assess its progress, and
(5) describing how data will be verified and validated.




Page 5                                       GAO/NSIAD-99-159 Chemical and Biological Defense
                          B-282699




Goals of the CB           Although DOD has taken the initial and necessary step of articulating CB
                          Defense Program goals, the goals are not articulated in a manner consistent
Defense Program Are       with Results Act principles. The stated goals are vague and unmeasurable,
Vague and                 and they fail to articulate specific desired impacts. A Results Act
                          framework requires that managers define a related set of long-term
Unmeasurable and Do       strategic goals, annual agency goals, and measurable performance goals for
Not Articulate Specific   each program. The five CB Defense Program goals are to
Desired Impacts
                          1. deter CB weapon use by denying military advantage to an enemy
                          through a combination of avoidance, protection, decontamination, and
                          medical support capabilities, allowing U.S. forces to operate largely
                          unimpeded by chemical and biological attacks and their subsequent
                          effects;

                          2. address the most probable CB weapon threats that could be
                          encountered in regional conflicts and field capabilities to the forces
                          required for two major theater wars;

                          3. ensure the CB weapon Threat Evaluation Projection drives CB defense
                          research, development, and acquisition programs;

                          4. emphasize a joint service approach to CB defense research and
                          development, and acquisition; and

                          5. complete critical RDT&E and acquisition of improved CB detection,
                          identification, and warning systems; individual and collective protection
                          systems; and medical support and decontamination systems.

                          Measuring the first goal is unachievable, determining a deterrence effect is
                          problematic, and attributing the specific rationale for the deterrence is
                          unrealistic. The second, third, and fourth goals address the size, focus, and
                          coordination of the program—not program outcomes. Together, these
                          goals direct that the program be sufficiently large to address the needs
                          resulting from two major theater wars; sufficiently focused to address the
                          likely validated threats; and sufficiently coordinated to capitalize on
                          efficiencies and other benefits of joint requirements determination,
                          research, development, and acquisition. The objective of the fifth goal is
                          measurable but speaks to program outputs without addressing program
                          outcomes or impacts (such as decreased defensive vulnerabilities or
                          increased operational capabilities). The completion of RDT&E or
                          procurement cannot be assumed to result in a positive impact on the




                          Page 6                           GAO/NSIAD-99-159 Chemical and Biological Defense
                       B-282699




                       defensive posture or operational flexibility of U.S. forces. While the
                       completion of these activities may well generate benefits for U.S. troops, in
                       the absence of valid, reliable measures, the contributions of RDT&E or
                       procurement cannot be determined.



CB Defense Program     The CB Defense Program is not evaluated on the impact of its activities on
                       the defensive or operational capabilities of U.S. forces, either individually
Performance Measures   or collectively.
Emphasize Activities
                       CB Defense Program planners use roadmaps, Defense Technology
Rather Than Outcomes   Objectives (DTO) and advanced concept technology demonstrations
and Impacts            (ACTD) to assess progress toward goals. Program planners collectively
                       prepare a number of strategic plans they describe as “in the spirit of the
                       Results Act,” if not specifically for the purpose of assessing outcomes and
                       impacts. For example, DOD’s Nuclear, Biological, and Chemical (NBC)
                       Defense Annual Report to Congress8 and the Joint Service NBC Defense
                       Research, Development, and Acquisition Plan are updated annually and
                       include detailed metrics and time lines reflecting the performance of the
                       program (such as the demonstration of a new technology). Roadmaps
                       track program progress toward DTOs that, when achieved, DOD claims will
                       create new operational capabilities. A number of DTOs are ACTDs, and
                       plans state that technology demonstrations provide a means for the rapid
                       field testing of technical options to solve operational needs. CB Defense
                       Program roadmaps explicitly link the completion of DTOs and ACTDs with
                       an increase in the demonstrated warfighting capabilities of U.S. forces. In
                       addition, CB Defense Program planners cited ongoing programmatic peer
                       reviews, such as Technology Area Reviews and Assessments (TARA), as
                       additional means to measure progress toward meeting program goals.

                       We do not concur that the conduct of an ACTD or a peer review of ongoing
                       work measures the impact of the CB Defense Program on the warfighter.
                       Both measures have limitations that make them inappropriate for
                       appraising progress toward achieving program objectives. ACTDs
                       represent a means for rapidly introducing new technologies and reducing
                       the time from the start of a program to the system’s initial operational
                       capability. However, the demonstration of a new technology may not by
                       itself result in the effective and safe deployment of a military capability in


                       8
                           Submitted to Congress annually pursuant to 50 U.S.C. 1523.




                       Page 7                                        GAO/NSIAD-99-159 Chemical and Biological Defense
                            B-282699




                            support of the warfighter. Moreover, as we previously reported, DOD has
                            not always emphasized the need to complete concept and product
                            development or testing before production, thus increasing the risk of
                            approving ACTDs in support of CB defense that include immature
                            technologies and then prematurely starting production.9


Technology Area Review      TARAs are peer reviews conducted by the Director, Defense Research and
Assessments Are Not a       Engineering on each of DOD’s 12 science and technology programs—one
                            being, CB defense. TARAs address progress toward achieving DTOs and
Measure of Program Impact
                            form the basis of the Results Act rating for DOD’s performance in science
                            and technology. 10 However, the application of the TARA to generate
                            performance measures of DOD’s science and technology programs—such
                            as CB defense—is limited by several factors. First, the scope of the TARA
                            Results Act ratings is limited because TARAs only address DTOs. Funding
                            for DTOs comprises less than 50 percent of total funding for applied and
                            advanced technology research and development. Thus, the Results Act
                            ratings do not capture the majority of the CB Defense Program’s RDT&E
                            activities. Second, the focus of TARAs is on budgets, schedules, and
                            technical performance. TARAs do not measure technology transition from
                            the laboratory to the battlefield. Lastly, TARAs do not measure
                            improvements in the ability of U.S. troops to survive, fight, and win in a CB
                            environment.



CB Defense Program          The organizations that execute or contribute to the research and
                            development goals of the CB Defense Program vary in their use of the
Research and                Results Act principles to plan and assess their activities.
Development
Organizations Have
Incorporated Results
Act Principles
Inconsistently


                            9
                             Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved
                            (GAO/NSIAD-99-4, Oct. 15, 1998).
                            10
                                 In fiscal year 1999, there were 23 chemical and biological defense DTOs, many in the form of ACTDs.




                            Page 8                                        GAO/NSIAD-99-159 Chemical and Biological Defense
                             B-282699




The Soldier and Biological   The Soldier and Biological Chemical Command has demonstrated that the
Chemical Command Is the      Results Act principles can be integrated into the planning and evaluation
                             process of an organization conducting research and development for the
Only RDT&E Organization      CB Defense Program.11 In 1996, the Chemical and Biological Defense
to Systematically Apply      Command (which in fiscal year 1999 merged with the Soldier Systems
Results Act Principles       Command to form the Soldier and Biological Chemical Command)
                             developed a Business Planning Guide aligned with the Results Act
                             guidelines. The guide linked planning guidance of higher headquarters
                             (the Army Materiel Command); the Army’s Planning, Programming,
                             Budgeting, and Execution System; and the Results Act. The guide stated
                             that the Results Act “is the basis for the three products that result from our
                             business planning process outlined in this document: a strategic plan, an
                             annual performance plan, and an annual report.” Subsequently, the
                             Chemical and Biological Defense Command issued strategic plans for the
                             fiscal years 1997–2003 and fiscal years 1998–2004 time frames, annual
                             performance plans for fiscal years 1997 and 1998, and an annual report for
                             fiscal year 1997.

                             The Chemical and Biological Defense Command strategic plan is driven by
                             and linked with the overarching planning architectures of DOD, the Army,
                             and the Army Materiel Command. The Chemical and Biological Defense
                             Command’s strategic planning model directly links the attainment of its
                             vision with the development of goals and enabling strategies—followed by
                             the execution of the strategies and measurement of performance. Separate
                             measures were developed to assess goal achievement as well as progress
                             toward goal achievement.

                             Specifically, the plan identified Chemical and Biological Defense Command
                             goals, strategies for achieving the goals, and measures of goal achievement.
                             Two of the Command’s six goals address their potential contributions to the
                             CB Defense Program. They are to

                             • research, develop, acquire, and field NBC defense, smoke, and
                               obscurant materiel that meets warfighter requirements and reduces
                               acquisition costs and timeliness (i.e., produces products faster and at
                               lower costs) and
                             • become the organization of choice for chemical, biological, and
                               smoke/obscurant research, development, and technology services.


                             11
                                  We did not assess the comprehensiveness, quality, or effectiveness of the command’s effort.




                             Page 9                                         GAO/NSIAD-99-159 Chemical and Biological Defense
                              B-282699




                              The performance plan identifies performance measures for each Command
                              goal and performance goals for each strategy. The performance measures
                              address both accomplishments and progress toward accomplishments.
                              Examples of quantitative measures of research and development
                              accomplishments include (1) the percentage of new NBC systems that
                              meet NBC survivability requirements, (2) the percentage of nonexempt
                              acquisitions receiving waivers from performance specifications, and (3) the
                              percentage of Soldier and Biological Chemical Command science and
                              technology programs transitioning to joint service and Army development
                              programs with user validation through modeling, wargames, or similar
                              methods.

                              Soldier and Biological Chemical Command officials explained that the
                              Command is still evaluating some of the fundamental dilemmas in applying
                              Results Act principles. For example, the Command has yet to agree on
                              what the right measures are. The identification of measures in the research
                              and development component of the Soldier and Biological Chemical
                              Command (and CB defense in general) has been an ongoing challenge and
                              continues to evolve.


Other CB Defense RDT&E        The remaining RDT&E organizations cited a variety of reasons for not
Organizations Have Not        incorporating the Results Act’s principles in their program planning or
                              evaluation systems.12 The two most common reasons cited were that
Applied Results Act           current DOD planning processes were equivalent to those of the act,
Principles                    resulting in plans that were “in the spirit” of the Results Act, and that the
                              unique nature of RDT&E activities did not lend itself to the act’s
                              performance measurement and evaluation.

CB Defense Program Planning   DOD officials explained that the Office of the Secretary of Defense, NBC
and Evaluation                Defense Steering Committee, conducts planning consistent with Results
                              Act principles for the CB Defense Program by issuing and executing the
                              Program Objectives Memorandum. They claimed that
                              memorandum-related planning documents, including reports that address
                              different aspects of the CB Defense Program (such as the Joint Operational
                              Concept; the Research, Development, and Acquisition Plan; the Joint
                              Logistics Support Plan; and the Joint Modernization Plan) as well as the


                              12
                                 Of the remaining principal federal organizations that conduct RDT&E on CB defense topics, only
                              DOE’s CB Nonproliferation Program has developed a strategic plan, and none have developed a
                              performance plan.




                              Page 10                                     GAO/NSIAD-99-159 Chemical and Biological Defense
                       B-282699




                       NBC Defense Annual Report to Congress, constitute the equivalent of a
                       strategic plan. Moreover, CB Defense Program managers stated that DOD’s
                       Planning, Programming, and Budgeting System is equivalent to the system
                       required by the act and that therefore no substantive changes are necessary
                       to comply with the spirit of the legislation. 13

Measurement of RDT&E   There is no consensus on the appropriateness of applying performance
Activities             measures to RDT&E activities. While the Soldier and Biological Chemical
                       Command has developed and applied measures of research and
                       development outcomes, other organizations conducting CB Defense
                       Program RDT&E have not. Neither DARPA, DOE, nor the Joint Program
                       Office for Biological Defense have taken the initiative to develop a
                       performance plan. In its strategic plan, DOE included a 5-year roadmap for
                       developing, demonstrating, and delivering technology that would lead to
                       major improvements in preparedness and capabilities. The Joint Program
                       Office cites the conduct of ACTDs as measures of its performance. DARPA
                       officials maintain that the nature of their RDT&E activities do not lend
                       themselves to the application of performance measurement.

                       DARPA conducts leading-edge research where the risks of failure are high
                       and the probability of success is low. Its mission is to pursue long-term,
                       far-reaching, and high-risk/high-payoff technology for military systems in
                       the distant future. DARPA officials argued that developing useable metrics
                       that are measurable, relevant, and timely for technology anticipated 10 or
                       more years into the future is impossible. Moreover, they stated that goals
                       and expectations are set at the project level and cannot be aggregated at
                       the program level. Therefore, according to the officials, it would be
                       inappropriate to develop programwide or agencywide measures of success
                       or performance.

                       Nonetheless, DARPA did try to develop a performance contract and
                       submitted a draft to the Defense Management Council in early 1998. It also
                       argued, however, that it did not fit the mold of most DOD agencies and
                       should therefore be exempt from the act’s requirements. DARPA drafted
                       performance metrics addressing its research operations as well as
                       administrative efficiency. The performance metrics proposed for the
                       research portion of its activities consisted of a series of assessments


                       13
                          The DOD Comptroller has noted that the Results Act is related to, but distinct from, DOD’s Planning,
                       Programming, and Budgeting System, and has stated that Results Act planning and program evaluations
                       need to be integrated with DOD’s Planning, Programming, and Budgeting System.




                       Page 11                                     GAO/NSIAD-99-159 Chemical and Biological Defense
              B-282699




              performed by an independent panel of experts.14 In December 1998, the
              Defense Management Council notified DARPA that it was exempt from the
              requirement. The draft performance contract was never finalized or
              implemented.

              Congress has recognized that successful implementation on the Results Act
              in science agencies would not come quickly or easily. Nonetheless,
              research organizations have concluded that the Results Act can or should
              be applied. The Research Roundtable, a group of federal researchers and
              managers representing a cross section of departments and agencies,
              concluded in 1995 that the results of a research program’s performance can
              be measured. The Army Research Laboratory was designated as a pilot
              project for performance measurement under the act and ultimately
              outlined an evaluation approach that made use of three pillars: metrics,
              peer review, and customer feedback. In 1999, the Committee on Science,
              Engineering, and Public Policy of the National Academy of Sciences, the
              National Academy of Engineering, and the Institute of Medicine reported
              on the results of their work on the issue of measuring and evaluating
              research in compliance with the act’s requirements. The committee
              concluded that both applied research and basic research programs
              supported by the federal government can be evaluated meaningfully on a
              regular basis.



Conclusions   The CB defense research and development outcomes and impacts are not
              being systematically measured. The CB Defense Program lacks both
              quantifiable performance measures and measurable objectives. In the
              absence of measures of program impacts and measurable objectives,
              progress toward achieving program goals cannot be determined. Program
              planning consists of a series of roadmaps leading to specific equipment
              items. Managers cite activity measures and technology demonstrations as
              measures of the program’s contribution. These planning and programming
              steps are appropriate and necessary, but they are insufficient for
              quantifying outcomes and impacts. Current measures do not assess the
              incremental changes attributable, in whole or in part, to the CB Defense




              14
                 The Defense Science Board was the proposed panel. It would have been tasked to review the
              portfolio of DARPA projects to assess projects with regard to (1) relevance to warfighters, (2) ratio of
              technology investments versus system development, (3) level of risk, (4) ratio of new versus continuing
              efforts, and (5) level of service and commercial sector participation.




              Page 12                                      GAO/NSIAD-99-159 Chemical and Biological Defense
                  B-282699




                  Program that improve warfighters’ ability to survive, fight, and win in a CB
                  environment.

                  The Results Act outcome-oriented principles have not been widely applied
                  by either CB Defense Program planners or executing organizations. The
                  utilization of these principles can enable managers and those overseeing
                  the program to quantify the relative success of the program and of
                  component projects in satisfying requirements across different activities
                  (e.g., point detection, early warning, warning and reporting, and modeling).
                  Impact measures can provide a planning tool to allocate finite CB Defense
                  Program resources among competing sets of unmet requirements.



Recommendations   We recommend that the Secretary of Defense take actions to develop a
                  performance plan for the CB Defense Program based on the
                  outcome-oriented management principles embodied in the Results Act.
                  The plan should be agreed to and supported by the relevant RDT&E
                  organizations and incorporated in DOD’s NBC Defense Annual Report to
                  Congress. Specifically, the plan should

                  • establish explicit and outcome-oriented goals linked to warfighters’
                    ability to survive, fight, and win in a CB environment;
                  • identify quantitative or qualitative performance measures that can be
                    used to assess progress toward goal achievement;
                  • describe how performance data would be validated;
                  • describe how RDT&E activities of participating DOD and non-DOD
                    organizations are coordinated to achieve program goals; and
                  • identify human capital, financial, and resource challenges or external
                    factors that limit the ability of the program to achieve its goals.



Agency Comments   In written comments on a draft of this report, DOD concurred with our
                  recommendation. DOD stated it will develop a strategic plan more closely
                  aligned with the tenets of the Results Act and publish the plan in the next
                  DOD NBC Defense Annual Report to Congress. DOD’s comments are
                  reprinted in appendix I. Our scope and methodology are explained in
                  appendix II, and CB Defense Program RDT&E organizations are described
                  in appendix III.




                  Page 13                          GAO/NSIAD-99-159 Chemical and Biological Defense
B-282699




As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 7 days after its
issue date. At that time, we will send copies to the Honorable William S.
Cohen, Secretary of Defense and other congressional committees. We will
also make copies available to others on request.

If you have any questions regarding this report, please contact me or
Sushil K. Sharma at (202) 512-3092. Key contributors to this assignment
were Jeffrey Harris and Weihsueh Chiu.




Kwai-Cheung Chan
Director, Special Studies and Evaluations




Page 14                           GAO/NSIAD-99-159 Chemical and Biological Defense
Page 15   GAO/NSIAD-99-159 Chemical and Biological Defense
Contents



Letter                                                                                                1


Appendix I                                                                                           18
Comments From the
Department of Defense

Appendix II                                                                                          20
Scope and
Methodology

Appendix III                                                                                         22
Chemical and
Biological Defense
Program Research,
Development, Test, and
Evaluation
Organizations

Figures                  Figure III.1: Primary Planning and Executing Organizations and
                           Programs of DOD’s Chemical and Biological Defense Program
                           Research, Development, Test, and Evaluation                               23

                         Abbreviations

                         ACTD      Advanced Concept Technology Demonstration
                         CB        chemical and biological
                         DARPA     Defense Advanced Research Projects Agency
                         DOD       Department of Defense
                         DOE       Department of Energy
                         DTO       Defense Technology Objective
                         NBC       Nuclear, Biological, and Chemical
                         QDR       Quadrennial Defense Review
                         RDT&E     research, development, testing, and evaluation
                         TARA      Technology Area Review Assessment



                         Page 16                        GAO/NSIAD-99-159 Chemical and Biological Defense
Contents




Page 17    GAO/NSIAD-99-159 Chemical and Biological Defense
Appendix I

Comments From the Department of Defense                                 Appenx
                                                                             Idi




             Page 18     GAO/NSIAD-99-159 Chemical and Biological Defense
Appendix I
Comments From the Department of Defense




Page 19                            GAO/NSIAD-99-159 Chemical and Biological Defense
Appendix II

Scope and Methodology                                                                                             AppenIx
                                                                                                                        Idi




              The scope of our study was limited to the nonmedical research,
              development, testing, and evaluation (RDT&E) activities of the Department
              of Defense’s (DOD)Chemical and Biological (CB) Defense Program. This
              study does not address any classified programs or projects.

              To determine whether the program goals were explicit and measurable and
              whether its performance measures assessed outcomes and impacts rather
              than outputs and activities, we reviewed the legislative record, interviewed
              agency officials, and analyzed program documents. To understand the
              requirements of the Government Performance and Results Act, we
              reviewed the legislation as well as implementation guidance issued by the
              Office of Management and Budget, DOD Comptroller, and the General
              Accounting Office.1 We queried representatives of the Office of the
              Secretary of Defense’s Nuclear, Biological and Chemical (NBC) Defense
              Steering Committee and the executing organizations (i.e., Joint Program
              Office, Soldier and Biological Chemical Command, Defense Advanced
              Research Projects Agency (DARPA)) regarding their familiarity with, and
              utilization of, the Results Act’s performance measurement principles. We
              reviewed strategic and performance plans implementing the act as well as
              documents characterized by DOD officials as complying with the “spirit of
              the legislation.”2

              To determine whether organizations executing the CB Defense RDT&E
              activities have incorporated Results Act principles in their program
              planning and evaluation, we interviewed program officials, examined
              program documents, observed program review meetings, and attended
              scientific conferences on CB defense technologies. Program documents
              we examined included program budgets, strategic and performance plans,
              annual reports, internal program planning documents, program briefings,
              and proceedings of program review meetings. We also observed the 1999
              DOD Technology Area Review and Assessment of chemical and biological
              defense. Scientific conferences we attended included the 1998 Joint
              Workshop on Standoff Detection for Chemical and Biological Defense and
              the 1998 Scientific Conference on Chemical and Biological Defense

              1
                See Executive Guide: Effectively Implementing the Government Performance and Results Act
              (GAO/GGD-96-118, June 1996); Agencies’ Annual Performance Plans Under the Results Act: An
              Assessment Guide to Facilitate Congressional Decisionmaking (GAO/GGD/AIMD-10.1.18, February
              1998); and The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans
              (GAO/GGD-10.1.20, April 1998).
              2
               These include the DOD NBC Defense Annual Report to Congress (submitted pursuant to 50 CFR
              1523), the Joint NBC Modernization Plan, the Joint Service NBC Defense Research, Development, and
              Acquisition Plan, and the Joint Service NBC Defense Logistics Plan.




              Page 20                                   GAO/NSIAD-99-159 Chemical and Biological Defense
Appendix II
Scope and Methodology




Research. We also obtained proceedings from these and other scientific
conferences.

To respond to all three objectives, we contacted the following
organizations: DARPA, Arlington, Virginia; Defense Threat Reduction
Agency, Dulles, Virginia; DOD Inspector General, Washington, D.C.;
Department of Energy, Washington, D.C.; Director, Defense Research and
Engineering, Washington, D.C.; Dugway Proving Ground, Dugway, Utah;
Edgewood Chemical and Biological Center, Aberdeen Proving Ground,
Maryland; Joint Program Office for Biological Defense, Falls Church,
Virginia; National Domestic Preparedness Office, Washington, D.C.;
National Ground Intelligence Center, Charlottesville, Virginia;
Nonproliferation and National Security Office, Department of Energy,
Washington, D.C.; Office of Science and Technology Policy, White House,
Washington, D.C.; Office of the Secretary of Defense, Washington, D.C.;
U.S. Army Soldier and Biological Chemical Command, Aberdeen Proving
Ground, Maryland; and U.S. Army Training and Doctrine Command, Fort
Monroe, Virginia.

We conducted our review from November 1998 to April 1999 in accordance
with generally accepted government auditing standards.




Page 21                        GAO/NSIAD-99-159 Chemical and Biological Defense
Appendix III

Chemical and Biological Defense Program
Research, Development, Test, and Evaluation
Organizations                                                                                                         AppeInx
                                                                                                                            Idi




               The CB Defense Program is overseen by the Office of the Secretary of
               Defense’s Nuclear, Biological, and Chemical Defense Steering Committee,
               which is comprised of the Directors of the Defense Threat Reduction
               Agency and Defense Research and Engineering as well as their top officials
               responsible for CB defense. The steering committee funds research and
               development at numerous laboratories in DOD, Department of Energy, and
               private industry.1

               As illustrated in figure III.1, key research and development organizations in
               the execution of the program include the SBCCOM, JPO-BD, DARPA.




               1
                The CB Defense Program addresses nonmedical research and development in the areas of chemical
               detection, biological detection, individual protection, collective protection, decontamination, modeling
               and simulation, core science and technology, and basic research. In addition, core science and
               technology includes threat assessment and aerosol technology; and basic research includes aerosol
               science, chemistry and toxicology, and analytical chemistry.




               Page 22                                      GAO/NSIAD-99-159 Chemical and Biological Defense
                                           Appendix III
                                           Chemical and Biological Defense Program
                                           Research, Development, Test, and Evaluation
                                           Organizations




Figure III.1: Primary Planning and Executing Organizations and Programs of DOD’s Chemical and Biological Defense Program
Research, Development, Test, and Evaluation


                                     Department of                                           Department of
                                       Defense                                                  Energy




                                  CB Defense Program                                      CB Nonproliferation
                                                                                           CB Nonproliferation
                                                                                              Program
                                                                                               Program
                  Defense Threat                     Director of Defense
                    Reduction                          Research and
                     Agency                              Engineering

                            NBC Defense Steering Committee

                    Director of                           DATSD
                   CB Defense                        Counterproliferation
                   Directorate                        and CB Defense



                                             $                                                       $




                                                                                              Department
                SBCCOM                     JPO-BD                      DARPA                   of Energy
                                                                                                  Labs


          DATSD       =   Deputy Assistant to the Secretary of Defense
          SBCCOM      =   Soldier and Biological Chemical Command
          JPO-BD      =   Joint Program Office for Biological Defense
          DARPA       =   Defense Advanced Research Projects Agency
                      =   Funding flows
                      =   Informal coordination




                                           SBCCOM is organized around two integrated business areas, one of which
                                           is research, development, and acquisition. Nearly half of SBCCOM
                                           research, development and acquisition funding supports the CB Defense
                                           Program. SBCCOM is engaged in the full range of research and
                                           development encompassing both biological and chemical systems.




                                           Page 23                               GAO/NSIAD-99-159 Chemical and Biological Defense
                   Appendix III
                   Chemical and Biological Defense Program
                   Research, Development, Test, and Evaluation
                   Organizations




                   SBCCOM business areas include chemical detection, biological detection,
                   decontamination, protection, and supporting science and technology.

                   JPO-BD was created in 1994 to manage the biological warfare agent
                   detection program. The office monitors emerging technologies for
                   advanced development, demonstration, and upgrades of fielded biological
                   detection systems.

                   The DARPA Biological Warfare Defense Program is an applied research
                   program established by the National Defense Authorization Act for Fiscal
                   Year 1997 (P.L. 103-160, as amended) to fund an applied research program
                   supporting revolutionary new approaches to biological warfare defense.
                   The Biological Warfare Defense Program pursues high-risk, high-potential
                   technologies from the demonstration of technical feasibility through the
                   development of prototype systems.

                   DOE’s CB Nonproliferation Program was established in 1997 in response to
                   the Defense Against Weapons of Mass Destruction Act of 1996, which
                   appropriated $17 million to DOE to conduct research and development to
                   develop new means for detecting the presence, transportation, production,
                   and use of weapons of mass destruction and related materials and
                   technologies. According to DOE, the purpose of this appropriation was to
                   ensure the full engagement of DOE national laboratories in responding to
                   the threat posed by CB weapons to U.S. civilians. DOE funds research and
                   development, from basic research to fieldable prototypes, in pursuit of
                   advanced technologies that can enable first responders to more effectively
                   prepare and respond to the use of CB agents.




(713039)   Ltert   Page 24                               GAO/NSIAD-99-159 Chemical and Biological Defense
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made
out to the Superintendent of Documents, when necessary, VISA and
MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any list
from the past 30 days, please call (202) 512-6000 using a touchtone
phone. A recorded menu will provide information on how to obtain
these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Correction Requested