United States General Accounting Office GAO Report to Congressional Requesters August 1999 CHEMICAL AND BIOLOGICAL DEFENSE Program Planning and Evaluation Should Follow Results Act Framework GAO/NSIAD-99-159 United States General Accounting Office National Security and Washington, D.C. 20548 International Affairs Division B-282699 Letter August 16, 1999 The Honorable Robert C. Byrd Ranking Minority Member Committee on Appropriations United States Senate The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate Since the Persian Gulf War, Members of Congress have raised concerns regarding the adequacy of technology used by the Department of Defense (DOD) to detect, identify, prepare for, and protect troops against chemical and biological (CB) weapons.1 In 1993, the National Defense Authorization Act for Fiscal Year 1994 (P.L. 103-160) directed the Secretary of Defense to take actions designed to improve the Department’s CB defense capabilities, including coordination and integration of all CB defense programs into what is now the CB Defense Program. More recently, concerns that terrorists might move beyond using conventional weapons to chemical or biological devices led Congress to authorize the federal government to improve domestic capabilities to respond to such incidents. With the initiation of these domestic preparedness programs in fiscal year 1997, federal research and development efforts to develop nonmedical CB defense technology expanded considerably, and they continue to grow.2 According to the White House, the President’s fiscal year 2000 budget request includes over $10 billion to combat terrorism. Almost $1.4 billion is for programs specifically aimed at terrorist threats from chemical, biological, radiological, or nuclear weapons, an amount which exceeds the 1 See Chemical and Biological Defense: Emphasis Remains Insufficient to Resolve Continuing Problems (GAO/NSIAD-96-103, Mar. 29, 1996) and Chemical Weapons: DOD Does Not Have a Strategy to Address Low-Level Exposures (GAO/NSIAD-98-228, Sept. 23, 1998). 2 Nonmedical technologies refer to technologies for detecting, identifying, protecting against, or decontaminating personnel and equipment of chemical and biological agents. By contrast, examples of medical research and development include the development of prophylactics such as vaccines, medical diagnostics for determining exposure to chemical or biological agents, and therapeutic drugs or procedures for countering the effects of exposure. Leter Page 1 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 funding of less than $1 billion for military programs to counter chemical and biological threats. In 1993 Congress enacted the Government Performance and Results Act (commonly referred to as the Results Act). The legislation was designed to have agencies focus on the performance and results of their programs rather than on program activities and resources, as they had traditionally done. Congress sought to shift federal management and oversight from its preoccupation with program staffing, activity levels, and tasks completed to program results—that is, to the real difference that federal programs make in people’s lives. Congressional reports and administrative guidance indicate that programs such as the CB Defense Program should follow the Results Act’s outcome-oriented principles, including the establishment of general goals as well as quantifiable, measurable, outcome-oriented performance goals and related measures. As you requested, we examined the extent to which DOD has applied the Results Act’s outcome-oriented principles to the CB Defense Program, focusing in particular on research, development, testing, and evaluation (RDT&E) activities that lead to new technologies and defensive capabilities. Specifically, we assessed whether (1) CB Defense Program goals are explicit and measurable, (2) the CB Defense Program has performance measures that assess outcomes and impacts rather than outputs and activities, and (3) organizations executing the CB Defense RDT&E activities have incorporated Results Act principles in their program planning and evaluation. A companion report Chemical and Biological Defense: Coordination of Nonmedical Chemical and Biological Research and Development Programs (GAO/NSIAD-99-160, Aug. 16, 1999) examines coordination on nonmedical CB defense research and development programs. Results in Brief DOD’s CB Defense Program in general, and its RDT&E activities in particular, have not incorporated key Results Act principles, as evidenced by the fact that the goals of the program are vague and unmeasurable and do not articulate specific desired impacts. Program planners do not explain, for example, the meaning of goals such as denying military advantage or allowing U.S. forces to operate largely unimpeded by chemical and biological attacks. In the absence of explicit and measurable goals, it is difficult to assess whether the program has been successful in achieving its goals. Leter Page 2 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 The performance measures of CB Defense Program RDT&E emphasize activities rather than impacts. The program is not being evaluated according to its impact on the defensive or operational capabilities of U.S. forces, either individually or collectively. CB Defense Program planners use roadmaps to track program progress toward meeting chemical and biological defense goals. These goals frequently take the form of advanced concept technology demonstrations.3 However, the demonstration of a new defensive technology or capability is not a measure of the program’s impact or contribution to the military’s ability to survive, fight, and win in chemical and biological environments. For example, these technology demonstrations may still need additional engineering and manufacturing development or product and concept development, as well as successful operational testing, before production begins and warfighters are equipped. CB Defense Program research and development organizations have incorporated Results Act principles inconsistently. Only one organization has adopted the Results Act planning and evaluation tools. The remaining research and development organizations cited either the utilization of equivalent planning tools or the unique challenges of evaluating research and development activities as reasons why they had not or could not adopt the Results Act processes. We are recommending that the Secretary of Defense take actions to develop a performance plan for the CB Defense Program based on the outcome-oriented management principles embodied in the Results Act. Background Consistent with the National Defense Authorization Act for Fiscal Year 1994,4 the Secretary of Defense assigned responsibility for the overall coordination and integration of the CB Defense Program to a single office, the Deputy Assistant to the Secretary of Defense for Counterproliferation and Chemical/Biological Defense. The office is responsible for approving all planning, programming, and budgeting documents, ensuring 3 Advanced concept technology demonstrations assess the military utility of mature technologies and their capabilities in realistic operational scenarios. CB defense capabilities that have been explored through these technology demonstrations include the capability to (1) provide early warning of remote biological warfare agents; (2) detect, warn, dewarn, identify, protect, and decontaminate air bases and seaports against biological attack; and (3) integrate biological and chemical detection and early warning capability at an air base or seaport. 4 P.L. 103-160, sec. 1701. Page 3 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 coordination between the medical and nonmedical CB defense efforts, and overseeing management oversight in accordance with the law. Several organizations within DOD conduct RDT&E activities on behalf of the CB Defense Program. These include the Defense Advanced Research Projects Agency (DARPA), the Soldier and Biological Chemical Command, and the Joint Program Office for Biological Defense. In addition, CB defense research and development is conducted at the national laboratories of the Department of Energy (DOE), funded both by DOD and DOE. The program addresses five defensive capabilities, three of which are nonmedical: contamination avoidance, protection, and decontamination,5 as well as medical chemical defense and medical biological defense. These areas comprise the framework that DOD uses to formulate nonmedical CB Defense Program requirements. When doctrinal, training, or organizational solutions cannot satisfy warfighters’ needs in these areas, DOD seeks new equipment through the research, development, and acquisition cycle. 6 CB defense funding is divided between the program’s two primary activities: RDT&E and procurement. Of the CB Defense Program budget of $717 million proposed for fiscal year 2000, $340 million (47 percent) would be for RDT&E and the remaining $377 million (53 percent) for procurement. The Results Act is the primary legislative framework through which agencies, at all levels, are required to set strategic goals, measure performance, and report on the degree to which goals are met. The outcome-oriented principles of the Results Act, which Congress anticipated would be institutionalized and practiced at all organizational levels in federal agencies, include (1) establishing general goals and quantifiable, measurable, outcome-oriented performance goals and related measures; (2) developing strategies for achieving the goals, including strategies for overcoming or mitigating major impediments to goal achievement; (3) ensuring that goals at lower organizational levels align with and support general goals; and (4) identifying the resources that will be required to achieve the goals. 5 Contamination avoidance includes detecting, avoiding, and bypassing contaminated areas; protection consists of individual and collective protection; decontamination is the restoration of combat power after a CB attack. 6 DOD categorizes RDT&E into five budget activities: basic research (6.1 account), applied research (6.2), advanced technology development (6.3), demonstration/validation (6.4), and engineering and manufacturing development (6.5). DOD refers to “activities 6.1 to 6.3” as Defense Science and Technology. Page 4 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 In its guidance on Results Act implementation, the Chief Financial Officers Council advised agencies that to comply with the spirit and intent of the act, the goals and measures used at lower organizational levels should be linked with the agency’s strategic goals. The Quadrennial Defense Review (QDR) serves as DOD’s overall strategic planning document. The QDR directs DOD organizations at all levels to review their strategic and mission objectives in order to ensure that they link to the goals and objectives of the QDR and that Results Act performance plans indicate progress toward meeting QDR goals. DOD implementing guidance states that the goals, objectives, measures of success, quantifiable performance measures, and program outcome evaluations of subordinate organizations should be linked to the DOD corporate goals articulated in the QDR and made operational in the DOD performance plan. DOD does not routinely link its performance measures to specific organizational units or individuals, which have sufficient flexibility, discretion, and authority to accomplish desired results. DOD’s performance plan for fiscal year 2000 does not specifically discuss the CB Defense Program. (CB Defense Program RDT&E activities are aggregated with those of other modernization activities to support DOD’s second corporate goal to “prepare now for an uncertain future by pursuing a focused modernization effort that maintains U.S. qualitative superiority in key warfighting capabilities.”) Congressional reports and administrative guidance indicate that programs such as the CB Defense Program should follow the outcome-oriented principles of the Results Act. In our assessment of the adherence of the program to the act, we only evaluate of the first of four principles— establishing general goals and quantifiable, measurable, outcome-oriented performance goals, and related measures.7 7 Tasks required for implementing the first principle include (1) identifying the organization’s mission and long-term strategic goals, (2) describing how the organization’s annual performance goals are related to its long-term goals, (3) specifying annual performance goals for each program activity, (4) identifying the performance measures the organization will use to assess its progress, and (5) describing how data will be verified and validated. Page 5 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 Goals of the CB Although DOD has taken the initial and necessary step of articulating CB Defense Program goals, the goals are not articulated in a manner consistent Defense Program Are with Results Act principles. The stated goals are vague and unmeasurable, Vague and and they fail to articulate specific desired impacts. A Results Act framework requires that managers define a related set of long-term Unmeasurable and Do strategic goals, annual agency goals, and measurable performance goals for Not Articulate Specific each program. The five CB Defense Program goals are to Desired Impacts 1. deter CB weapon use by denying military advantage to an enemy through a combination of avoidance, protection, decontamination, and medical support capabilities, allowing U.S. forces to operate largely unimpeded by chemical and biological attacks and their subsequent effects; 2. address the most probable CB weapon threats that could be encountered in regional conflicts and field capabilities to the forces required for two major theater wars; 3. ensure the CB weapon Threat Evaluation Projection drives CB defense research, development, and acquisition programs; 4. emphasize a joint service approach to CB defense research and development, and acquisition; and 5. complete critical RDT&E and acquisition of improved CB detection, identification, and warning systems; individual and collective protection systems; and medical support and decontamination systems. Measuring the first goal is unachievable, determining a deterrence effect is problematic, and attributing the specific rationale for the deterrence is unrealistic. The second, third, and fourth goals address the size, focus, and coordination of the program—not program outcomes. Together, these goals direct that the program be sufficiently large to address the needs resulting from two major theater wars; sufficiently focused to address the likely validated threats; and sufficiently coordinated to capitalize on efficiencies and other benefits of joint requirements determination, research, development, and acquisition. The objective of the fifth goal is measurable but speaks to program outputs without addressing program outcomes or impacts (such as decreased defensive vulnerabilities or increased operational capabilities). The completion of RDT&E or procurement cannot be assumed to result in a positive impact on the Page 6 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 defensive posture or operational flexibility of U.S. forces. While the completion of these activities may well generate benefits for U.S. troops, in the absence of valid, reliable measures, the contributions of RDT&E or procurement cannot be determined. CB Defense Program The CB Defense Program is not evaluated on the impact of its activities on the defensive or operational capabilities of U.S. forces, either individually Performance Measures or collectively. Emphasize Activities CB Defense Program planners use roadmaps, Defense Technology Rather Than Outcomes Objectives (DTO) and advanced concept technology demonstrations and Impacts (ACTD) to assess progress toward goals. Program planners collectively prepare a number of strategic plans they describe as “in the spirit of the Results Act,” if not specifically for the purpose of assessing outcomes and impacts. For example, DOD’s Nuclear, Biological, and Chemical (NBC) Defense Annual Report to Congress8 and the Joint Service NBC Defense Research, Development, and Acquisition Plan are updated annually and include detailed metrics and time lines reflecting the performance of the program (such as the demonstration of a new technology). Roadmaps track program progress toward DTOs that, when achieved, DOD claims will create new operational capabilities. A number of DTOs are ACTDs, and plans state that technology demonstrations provide a means for the rapid field testing of technical options to solve operational needs. CB Defense Program roadmaps explicitly link the completion of DTOs and ACTDs with an increase in the demonstrated warfighting capabilities of U.S. forces. In addition, CB Defense Program planners cited ongoing programmatic peer reviews, such as Technology Area Reviews and Assessments (TARA), as additional means to measure progress toward meeting program goals. We do not concur that the conduct of an ACTD or a peer review of ongoing work measures the impact of the CB Defense Program on the warfighter. Both measures have limitations that make them inappropriate for appraising progress toward achieving program objectives. ACTDs represent a means for rapidly introducing new technologies and reducing the time from the start of a program to the system’s initial operational capability. However, the demonstration of a new technology may not by itself result in the effective and safe deployment of a military capability in 8 Submitted to Congress annually pursuant to 50 U.S.C. 1523. Page 7 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 support of the warfighter. Moreover, as we previously reported, DOD has not always emphasized the need to complete concept and product development or testing before production, thus increasing the risk of approving ACTDs in support of CB defense that include immature technologies and then prematurely starting production.9 Technology Area Review TARAs are peer reviews conducted by the Director, Defense Research and Assessments Are Not a Engineering on each of DOD’s 12 science and technology programs—one being, CB defense. TARAs address progress toward achieving DTOs and Measure of Program Impact form the basis of the Results Act rating for DOD’s performance in science and technology. 10 However, the application of the TARA to generate performance measures of DOD’s science and technology programs—such as CB defense—is limited by several factors. First, the scope of the TARA Results Act ratings is limited because TARAs only address DTOs. Funding for DTOs comprises less than 50 percent of total funding for applied and advanced technology research and development. Thus, the Results Act ratings do not capture the majority of the CB Defense Program’s RDT&E activities. Second, the focus of TARAs is on budgets, schedules, and technical performance. TARAs do not measure technology transition from the laboratory to the battlefield. Lastly, TARAs do not measure improvements in the ability of U.S. troops to survive, fight, and win in a CB environment. CB Defense Program The organizations that execute or contribute to the research and development goals of the CB Defense Program vary in their use of the Research and Results Act principles to plan and assess their activities. Development Organizations Have Incorporated Results Act Principles Inconsistently 9 Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved (GAO/NSIAD-99-4, Oct. 15, 1998). 10 In fiscal year 1999, there were 23 chemical and biological defense DTOs, many in the form of ACTDs. Page 8 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 The Soldier and Biological The Soldier and Biological Chemical Command has demonstrated that the Chemical Command Is the Results Act principles can be integrated into the planning and evaluation process of an organization conducting research and development for the Only RDT&E Organization CB Defense Program.11 In 1996, the Chemical and Biological Defense to Systematically Apply Command (which in fiscal year 1999 merged with the Soldier Systems Results Act Principles Command to form the Soldier and Biological Chemical Command) developed a Business Planning Guide aligned with the Results Act guidelines. The guide linked planning guidance of higher headquarters (the Army Materiel Command); the Army’s Planning, Programming, Budgeting, and Execution System; and the Results Act. The guide stated that the Results Act “is the basis for the three products that result from our business planning process outlined in this document: a strategic plan, an annual performance plan, and an annual report.” Subsequently, the Chemical and Biological Defense Command issued strategic plans for the fiscal years 1997–2003 and fiscal years 1998–2004 time frames, annual performance plans for fiscal years 1997 and 1998, and an annual report for fiscal year 1997. The Chemical and Biological Defense Command strategic plan is driven by and linked with the overarching planning architectures of DOD, the Army, and the Army Materiel Command. The Chemical and Biological Defense Command’s strategic planning model directly links the attainment of its vision with the development of goals and enabling strategies—followed by the execution of the strategies and measurement of performance. Separate measures were developed to assess goal achievement as well as progress toward goal achievement. Specifically, the plan identified Chemical and Biological Defense Command goals, strategies for achieving the goals, and measures of goal achievement. Two of the Command’s six goals address their potential contributions to the CB Defense Program. They are to • research, develop, acquire, and field NBC defense, smoke, and obscurant materiel that meets warfighter requirements and reduces acquisition costs and timeliness (i.e., produces products faster and at lower costs) and • become the organization of choice for chemical, biological, and smoke/obscurant research, development, and technology services. 11 We did not assess the comprehensiveness, quality, or effectiveness of the command’s effort. Page 9 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 The performance plan identifies performance measures for each Command goal and performance goals for each strategy. The performance measures address both accomplishments and progress toward accomplishments. Examples of quantitative measures of research and development accomplishments include (1) the percentage of new NBC systems that meet NBC survivability requirements, (2) the percentage of nonexempt acquisitions receiving waivers from performance specifications, and (3) the percentage of Soldier and Biological Chemical Command science and technology programs transitioning to joint service and Army development programs with user validation through modeling, wargames, or similar methods. Soldier and Biological Chemical Command officials explained that the Command is still evaluating some of the fundamental dilemmas in applying Results Act principles. For example, the Command has yet to agree on what the right measures are. The identification of measures in the research and development component of the Soldier and Biological Chemical Command (and CB defense in general) has been an ongoing challenge and continues to evolve. Other CB Defense RDT&E The remaining RDT&E organizations cited a variety of reasons for not Organizations Have Not incorporating the Results Act’s principles in their program planning or evaluation systems.12 The two most common reasons cited were that Applied Results Act current DOD planning processes were equivalent to those of the act, Principles resulting in plans that were “in the spirit” of the Results Act, and that the unique nature of RDT&E activities did not lend itself to the act’s performance measurement and evaluation. CB Defense Program Planning DOD officials explained that the Office of the Secretary of Defense, NBC and Evaluation Defense Steering Committee, conducts planning consistent with Results Act principles for the CB Defense Program by issuing and executing the Program Objectives Memorandum. They claimed that memorandum-related planning documents, including reports that address different aspects of the CB Defense Program (such as the Joint Operational Concept; the Research, Development, and Acquisition Plan; the Joint Logistics Support Plan; and the Joint Modernization Plan) as well as the 12 Of the remaining principal federal organizations that conduct RDT&E on CB defense topics, only DOE’s CB Nonproliferation Program has developed a strategic plan, and none have developed a performance plan. Page 10 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 NBC Defense Annual Report to Congress, constitute the equivalent of a strategic plan. Moreover, CB Defense Program managers stated that DOD’s Planning, Programming, and Budgeting System is equivalent to the system required by the act and that therefore no substantive changes are necessary to comply with the spirit of the legislation. 13 Measurement of RDT&E There is no consensus on the appropriateness of applying performance Activities measures to RDT&E activities. While the Soldier and Biological Chemical Command has developed and applied measures of research and development outcomes, other organizations conducting CB Defense Program RDT&E have not. Neither DARPA, DOE, nor the Joint Program Office for Biological Defense have taken the initiative to develop a performance plan. In its strategic plan, DOE included a 5-year roadmap for developing, demonstrating, and delivering technology that would lead to major improvements in preparedness and capabilities. The Joint Program Office cites the conduct of ACTDs as measures of its performance. DARPA officials maintain that the nature of their RDT&E activities do not lend themselves to the application of performance measurement. DARPA conducts leading-edge research where the risks of failure are high and the probability of success is low. Its mission is to pursue long-term, far-reaching, and high-risk/high-payoff technology for military systems in the distant future. DARPA officials argued that developing useable metrics that are measurable, relevant, and timely for technology anticipated 10 or more years into the future is impossible. Moreover, they stated that goals and expectations are set at the project level and cannot be aggregated at the program level. Therefore, according to the officials, it would be inappropriate to develop programwide or agencywide measures of success or performance. Nonetheless, DARPA did try to develop a performance contract and submitted a draft to the Defense Management Council in early 1998. It also argued, however, that it did not fit the mold of most DOD agencies and should therefore be exempt from the act’s requirements. DARPA drafted performance metrics addressing its research operations as well as administrative efficiency. The performance metrics proposed for the research portion of its activities consisted of a series of assessments 13 The DOD Comptroller has noted that the Results Act is related to, but distinct from, DOD’s Planning, Programming, and Budgeting System, and has stated that Results Act planning and program evaluations need to be integrated with DOD’s Planning, Programming, and Budgeting System. Page 11 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 performed by an independent panel of experts.14 In December 1998, the Defense Management Council notified DARPA that it was exempt from the requirement. The draft performance contract was never finalized or implemented. Congress has recognized that successful implementation on the Results Act in science agencies would not come quickly or easily. Nonetheless, research organizations have concluded that the Results Act can or should be applied. The Research Roundtable, a group of federal researchers and managers representing a cross section of departments and agencies, concluded in 1995 that the results of a research program’s performance can be measured. The Army Research Laboratory was designated as a pilot project for performance measurement under the act and ultimately outlined an evaluation approach that made use of three pillars: metrics, peer review, and customer feedback. In 1999, the Committee on Science, Engineering, and Public Policy of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine reported on the results of their work on the issue of measuring and evaluating research in compliance with the act’s requirements. The committee concluded that both applied research and basic research programs supported by the federal government can be evaluated meaningfully on a regular basis. Conclusions The CB defense research and development outcomes and impacts are not being systematically measured. The CB Defense Program lacks both quantifiable performance measures and measurable objectives. In the absence of measures of program impacts and measurable objectives, progress toward achieving program goals cannot be determined. Program planning consists of a series of roadmaps leading to specific equipment items. Managers cite activity measures and technology demonstrations as measures of the program’s contribution. These planning and programming steps are appropriate and necessary, but they are insufficient for quantifying outcomes and impacts. Current measures do not assess the incremental changes attributable, in whole or in part, to the CB Defense 14 The Defense Science Board was the proposed panel. It would have been tasked to review the portfolio of DARPA projects to assess projects with regard to (1) relevance to warfighters, (2) ratio of technology investments versus system development, (3) level of risk, (4) ratio of new versus continuing efforts, and (5) level of service and commercial sector participation. Page 12 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 Program that improve warfighters’ ability to survive, fight, and win in a CB environment. The Results Act outcome-oriented principles have not been widely applied by either CB Defense Program planners or executing organizations. The utilization of these principles can enable managers and those overseeing the program to quantify the relative success of the program and of component projects in satisfying requirements across different activities (e.g., point detection, early warning, warning and reporting, and modeling). Impact measures can provide a planning tool to allocate finite CB Defense Program resources among competing sets of unmet requirements. Recommendations We recommend that the Secretary of Defense take actions to develop a performance plan for the CB Defense Program based on the outcome-oriented management principles embodied in the Results Act. The plan should be agreed to and supported by the relevant RDT&E organizations and incorporated in DOD’s NBC Defense Annual Report to Congress. Specifically, the plan should • establish explicit and outcome-oriented goals linked to warfighters’ ability to survive, fight, and win in a CB environment; • identify quantitative or qualitative performance measures that can be used to assess progress toward goal achievement; • describe how performance data would be validated; • describe how RDT&E activities of participating DOD and non-DOD organizations are coordinated to achieve program goals; and • identify human capital, financial, and resource challenges or external factors that limit the ability of the program to achieve its goals. Agency Comments In written comments on a draft of this report, DOD concurred with our recommendation. DOD stated it will develop a strategic plan more closely aligned with the tenets of the Results Act and publish the plan in the next DOD NBC Defense Annual Report to Congress. DOD’s comments are reprinted in appendix I. Our scope and methodology are explained in appendix II, and CB Defense Program RDT&E organizations are described in appendix III. Page 13 GAO/NSIAD-99-159 Chemical and Biological Defense B-282699 As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 7 days after its issue date. At that time, we will send copies to the Honorable William S. Cohen, Secretary of Defense and other congressional committees. We will also make copies available to others on request. If you have any questions regarding this report, please contact me or Sushil K. Sharma at (202) 512-3092. Key contributors to this assignment were Jeffrey Harris and Weihsueh Chiu. Kwai-Cheung Chan Director, Special Studies and Evaluations Page 14 GAO/NSIAD-99-159 Chemical and Biological Defense Page 15 GAO/NSIAD-99-159 Chemical and Biological Defense Contents Letter 1 Appendix I 18 Comments From the Department of Defense Appendix II 20 Scope and Methodology Appendix III 22 Chemical and Biological Defense Program Research, Development, Test, and Evaluation Organizations Figures Figure III.1: Primary Planning and Executing Organizations and Programs of DOD’s Chemical and Biological Defense Program Research, Development, Test, and Evaluation 23 Abbreviations ACTD Advanced Concept Technology Demonstration CB chemical and biological DARPA Defense Advanced Research Projects Agency DOD Department of Defense DOE Department of Energy DTO Defense Technology Objective NBC Nuclear, Biological, and Chemical QDR Quadrennial Defense Review RDT&E research, development, testing, and evaluation TARA Technology Area Review Assessment Page 16 GAO/NSIAD-99-159 Chemical and Biological Defense Contents Page 17 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix I Comments From the Department of Defense Appenx Idi Page 18 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix I Comments From the Department of Defense Page 19 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix II Scope and Methodology AppenIx Idi The scope of our study was limited to the nonmedical research, development, testing, and evaluation (RDT&E) activities of the Department of Defense’s (DOD)Chemical and Biological (CB) Defense Program. This study does not address any classified programs or projects. To determine whether the program goals were explicit and measurable and whether its performance measures assessed outcomes and impacts rather than outputs and activities, we reviewed the legislative record, interviewed agency officials, and analyzed program documents. To understand the requirements of the Government Performance and Results Act, we reviewed the legislation as well as implementation guidance issued by the Office of Management and Budget, DOD Comptroller, and the General Accounting Office.1 We queried representatives of the Office of the Secretary of Defense’s Nuclear, Biological and Chemical (NBC) Defense Steering Committee and the executing organizations (i.e., Joint Program Office, Soldier and Biological Chemical Command, Defense Advanced Research Projects Agency (DARPA)) regarding their familiarity with, and utilization of, the Results Act’s performance measurement principles. We reviewed strategic and performance plans implementing the act as well as documents characterized by DOD officials as complying with the “spirit of the legislation.”2 To determine whether organizations executing the CB Defense RDT&E activities have incorporated Results Act principles in their program planning and evaluation, we interviewed program officials, examined program documents, observed program review meetings, and attended scientific conferences on CB defense technologies. Program documents we examined included program budgets, strategic and performance plans, annual reports, internal program planning documents, program briefings, and proceedings of program review meetings. We also observed the 1999 DOD Technology Area Review and Assessment of chemical and biological defense. Scientific conferences we attended included the 1998 Joint Workshop on Standoff Detection for Chemical and Biological Defense and the 1998 Scientific Conference on Chemical and Biological Defense 1 See Executive Guide: Effectively Implementing the Government Performance and Results Act (GAO/GGD-96-118, June 1996); Agencies’ Annual Performance Plans Under the Results Act: An Assessment Guide to Facilitate Congressional Decisionmaking (GAO/GGD/AIMD-10.1.18, February 1998); and The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans (GAO/GGD-10.1.20, April 1998). 2 These include the DOD NBC Defense Annual Report to Congress (submitted pursuant to 50 CFR 1523), the Joint NBC Modernization Plan, the Joint Service NBC Defense Research, Development, and Acquisition Plan, and the Joint Service NBC Defense Logistics Plan. Page 20 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix II Scope and Methodology Research. We also obtained proceedings from these and other scientific conferences. To respond to all three objectives, we contacted the following organizations: DARPA, Arlington, Virginia; Defense Threat Reduction Agency, Dulles, Virginia; DOD Inspector General, Washington, D.C.; Department of Energy, Washington, D.C.; Director, Defense Research and Engineering, Washington, D.C.; Dugway Proving Ground, Dugway, Utah; Edgewood Chemical and Biological Center, Aberdeen Proving Ground, Maryland; Joint Program Office for Biological Defense, Falls Church, Virginia; National Domestic Preparedness Office, Washington, D.C.; National Ground Intelligence Center, Charlottesville, Virginia; Nonproliferation and National Security Office, Department of Energy, Washington, D.C.; Office of Science and Technology Policy, White House, Washington, D.C.; Office of the Secretary of Defense, Washington, D.C.; U.S. Army Soldier and Biological Chemical Command, Aberdeen Proving Ground, Maryland; and U.S. Army Training and Doctrine Command, Fort Monroe, Virginia. We conducted our review from November 1998 to April 1999 in accordance with generally accepted government auditing standards. Page 21 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix III Chemical and Biological Defense Program Research, Development, Test, and Evaluation Organizations AppeInx Idi The CB Defense Program is overseen by the Office of the Secretary of Defense’s Nuclear, Biological, and Chemical Defense Steering Committee, which is comprised of the Directors of the Defense Threat Reduction Agency and Defense Research and Engineering as well as their top officials responsible for CB defense. The steering committee funds research and development at numerous laboratories in DOD, Department of Energy, and private industry.1 As illustrated in figure III.1, key research and development organizations in the execution of the program include the SBCCOM, JPO-BD, DARPA. 1 The CB Defense Program addresses nonmedical research and development in the areas of chemical detection, biological detection, individual protection, collective protection, decontamination, modeling and simulation, core science and technology, and basic research. In addition, core science and technology includes threat assessment and aerosol technology; and basic research includes aerosol science, chemistry and toxicology, and analytical chemistry. Page 22 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix III Chemical and Biological Defense Program Research, Development, Test, and Evaluation Organizations Figure III.1: Primary Planning and Executing Organizations and Programs of DOD’s Chemical and Biological Defense Program Research, Development, Test, and Evaluation Department of Department of Defense Energy CB Defense Program CB Nonproliferation CB Nonproliferation Program Program Defense Threat Director of Defense Reduction Research and Agency Engineering NBC Defense Steering Committee Director of DATSD CB Defense Counterproliferation Directorate and CB Defense $ $ Department SBCCOM JPO-BD DARPA of Energy Labs DATSD = Deputy Assistant to the Secretary of Defense SBCCOM = Soldier and Biological Chemical Command JPO-BD = Joint Program Office for Biological Defense DARPA = Defense Advanced Research Projects Agency = Funding flows = Informal coordination SBCCOM is organized around two integrated business areas, one of which is research, development, and acquisition. Nearly half of SBCCOM research, development and acquisition funding supports the CB Defense Program. SBCCOM is engaged in the full range of research and development encompassing both biological and chemical systems. Page 23 GAO/NSIAD-99-159 Chemical and Biological Defense Appendix III Chemical and Biological Defense Program Research, Development, Test, and Evaluation Organizations SBCCOM business areas include chemical detection, biological detection, decontamination, protection, and supporting science and technology. JPO-BD was created in 1994 to manage the biological warfare agent detection program. The office monitors emerging technologies for advanced development, demonstration, and upgrades of fielded biological detection systems. The DARPA Biological Warfare Defense Program is an applied research program established by the National Defense Authorization Act for Fiscal Year 1997 (P.L. 103-160, as amended) to fund an applied research program supporting revolutionary new approaches to biological warfare defense. The Biological Warfare Defense Program pursues high-risk, high-potential technologies from the demonstration of technical feasibility through the development of prototype systems. DOE’s CB Nonproliferation Program was established in 1997 in response to the Defense Against Weapons of Mass Destruction Act of 1996, which appropriated $17 million to DOE to conduct research and development to develop new means for detecting the presence, transportation, production, and use of weapons of mass destruction and related materials and technologies. According to DOE, the purpose of this appropriation was to ensure the full engagement of DOE national laboratories in responding to the threat posed by CB weapons to U.S. civilians. DOE funds research and development, from basic research to fieldable prototypes, in pursuit of advanced technologies that can enable first responders to more effectively prepare and respond to the use of CB agents. (713039) Ltert Page 24 GAO/NSIAD-99-159 Chemical and Biological Defense Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Correction Requested
Chemical and Biological Defense: Program Planning and Evaluation Should Follow Results Act Framework
Published by the Government Accountability Office on 1999-08-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)