United States General Accounting Office GAO Report to Congressional Requesters June 1997 MANAGING FOR RESULTS EPA’s Efforts to Implement Needed Management Systems and Processes GAO/RCED-97-156 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-276608 June 18, 1997 The Honorable Christopher S. Bond Chairman, Subcommittee on VA, HUD, and Independent Agencies Committee on Appropriations United States Senate The Honorable Jerry Lewis Chairman, Subcommittee on VA, HUD, and Independent Agencies Committee on Appropriations House of Representatives For over a decade, internal and external studies have called for the Environmental Protection Agency (EPA) to “manage for environmental results” as a way to both improve and better account for its performance. To implement this type of management, EPA must establish goals and objectives identifying the improvements in environmental conditions that it is seeking, develop strategies and allocate resources to achieve these goals and objectives, and assess and report the results achieved. A 1995 study by the National Academy of Public Administration (NAPA) recommended a series of steps for EPA to take to manage for results, including actions to improve and integrate its processes for planning, budgeting, and ensuring accountability.1 In March 1996, EPA responded to NAPA’s recommendations by announcing its plans to create a new office that would develop and implement an integrated planning, budgeting, and accountability system for the agency. Because of your continued interest in how EPA sets its priorities and plans its environmental activities, you asked us to review the status of EPA’s efforts to (1) establish the new office and develop and implement an integrated system and (2) ensure that the agency has comprehensive scientific and environmental data and appropriate environmental measures of progress for use in strategic planning, budgeting, and accountability.2 In addition, you asked us to discuss EPA’s previous efforts to implement this type of system and identify elements of an effective 1 Setting Priorities, Getting Results: A New Direction for EPA, NAPA (Apr. 1995). 2 The preliminary results of our review of these issues were reported to the Subcommittees on VA, HUD, and Independent Agencies, Senate and House Committees on Appropriations in Managing for Results: EPA’s Efforts to Implement Needed Management Systems and Processes (GAO/T-RCED-97-116, Apr. 8, 1997, and GAO/T-RCED-97-133, Apr. 15, 1997). Page 1 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 system that could be used as benchmarks to gauge progress in this current effort. In January 1997, the EPA Administrator approved the organizational Results in Brief structure and staffing for a new office—called the Office of Planning, Analysis, and Accountability—which is responsible for developing and implementing an integrated planning, budgeting, and accountability system. Earlier, in March 1996, EPA had established a work group, composed of employees on temporary assignment, to begin developing the new system. Although the work group was never fully staffed and made limited progress in developing the system, it developed a conceptual framework that the new office is beginning to implement. Members of the work group were temporarily assigned to the new office, which will not be fully staffed before July 1997. EPA faces long-term challenges to obtain the scientific and environmental information needed to fully support its new system. Although much scientific and environmental information has already been collected, many gaps exist, and the data are often difficult to compile because different data collection methods have been used. Likewise, much effort is still needed to identify, develop, and reach agreement on a comprehensive set of environmental measures to link the agency’s activities to changes in human health and environmental conditions. Currently, EPA has to rely mainly on administrative measures, such as the number of permits issued or inspections made, to measure its performance or success. EPA has been trying since the early 1970s to revise its management systems to better manage for results. In the early 1990s, for example, EPA announced a strategic planning, budgeting, and accountability effort that was very similar to the system being proposed today. However, such efforts proved to be difficult and fell short of EPA’s expectations. EPA’s current plans are no less complex and challenging. Given this complexity, we identified various elements of an effective planning, budgeting, and accountability system that EPA can use as benchmarks to monitor its progress as it works to implement its plans for an integrated system. These elements describe organizational, informational, and procedural capabilities needed to improve and integrate EPA’s management processes. For example, EPA needs comprehensive data on health and environmental risks and a process to compare these risks across environmental problems and programs. EPA also needs performance partnership agreements with the states to set out expectations for their contributions to meeting Page 2 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 national environmental and agencywide goals and to specify measures for assessing their performance. NAPA’s April 1995 report recommended, among other things, that EPA Background establish specific environmental goals and strategies to attain them, use comparative risk analyses to inform the selection of priorities and the development of specific program strategies, consolidate the planning and budgeting functions, use the budget process to allocate resources to the agency’s priorities, and establish accountability by setting and tracking benchmarks and evaluating performance. NAPA’s recommendations on setting goals and objectives and ensuring accountability for meeting them are similar to the requirements for federal agencies established by the Government Performance and Results Act of 1993 (GPRA). Under GPRA, agencies are to develop strategic plans by September 30, 1997. In addition, agencies are to submit to the Office of Management and Budget annual performance plans, including annual performance goals and performance measures. The first annual performance plans are to be submitted in the fall of 1997 for fiscal year 1999. Agencies are also required to submit annual performance reports to the President and the Congress on the extent to which their programs are meeting annual performance goals. The act requires the first performance reports, which are to cover fiscal year 1999, by March 31, 2000. In establishing these planning and performance reporting requirements, GPRA did not specify the management processes that the individual agencies are to use to implement them. In response to NAPA’s recommendations, EPA created a task force in July 1995 to identify ways to improve and better integrate its processes for planning, budgeting, and ensuring accountability. The task force’s February 1996 report recommended a major restructuring of these processes into a “strong, centrally managed” system. The recommended system was envisioned to provide for (1) the more consistent and comprehensive use of science and analysis in setting the agency’s priorities and (2) greater management and fiscal accountability for implementing programs and activities according to budgets and operating plans. In March 1996, the EPA Administrator and Deputy Administrator directed the agency to implement the task force’s recommendations. They announced plans to create a new office by January 1997 to develop and Page 3 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 implement the integrated planning, budgeting, and accountability system proposed by the task force. In the interim, EPA established the Planning, Budgeting, Accountability, and Analysis Work Group composed of employees on temporary assignment, to begin to develop the new system. Although approved by the Administrator in January 1997, EPA’s new Office EPA’s New Office Is of Planning, Analysis, and Accountability will not be fully staffed until July Being Formed to 1997. The Planning, Budgeting, Accountability, and Analysis Work Group Develop an Integrated began to develop the new system in March 1996, and its staff has been detailed to the new office until permanent employees are selected. System However, the work group was not fully staffed, and much of its time was devoted to preparing the strategic plan due under GPRA by September 30, 1997. Given this level of staffing and focus on meeting the requirements of GPRA, EPA has made progress in designing an integrated system but has not yet implemented all the needed processes and procedures. Staffing the New Office The Office of Planning, Analysis, and Accountability, which is located in the Office of the Chief Financial Officer, is authorized 47 employees.3 (An organizational chart for the Office of the Chief Financial Officer is shown in fig. 1.) 3 As part of the reorganization creating the Office of Planning, Analysis, and Accountability, EPA’s budget and financial divisions in the Office of Administration and Resource Management were moved to the Office of the Chief Financial Officer. The Budget Division was made responsible for both the annual planning and budgeting processes because of their close link under EPA’s new system. The Chief Financial Officer reports to the Administrator. Page 4 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 Figure 1: Organizational Chart for EPA’s Office of the Chief Financial Officer Administrator Deputy Administrator Chief Financial Officer Deputy CFO Office of Planning, Office of the Analysis, and Comptroller Accountability Financial Financial Services Annual Planning and Management Division Budget Division Division Analysis staff Planning staff Accountability staff Page 5 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 As of May 1997, EPA had filled 17 of the new positions—16 from within EPA and 1 from outside the agency. The remaining positions are anticipated to be filled by July 1997. The Acting Chief Financial Officer told us that staffing could not begin until the Administrator had approved the plans for the office. In addition, time is required to select staff using a competitive selection process. After its formation in March 1996, the Planning, Budgeting, Accountability, and Analysis Work Group began to develop the new planning, budgeting, and accountability system. The work group’s staff, who were detailed to the Office of Planning, Analysis, and Accountability until the permanent staff could be selected, have devoted much of their effort to preparing the strategic plan that GPRA requires by September 30, 1997. Although the work group was to have 40 to 50 temporary employees assigned from other parts of EPA, it has had no more than 27 employees assigned at any one time. Currently, 10 of the work group’s members remain on detail to the Office of Planning, Analysis, and Accountability. According to the Acting Chief Financial Officer, the work group had to compete with other high-priority activities for staff. Developing an Integrated Because of the work group’s staffing level and focus on meeting GPRA’s System deadlines, EPA is in the early stages of developing an integrated system. EPA has a conceptual framework for the new system and has begun to develop the individual components. The agency is reviewing its former accountability process to find out what did and did not work well, contacting other federal agencies to determine how they account for progress in meeting their goals, and examining reporting systems in the agency’s program offices to identify their potential use in the new system. EPA hopes to have the accountability component in place by September 1999. EPA is also preparing the strategic plan that is to be completed in September 1997. On April 25, 1997, the agency provided the Congress with a detailed outline of the plan, including a draft of its mission statement and goals. It provided its draft objectives on May 21, 1997. The Congress is scheduled to receive a complete draft of the plan for review and consultation in July 1997. Once the strategic plan is completed, the development of the new budgeting component will begin. According to EPA officials, the agency plans to seek approval from the Office of Management and Budget and the Congress to restructure its fiscal year 1999 budget along the lines of the goals in the strategic plan. (A more detailed Page 6 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 discussion of the framework and the status of EPA’s implementation of the system is contained in app. I.) EPA’s new planning, budgeting, and accountability system depends on the EPA Faces availability of credible information to identify environmental problems, Information establish priorities among competing needs, develop effective strategies to Challenges to achieve goals and objectives, and assess progress or performance in meeting the goals and objectives. Although EPA continues to make Implement Its New improvements, the environmental data that it compiles have substantial System shortcomings, including gaps and inconsistencies in quality. In addition, the agency’s information systems are not integrated. To evaluate the impact of its programs on the environment and determine whether they are achieving the desired results, EPA will need a comprehensive set of environmental measures. Although the agency’s efforts to develop such measures have been valuable, focusing these and future efforts could enable EPA to more effectively address scientific and technical obstacles and reach agreement with the states on the measures that will be used to assess their performance in carrying out environmental programs at the local level. Shortcomings in EPA’s The need to assess EPA’s performance in terms of changes in Environmental Data and environmental conditions substantially increases the demand for Systems high-quality environmental data. Such data are also needed to identify and respond to emerging problems before significant damage is done to the environment. Although EPA has tried to improve the quality of its data, these data are often unreliable, and the agency’s many disparate information systems are not integrated. These shortcomings have been raised in various external and internal reports on EPA, including the Vice President’s report on reinventing government.4 In its April 1995 report, NAPA identified the lack of high-quality data on environmental conditions as a particularly important problem for EPA. NAPA specifically noted the limited amount of information based on the real-time monitoring of environmental conditions. NAPA also concluded that much remains to be done to improve the agency’s overall management of environmental information. It noted that EPA had numerous information systems and that the agency’s program offices, which are responsible for their own data, use different methods and 4 Reinventing Environmental Regulation, National Performance Review (Mar. 16, 1995). Page 7 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 definitions to gather data. In addition, EPA relies on data compiled by other federal agencies and the states. According to NAPA, these agencies and the states also use divergent methods of collecting data. More recently, a 1996 EPA report concluded that the agency needs to redesign its many disparate fiscal and environmental data systems so that it and others can measure its success in meeting environmental goals and can determine the costs of doing so.5 The difficulty EPA has had in demonstrating its performance or the impact of its actions is illustrated by the findings of a team of agency personnel, which was formed in 1995 to evaluate the agency’s needs for environmental information. The team identified various problems with the information needed to report on environmental goals, such as gaps in the data and inconsistencies in the methods of collecting and/or reporting data across states or federal agencies. Specific examples include the lack of (1) national reporting on risk reduction at waste sites, (2) reliable data on the nature and cause of pesticide poisonings, (3) effective reporting on progress in improving the nation’s water quality, and (4) complete data on air pollutants.6 Need to Focus EPA’s Although EPA and state officials recognize that measures of activities are Efforts to Develop still needed, they believe that environmental measures are more useful for Environmental Measures assessing programs’ performance and for informing the public about environmental conditions. However, they understand that scientific and technical issues have to be addressed before EPA can widely use indicators of environmental conditions and trends. Greater use of these measures also depends on agreement with the states—EPA’s partners in implementing environmental programs—specifying the measures that will be used to assess their performance. EPA and the states have various efforts under way to develop and use environmental measures. According to EPA officials, these efforts have been valuable but disparate. Furthermore, at a conference convened by EPA in September 1996 to better coordinate these efforts, as well as in interviews conducted by EPA staff to prepare for the conference, regional and state representatives cited several concerns. They said, for example, that (1) clarification is needed on EPA’s and the states’ direction in 5 Managing for Results, EPA’s Planning, Budgeting, and Accountability Task Force (Feb. 23, 1996). 6 The Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996 establish an investment framework for having agencies better plan and manage their technology efforts and link these efforts directly to the achievement of their programs’ goals and missions. If successfully implemented, these acts hold a key to ensuring that agencies put systems in place to collect needed performance data. Page 8 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 developing indicators; (2) some qualities of a good indicator are not well understood; and (3) in some cases, determining whether the best indicators have been chosen will take many years. The representatives also believed that the data and resources needed to develop and use environmental indicators are inadequate. Focusing these disparate efforts could enable EPA to better address (1) scientific and technical issues associated with developing measures and (2) the states’ concerns about the appropriate measures to use. A major technical challenge in developing and using environmental measures is to be able to scientifically establish a direct cause-and-effect relationship between a program’s activities and changes in environmental conditions. This relationship is especially difficult to establish because factors beyond a program’s control, such as changes in weather patterns and economic conditions, can substantially affect environmental conditions. Furthermore, the data needed to understand these relationships can be extensive. Another challenge is having sufficient monitoring data available to document trends or changes in the environment. In many cases, data or indicators are not available for a specific aspect of the environment because of high costs or technical difficulties. Thus, it will be some time before EPA is able to develop and use a set of environmental indicators that accurately and comprehensively reflect the impact of its programs or their results.7 Reaching agreement within EPA and among the states and the agency’s other stakeholders on the specific environmental indicators to use will also be a challenge. A consensus may be difficult to achieve because of the potential for disagreement about what should be measured and whether a relatively small number of measures can adequately reflect the effects of an agency’s or a program’s activities. For example, EPA will need a set of measures common to all the states to report to the Congress and the public on the agency’s performance and the condition of the nation’s environment. At the same time, the development of national measures, to the extent that such measures drive the implementation of environmental programs, will reduce the states’ flexibility to tailor the programs to meet local needs and conditions, a major concern of the states. Reporting on new measures will also increase the states’ costs unless other reporting requirements are eliminated or reduced. 7 Our report Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997) discusses in more detail the analytic and technical challenges that EPA and other federal agencies are experiencing as they try to measure their programs’ performance under GPRA. Page 9 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 EPA is working with the states to develop a set of core performance measures, including some environmental indicators, that the agency’s regional offices are to use in negotiating annual work plans and performance partnership agreements with the states.8 The core measures are to be focused and limited in number, representing measurable priorities for each of EPA’s national program managers. They are to serve as the minimum measures in performance agreements with the states, which may develop additional measures to represent their own environmental or programmatic issues. In addition, a particular core measure may not be required if a state can demonstrate that the measure does not apply or cannot be addressed. According to EPA, its national program managers finalized their core measures in April 1997, and its regional staff began negotiations with the states to incorporate these measures into the performance partnership agreements for fiscal year 1998. However, because of various state concerns, the core measures were issued as interim measures and stamped as “draft” to permit work to continue on the measures while negotiations on the partnership agreements take place. For example, the environmental commissioners of the New England states have expressed to EPA their opposition to making any national indicator or core performance measure mandatory. It is too soon to know how extensively EPA’s regional offices will be negotiating measures that reflect the direct effects of programs on human health and the environment. According to Office of Planning, Analysis, and Accountability officials, additional efforts to develop environmental measures are needed, but the key is to find the right balance of environmental and activity measures. The officials said that environmental measures can be costly to develop and use because they require so much environmental information. In their view, activity measures may be the only available or appropriate measures for some programs or activities. 8 EPA enters into performance partnership agreements with states that are participating in the National Environmental Performance Partnership System. This new system of federal oversight for state environmental programs, which is being established under a 1995 agreement with state leaders, places greater emphasis on the use of environmental goals and performance indicators and provides opportunities for reducing EPA’s oversight of state programs that exhibit high performance in certain areas. About half the states have signed performance partnership agreements to participate in the system. Page 10 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 For EPA, revising its management processes to implement an integrated Previous EPA Efforts planning, budgeting, and accountability system that will enable it to Suggest That manage for environmental results is not easy. According to EPA officials, Expectations or implementing the system as currently planned is a major challenge that could take several years to complete. This is also not the first time that the Benchmarks Are agency has tried to develop and implement such a system. EPA’s attempts Needed to Guide to develop and use environmental measures go back to 1974, and when we performed our general management review of the agency in 1988, we System’s reported that one of its major management goals emphasized managing for Implementation measurable environmental results.9 In a March 1990 response to our report, EPA said that the new Administrator had made managing for environmental results a major policy and operational focus for the agency and was initiating a strategic planning, budgeting, and accountability effort to improve the agency’s ability to protect human health and the environment. The goals of that effort were very similar to those of the system being proposed today. Given the complexity of the task and time likely needed to fully implement the proposed new system, it is important for EPA to identify the processes, organization, and information that the system will need to work effectively when implemented. These elements can be used as benchmarks to help guide the development of the system and to monitor the progress of its implementation. Over time, support for fully implementing the system may be eroded by personnel changes in top management or other key positions or by constraints on resources. The major elements of an effective planning, budgeting, and accountability system for EPA that we identified are shown in appendix II. We identified these key elements through our prior reviews of EPA’s management and programs, NAPA’s report on EPA, and various other studies, including the EPA task force’s study recommending the conceptual framework that the agency is currently implementing. We also considered the requirements of GPRA in developing the system’s elements. While GPRA requires federal agencies to set goals, measure performance, and report on results, it does not discuss the characteristics of the management processes needed to meet these requirements. Thus, the specific elements that we developed supplement GPRA’s general management framework. We also obtained and incorporated comments on our descriptions of the key elements from four former EPA officials who were responsible for 9 Environmental Protection: Protecting Human Health and the Environment Through Improved Management (GAO/RCED-88-101, Aug. 16, 1988). Page 11 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 planning, budgeting, and/or ensuring accountability while at the agency.10 These former officials generally agreed with the key elements but suggested various additions and clarifications. We also obtained and incorporated comments from representatives of NAPA and EPA’s Science Advisory Board. The Board, which was established in the Office of the Administrator to provide advice on scientific matters, has evaluated various aspects of EPA’s priority-setting and other management systems that make use of scientific and technical information and analyses. As shown in appendix II, we identified key elements for each of the planning, budgeting, and accountability components of an effective system. These elements describe the organizational, informational, and procedural capabilities that EPA needs to improve and integrate its management processes. For example, EPA needs to be able to establish priorities on the basis of human health and environmental risks. Key elements of the planning component would include (1) comprehensive data on risks and (2) a process to compare these risks across environmental problems and the agency’s programs. Similarly, the agency needs to work with the states as partners in carrying out environmental programs. Hence, performance partnership agreements with the states are part of the agency’s accountability component. These agreements would set out expectations for the states’ contributions to meeting national environmental and agencywide goals and would specify measures for assessing the states’ performance. In addition to the planning, budgeting, and accountability elements, we included the elements of an effective analysis component to reflect the importance of information and analytical capability to the system’s effectiveness. For example, one of the elements of an effective central planning, budgeting, and accountability office is an analysis function to independently compile and assess relevant information on health and environmental risks and on the performance of programs across the agency. Our list of elements for the analysis component describes the environmental, risk, and performance information needed to make decisions about planning, budgeting, and accountability. In total, the key elements that we identified would provide EPA with a structured approach for establishing direction, setting priorities, and measuring performance. They would aim to ensure that the agency (1) has adequate environmental, risk, and performance data to support 10 These former EPA officials include a deputy administrator and three assistant administrators—two for policy, planning, and evaluation and one for administration and resource management. Page 12 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 decisionmakers and to inform stakeholders and (2) can link its mission and priorities to programs’ actions by setting and reporting on its progress toward accomplishing measurable goals and objectives. Thus, the elements would be consistent with the plans for EPA’s new system and the requirements of GPRA. EPA is likely to need several years to develop and fully implement an Conclusions integrated planning, budgeting, and accountability system. Even given this much time, the agency will have difficulty obtaining the scientific and environmental data and developing and reaching agreement on the appropriate environmental measures of its programs’ and its own performance called for by the new system. Furthermore, the efforts EPA has made over the years to improve and integrate its planning, budgeting, and accountability processes illustrate both the importance and the difficulty of these efforts. Given the complexity of the efforts and the time required to complete them, clear expectations or benchmarks for fully implementing an integrated system would better enable the agency’s management, the Congress, and other stakeholders to monitor progress toward implementing such a system. To help ensure the full implementation of the agency’s plans for an Recommendation integrated planning, budgeting, and accountability system, we recommend that the EPA Administrator, in consultation with key stakeholders, establish expectations or benchmarks for how the new system is to operate when fully implemented and use them to monitor the agency’s progress in implementing the system. We provided a draft of this report to EPA for review and comment. EPA Agency Comments officials, including the Acting Deputy Director of the Office of Planning, Analysis, and Accountability, said that the report was well done and accurately described the status of the agency’s efforts to implement the new planning, budgeting, and accountability system. In addition to suggesting clarifications, which we have incorporated where appropriate, the officials noted that the ultimate success of EPA’s new system will largely depend on others, especially the states, which primarily implement the agency’s programs. According to the officials, in working with the states to implement the system, the agency will face the inherent conflict between the states’ desire for flexibility and EPA’s need to hold them accountable for their performance. Page 13 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System B-276608 The EPA officials agreed with our recommendation and the need to identify some measures of success for implementing the new planning, budgeting, and accountability system—that is, some ways for them to know whether they are progressing toward the type of system intended. They stated that the key elements listed in appendix II, including the heavy involvement of stakeholders, are consistent with their plans for implementing GPRA. We conducted our audit work from June 1996 through May 1997 in accordance with generally accepted government auditing standards. A detailed discussion of our objectives, scope, and methodology appears in appendix III. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 15 days from the date of this letter. At that time, we will send copies to the EPA Administrator and other interested parties. We will also make copies available upon request. Please call me at (202) 512-6111 if you or your staff have any questions. Major contributors to this report are listed in appendix IV. Peter F. Guerrero Director, Environmental Protection Issues Page 14 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Page 15 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Contents Letter 1 Appendix I 18 EPA’s Framework 18 EPA’s Conceptual Implementation Activities 20 Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System Appendix II 25 Planning Component 25 Key Elements of an Budgeting Component 27 Effective Planning, Accountability Component 27 Analysis Component 28 Budgeting, and Accountability System for the Environmental Protection Agency Appendix III 31 Objectives, Scope, and Methodology Appendix IV 33 Major Contributors to This Report Figures Figure 1: Organizational Chart for EPA’s Office of the Chief 5 Financial Officer Figure I.1: General Framework for EPA’s New Planning, 19 Budgeting, and Accountability System Page 16 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Contents Abbreviations EPA Environmental Protection Agency GAO General Accounting Office GPRA Government Performance and Results Act of 1993 NAPA National Academy of Public Administration OPAA Office of Planning, Analysis, and Accountability Page 17 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System The Environmental Protection Agency (EPA) has developed an outline, or conceptual framework, for its new planning, budgeting, and accountability system and has begun to develop the individual components.11 This appendix describes the framework and the agency’s implementation activities. Under its new system, EPA is to (1) establish strategic or long-term goals EPA’s Framework and objectives; (2) develop strategies to achieve the goals and objectives; (3) identify the desired outcomes from these strategies; (4) translate these goals, objectives, strategies, and desired outcomes into annual goals, planned activities, and performance measures; and (5) use the performance measures to assess the agency’s and individual programs’ performance in achieving the desired outcomes. This information on performance is to be considered whenever the agency’s goals, objectives, and strategies are reexamined. As figure I.1 shows, EPA’s framework calls for the agency to first prepare a strategic plan that defines its mission and sets out its goals and objectives for the next 5 years or so. This plan is also to describe the principles and general strategies that are to be employed to achieve the goals and objectives. In turn, a multiyear action plan is to be prepared for each objective in the strategic plan, setting out, for a specific period, outcomes or results to be achieved, ways to measure progress towards these outcomes, alternative strategies to achieve the outcomes, the costs to EPA for each of the strategies, and measures of costs and benefits or cost-effectiveness for each of the alternatives. 11 EPA refers to its new system as the planning, budgeting, accountability, and analysis system to highlight the greater role planned for the use of scientific information and analysis in these management processes. Page 18 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System Figure I.1: General Framework for EPA’s New Planning, Budgeting, and Accountability System Strategic plan Goals Objectives Principles General strategies Multiyear action plans Resource "envelope" projections Performance feedback Annual performance plans Annual budget Annual request appropriation Annual operating plan Actual resources Actual performance Achievement of performance measures Accountability Resource use process Performance evaluation Performance reports Source: EPA’s Office of Planning, Analysis, and Accountability. Page 19 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System Annual performance plans would then link strategic planning to annual planning and budgeting by setting out annual performance goals and annual measures of performance tied to strategic goals and objectives. These plans, which are to be developed for each objective in the strategic plan, also describe specific activities to be carried out during the year (outputs), specific results to be achieved through these activities (outcomes), and the effects of reductions or increases in available resources on outputs and outcomes. The annual plans, especially the sections on needed resources, are to be the basis for preparing annual budget requests. The system’s accountability component provides for examining actual performance and costs against goals and objectives and cost projections. In addition to evaluating achievement against the performance measures set out in the annual performance plans, EPA is to conduct broader performance evaluations. The results of both types of evaluations are to be fed back into the planning and budgeting processes and used to produce the annual performance reports required by the Government Performance and Results Act of 1993 (GPRA) and other internal management reports on performance.12 As discussed below, EPA has activities under way to develop the planning, Implementation budgeting, and accountability components of its new system. Most of Activities these efforts thus far have been directed to preparing the strategic plan that GPRA requires by September 30, 1997. Planning Although EPA has experienced some delays, it plans to complete its strategic plan by September 30, 1997, as required. On April 25, 1997, EPA provided the Congress with an outline of its strategic plan, including a draft of its mission statement, goals, and strategic principles.13 This outline is to serve as the basis for discussion with the Congress during formal 12 Over the longer term, improved financial reporting and auditing, as required by the Chief Financial Officers Act of 1990, is to strengthen the cost basis and reliability of performance information. Reporting concepts and standards developed by the Federal Accounting Standards Advisory Board are intended to provide congressional and other decisionmakers for the first time with annual “report cards” on the costs, management, and effectiveness of federal agencies in order to help strengthen federal decision-making and agencies’ accountability. 13 The term “strategic principles” refers to statements describing the critical points to be considered when making decisions about EPA’s priorities and activities. These principles—for example, an equal focus on human health and ecological health and the use of risk reduction as the first criterion for establishing ecological and human health priorities—are to provide guidance to decisionmakers and to serve as a common basis for making decisions across the agency’s programs. Page 20 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System consultation on the plan. A list of the agency’s draft objectives, which are to be measurable statements of what EPA plans to accomplish under each goal, was provided to the Congress on May 21, 1997. EPA plans to give the Congress sections of the plan as they become available rather than waiting until they all are completed. According to the agency’s projections, a complete draft of the plan will be available by early July 1997. The Office of Planning, Analysis, and Accountability (OPAA) has compiled a list of EPA’s strategic objectives from lists of objectives prepared by the agency’s individual offices, using the agency’s draft goals. In developing their objectives, the program offices were directed to obtain input from states, Indian tribes, associations, and other principal stakeholders. According to the Acting Deputy Director of OPAA, the office will ask these stakeholders for their input on the mission statement, goals, and objectives while the agency is consulting with the Congress on the strategic plan. Two studies that could affect the selection of EPA’s goals and objectives are not scheduled to be completed in time for this initial strategic plan. One study, EPA’s National Environmental Goals Project, is being performed to establish a set of long-range national environmental goals with realistic and measurable milestones for the year 2005. In January 1997, EPA made a draft of the national goals available for “full government review,” including reviews by the legislative branch, other federal agencies, and the states. Project officials anticipate that the goals will be available for public comment in late 1997 and will be issued in 1998. The other study, the Integrated Risk Project, is being done by EPA’s Science Advisory Board, which advises the Administrator on scientific matters. A draft report on the results of this study, which is being done to rank the relative risks of environmental problems and to develop methodologies that EPA can use to rank risks in the future, is to be provided to other scientists and experts for peer review in August 1997. According to EPA officials, strategic planning is an iterative process and the plan will be updated, as appropriate, to reflect the final results of these and other studies or factors. The officials said that a draft of the national goals was considered in developing the agency’s draft goals, and program offices were instructed to consider risk and the potential for risk reduction in developing their objectives. According to EPA officials, the agency is beginning to develop the data that it will need to compare risks across programs. Page 21 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System The framework for EPA’s new system calls for the agency’s program offices to prepare multiyear action plans after the strategic plan is completed and before annual performance plans are begun. These multiyear plans are to bridge the strategic plan and the annual performance plans by describing for each objective in the strategic plan the results or outcomes that are to be achieved, the methods of measuring progress toward achieving these outcomes, and the costs of achieving the outcomes under alternative strategies. This information on the outcomes, strategies, and costs associated with achieving goals and objectives is then to be used for annual planning and budgeting. However, EPA officials believe that they need to begin preparing the annual performance plans as soon as the strategic goals and objectives are selected if the annual plans are to be completed by September 30, 1997, as required by the Office of Management and Budget. According to agency officials, the preparation of multiyear action plans, which are not required by GPRA, will be delayed until the fiscal year 2000 planning cycle. The officials said that some items that would have been discussed in the action plans may be incorporated into the strategic plan that is currently being prepared. Once the strategic goals and objectives are approved by EPA management, the program offices are to prepare annual performance plans for fiscal year 1999. These plans are to be prepared for each of the objectives in the strategic plan. OPAA is currently developing guidance for the program offices on how these plans are to be prepared. Budgeting According to EPA officials, the agency plans to organize its budget, starting with the one for fiscal year 1999, on the basis of its strategic goals and objectives. Under GPRA, this change has to be negotiated with the Office of Management and Budget and the agency’s appropriations subcommittees. EPA officials told us that they are currently deciding how best to restructure the budget and have discussed their views with Office of Management and Budget officials. Agency officials said that more than one budgeting cycle may be needed to determine the best structure for the budget. GPRA requires federal agencies to use the program activities appearing in the President’s Budget as the basis for performance planning and measurement. This requirement aims at ensuring a simple, straightforward link among plans, budgets, and performance information and the related congressional oversight and resource allocation processes. However, program activity structures and their suitability for planning and Page 22 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System measurement purposes under GPRA vary from agency to agency. Under GPRA, agencies can consolidate, aggregate, or disaggregate program activity structures for performance planning purposes, where needed. (Using this approach requires subsequent crosswalks between performance plans and the program activities appearing in the budget request.) Agencies may also attempt to renegotiate program activities with their appropriations subcommittees and the Office of Management and Budget.14 According to EPA officials, the ongoing strategic and annual performance planning will influence budget decisions for fiscal year 1999. The Office of Management and Budget will review EPA’s and other agencies’ fiscal year 1999 annual performance plans when it considers their budget requests for that year. Accountability EPA’s two major activities to develop this component are the implementation of accountability pilot projects and the development of an agencywide reporting system. EPA is implementing 46 accountability pilot projects across the agency’s programs, offices, and regions. According to agency officials, these projects are being conducted to provide organizational units with experience in using performance measures and to obtain input into the development of an agencywide accountability system. Under the pilot projects, the responsible units are to prepare performance plans, identify performance measures, and use the measures to assess performance in achieving the goals set out in the plans. The final results of the pilot projects are due to OPAA in September 1997. In addition, during fiscal years 1994 through 1996, EPA, as well as other agencies, participated with the Office of Management and Budget in carrying out projects to pilot-test GPRA’s performance planning and reporting requirements. EPA’s six pilot projects were in the following programs: (1) Acid Rain, (2) Leaking Underground Storage Tanks, (3) Public Water System Supervision, (4) Environmental Technology Initiative, (5) Chesapeake Bay, and (6) Superfund. 14 In a March 1997 report, Performance Budgeting: Past Initiatives Offer Insights for GPRA Implementation (GAO/AIMD-97-46, Mar. 27, 1997), we noted a potential tension between legislative staff and agency officials on using program activities as a basis for agencies’ performance planning and measurement. Agency officials generally saw a need to be flexible in using program activities for planning and considered changes to program activity structures likely and desirable to better align programs with GPRA’s goals and objectives. Legislative staff generally viewed these structures as fundamental to congressional oversight of agencies’ activities. The staff were generally comfortable with the existing structures and suggested that changes might frustrate congressional oversight. Page 23 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix I EPA’s Conceptual Framework for and Activities to Implement a New Planning, Budgeting, and Accountability System EPA plans to have a formal reporting system as part of the accountability process. The target date for having the system in place is September 1999, when the agency will need to begin preparing the annual performance reports required under GPRA. The act requires agencies to submit these reports covering fiscal year 1999 by March 31, 2000. According to the Acting Chief Financial Officer, designing the accountability system will take some time because the system needs to be linked to the financial management and personnel rewards systems. Decisions on certain issues, such as the extent to which the system will be centralized or decentralized, have not yet been made. OPAA is reviewing previously used agencywide reporting systems to find out what did and did not work well, contacting other federal agencies to determine how they account for progress in meeting their goals, and examining reporting systems in the agency’s program offices to identify their potential use in the new system. Page 24 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency GAO has identified the following elements of a planning, budgeting, and accountability system for the Environmental Protection Agency (EPA) as important to its efforts to improve its processes for establishing its direction, setting priorities, allocating resources, and assessing results. These elements are based on prior GAO work; the findings and recommendations of various external and internal reports on EPA, including those of the National Academy of Public Administration; the requirements of the Government Performance and Results Act of 1993; and input from several former high-level officials responsible for planning, budgeting, and/or accountability while at the agency and others knowledgeable about EPA’s management. The elements are grouped by major management process or system component, that is, planning, budgeting, and accountability. They also are shown for an analysis component. The individual analysis elements support and could have been included under one or more of the other components. They have been segregated and are shown together to emphasize and highlight their importance to the effectiveness of the system. OBJECTIVE: To establish direction and priorities for EPA by clarifying its Planning Component mission; setting measurable, outcome-oriented goals and objectives; and determining how these goals and objectives will be met and how performance/results will be measured. Central office of planning, budgeting, accountability, and analysis to oversee, manage, and facilitate an integrated, agencywide planning, budgeting, and accountability structure that focuses on the (1) identification of, (2) appropriate allocation of resources for, and (3) measurement of results in achieving, priorities related to the agency’s mission. Strategic plan to define the agency’s mission and clearly establish its priorities in the longer term, that is, over the next 5 to 6 years. The priorities are to be expressed in measurable, outcome-related goals and objectives and linked to the agency’s mission, national environmental goals, comparative or relative rankings of health and environmental risks, and potential for reducing risk. The adequacy and suitability of the goals and objectives are to be periodically reviewed. At a minimum, this review could take place during updates of the action plans described below, which would occur about half way through the period covered by the strategic plan and again during the preparation of a strategic plan for the next 5 to 6 years. Page 25 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency The plan is to describe how the agency intends to achieve its goals and objectives, including how external factors and resources may affect what can be achieved. The plan is also to discuss how nonregulatory, incentive-based, and traditional regulatory approaches will be considered and what changes in the agency’s staffing or mix of skills will be required. Furthermore, the plan is to have a “futures” component that looks 20 to 30 years ahead to identify emerging environmental problems and new abatement strategies or techniques that may warrant research or other actions during the period covered by the plan. Because abatement and control actions can move pollutants from one environmental medium to another, the goals, objectives, and strategies set out in the plan are to apply to multiple media, rather than to a single medium—such as air, water, or land—wherever possible and appropriate. Stakeholders, particularly the Congress, the states, Indian tribes, and appropriate public interest groups, are to participate in the strategic plan’s development. Multiyear action plan for each objective in the strategic plan to detail decisions for the shorter term—over the next 2 to 3 years—on what the agency intends to achieve, how it will be achieved, and how progress will be measured. The plans are to also (1) establish milestones for completing the specified actions or activities, (2) determine the resources needed to achieve each milestone, and (3) define the roles and responsibilities of EPA and its partners, such as the states, for achieving the milestones.15 Annual performance plan for each objective in the strategic plan to describe the specific activities that are to be carried out and the specific results that are to be achieved in a particular fiscal year in support of the strategies and milestones in the action plan.16 The annual plan is to establish specific performance goals for these activities, summarize the necessary resources to conduct the activities, and set out the performance indicators to be used to measure performance. The plan links day-to-day program activities and outcomes to longer-term agency goals through the annual performance goals. It supports the budgeting process by providing information on the outputs and outcomes that could not be achieved if 15 These functions of the multiyear action plans could also be carried out as part of the strategic planning process and incorporated into the strategic plan. In this case, separate documents would not be prepared. 16 GPRA requires the use of program activities appearing in the President’s Budget as the basis for performance planning. GPRA allows agencies to consolidate, aggregate, or disaggregate program activities in conducting performance planning. Agencies may also attempt to renegotiate the program activity structures of their budgets with their appropriations subcommittees and the Office of Management and Budget. Page 26 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency resources were reduced and on the additional outputs and outcomes that could be achieved with additional resources. OBJECTIVE: To allocate the agency’s resources to the highest priorities so Budgeting Component as to achieve the greatest reduction in risks to human health and the environment. Annual budget document to finalize resource allocation decisions made during the internal planning process. The document is essentially the resources section of the annual performance plan prepared to present and defend the agency’s allocation decisions to the Office of Management and Budget and the Congress. Base program analysis to evaluate the contribution of individual programs to meeting the agency’s goals and identify areas of lower priority for potential disinvestment and shifting of resources to new initiatives or higher priorities in other programs. These analyses are to be performed by individual program managers under the guidance of the central planning, budgeting, accountability, and analysis office for review by the agency’s management. Goals-based program element budget structure to organize the presentation and execution of budget decisions around the agency’s goals, performance plans, and measures of success. OBJECTIVE: To provide EPA managers, the Congress, and the public with Accountability the information on programs’ finances and results needed to judge the Component agency’s performance in making efficient and effective use of resources to carry out its mission. Formal agencywide accountability system to compile—through internal reporting requirements, program evaluation results, and other analyses—relevant and essential information on the accomplishments and resources expended relative to the commitments made in annual performance plans, focusing on key measures of success for each goal. Performance measures to indicate the degree to which strategic goals are being met and to gauge the impact of programs by linking their activities or actions to environmental results. Include measures of the programs’ activity levels as necessary but focus on indicators of ambient Page 27 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency (environmental) conditions. The key performance measures, especially the technical measures, are to be peer-reviewed by independent sources, such as the Science Advisory Board. Program evaluations to provide information on the agency’s implementation of individual programs, including both what the agency is achieving given its mandate and how the mandate is contributing to the achievement of national goals. These evaluations are to be performed periodically, preferably near the end of the period covered by the current multiyear action plans and before the new plans are prepared. At a minimum, the evaluations are to be completed for major programs in time to be used in preparing new strategic plans. National environmental performance partnership agreements with the states to set out expectations for their contribution to achieving national environmental and agencywide goals and measures for assessing their performance in achieving these goals. Expectations are to be in the form of negotiated, goal-oriented commitments for EPA-funded and delegated program responsibilities and linked to individual national and agency goals. The states are to have input into the agency’s processes for setting these goals, selecting the strategies to achieve them, and allocating the resources to implement the strategies. Annual performance reports to evaluate and compare a program’s achievements to the goals set out in the annual performance plans. If a goal is not met, the report is to explain why and identify the actions needed to achieve the goal. These reports are to be provided to the Office of Management and Budget and the Congress. They are also to be made available to the public. OBJECTIVE: To provide the data and analyses decisionmakers need to Analysis Component make informed judgments about the environmental problems or concerns that the agency should address, its strategies for dealing with them, and the resources to be allocated for these purposes. ORGANIZATIONAL/PROCESS NEEDS: Bureau or center of environmental information and statistics that is independent of program/project offices and the central planning, budgeting, and accountability office. This entity is responsible to the Administrator for defining data quality and collection methods; identifying Page 28 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency data gaps; and independently assembling, analyzing, and reporting comprehensive, objective, and reliable statistics that reflect environmental quality and trends. Independent analytical capability to compile and assess relevant information on health and environmental risks, risk reduction potential, and performance across the agency’s programs. This capability is to be located within the central planning, budgeting, and accountability office. The information is to be provided by the program offices; other units within the agency, such as the research and development office; and outside sources. “Lookout” panel of scientists and other experts to identify future or emerging environmental problems for consideration by the agency’s management during the planning and budgeting process. Conflict of interest rules are to apply to panel members. Participants in the agency’s grants and contracts are generally not to participate on the panel. INFORMATION NEEDS: Monitoring data to comprehensively document environmental conditions and trends on a regional and national basis. In describing how performance is to be measured, multiyear action plans, annual performance plans, and national environmental performance partnership agreements are also to describe the extent to which ambient monitoring data are available for use in assessing programs’ results and determining how critical gaps in the data are to be filled. National environmental goals to express intended, long-range results of national (federal, state, local, industry, and private) efforts to protect the environment, reflecting stakeholders’ judgment as to the measurable progress the nation can and should achieve. Comparative or relative risk information to permit the assessment or weighing of health and environmental risks and the potential for reducing risks across environmental problems and programs. Options information to help identify cost-effective measures or actions to address environmental problems. This information is to include data on the benefits and costs of alternative ways of achieving similar results and the incremental social costs of given actions compared with their incremental benefits. Page 29 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix II Key Elements of an Effective Planning, Budgeting, and Accountability System for the Environmental Protection Agency Performance information, including risk reductions achieved; program evaluation results; and reporting on environmental indicators/outcome measures, program activity levels, and financial data, to identify how well and at what cost programs and projects are meeting their objectives. Page 30 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix III Objectives, Scope, and Methodology The Chairmen of the Subcommittees on VA, HUD, and Independent Agencies, Senate and House Committees on Appropriations, requested that we review EPA’s plans to implement a new planning, budgeting, and accountability system. Specifically, the Chairmen asked us to review the status of EPA’s efforts to (1) establish a new planning, budgeting, and accountability office and develop and implement an integrated system and (2) ensure that the system uses comprehensive scientific and environmental data and appropriate environmental measures. They also asked us to discuss EPA’s previous efforts to implement this type of system and to identify elements of an effective system that could be used as benchmarks to gauge progress in this current effort. To determine the status of EPA’s efforts to establish the new office and to develop and implement an integrated system, we reviewed relevant documents on EPA’s plans, target dates for these efforts, and actions taken; the staffing provided for the Planning, Budgeting, Accountability, and Analysis Work Group that was set up until the new planning, budgeting, and accountability organization could be established; and the status of the new office’s approval and staffing. We also discussed the status of EPA’s plans with the Acting Chief Financial Officer; the Comptroller; and other personnel of the Planning, Budgeting, Accountability, and Analysis Work Group and the new Office of Planning, Analysis, and Accountability. In addition, we compared activities for implementing the system with the recommendations of EPA’s Planning, Budgeting, and Accountability Task Force, which were accepted by the EPA Administrator. We also reviewed relevant documents and held discussions with staff of the Office of Policy, Planning, and Evaluation to determine the status of the National Environmental Goals Project. In addition, we discussed the purpose and status of the Integrated Risk Project with staff of the Science Advisory Board, reviewed pertinent documents on the project, and attended various sessions of the Board’s committees and subcommittees established to carry out the project. To determine EPA’s efforts to ensure that its new system uses comprehensive scientific and environmental data and appropriate environmental measures, we reviewed pertinent studies and other documents to identify improvements needed in the data and measures. We compared the needed improvements to EPA’s plans for the use of data and measures in the new system. In addition, we discussed the status of EPA’s efforts to identify and develop environmental measures with various EPA regional and state officials, including representatives of the Environmental Page 31 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix III Objectives, Scope, and Methodology Council of the States. Furthermore, we attended two EPA workshops on environmental data and measures. These were “Managing for Environmental Results: Using Goals and Indicators,” held in September 1996, and “Environmental Management in the Nineties: Using the Right Tools,” held in January 1997. To identify elements that could be used as benchmarks for monitoring EPA’s progress in implementing an integrated planning, budgeting, and accountability system, we reviewed the findings and recommendations of the NAPA report on EPA, the requirements of the Government Performance and Results Act, our prior reviews of EPA, and other external and internal EPA studies and evaluations to identify key elements of an effective planning, budgeting, and accountability system for EPA. We then obtained and incorporated comments on these key elements from four former top EPA officials who had responsibilities for the planning, budgeting, and/or accountability functions when at the agency—a deputy administrator and three assistant administrators (two for policy, planning, and evaluation and one for administration and resource management). We also obtained comments from representatives of NAPA and EPA’s Science Advisory Board. We conducted our review from June 1996 through May 1997 in accordance with generally accepted government auditing standards. Page 32 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Appendix IV Major Contributors to This Report Stanley J. Czerwinski, Associate Director Resources, Edward A. Kratzer, Assistant Director Community and Raymond H. Smith, Jr., Evaluator-in-Charge Economic Frank J. Gross, Senior Evaluator Jennifer W. Clayborne, Staff Evaluator Development Division, Washington, D.C. Bruce Skud, Senior Evaluator Boston Field Office (160359) Page 33 GAO/RCED-97-156 EPA’s Planning, Budgeting, and Accountability System Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: email@example.com or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Managing for Results: EPA's Efforts to Implement Needed Management Systems and Processes
Published by the Government Accountability Office on 1997-06-18.
Below is a raw (and likely hideous) rendition of the original report. (PDF)