United States General Accounting Office GAO Report to the Chairman, Committee on Banking and Financial Services, House of Representatives October 1999 MANAGING FOR RESULTS Strengthening Regulatory Agencies’ Performance Management Practices GAO/GGD-00-10 United States General Accounting Office General Government Division Washington, D.C. 20548 B-281899 October 28, 1999 The Honorable James A. Leach Chairman, Committee on Banking and Financial Services House of Representatives Dear Mr. Chairman: As part of your oversight of federal agencies that are responsible for 1 regulating financial institutions, you expressed concern over the progress these agencies have made in implementing the Government Performance and Results Act of 1993 (Results Act). At your request, in September and November 1998, we provided you with our observations on how the financial institution regulatory agencies could improve their annual 2 performance plans. Subsequently, on November 10, 1998, you requested that we undertake a performance management and measurement “best 3 practices” study to identify model approaches that might help these agencies in their efforts to better implement the Results Act. This report summarizes information provided in a May 26, 1999, briefing to representatives from the federal financial institution regulatory agencies, your office, and other federal representatives. In the briefing, we presented performance management and measurement practices that could position the agencies to more effectively implement the Results Act. Details on these practices are presented in appendixes I through V. Additionally, a bibliography and a list of related GAO products are included at the end of this report. 1 For purposes of this report, the federal financial institution regulatory agencies include the Federal Deposit Insurance Corporation, the Federal Reserve Board, the National Credit Union Administration, the Office of the Comptroller of the Currency, and the Office of Thrift Supervision. 2 The Results Act: Observations on FDIC’s Annual Performance Plan (GAO/GGD-98-190R, Sept. 15, 1998), The Results Act: Observations on the Federal Reserve’s 1998-99 Biennial Performance Plan (GAO/GGD-99-9R, Nov. 9, 1998), The Results Act: Observations on NCUA’s Annual Performance Plan (GAO/GGD-98-192R, Sept. 15, 1998), The Results Act: Observations on OCC’s Annual Performance Plan for Calendar Year 1998 (GAO/GGD-98-189R, Sept. 15, 1998), and The Results Act: Observations on OTS’ Annual Performance Plan (GAO-GGD-98-191R, Sept. 29, 1998). 3 “Best practices” are generally recognized as the processes, practices, and systems identified in organizations that could provide models for other organizations. Page 1 GAO/GGD-00-10 Managing for Results B-281899 We gathered information from 23 federal and state organizations that we Results in Brief or other credible sources identified as using or planning to use a variety of useful practices to enhance specific aspects of their performance management and measurement processes. We have grouped these practices into the following five categories, which are also detailed in appendixes I through V: • Restructure the organization’s management approach to become more performance-oriented. • Establish relationships outside of the organization to enhance performance. • Refine performance goals, measures, and targets to better translate activities into results. • Strengthen analytical capabilities and techniques to better meet performance management information needs. • Assess performance-based management efforts on a continuous basis to identify areas for improvement. Each appendix describes individual practices within each category and provides examples of how one or more of the state and federal organizations we reviewed said they have implemented or plan to implement these practices. The organizations, although they had different missions, sizes, and organizational structures, said they consistently recognized that these practices are important in their efforts to develop a stronger results orientation. The organizations used the practices to varying degrees. Thus, the practices did not appear to be a function of any particular organizational characteristic. We believe the practices would be readily transferable to the federal financial institution regulatory agencies or other governmental agencies seeking to improve their implementation of the Results Act. In addition, the practices are consistent with those identified in our previous reports, which are listed at the end of this report. These reports, for example, described approaches agencies were taking to address analytical and technical challenges in measuring program performance, align employee performance with organizational missions and goals, and address the influence of external factors in developing performance measures. Congress passed the Results Act as part of a legislative framework to Background instill performance-based management in the federal government. The Results Act establishes a management system to set agency goals for program performance and to measure results against those goals. In Page 2 GAO/GGD-00-10 Managing for Results B-281899 enacting the Act, Congress and the administration realized that the transition to results-oriented management would not be easy. For that reason, the Act provided for a phased approach to implementation. Implementing the Results Act in a regulatory environment is particularly challenging. In the past, regulatory agencies have cited numerous barriers to their efforts to establish results-oriented goals and measures. These barriers included problems in obtaining data to demonstrate results, accounting for factors outside of the agency’s control that affect results, and dealing with the long time periods often needed to see results. Over the past several years, we have issued reports that identified practices for improving Results Act implementation in federal agencies. These reports have focused on, among other things, overcoming barriers specific to regulatory agencies, improving the usefulness of annual performance plans to decisionmakers, and measuring program results that are under limited federal control. These reports point out the depth and scope of management practices needed to successfully implement performance-based management as envisioned under the Results Act. At your request, during the last year, we reviewed the annual performance plans of the federal financial institution regulatory agencies. Our review identified several ways in which the agencies could improve their plans, including making performance goals more results-oriented; more clearly linking performance goals and measures with strategic goals; more fully describing crosscutting efforts with other agencies; more fully explaining how strategies and resources would be used to achieve agency goals; and providing more details on data verification and validation efforts and data 4 limitations. To identify practices potentially useful to the federal financial institution Scope and regulatory agencies, we first gathered views from these agencies to Methodology identify issues they believed we should consider in our work. Overall, the agencies were interested in the specific activities, organizational support, and incentives that would comprise a successful performance management approach, and they wanted us to use specific examples whenever possible. To develop this information, we selected several government organizations—mostly regulatory in nature—that our past work or other credible sources identified as having, or planning to implement, all or parts of performance management and measurement practices that other agencies might find useful. 4 GAO/GGD-98-190R, GAO/GGD-99-9R, GAO/GGD-98-192R, GAO/GGD-98-189R, and GAO-GGD-98-191R. Page 3 GAO/GGD-00-10 Managing for Results B-281899 At the federal level, the organizations we reviewed included the Federal Aviation Administration, the Food and Drug Administration, the U.S. Customs Service, the U.S. Coast Guard, the National Highway Traffic Safety Administration, the Environmental Protection Agency, the Occupational Safety and Health Administration, the Food Safety and Inspection Service, the National Science Foundation, and the Nuclear Regulatory Commission. In addition, we reviewed the performance management approach of Canada’s Office of the Superintendent of Financial Institutions. At the state level, the organizations we reviewed included the following five Florida state agencies: the Office of Program, Policy Analysis, and Government Accountability; the Department of Banking and Finance; the Department of Insurance; the Game and Freshwater Fish Commission; and the Agency for Health Care Administration. We also reviewed the following six Minnesota state agencies: the Department of Economic Security, the Pollution Control Agency, the Department of Transportation, the Office of Environmental Assistance, the Department of Corrections, and the Department of Revenue. Lastly, we reviewed the Texas Department of Banking. We collected data on the performance management and measurement efforts of these organizations. We did not directly verify the accuracy of these data. Because of the recent adoption of the practices, we were unable to determine the extent to which the organizations’ emphasis on results was directly traceable to specific practices. However, we did ask the organizations to review our results for accuracy and completeness. Each organization agreed with our characterization of the information discussed in this report. To supplement the organizational information, we reviewed available public and private sector performance management and measurement literature. From the organizations we studied, supplemented by our literature review, we identified organizational performance management and measurement practices that appeared useful for the federal financial institution regulatory agencies. As part of our methodology, we also asked several government performance management experts to review these practices and our interpretation of the information. These experts generally concurred with our observations and characterizations. We did our work between September 1998 and May 1999 in accordance with generally accepted government auditing standards. Page 4 GAO/GGD-00-10 Managing for Results B-281899 We are providing copies of this report to Representative John J. LaFalce, Ranking Minority Member of this Committee; the Honorable John D. Hawke, Jr., Comptroller of the Currency; the Honorable Alan Greenspan, Chairman, Board of Governors of the Federal Reserve System; the Honorable Donna Tanoue, Chairman, Federal Deposit Insurance Corporation; the Honorable Norman E. D’Amours, Chairman, National Credit Union Administration; and the Honorable Ellen Seidman, Director, Office of Thrift Supervision. We will also make copies available to others on request. This report was prepared under the direction of Kane Wong, Assistant Director. Key contributors to this assignment were Sharon Caudle and Patrick Ward. Please contact me at (202) 512-8678 or Mr. Wong at (415) 904-2000 if you or your staff have any questions. Sincerely yours, Richard J. Hillman Associate Director, Financial Institutions and Markets Issues Page 5 GAO/GGD-00-10 Managing for Results Contents Letter 1 Appendix I 10 Summary 10 Practice Category 1: Restructure Management Approach Appendix II 24 Summary 24 Practice Category 2: Establish Relationships Outside of the Organization Appendix III 32 Summary 32 Practice Category 3: Refine Performance Goals, Measures, and Targets Appendix IV 45 Summary 45 Practice Category 4: Strengthen Analytical Capabilities and Techniques Appendix V 55 Summary 55 Practice Category 5: Continue Improving Performance-Based Management Page 6 GAO/GGD-00-10 Managing for Results Contents Bibliography 57 Related GAO Products 62 Tables Table IV.1: Florida Measure Definitions 47 Figures Figure I.1: Example of Mission Statement Used to 16 Develop Strategic Goals—U.S. Coast Guard Figure I.2: Example of Relating Efforts to Outcomes— 17 U.S. Coast Guard Figure II.1: Example of a Survey for Client Input on 27 Agency Performance—Florida Department of Banking and Finance Figure II.2: Matrix Showing Crosscutting Program 28 Relationships—Nuclear Regulatory Commission Figure III.1: Constructing National Highway Traffic 34 Safety Administration’s Program Logic Model Figure III.2: Decisionmaking Tiers 39 Figure III.3: Examples of Performance Comparisons 41 Figure III.4: Examples of Targets Set Using Baseline 44 Data—U.S. Coast Guard Figure IV.1: National Highway Traffic Safety 51 Administration Matrix to Define Problems and Strategies Page 7 GAO/GGD-00-10 Managing for Results Contents Abbreviations BCI Border Coordination Initiative DOT Department of Transportation EPA Environmental Protection Agency FAA Federal Aviation Administration FDA Food and Drug Administration FDBF Florida Department of Banking and Finance FDI Florida Department of Insurance FGFFC Florida Game and Freshwater Fish Commission FHCA Florida Agency for Health Care Administration FSIS Food Safety and Inspection Service INS Immigration and Naturalization Service MDES Minnesota Department of Economic Security MDOC Minnesota Department of Corrections MDOR Minnesota Department of Revenue MDOT Minnesota Department of Transportation MNA Mission Needs Analysis Process MOEA Minnesota Office of Environmental Assistance MPCA Minnesota Pollution Control Agency NHTSA National Highway Traffic Safety Administration NOAA National Oceanic and Atmospheric Administration NRC Nuclear Regulatory Commission NSF National Science Foundation OPPAGA Office of Program, Policy Analysis and Government Accountability OSFI Office of the Superintendent of Financial Institutions OSHA Occupational Safety and Health Administration PBPM planning, budgeting, and performance management TDOB Texas Department of Banking MOE Measures of Effectiveness Page 8 GAO/GGD-00-10 Managing for Results Page 9 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach 1 In the first practice category, the organizations said they restructured Summary their management approach to become more performance-oriented. Practices in this area included the following: 1. Strengthen the organizations’ performance-based management approach. 2. Enhance organization ownership and coordination of performance management efforts. 3. Redesign responsibility and accountability structures. Practice 1: Strengthen As a first practice, the organizations said they strengthened their performance-based management approach. The characteristics of this Management Approach practice included the following: • Make implementation of performance-based management and achievement of outcomes top organizational priorities. • Conduct a comprehensive assessment of the organizations’ internal and external environments. • Establish a high-level performance management support capability. • Develop core performance-based management competencies through skill development and operational pilots. Make Implementation and The organizations said top management in the organizations was Achievement Top Priorities committed to successful implementation of performance-based management and the achievement of agreed-upon outcomes, making them top agency priorities. Top management buy-in and commitment, a high level of involvement, and consistency in leadership characterized the organizations. According to the organizations, top managers were involved in all aspects of performance-based management, from developing a performance monitoring and evaluation system to identifying and assessing key measures. For example: • The Federal Aviation Administration (FAA) said that to demonstrate support, the agency head needed to attend key meetings, support requests for the time and resources to work on strategic planning, and be willing to talk with managers and staff about performance issues and processes. • To show high-level commitment, the U.S. Customs Service (Customs Service) said the agency put an Assistant Commissioner in charge of the 1 The 5 practice categories in appendixes I through V contain 12 practices that are numbered consecutively throughout the appendixes. Page 10 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach performance management redesign in its trade compliance program. The organizations we studied also said they used several methods of communication to explain the purpose, processes, implementation strategies, and staff responsibilities for performance management and measurement. For example: • The Minnesota Office of Environmental Assistance (MOEA) said that the agency stressed outcomes in major policy reports. MOEA shared performance information with staff and detailed how the information supported major agency decisions. • To set the tone for major changes in its management approach, the Minnesota Pollution Control Agency (MPCA) said it developed descriptions of how the organization would be changing its management approach by, for example, developing shared goals, using environmental outcomes, creating alliances with others, and becoming a learning organization. In addition, the organizations said top management ensured that the organizations were accountable for the implementation of performance- based management by rigorously tracking and evaluating action items designed to implement strategic plans and meet performance expectations. Variances between actual performance and expected performance targets were promptly identified and acted upon. For example: • The Food Safety and Inspection Service (FSIS) said line managers were (1) involved in preparing plans for their own areas and (2) regularly discussed with top management the activities under way to implement the plans. • The National Science Foundation (NSF) and the Minnesota Department of Economic Security (MDES) said they used management meetings to discuss performance management. NSF said it had a permanent slot on the Director’s agenda to address Results Act implementation. MDES said the agency ensured that performance management issues were discussed regularly at management meetings. Conduct a Comprehensive As a first step toward developing a tightly integrated and comprehensive Assessment of Internal and strategic management approach, the organizations said they performed External Environments comprehensive internal and external assessments. These assessments generally included identifying customers and stakeholders and assessing the agency’s mission, vision statement, and operating principles. For example: Page 11 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach • FAA said its overall assessment process was to examine FAA’s legislation, define mission areas, and define measures for the intended outcomes. • At the beginning of its performance management efforts, MDES said it conducted a self-assessment using the Minnesota Quality Award criteria, which were based directly on Malcolm Baldrige award criteria. According to MDES, this effort allowed the agency to systematically assess core processes and, at the same time, more directly engage agency leadership in performance-based management. • The Food and Drug Administration (FDA) said that after passage of the FDA Modernization Act of 1997, it conducted an assessment of its external and internal environments to identify challenges the agency would face over the next several years. According to the agency, the assessment included identifying FDA’s statutory requirements and public expectations, evaluating environmental factors affecting the agency’s future actions, and reviewing current program performance. Establish a High-Level The organizations said they recognized that even when performance-based Organizational Capability management had the involvement and support of top management, there was still a need for a central point to facilitate and support performance management. Therefore, they established central units or individuals to help coordinate the organizations’ performance management efforts, ensure consistency across the organizations, and provide training on methodologies and approaches. These individuals or units were to act as internal consultants, and they were not responsible for developing plans, goals, targets, or measures. Those aspects were clearly the responsibility of program managers. For example: • The Customs Service said its Office of Budget Formulation and Planning served as the facilitator for Results Act implementation, with function managers and process owners developing their own performance plans. • The Minnesota Department of Transportation (MDOT) said it used its Data Services Office to facilitate the agency’s performance measurement efforts. In addition, the office acted as a clearinghouse for best practices in measurement and a source for benchmarking data. Develop Core Performance- The organizations said they recognized that to successfully implement Based Management results-oriented organizational strategies, they needed managers and staff Competencies to be competent in at least the basics of performance management. According to the organizations, the competencies were needed for two purposes. The first purpose was to understand the rationale of performance management and how measurement could be used. The second purpose was to go beyond understanding and actually put performance management and measurement to use in directly improving Page 12 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach organizational and program performance. The organizations said they sought to build the necessary competencies through training, on-the-job activities, and the use of pilot efforts in lead programs or organizational units. The organizations said they offered in-depth training on performance management. For example: • The National Highway Traffic Safety Administration (NHTSA) said it conducted a 3-day “train the trainer” seminar on Results Act-related areas early in its Results Act efforts. NHTSA also joined the Consortium for Continuous Improvement, which the agency has found to be a useful source of training materials. • MDES said that it used a consultant to provide a 3-day training session for all managers and supervisors on various topics, such as quality concepts and performance requirements. Later, the agency provided 40 hours of similar training for the remaining 1,500 employees. • The Florida Department of Banking and Finance (FDBF) said it provided performance measurement training as a part of new employee orientations and provided supervisors with training in performance management basics. FDBF also said it conducted yearly refresher training for staff. According to the organizations, pilot efforts within the organizations were used to motivate and provide “lessons learned” for other parts of the organizations or to focus on specific performance management tasks. For example: • FAA said one FAA line of business with more experience in performance 2 management served as a model for the rest of the agency’s lines of business for Results Act implementation. • MDOT said it used divisions “ahead of the curve” in its performance management efforts. The agency said these divisions were more practiced in using performance data to make decisions and support funding requests, and they motivated other divisions to improve their performance management efforts. • The Environmental Protection Agency (EPA) said its Region 10 was leading a pilot to establish baselines for water quality and measure the effects of EPA activities on dairy farmer behavior and water quality. 2 Within FAA, there are six lines of business: Air Traffic Services, Research and Acquisition, Regulation and Certification, Airports, Civil Aviation Security, and Commercial Space Transportation. Page 13 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach Practice 2: Enhance As a second practice, the organizations said they enhanced the ownership of and commitment to their performance management efforts across their Ownership and organizations. The characteristics of this practice included the following: Commitment • Have management and staff fully participate in performance management decisionmaking and agree on program mission areas, goals, measures, and targets. • Develop a well-defined and tightly focused mission statement to direct agency efforts. • Develop a “family” of plans to implement strategies across the organization. • Align the organizational structure and coordinating management interrelationships. Have Management and Staff The organizations said they ensured that managers and staff at all levels Fully Participate extensively participated in the development of goals, targets, and measures, seeking to secure agreement between management and staff on the program mission areas, goals, targets, and measures that would be used for detailed planning and program management. In addition, the managers and staff actively participated in the implementation and tracking of performance results. For example: • FAA said that its unions and key managers used a National Partnership Council to address issues of common concern, and that the Council participated in the development of the agency’s strategic plan. In addition, FAA said all of its employee associations received the plan for comments. FAA program managers also had to agree on agency mission areas and measures. Further, the agency said there was a conscious effort to focus on the most important results or outcomes, resulting from a long process of negotiation with lines of business to agree on the goals and measures. • The Texas Department of Banking (TDOB) said it used an iterative process to share a draft strategic plan with field staff, obtain comments, and then redistribute the draft. TDOB’s process was repeated three times before final agreement was reached. The agency also said it included key divisional staff on a strategic planning task force. • Canada’s Office of the Superintendent of Financial Institutions (OSFI) said the agency’s Performance Measure Advisory Committee—tasked with developing a performance measurement framework—met with directors and executives to understand what measures would be useful to them. • The Florida Agency for Health Care Administration (FHCA) said it made extensive use of workgroups, using them to develop measures, educate staff, enhance interdivisional communication, and obtain feedback from the legislature on FHCA’s measures. Page 14 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach Use a Well-Defined and The organizations said they built mission statements that were focused, yet Tightly Focused Mission would encompass all of the programs within the organizations. The Statement organizations said they found that a clear, concise mission statement formed the foundation for a coordinated, balanced set of strategic goals, performance measures, and strategies to implement the goals. Without such a mission statement, the organizations said they found it difficult to develop an appropriate hierarchy of goals and strategies across the organizations and to clearly relate the associated outputs and outcomes to the organizations’ missions. For example: • The Nuclear Regulatory Commission (NRC) said it started its Results Act efforts with the agency’s mission statement, using it to formulate “what if” questions in linking strategic goals to its mission. That is, the agency said it assessed whether pursuing a particular strategic goal would lead to fulfilling its mission. • TDOB said it reduced its mission statement from two paragraphs to one sentence by focusing on key outcomes and removing any discussion of goals and strategies. The new mission statement was “to promote a stable state banking and financial services environment and provide the public with convenient, safe, and competitive financial services.” • FAA said its mission statement—to provide a safe, secure, and efficient global aerospace system that contributes to national security and the promotion of U.S. aerospace safety—translated into its three mission- based strategic goals on safety, security, and system efficiency. • As shown in figure I.1, the U.S. Coast Guard said it translated its mission statement—to protect the people, the environment, and the maritime security of the United States—into five strategic goals that described the outcomes the agency sought to achieve or influence over the long term. These goals included the major outcome areas of safety, protection of natural resources, mobility, maritime security, and national defense. The Coast Guard said that the agency’s programs, policies, facilities, processes, procedures, activities, and requirements all should ultimately be linked to achieving the agency’s mission, vision, and strategic goals. Figure I.2 shows a diagram used by the Coast Guard to illustrate the relationship of effort—what the Coast Guard does—to outcomes—why the Coast Guard undertakes its efforts. The Coast Guard said it had five traditional roles that are categorized into specific mission areas for organizational and administrative purposes. Activities performed in each mission area were to contribute to one or more of the agency’s strategic goals, leading to accomplishment of the Coast Guard’s mission. Page 15 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach Figure I.1: Example of Mission Statement Used to Develop Strategic Goals—U.S. Coast Guard Source: U.S. Coast Guard. Page 16 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach Figure I.2: Example of Relating Efforts to Outcomes—U.S. Coast Guard Source: U.S. Coast Guard. Design a Family of Plans The organizations said they saw a need to direct and coordinate Across the Organization performance management and measurement efforts within each of their organizations, and found that using a family of plans encouraged the direct linkage of strategic goals and measures to operational and support goals, measures, and related activities. By using performance plans at all levels of an organization, each level’s goals could be carefully integrated with those of the other levels, allowing all of the organization’s strategies and activities to be oriented toward achieving the principal strategic goals. The organizations also said that a family of plans helped prevent the occurrence of contradictory goals across the organization. In addition, they said a hierarchy of plans was developed to recognize that different management levels need different performance information, and that goals, objectives, and measures become more meaningful if they relate to Page 17 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach the appropriate level of responsibility and control. The use of measures targeted at different decisionmaking tiers is discussed further in practice 7. Following are examples of organizations’ use of a family of plans: • The Coast Guard said it had a complete family of plans illustrating how planning efforts at various levels of the Coast Guard were intended to be linked. The agency said the family of plans was designed to facilitate the communication of vital information for decisionmaking purposes between the Coast Guard’s operational and logistics components in the field and at headquarters. According to the agency, the family of plans started with the Coast Guard 2020—a broad internal and external environmental scan— which outlined probable challenges and opportunities that the Coast Guard may face in the coming decades. The strategic plan served as the implementation vehicle for Coast Guard 2020 and was developed to provide focus and alignment for the development of business plans and specialized plans. The performance plan showed how the agency intended to translate the resources it has, and those it was requesting, into performance outcomes for a specific budget year. Other parts of the family of plans included (1) the performance report that detailed the annual level of performance actually achieved; (2) the annual budget that detailed the resources needed to fund operations and logistics activities; (3) specialized plans for major capital assets, such as workforce and systems; (4) business plans for headquarters strategies, measures, objectives, and resources; and (5) regional strategic assessments that were assessments of risk, threat, opportunity, and demand as well as resource requirements and major issues from area and district commanders for incorporation into headquarters business plans and the agency strategic plan. • FAA said it used a linkage across several primary documents, as follows: (1) FAA’s strategic plan coordinated with the DOT plan, (2) FAA’s annual performance plan, (3) FAA’s line of business and performance plans, and (4) FAA’s performance agreement with the DOT Secretary. According to FAA, the agency’s lines of business prepared their own business plans, complete with their own performance measures. The plans described the work and work-related activities that each major organizational unit would undertake in the next several years. • NRC said it developed operating plans for each of its programs, covering program commitments, significant information technology initiatives, program assumptions, self-assessments and evaluations, and a summary of quarterly resource changes. The program commitments included various items, such as planned accomplishments, resources, milestones, measures, status, activities, and quarterly targets. Page 18 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach • MDOT said it operated from a family of plans that included a statewide strategic plan, an agencywide strategic plan, district strategic plans, and an upcoming agencywide business plan. Change Organizational The organizations we studied said they followed the principal that “form Structure and/or Management follows function,” or in this case, form follows the intended results Interrelations described in the mission statement and the strategic goals. The organizations said they evaluated whether their organizational structures aided or hindered meeting performance expectations, and made the necessary changes. For example: • Using a business process improvement methodology, the Customs Service said it identified its core and mission support processes and managed through those processes. Its outcome-oriented core processes included trade compliance (the commercial importation of merchandise), passenger processing (the processing of passengers entering and leaving the United States by vehicle, vessel, and air), and outbound processing (the commercial exportation of merchandise). Mission support processes included information and technology, financial, and human resources processes. The agency said it assigned a process owner to each of its processes to ensure accountability. • FDBF said it defined separate program areas by first identifying the ultimate outcomes desired by the agency, such as consumer protection or bank safety and soundness. • The Minnesota Department of Corrections (MDOC) said the agency organized its divisions by customer, such as adult facilities and juvenile facilities, with each division required to relate its efforts to the agencywide outcome goals. • MPCA said it was restructuring its divisions so that it could attack environmental problems more holistically and avoid duplication. Instead of being organized around four “media” divisions—Air Quality, Ground Water and Solid Waste, Hazardous Waste, and Water Quality—the agency was becoming geographically organized, allowing different district offices to focus their resources on the most serious problems faced in their area. The organizations said they also recognized that coordination among managers was important in achieving performance goals and, therefore, crafted operating procedures and activities to foster this coordination, especially when several program or functional areas shared similar goals. For example: • FAA said the success of the FAA plan required assigning both accountability and coordination within the organization. FAA lead and Page 19 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach support organizations were designated for each goal and project. According to FAA, each lead organization called together managers of key supporting organizations to discuss interrelationships, what is required by whom and when, and how the goal or project would be accomplished. • FSIS said that the performance management efforts helped program managers work together by laying out common goals and describing how managers should interact with each other to meet the goals. This provided a structure to implement the goals and identified assignments and tasks to implement them. Practice 3: Redesign As a third practice, the organizations said they redesigned organizational responsibility and accountability structures. The characteristics of this Responsibility and practice included the following: Accountability Structures • Target daily activities and projects to support strategies to implement goals. • Integrate performance goals, measures, and costs into budget structures and decisionmaking. • Link performance appraisals to performance accountability and responsibility. Target Daily Activities to The organizations said the family of plans previously mentioned was to Implement Performance align strategic goals and lower-level organizational goals. As a further Goals alignment step, the organizations said they clearly linked organizational strategies—which were to reflect daily activities and projects—to strategic goals. According to the organizations, this approach encouraged an outcome orientation down to the task level in the organization, thereby building a clearly linked hierarchy of goals, objectives, measures, and implementation activities. Performance management became a clear part of how the organizations were to manage day to day in such areas as securing and justifying resources and defining accountability. For example: • EPA said it used more detailed strategic plan objectives with subobjectives for annual performance goals. Shorter term activities indicated the contribution of lower levels of the organization to the strategic goals. • FSIS said its strategic plan was coordinated with an agency database containing 26 activities that were used to track, on a monthly basis, specific implementation tasks assigned to staff for the agency’s new meat and poultry safety program. • NHTSA said each program office had defined milestones for activities and projects supporting strategies for the annual performance plan, such as completing rulemaking by a certain date. Page 20 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach • The Minnesota Department of Revenue (MDOR) said the agency used information from each of its functional areas in its daily management to compare various measures to current targets as well as the results achieved in prior years. According to the agency, the performance information was contained in reports available to all employees over the agency’s network, all updated at specific times. For example, the agency said it weekly updated the report on the number of days taken to process refunds for paper tax returns. The report provided information on the year- to-date count for specific processing times, including the target of processing the returns within 90 days if received by April 1, and 120 days if received after April 1. The report contrasted this current information with the processing time performance of the past 2 calendar years. Integrate Performance The organizations said they recognized that one of the biggest incentives Management Into Budgetary for performance-based management was seeing information on results Processes integrated into budgetary structures and decisionmaking. The organizations said they aligned their budgets with program activities which, in turn, were tied to program goals, targets, and measures. The organizations said they were very specific in defining the direct and indirect costs of the activities that produced outputs, especially those that could be linked to intermediate or final outcomes. For example: • TDOB said it used performance measurement to allocate and prioritize the use of its limited resources. When TDOB did not meet targets in one area, the agency said it could divert resources from an area where the agency had exceeded its performance targets. • The Coast Guard said it was using activity-based costing to identify direct and indirect costs in program areas and developing standard rates for time and for fully loaded costs. According to the Coast Guard, this will enable it to identify all costs to the activity level and link them to outcome areas. The agency said it was also refining its resource allocation process to align its assets with mission requirements. • NRC said it was implementing an integrated and disciplined system to improve its processes for planning, budgeting, and performance management (PBPM). According to NRC, the PBPM system was to establish a process for defining agency goals, develop cost-effective strategies to achieve those goals, determine the resources needed to implement the agency’s strategic direction, and measure and assess the agency’s progress. • FAA said it was in the process of developing a cost accounting system that linked outputs to activities and costs. FAA said it already had a formal agency needs assessment process in place, called the Mission Needs Analysis Process (MNA). This formal process, for example, called for an Page 21 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach assessment of the need for new systems and equipment to meet the mission and strategic goals of the agency that related to future National Airspace System capital requirements. According to FAA, each organization within the agency was to conduct its own assessments to establish its mission needs. In addition, FAA said it was developing a cost and performance management system to track the costs incurred in performing services, to allow management to measure the effectiveness of its workforce, and to rank return on assets. • The Customs Service said it also was developing a resource allocation model to identify the effect of increases in operational staff on requirements for support staff. According to the agency, the model was to be used to develop a budget and a basis for requesting funding. The model allowed the user to perform an “what-if” analysis, such as changes in the amount of overtime, minimum staffing constraints, or staffing allocations. Use Performance Agreements The organizations said they incorporated performance management into and Appraisals performance agreements and appraisals by clearly specifying performance responsibilities and accountabilities. The organizations said they did this to (1) increase the visibility and importance of performance management results and (2) encourage managers and staff to pay attention to performance information and outcomes. In addition to the performance agreements and appraisals, the organizations said they established a firm link, at the individual and/or unit level, between the performance expectations and incentives. The incentives included both monetary and nonmonetary incentives. Nonmonetary incentives included recognition awards, regular performance reports to the organization, and managerial flexibility in exchange for more performance accountability. For example: • NSF said agency staff were determined that whatever performance management system was implemented, it would be consistent with normal program management and a useful management tool. As a result, senior executive service performance objectives contained at least one Results Act-related element. • FAA said that each month its lines of business managers reported on their progress in reaching the goals set forth in the performance agreement between the FAA head and the Department of Transportation (DOT) Deputy Secretary. The agency said the performance agreement contained measures used in the performance plan and specific projects and milestones that were not in the performance plan. FAA also said it was piloting, at the senior executive service level, a program tying incentives, such as bonuses, to key performance measures in the strategic plan. Agreements with executive-level staff were to specify how the executives Page 22 GAO/GGD-00-10 Managing for Results Appendix I Practice Category 1: Restructure Management Approach would contribute to FAA’s three primary goals, weighted by the areas in which the executives had responsibilities. • NHTSA said it used an employee performance management program that focused on results, not behaviors and characteristics, and provided a linkage between individual performance results and required outcomes of organizational performance. According to NHTSA, outcomes in the employee’s performance plan were desired end-results that the employee and the organization were trying to achieve. The employee performance plan was to link the work of the employee to the organization’s goals. NHTSA said the outcomes were generally beyond the employee’s ability to control, but defined what the employee and the organization were working toward. The employee performance plan contained performance targets (equivalent to performance standards) for each outcome. The agency said the targets were those activities that the employee should accomplish by the end of the performance period that contributed to the attainment of the outcome or desired end-result. The employee was accountable for the performance targets, not the outcomes per se. Page 23 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization In the second practice category, the organizations said they established Summary relationships outside of the organization for the purpose of enhancing performance. They then worked through these relationships to implement specific programs. Practices in this area included the following: 4. Establish results-oriented collaborative relationships with regulated entities and program delivery partners. 5. Establish partnerships with other organizations involved in implementing crosscutting programs. Practice 4: Establish In the fourth practice, the organizations said they established, at a program level, results-oriented collaborative relationships with regulated Results-Oriented institutions and program delivery partners. Specific practices included the Collaborative following: Relationships • Work with regulated entities in identifying the organization’s mission, performance goals, targets, and measures. • Involve regulated entities in the prevention aspect of performance. • Build consensus among program delivery partners on performance targets, measures, and data use. • Obtain periodic input from regulated entities and delivery partners on the organization’s performance management efforts. Involve Regulated Entities in The organizations said they recognized the need for the entities they Identifying Mission, Goals, regulated to be involved in developing the organizations’ mission, goals, Targets, and Measures and targets, as well as selecting performance measures and helping determine their use in program operations. According to the organizations, the consultative process was meaningful and involved extensive dialogue with the regulated entities. For example: • FAA held annual sessions with its stakeholders, called “Challenger Sessions,” in which FAA discussed its mission and goals with representatives from industry groups, such as airlines and manufacturers, as well as user groups, such as pilots. The agency said the stakeholders also played a role in defining FAA’s strategic goals and strategies. For example, FAA said that the agency established air traffic control preferred routes to minimize conflicts in congested airspace. However, the FAA routes often differed significantly from the routes that airline pilots and flight planners would prefer to optimize their operations on the basis of their own objectives and constraints. In response, FAA said it began working closely with airlines to share air traffic information so that collaborative decisions could be made. Page 24 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization • The Coast Guard said it conducted regional strategic assessments that examined the demand for agency services and attempted to anticipate current and future demands for services and resources. These assessments examined partnerships with such entities as ports and waterways. • The Customs Service said it worked with industries and communities to achieve outcomes. To illustrate, the agency said it worked with local community and Mexican government officials to measure delays entering the United States from Mexico and develop strategies to reduce the delays but still ensure compliance. • MDOR said the agency validated its initial strategic objectives with over 70 customers and external stakeholders and used focus groups as part of its target-setting processes, thereby getting customer feedback on what constituted acceptable targets. Involve Regulated Entities The organizations said they took the approach that regulated entities in the Prevention Aspect of should join with the regulators in the prevention of problems and the Performance “coproduction” of performance results. The organizations said they recognized that the regulated entities themselves should assume more responsibility for identifying and addressing performance issues. According to the organizations, this approach was to allow them to target resources more effectively. For example: • FAA said the agency wanted to reduce the fatal commercial aviation accident rate by 80 percent, but FAA could not do so solely through its role as regulator and enforcer. The agency said it worked with the aviation community, analyzing data to agree on the top accident categories and then conducting a detailed root-cause analysis. According to FAA, this approach had been instrumental in helping stakeholders throughout the aviation community—both the regulators and the regulated entities—reach agreement on the key accident categories on which to focus. FAA said that most important was the subsequent stakeholder commitment to effective implementation of intervention strategies to prevent future accidents in these categories. • FSIS said the agency’s new meat and poultry food safety program provided a more specific and critical approach to the control of microbiological hazards in foods than the approaches provided by traditional inspection and quality control. The program required meat and poultry slaughter and processing plants to adopt a system of process controls to prevent chemical, physical, and biological food safety hazards. According to FSIS, the program allowed FSIS inspectors to use a combination of a scientific approach for monitoring meat company management practices and traditional carcass-by-carcass inspections. FSIS said its employees analyzed company meat and poultry safety practices, monitored company Page 25 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization records, and conducted laboratory tests to ensure that company testing was accurate. • The Occupational Safety and Health Administration (OSHA) said the agency’s pilot Maine Top 200 Program encouraged employers to identify hazards themselves and take corrective action before the hazards led to injury and illness. According to the agency, this pilot program enabled OSHA to focus on workplaces where the largest number of serious injuries and illnesses occurred. Build Consensus Among Similar to their approach with regulated entities, the organizations said Program Delivery Partners they recognized the need for the involvement of the organizations on on Targets, Measures, and which they relied in delivering program products and services, such as Data Use state regulatory agencies. The organizations said they saw the delivery partners as key stakeholders and sought to establish agreement on program mission and objectives, the most useful performance measures, and the potential use of performance measures. For example: • EPA said its Office of Enforcement and Compliance Assurance initiated the agency’s National Performance Measures Strategy in January 1997 to develop an enhanced set of performance measures for EPA’s enforcement and compliance assurance program. EPA said it consulted with state environmental agencies on the measures, which resulted in several key ideas. For example, the states believed that EPA’s enforcement and compliance assurance program should (1) place more emphasis on the use of outcomes and environmental indicators to measure performance and (2) reduce its emphasis on outputs as a measure of performance. • OSHA said it relied on 23 states and 2 U.S. territories to operate their own OSHA-approved occupational safety and health programs. According to OSHA, these states were integral partners in OSHA’s mission of ensuring the safety and health of workers. In developing its strategic plan, OSHA said it directly involved these organizations in reviewing the plan. The agency said this approach clarified the role of states and how OSHA’s strategic plan applied to state programs. Obtain Periodic Input on On an ongoing basis, the organizations said they secured input from their Performance Management regulated entities and delivery partners on the effectiveness of program Efforts operations. In addition, they gathered information from these partners on program mission statements and goals. For example: • MDES said it conducted quarterly customer satisfaction surveys through its workforce centers, with some centers going even further and conducting focus groups with staff and clients. MDES said it used the survey data to update its mission, values, and goals. Page 26 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization • FDBF said it obtained feedback from its customers—the banks it regulated—on how they were regulated by conducting short surveys on various areas, such as the bank examination process, examination reporting, and examination team competency. A portion of the survey is shown in figure II.1. Figure II.1: Example of a Survey for Client Input on Agency Performance— Florida Department of Banking and Finance Source: Florida Department of Banking and Finance. • MDOR said the agency conducted postaudit customer surveys to obtain information on the organization’s audit process. The agency said the surveys solicited responses on how auditors treated the agency’s customers, not the outcome of the audit, and revealed that most employers do want to comply with tax laws. Practice 5: Establish Under the fifth practice, the organizations said they sought to identify other organizations involved in implementing crosscutting programs, then Rigorous Partnerships establish rigorous partnerships with those organizations. Specific practices Across Organizations included the following: • Identify the contributions of each organization and develop common outcomes. • Establish a leadership role to coordinate cooperative efforts across common goal areas. • Use tools to facilitate common data sharing across partner organizations. Page 27 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization Identify Crosscutting The organizations said they recognized the importance of interactions Relationships, Common among agencies involved in similar or related programs. They identified Outcomes, and Agency organizations with related programs, their relationship with those Contributions organizations, and common outcome expectations. For example, FSIS said it included a specific goal to establish effective working partnerships with other public health agencies and stakeholders to support the President’s National Food Safety Initiative, which called for a reduction in foodborne illnesses. The agency said specific activities were to include (1) collaborative monitoring with the Center for Disease Control, FDA, and state public health departments; (2) establishing cooperative agreements with states for risk assessment; (3) developing standard operating procedures for coordination in cases of foodborne illness outbreaks and other food safety emergencies; and (4) coordinating strategies with the Department of Health and Human Services, the United States Department of Agriculture, and private sector groups to expand communication of food safety information to the general public. According to the organizations, their crosscutting efforts also involved coordinating their program activities and linking their goals and strategies with those of the partner organizations. For example, NRC’s annual performance plan identified areas of mutual interest with other agencies, the related NRC programs, and the crosscutting strategies NRC would use to address the shared responsibilities with these other agencies. NRC presented most of this information in the form of a matrix in its annual plan. A portion of that matrix, describing part of NRC’s interaction with the Department of Energy, is shown in figure II.2. Figure II.2: Matrix Showing Crosscutting Program Relationships—Nuclear Regulatory Commission Source: U.S. Nuclear Regulatory Commission. Page 28 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization The organizations said they involved crosscutting partners, jointly and equally, in developing common or consistent outcome goals, associated measures, targets, and data collection procedures as well as in determining the types of decisions they would be used to make. The organizations said they also used interagency agreements and performance partnerships to clarify each partner’s roles and responsibilities. For example: • NRC said that in most instances, it had or was developing memorandums of understanding or other agreements with other agencies to ensure that areas of mutual interest and cooperation were treated in a consistent, coordinated, and complementary way that avoided unnecessary duplication or conflict. • MDOC said that it completed literature reviews and other secondary research to ensure that the agency was measuring many of the same indicators as other criminal justice agencies. As a result, the agency said it had the capability to benchmark with other agencies. In addition, the organizations said they worked to identify, to the maximum extent possible, each agency’s contribution to the common outcome expectations and whether those contributions were unique or common to other organizations. The organizations said they also took other organizations’ efforts into consideration when making resource plans. According to the organizations, working with other agencies also helped the organizations to leverage resources by combining project efforts. For example: • NHTSA said it developed a beneficial relationship in situations where another agency could regulate something NHTSA could not. For example, the draw strings on children’s jackets were getting caught on school bus doors, and NHTSA worked with the Consumer Product Safety Commission, which provided guidelines on children’s clothing. • The Coast Guard said the agency evaluated where it fit in a crosscutting program logic model (described in practice 6) and the level of agency outputs or outcomes that might be appropriate. In one case, the Coast Guard said the National Oceanic and Atmospheric Administration (NOAA) determined fishing regions and was interested in the intended outcome of protecting fish habitats. To contribute to NOAA’s intended outcome, the Coast Guard produced an output—patrolling miles of ocean. The Coast Guard, said the agency, was not responsible for protecting fish habitats, but could support NOAA in doing so. Page 29 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization Establish a Leadership The organizations said they recognized that successfully implementing Role to Coordinate crosscutting efforts among “equal” partners could be difficult without Cooperative Efforts specific leadership. To remedy this, the organizations said they established one lead organization or a specific leadership role among the partners. For example: • The Customs Service said it and the Immigration and Naturalization Service (INS) developed the Border Coordination Initiative (BCI) to increase cooperation on the southwest border for the interdiction of drugs, illegal aliens, and other contraband. The agency said the BCI was intended to be a comprehensive, integrated border management system at the southwest border, achieving the mission of both the Customs Service and INS. According to the Customs Service, the agencies established a joint Office of Border Coordination, comprising two border coordinators representing INS and the Customs Service, that managed the BCI and was responsible for overseeing border operations. • FSIS said its strategic plan focused on national and international food safety and was targeted at all food safety efforts at the federal and state levels. Relying on crosscutting efforts, FSIS said it was working with federal and state officials to optimize the food safety resources available at all levels. The agency planned to exercise leadership through memorandums of understanding and cooperative agreements. Use Tools to Facilitate The organizations said that to assist in crosscutting efforts, the partner Common Data Sharing organizations increased the usefulness of their common data sharing by establishing common data definitions and information systems. Common data definitions were to help ensure that data used for common purposes would be consistently defined, collected, calculated, and interpreted. In addition, the organizations said they (1) identified existing information systems within each organization that might serve common interests and (2) shared information with their partner organizations. For example: • NHTSA said it recognized the need to share information with other agencies and establish information systems to do so. For example, the agency said that the Department of Health and Human Services asked NHTSA for information regarding the use of drugs and driving. According to the agency, information sharing such as this bolstered NHTSA’s requests for additional performance data funding. • FDA said the agency’s focus on outcomes was tied to building information systems linked with other agencies, such as the Center for Disease Control, with the same goal, thereby making it easier to integrate and share performance information with these agencies. Page 30 GAO/GGD-00-10 Managing for Results Appendix II Practice Category 2: Establish Relationships Outside of the Organization • The Customs Service said that it and INS, collaborating on the previously mentioned BCI initiative, worked to develop common data definitions and data collection procedures. Page 31 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets In the third category, the organizations said they refined their performance Summary goals, measures, and targets to better translate activities into results. The practices included the following: 6. Establish a rationale of how individual programs delivered results. 7. Select meaningful goals and measures consistent with the rationale. 8. Set appropriate targets for performance goals. Practice 6: Establish Under the sixth practice, the organizations said they sought to establish a rationale of how their individual programs delivered results. Specific a Rationale of practices included the following: How the Program Delivers Results • Take a holistic or “systems” approach. • Build a program logic model that described how activities translated to outcomes. • Expand program assessments and evaluation efforts to validate model linkages and rationale. Look at Programs Using The organizations said they recognized that performance management and a Holistic or “Systems” measurement efforts could not be viewed from a limited perspective, such Approach as that of an individual program, but must be seen in terms of the operation as a whole. The organizations said they concentrated on the relationships and interactions of whole systems, as opposed to managing parts of a system. The organizations said they took a holistic approach to identify and evaluate factors that affected their outcomes, determine appropriate strategic goals, and assess how a change in one goal might affect another. For example: • OSHA said that the agency’s strategic planning process recognized that all of the agency’s strategic goals would be interrelated, and that working in one goal area would affect the work in another area. • FDA said that the agency was improving its efforts to describe total program relationships and systems. For instance, FDA said it was moving from setting only measurable performance goals to setting goals whose measures were important as strategic points in an overall system to respond to the agency’s mission. • MPCA said that the agency was dealing with pollution problems in a more holistic way. For example, air pollution resulting from mobile sources— cars, trains, airplanes, and boats—touched a number of MPCA program areas. In the past, the agency said it focused on specific media, such as air Page 32 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets and water. The holistic approach, using environmental impact data, was designed to help decisionmakers focus on the environment as a whole. • OSFI said that it used the image of a scale to demonstrate the interplay of its five strategic objectives. On one side of the balance was “safeguarding from undue loss” (representing the agency’s identification of specific risks and trends and intervention in a timely manner) and “public confidence” (evaluation of systemwide risks and promotion of the adoption of sound business and financial practices). On the other side was “competition” (the agency’s due regard for the regulated institutions’ need to compete effectively) and “cost effectiveness” (maintaining full and open dialogue with stakeholders on the costs and benefits of the agency’s work). The balance’s fulcrum was “quality,” representing OSFI’s objective of carrying out its mission with quality people, processes, and technology. OSFI said that it recognized some objectives would counterbalance others. For example, if greater emphasis was placed on “safeguarding from undue loss” by introducing tougher rules for financial institutions, there might be an adverse impact on the ability of financial institutions to be innovative and competitive. Develop Basic Program Logic The organizations said they sought to develop a better understanding of Models how their programs worked so that they could select appropriate performance goals and measures. To do so, they said they described the logic or rationale of how individual programs used inputs, such as resources and workload, in program components, such as activities and processes, to produce outputs. In turn, those outputs were connected to intermediate and final outcomes. These descriptions, often called program logic models, were not necessarily the more extensive models that might be used in more comprehensive program evaluations, but they were concise descriptions of the basic flow from inputs to outcomes. The organizations said they did not necessarily intend to describe causality, but to develop a description of a reasonable correlation or association showing how inputs were converted to outputs and outcomes. According to the organizations, the exercise of developing logic models can help internal and external stakeholders (1) see the progression from outputs to end outcomes, (2) see how changes in program components and outputs might better impact outcomes, and (3) better understand their contributions to desired results. For example: • NHTSA said the agency was using program impact models that showed linkages between inputs, outputs or process, intermediate outcomes, and end outcomes. NHTSA’s annual performance plan described the linkages to support the presentation of outcome and intermediate goals and measures in the body of the plan. Output measures, which were included Page 33 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets in the plan’s appendix, were to be used internally for departmental budget justifications and management decisions. A program logic model using components from NHTSA’s annual performance plan is shown in figure III.1. Figure III.1: Constructing National Highway Traffic Safety Administration’s Program Logic Model Source: NHTSA. • The Coast Guard said the agency’s Marine Safety area’s business plan followed the program logic model from inputs to outputs to outcomes. In addition, the agency said that in its strategic planning efforts, the agency used a basic model of internal processes leading to activities, which led to outcomes. Management goals reflected internal processes, while activities reflected performance goals. In this model, the Coast Guard noted that its performance outcomes were largely affected by where the Coast Guard Page 34 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets decided to intervene in a problem, by its particular strategies and activities, and by its internal management goals. • OSHA said it was also using formal program logic models in developing performance measures. The agency viewed measures of activities, such as the number of inspections, as intermediate measures leading to the desired outcome of fewer workplace accidents. The agency said that OSHA’s model flowed from performance goals to outcome goals to strategic goals to Department of Labor outcome goals and indicated the basic assumptions and strategies that were to be used to achieve desired outcomes. For example, implementing safety and health program promotional strategies would lead to employers’ having an effective safety and health program; leading to better safety and health programs; leading to changing the workplace culture to increase awareness and commitment of safety and health; and, finally, leading to reducing workplace injuries, illnesses, and fatalities. Expand Program Assessment The organizations said they expanded their evaluation efforts to explore and Evaluation Efforts and confirm the linkages and rationale within their program logic models. These efforts included questioning and testing the assumptions in the program logic model, the processes and activities selected to implement the models, and the way in which program impacts might be separated from external factors. • The Coast Guard said it planned to use program evaluation to examine the agency’s activities of interaction, influence, and impact after the agency developed a program policy. The agency’s view was that program evaluation should serve performance management in validating the program logic model and its rationale, and help determine if performance- based management was worthwhile. • FDA emphasized that a program evaluation function was needed to encourage program managers to think from inputs to the end of the logic model continuum. Practice 7: Select Goals Under the seventh practice, the organizations said they sought to select vital program and support goals, as well as performance measures, that and Measures Consistent were consistent with the program rationale they had developed. Specific With the Program practices included the following: Rationale • Select vital program and support goals directly or indirectly from a program logic model. • Use rigorous criteria to assess and select the most important measures. • Develop a comprehensive suite of measures representing total program performance. Page 35 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets Select Vital Program and According to the organizations, program logic models formed a foundation Support Goals From Program for selecting vital program performance goals, including management and Logic Models support goals. The organizations selected outcome goals, measures, and targets that reflected the influence of external factors and the agency’s varying levels of influence and control at different points in the program logic model. For example: • The Florida Game and Freshwater Fish Commission (FGFFC) said the agency tried to use only intermediate measures where the link to desired outcomes was established. As an example, FGFFC said it measured the number of cases successfully closed—referred to the state’s attorney— instead of reduction in crime rates because the agency (1) had more control over the number of cases closed and (2) believed, from experience, that closed cases generally resulted in conviction and therefore a reduction in the crime rate. • MPCA said that in cases where the agency could not develop or obtain data on an environmental outcome measure, it tracked changes in behavior it believed led to the desired outcome. For example, tracking mercury reduction required a long time frame, so the agency said it would track behaviors that lead to decreases in the release of mercury into the environment. • Florida’s Office of Program, Policy Analysis and Government Accountability (OPPAGA) stated that agencies could consider, as an intermediate measure, detailing the characteristics of a healthy regulated population, then explain how the agency contributed to the development and maintenance of a healthy population. The organizations said they considered administrative or management support goals as supportive of, and thus subordinate to, strategic program goals. For example: • NRC said it selected strategic goals on the basis of the importance to the agency’s mission areas and made support efforts subordinate to the strategic areas. • FAA said the agency distinguished between goals that were directly aimed at achieving the agency’s mission and other goals, such as administrative or management support goals, that enabled or supported the agency in achieving its mission. Use Rigorous Criteria to Select The organizations said they used rigorous criteria to assess and select the the Most Important Measures actual measures. The selection criteria—such as availability, accuracy, validity, potential adverse consequences, balance, and relevance— recognized that meaningful performance-based management required the Page 36 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets use of a manageable number of useful measures. According to the organizations, tracking more measures resulted in an increased data collection burden. Organizations also said that not carefully screening measures resulted in measures that were similar to others or that might be irrelevant to program results and operational needs. According to the organizations, the result might be a large volume of measures that would overwhelm those measures considered truly important for decisionmaking and guiding organizational operations. In addition, the organizations said they used the criteria to regularly review and modify the measures over time. The selection criteria also recognized the importance of selecting a suite of measures, as discussed later, reflecting a balance of measures across the logic model and for different decisionmaking tiers. The organizations said the logic models also allowed them to define what activity information—workload and process—would best support movement toward the ultimate outcomes. The following are examples of the organizations using criteria to select the most important measures: • FSIS said its strategic plan focused primarily on its fundamental public health mission by limiting the number of goals and accompanying measures and tying them to the primary goal of protecting public health. • FGFFC said it selected only measures that were easily explainable to the legislature and did a “reality check” with all measures to ensure they would be useful for both internal and external decisionmaking. • Several organizations said they took a customer-based focus. MDES recommended identifying customers for each measure, ensuring that the measures were clear, understandable, meaningful, and measurable. MPCA and MDOT said they primarily focused on outcomes and measures from the citizen or customer point of view, and they secondarily focused on outputs, such as the number of tests conducted. • OSFI said that before implementing a measure, the agency emphasized having a full understanding of the impact of the measure on stakeholders as well as the potential repercussions of communicating the measure. • The Customs Service said the agency’s fiscal year 2000 annual performance plan had 31 performance measures, down from 61 in 1999, and focused on the most meaningful and vital few measures. At one time, the agency said it used the number of inspections to measure the performance of its inspectors, but dropped the measure because it forced inspectors into “body counts.” Other measures, such as the number of indictments, were dropped because the Customs Service had no control over them. Page 37 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets Develop a Comprehensive The organizations said they recognized that developing a suite of measures Suite of Measures across the logic models and addressing different decisionmaking tiers Representing Total within the organization, best measured program and overall organizational Program Performance performance. Thus, the organizations chose performance measures from the different stages in the model, covering inputs, outputs, outcomes, and activities or processes. According to the organizations, the suite of measures, representing the interrelationship of multiple measures, helped the organizations take a broader perspective of program performance. For example, EPA said it categorized its measures into three types. These types included the following: (1) program output measures that represented actions taken by the agency, such as the number of permits issued/revised or the number of enforcement actions; (2) program outcome measures that reflected the direct results that lead to environmental improvements, such as the decreased use of higher risk pesticides or reduced emissions of toxics from manufacturing facilities; and (3) core environmental indicators that represented ultimate results, such as increases in the number of rivers and lakes in healthy ecological condition or decreases in the number of people with air quality-related illnesses/deaths. In addition, the organizations said that developing a suite of measures allowed them to select a subset of measures targeted at the different needs of specific audiences. For example, the measures could meet the information needs of both internal managers, who needed information for operational control from the bottom of the organization to the top, and external policymakers, who were interested in measures evaluating the program’s success. Figure III.2 is a diagram highlighting the types of information needed to make decisions at different levels within an organization and how the granularity increases from the top to the bottom. That is, the most aggregated information is at the top level, and the most detailed information is at the bottom. Figure III.2 also describes the timing and focus of information at these different levels. Page 38 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets Figure III.2: Decisionmaking Tiers Source: GAO summary of decisionmaking tiers. The organizations said they used the decisionmaking information to select measures and, in situations where final outcome information was not available or would be delayed, to provide justification for the use of output or intermediate outcome information. For example: • For each of its five strategic objectives, OSFI said the agency was developing a suite of performance measures designed to evaluate progress in reaching its objectives. For the strategic objective of safeguarding depositors from undue loss, the agency established the following measures: (1) the Level of Intervention Index, which measured the level of OSFI intervention and tracked individual financial institutions as their financial condition changed; (2) the Loss Recovery Index, which measured the amounts depositors of liquidated institutions could expect to receive; (3) the Risk Exposure Index, which was a composite measure of OSFI’s assessment of the level of risk facing the financial industry at a given time; and (4) the Intervention Effectiveness Measure, which measured OSFI’s effectiveness in identifying problem institutions and intervening in a timely way to address regulatory concerns. In each of these areas, OSFI said it identified the measures as in either an early or advanced stage of development or maturity. For example, the agency considered its Page 39 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets employee satisfaction level measuring quality as advanced. However, another quality measure, the extent to which OSFI staff met identified core competencies, was less mature and was in an earlier stage of development. • For each of its agencywide goals, MDOR said it used several different measures to evaluate the agency’s success in achieving its goals. For example, one of its goals was that “everyone is paying what is owed, no more, no less.” To evaluate progress toward achieving this goal, MDOR said it measured voluntary compliance rates, nonfiler discovery rates, tax filing accuracy, use and sales tax compliance levels, and the number of corporate audits completed. The agency said this mix of measures provided a complete performance picture. • The Coast Guard said the agency’s aeronautical engineers identified critical dimensions, or key issues, concerning Coast Guard aircraft. These dimensions were reliability, maintainability, supportability, and affordability. Using these dimensions, the agency said it created a system of Measures of Effectiveness (MOE) to track performance against aviation system goals. Daily, all air stations were to document when an aircraft was available to fly a mission. The MOE produced index information on (1) the percentage of time that aircraft at Coast Guard air stations were available to perform missions, (2) how often air stations had needed aircraft parts, (3) how much unit effort it took to generate each flight hour, and (4) an overall maintenance effort index. The agency said that a suite of measures could be evaluated to help identify common causes and trends, and were used to determine workforce levels, program flight hours, maintenance costs, and budgetary considerations. Practice 8: Set Appropriate Under the eighth practice, the organizations said they used different methods to set appropriate targets for their performance goals. Specific Targets for Performance practices included the following: Goals • Use different types of performance comparisons to match performance goals. • Provide multiyear and subgoal performance targets. • Use targets for distinct populations or comparison categories that are meaningful to the organization. • Use baselines to set realistic, but challenging targets. Use Different Types of The organizations said they used a diversity of performance comparisons, Comparisons depending on the goal, to set performance targets. The comparisons included (1) predefined performance specifications, (2) future performance levels or changes in levels to be achieved at a later date, (3) best practice benchmarks from other organizations, and (4) program implementation milestones. For example, the Coast Guard said it based its Page 40 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets performance targets on historical performance, trend analysis, and improvements currently under way. In addition, some targets were based on a defined performance level, such as an absolute readiness index score required by the Department of Defense. Figure III.3 provides examples of several types of performance comparisons. Figure III.3: Examples of Performance Comparisons Sources: FDA, the Coast Guard, NRC, FAA, and OSHA. The organizations said they also recognized that because of the nature of some goals, they could not always set absolute targets. For example, in some cases, the organizations said they did not have a baseline or benchmark to set a target, such as when a measure was new, a baseline had never been established, or benchmarks were not readily available. When this occurred, the organizations said they either set a preliminary target or directional target, or stated that a baseline would be set with the initial collection of data. For example: • FDA said that the targets for its strategic investment goals were a series of milestones for achieving the desired capability, such as developing modeling techniques to assess human exposure and dose response to certain foodborne pathogens. For program result or outcome goals, the agency said targets were quantitative or productivity goals. For example, Page 41 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets one target was to have 80 percent of the domestic seafood industry operating preventative controls for safety. • FAA said it gathered data to set a baseline for assessing future needs and desired accomplishments, such as FAA’s efforts to reduce delays, accidents, and incidents. • The Customs Service said the agency often set targets as a certain percentage increase in a measure, rather than setting a specific number as a target. For example, instead of setting a target of 91 percent, the agency said the target would be set as a certain percentage change over the previous year. Once baseline information was available, the agency would set specific target numbers. • NHTSA and MPCA said that they did not set specific targets in some cases. For example, NHTSA said it had a difficult time deciding on appropriate targets in the face of unpredictable trends and many external factors, and ended up agreeing on a target of “no increase.” For some measures, MPCA said the agency did not set a specific target, but only indicated whether it wanted the measure to increase or decrease. Use Multiyear and Subgoal The organizations said they faced situations where they could not identify Targets specific annual targets. In these cases, the organizations established multiyear goals and targets and conducted annual progress checks. In addition, they set targets at the subgoal level that were to cumulatively reach an overarching goal over time. For example: • FAA said it used 3-year or even 10-year baselines in such areas as commercial aviation fatal accidents because there were so few of them in any given year. An increase or decrease of even one or two accidents could skew the data. • NHTSA said it set a long-term goal of reducing highway fatalities and injuries by 20 percent by the year 2008 and tied in annual and multiyear efforts. The target was set by using an analysis of factors the agency could influence, and how it could influence them, to estimate the cumulative effect of reaching goals with various interventions (intermediate outcomes). The agency said it developed subgoal targets that were expected to produce the 20-percent reductions by the year 2008. The agency thus worked on intermediate outcomes at the subgoal level and tied that to an overall outcome goal. NHTSA also said that the agency tried to avoid straight-line annual targets, preferring floating targets that the agency could reassess each year, depending on progress towards the final outcome goal. • FAA said that the agency also used varying targets. For instance, FAA’s strategic plan contained a safety mission goal of reducing the United States aviation fatal accident rates by 80 percent from 1996 levels by the year Page 42 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets 2007. According to the agency, the fiscal year 2000 performance plan had several performance goals to achieve this overall goal with either annual or multiyear targets. For example, performance goals for the safety mission goal included (1) reducing the fatal aviation accident rate for commercial air carriers from a 1994-96 baseline of 0.037 fatal accidents per 100,000 flight hours; (2) decreasing the rate of air shipment hazardous materials incidents by the year 2000 from a 1998 base; (3) by the year 2007, reducing (by a to be determined percentage from baseline levels) the rate of airport accidents/incidents that result in injury to persons or damage to aircraft; and (4) reducing the rate of operational errors and deviations by 10 percent from 1994 baselines. The agency said it had or was developing a fiscal year 2000 target for each performance goal, such as reducing the fatal accident rate for commercial air carriers to 0.033 per 100,000 flight hours. Use Distinct Populations or The organizations, with input from stakeholders, said they carefully Comparison Categories selected performance categories down to the lowest, meaningful level of disaggregation of data and used these categories to set appropriate targets. For example, the categories included geographical areas, workload or customer groups, or types of services. The organizations said they sought to set targets at the lowest, most disaggregated level so that they would be meaningful to managers and staff at the activity level within the organization. In addition, the organizations said that setting goals at the appropriate level of disaggregation aided decisionmakers in evaluating how successful a program was working with respect to different categories. For example: • OSHA and FDA said that their agencies set performance targets for different industries. For example, OSHA said the agency set targets for nursing homes, logging, food processing, and shipyards to focus agency efforts on the most hazardous industries and workplaces. FDA said it set targets for such industries as the domestic seafood industry and foreign food establishments. • MDOR said it segmented its client population into those with minor offenses and those who were considered repeat offenders, and the agency developed measures for each category. • MDOT noted that it had the agency’s divisions categorize or segregate roads by type and develop measures specific for each type of road. The agency also set targets for different geographical areas within the state. Use Baselines to Set Realistic The organizations said they used baselines to set performance targets that But Challenging Targets were realistic, but they expected the baselines to challenge the organization to continually improve. The organizations said they used the Page 43 GAO/GGD-00-10 Managing for Results Appendix III Practice Category 3: Refine Performance Goals, Measures, and Targets baselines as the lowest acceptable performance expectation and set targets at higher levels. For example: • OSHA recommended selecting a “typical” year and developing averages, rolling averages, or other statistical measures to set the challenging targets. • FDA said it planned to reduce the percentage of food and color additive petitions under review for more than 360 days to 20 percent in fiscal year 2000. The baseline data were 44 percent in fiscal year 1997. • The Coast Guard said the agency used actual information to develop a trend line and set a target that was based on planned strategies. For example, figure III.4 shows a graphic from the Coast Guard’s fiscal year 2000 annual performance plan illustrating a goal and target to reduce the passenger vessel casualty rate. Figure III.4: Examples of Targets Set Using Baseline Data—U.S. Coast Guard Source: U.S. Coast Guard. Page 44 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques In this practice category, the organizations said they built on and Summary strengthened their analytical capabilities and techniques to better meet performance management information needs. The practices included the following: 9. Ensure that data resources and analytical capabilities were sufficient to provide performance management information. 10. Target analysis at regulatory intervention. 11. Account for external and contextual factors. Practice 9: Ensure Adequate In the ninth practice, the organizations said they sought to ensure that their data resources and analytical capabilities were sufficient to provide Data Resources and information necessary for formulating and assessing strategies. Specific Analytical Capabilities practices included the following: • Ensure staff analytical capabilities met performance management needs. • Develop a data infrastructure and information systems to generate useful performance data. • Use specific tools to support the use of performance information. • Ensure the quality, timeliness, and continuity of performance information. Ensure Staff Capabilities The organizations said they assessed their internal analytical capacity to Meet Performance deal with performance management—both in terms of staff available to do Management Needs analysis and the types of analytical skills possessed by the staff. They recognized that analytical capabilities were critical in assessing risks, aggregating and disaggregating performance information in a meaningful fashion, and allocating appropriate resources to activities linked to strategic goals. For example: • FAA found that it was critical to have statistical and financial analysis experts available to provide data needed by managers to change practices and make decisions. According to the agency, FAA analysis units also provided information to other FAA units for performance measurement. • The Customs Service said it hired a statistician to oversee sampling efforts related to trade compliance measurement and assessment. According to the agency, compliance measurement and assessment helped the Customs Service identify problems and make a specific evaluation of the risk of noncompliance posed by an individual company and the industry overall. Page 45 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques Develop Data Resource The organizations said they did not firmly establish data sources and data Infrastructures and Supporting collection processes until they had identified their program rationale and Information Systems accompanying goals, measures, and targets. Once they knew what data they would need, they said they either used existing data resources and information systems or upgraded or built new systems to collect the necessary performance information and conduct quicker and better data analyses. For example: • NHTSA said that even though the agency had tracked performance data for several years—even producing a data book to support agency decisionmaking and performance management—the agency was reassessing its existing data. NHTSA said it wanted to look at existing data and evaluate which factors lead to desirable outcomes and then develop interventions to achieve intermediate outcomes. • NRC said it initiated the development of an agencywide, integrated financial and resource management system called STARFIRE, which was to serve as the single authoritative source for financial and resource information and support the alignment of agency resources with program outputs, strategies, and strategic goals. • MDOC said it was using the development of a new data system as an opportunity to include routine collection of more useful performance measures and not just a new means to collect the same information that has always been collected. MDOC said the system was intended to increase its ability to provide accurate, descriptive statistics and to support program evaluations. • MDOT said it was integrating different management systems within the agency to ensure consistency in measurement through common definitions and measurement timing. Use Specific Tools to Support To enhance their performance measurement efforts, the organizations said the Use of Performance they used different tools to help support the understanding, Information communication, tracking, and reporting of performance information. They said they used performance measure definitions to identify—for both external and internal audiences and partners—the measure being used, its purpose, its source, how it would be calculated, how it would be verified and validated, who was responsible for its collection and analysis, and any limitations it might have. According to the organizations, they also used specific definitions to remove ambiguity regarding what was being measured and to ensure that measurements could be replicated. For example: Page 46 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques • MDOC said it developed a glossary with performance measurement terms and agency acronyms that accompanied its performance report and used extensive footnoting to better explain measures and data to the reader. • The Coast Guard said that for each of its performance goals, it included a description of why the agency acts, key factors, strategies, coordination, analysis and evaluation—including a graphic of the target, information for the past several years, the trend line, and key initiatives. • Florida state agencies said they were required to provide specific performance measure information as part of their performance management. The measures and their definitions are shown in table IV.1. Table IV.1: Florida Measure Definitions Measure information Measure definition Measure number Unique alpha/numeric identifier that is assigned to each measure Type of measure Input, output, or outcome Name of responsible individual(s) Individual responsible for creating/developing and defining the measure Purpose of measure Brief description of what information the measure will provide Definition of measure Thorough description of the measure that includes defining each word in the measure and what will be included Formula description If used to calculate the measure, the formula is written out and each aspect of the formula is defined Measurement period Time period that will be covered by the measure Relationship to mission How the measure relates to the unit’s overall mission Output/Outcome relationship If applicable, the relationship between output measures and outcome measures; identification of any associated measures that relate to the measure Reporting requirements Who the measure is reported to and the frequency of reporting (e.g., monthly, quarterly, etc.) Source: Florida Agency for Health Care Administration. In addition, the organizations said they used performance measure databases to track performance. For example: • FDA and EPA said they collected information in a performance database. FDA said the agency maintained a performance goal database showing inputs and outputs, dollars associated with specific goal clusters, and the part of the agency associated with the goals. FDA said it also used a database warehouse to track goals and resources over time and link FDA goals and measures to other legislative requirements, such as FDA’s Page 47 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques modernization act and the Chief Financial Officers Act. According to FDA, this approach was designed to allow the agency to maintain just one set of performance information, instead of several sets. EPA said its database was to be used by EPA goal teams and national program managers. • OSHA said it was developing a database for strategic plan measures and, when completed, the database would be available to all OSHA employees through its Intranet. • MOEA and MDES were using data matrices as part of their performance management efforts. MOEA said it used a set of matrices that described the linkages between the agency’s environmental outcomes and measures and the agency’s primary and secondary staff responsible for collecting and evaluating data and information for the measures. The agency said the matrices had been a useful tool at all levels in the organization to understand staff accountability. MDES said the agency’s data matrix listed the customer(s), the customer needs, what the agency would measure, the type of measurement, and how data would be collected for each service that the agency provided. • MDOR said the agency set up a performance measure database that included the measure definition, the purpose of the measure, the measure owner, and up-to-date trend data. According to the agency, this standardized format allowed the agency to use one set of data to generate reports for each program area. MDOR also said the agency used this database to generate an executive summary of measures that listed agencywide performance measures for each of the agency’s four goals. Ensure the Quality, The organizations said they supported their data analysis capabilities with Timeliness, and Continuity accessible, high-quality and timely data. They said they made a senior of Performance Information manager or team responsible for data resources and used data quality evaluative or auditing functions to assess data integrity and its sufficiency for supporting the program logic models. According to the organizations, the evaluative and auditing functions assessed the choice of measures, the collection and processing of data, the quality of information, the interpretation and explanation of results, and the relevance and adequacy for decisionmaking. For example: • The Customs Service said that when field staff realized management was using performance data to make decisions, they began providing accurate information and explanations for any incorrect data. The agency said it required each office to establish a data quality function, responsible for verification and validation, that would be inspected annually. The Customs Service also said it established specific measure owners who were responsible for handling measures in their own process areas. Page 48 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques • FGFFC said it had a key person in each division or functional area responsible for developing measures, educating staff, and gathering performance information. Practice 10: Target Under the tenth practice, the organizations said they targeted analytical techniques for their regulatory intervention efforts. Specific practices Analysis at Regulatory included the following: Intervention • Use risk management techniques to target resources for the maximum results. • Take a balanced approach to regulatory efforts by targeting actions before, during, and after a problem event. Use Risk Management The regulatory organizations said they developed risk management Techniques to Target techniques to target the largest regulatory problems, attempting to achieve Resources for Maximum maximum results for their resource allocation. For example: Results • The Coast Guard, OSHA, and NRC said they were targeting risk areas. The Coast Guard said the agency used risk analysis to define risk groups and levels of risk and then put investments toward prevention in high-risk areas. OSHA noted that the agency changed its enforcement strategy to target high-hazard workplaces, focusing on preventing accidents. NRC said it was developing recommendations for improving NRC’s inspection, assessment, and enforcement processes to focus on the most important safety issues. • The Customs Service said it formalized a Trade Compliance Risk Management Process that collected data and information, analyzed and assessed risk, prescribed action, and tracked and reported information. The agency said that it then used this data to analyze historical compliance data and trends for various industries, specific commodities, and certain importers and, by applying definitions of significance and materiality, to focus on areas with the greatest potential risk. For example, identifying whether the source of risk was an importer’s lack of knowledge, complex trade laws, or willful disregard for importer laws would result in different, specific action plans to assign resources and address the risk. • OSFI said it developed a risk exposure index as a composite measure of the agency’s assessment of the level of risk facing the financial industry at a given time, weighted by the condition of the institution, the value of assets, and the type of institution. Over time, OSFI said it planned to compare and track the impact of major events on overall system risk. Take a Balanced Approach The organizations said they carefully balanced their regulatory efforts to to Regulatory Efforts address both prevention and mitigation. According to the agencies, they Page 49 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques did this by analyzing how they influenced and directly impacted the entities they regulated and by targeting regulatory actions before, during, and after a potential or actual problem event. The organizations said they also supplemented their goals and measures related to mitigation with goals and measures designed to track and reduce risk factors, thus seeking to anticipate and prevent problem events where possible. For example: • NHTSA said the agency sought to achieve progress in the following two intermediate outcome areas: (1) reducing the occurrence of crashes and (2) mitigating the consequences of crashes. NHTSA said programs used performance measures to help achieve the intermediate outcomes, which, in turn, influenced the outcomes. The agency said it used a matrix, shown in figure IV.1, that addressed three time phases—precrash, crash, and postcrash, and where each of its programs had an impact. The matrix was a tool to use in defining problems and posing strategies. For example, the agency said NHTSA’s National Advanced Driving Simulator was a specific effort to conduct research on driver performance and behavior during the precrash sequence of events. For postcrash events, the Intelligent Vehicle Initiative was targeted at understanding the causes of highway collisions, and the Emergency Medical Services effort was designed to enhance the comprehensive emergency medical service systems to care for victims of crashes. Page 50 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques Figure IV.1: National Highway Traffic Safety Administration Matrix to Define Problems and Strategies Source: NHTSA. • The Coast Guard said the agency targeted its activities to (1) improve operational methods for a quicker response—mitigation—and (2) prevent problems from occurring—prevention. In the Coast Guard, the agency said the focus on prevention was preceded by a careful analysis of cause and effect and what the agency could influence. For example, the Coast Guard said it developed an analysis that recommended preventing Alaskan commercial fishing industry disasters before they occurred, as well as preparing to react to them if they should occur. The analysis described the fatal events and recommended strategies to prevent or reduce the seriousness of these events. The prevention efforts revolved around critical factors, such as vessel stability and hull integrity, skipper and crew training and licensing, avoidance of harsh sea and weather conditions, preventing falls overboard, and safer diving practices. Instead of saving lives after a vessel casualty, the Coast Guard said it could take action in these areas to improve the prevention of fatalities. • The Customs Service said the agency found that it could no longer depend on enforcement and interdiction efforts because the volume of transactions required an ever-increasing amount of resources. Realizing that most importer errors were not deliberate, but the result of trying to Page 51 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques follow complex requirements, the agency said it began to focus on accountability management, where the agency had an obligation to inform importers about compliance, and thus prevent compliance problems. However, when importers continued to violate requirements, the agency said it pursued appropriate enforcement actions. In addition, the agency said it examined measures that could provide some data on the effectiveness of its border interdiction efforts, such as drug transport costs. The assumption was that if the agency could disrupt and dismantle drug transportation attempts, then drug suppliers would find it too costly to operate. In this case, the agency said measures of disruption would compliment measures of drug seizures. Practice 11: Account for Under the eleventh practice, the organizations said they sought to account for factors beyond their control that might have an impact on their efforts External and Contextual to achieve outcomes. Specific practices included the following: Factors • Identify and track external and contextual factors to explain their influence on program results. • Use smaller units of analysis to better understand program effects. • Use statistical techniques and program evaluation to adjust for and isolate the influence of external factors. Identify and Track External The organizations said they grappled with the issues of responsibility and and Internal Contextual accountability for performance in those areas affected by external factors, Factors such as the state of the economy or internal changes in program operations and technological support. They said they identified these factors and tracked them over time, analyzing their impact on specific performance goals and targets. The organizations said they used this information to help internal and external stakeholders understand the influence of these factors on program results. In addition, this information was used to refine the rationale of the program logic models. For example: • NHTSA said it found that discussing the influence of external factors in the performance plan helped overcome agency fears of being held accountable for measures over which the agency had limited control. The agency said it examined the effect of external factors using internal and external resources, such as local law enforcement studies. In addition, tracking external factors lent credibility to other data, making it easier to explain the impact of the agency. The external factors were discussed in detail in the agency’s strategic plan. • The Florida Department of Insurance (FDI) said it identified the contributing factors that could create a demand for services, such as insurance or fire marshal services. These factors, such as arson, defective Page 52 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques products, acts of violence, and inadequately trained emergency personnel, provided the fire marshal with a starting point for identifying strategies for influencing or controlling specific factors. • The Coast Guard said it analyzed key factors related to its goals. For example, the Coast Guard said it identified several key factors that increased the difficulty of successfully responding to mariners in distress, such as untimely distress notification, severe weather, poor communication, and poor information about the distress. The agency said its strategies were targeted at preventing the distress, but also at maximizing the survival chances by addressing these key factors. • FAA said the agency determined that weather was a factor in 40 percent of aviation accidents and 50 percent of aviation fatalities; therefore, FAA focused on influencing the impact of the weather conditions. For example, the agency said it had strategies to invest in an integrated terminal weather system and a weather and radar processor. These would give air traffic controllers instant access to current weather data. FAA said the agency was also implementing and improving existing weather sensors, targeting a weather research program to demonstrate storm growth, and strengthening aviation delay forecasting technology. • FGFFC said the agency tracked external factors related to boating accidents, such as alcohol use, to be able to make a thoughtful analysis of the factors contributing to an accident, even though the factors were beyond the agency’s control. • TDOB said it tracked explanatory measures, such as the number of state- chartered banks in Texas and the total assets they represent, which provided the agency with the opportunity to explain the influence of external factors on agency performance measures. The agency said it also included a separate section in the strategic plan that discussed relevant external and internal factors. Use Smaller Units of Analysis The organizations said they recognized that program effects were often difficult to evaluate if analyzed on a national or statewide basis. In some cases, they said they used a smaller unit of analysis that was more clearly defined and in which program efforts and impacts could be more clearly identified and understood. For example: • Florida’s OPPAGA recommended that agencies segment the population they regulate into smaller segments and evaluate their efforts on that segment, where they were better able to control for external factors. • FGFFC said it conducted detailed observations on a smaller population, such as a portion of a river, attempting to identify outputs that contribute to outcomes in a habitat. The agency said it was able to carefully measure its efforts, detailing all of the inputs and outputs, such as number and Page 53 GAO/GGD-00-10 Managing for Results Appendix IV Practice Category 4: Strengthen Analytical Capabilities and Techniques location of officers, weather conditions, accidents, and violations. In doing this, the agency said it could take a snapshot of the agency’s impact and use it for future planning. Use Statistical Techniques The organizations said they used statistical techniques, such as and Program Evaluation normalization, ratios, or trend analysis, to better identify the influence of external factors. For example, using calculations that normalized information as a standard rate, such as “per capita,” provided a standard that could be compared from year to year. In addition, the organizations said they used program evaluations to examine very long-term outcomes and to isolate program effects from other factors. For example: • FAA said it accounted for the growth of the airline industry by normalizing its measures, stating them in terms of “per flight hours.” In addition, FAA said the agency planned to use logic models to capture a uniform picture of external factors and share them with stakeholders. • OSHA said it found that the decline in occupational injury and illness rates in the early to mid-1990s was attributable to legislative reforms motivated by increases in workers’ compensation payments and a growing awareness of workplace hazards among unions, employers, and the insurance industry. According to OSHA, it found such factors as employment shifts into low-hazard industries and underreporting of injury and illness rates were not contributory. OSHA said its reform efforts during this period affected the agency’s inspection strategy and resulted in a renewed emphasis on outreach, partnering, and working cooperatively with employers to address workplace hazards. According to the agency, the new approach complemented market influences affecting industry, namely, escalating costs for workers’ compensation programs and the dawning realization that corrective action was needed to reduce workplace accidents. OSHA said its reforms reinforced and supported industry initiatives and contributed to the decline in occupational injury and illness rates. • MOEA said the agency contracted with a consultant to do an evaluation of the effect of municipal solid waste management on resource conservation and greenhouse gas emissions to get some idea of the impact of the agency’s efforts on its outcome goals. The agency said it also conducted an internal study of the economic benefits of recycling. • MPCA said the agency discovered that its “intuitive” linkages regarding mercury contamination were incorrect when tested by an independent institute. MPCA adjusted the agency’s strategies accordingly. Page 54 GAO/GGD-00-10 Managing for Results Appendix V Practice Category 5: Continue Improving Performance-Based Management In this category, the organizations said they continuously assessed their Summary performance-based management efforts and results to identify areas for improvement. The specific practice in this area was as follows: 12. Continuously assess and strengthen performance-based management efforts. Practice 12: Continuously The organizations said they continuously assessed their performance- based management approach and results, and modified their approach as Assess and Strengthen necessary. In addition, they said they recognized that the evolution of an Performance-Based organization’s performance management efforts over time involved Management changes across the other four practice areas. For example, as previously discussed, the leading organizations said they regularly reviewed their measures, goals, and targets and adjusted them as necessary. • FHCA started with key measures that were simple and easy to understand. Once performance measurement was accepted in the agency and processes were well established, the agency then could move to experiment with more sophisticated measures. • FDA said the agency changed its performance plan presentation to show the integration between initiatives and performance goals, more clearly stating how FDA planned to close the gap between strategic priorities and current performance. FDA also said it provided additional information on baselines and contexts for the agency’s performance goals. The organizations said they also sought to improve their management approach by moving on to more sophisticated methodologies, such as the 1 balanced scorecard, once initial experience was gained. For example: • FDBF said the agency used simple program logic chains at the beginning of its performance management efforts. Later, the agency considered other methodologies, such as the balanced scorecard. Over time, FDBF said it looked to further develop and reinforce performance management capabilities. • The Customs Service, NRC, MDES, OSHA, MDOT, and MDOR said they were either using, or planning to use, all or part of a balanced scorecard approach, building on their past performance management efforts to create a balanced set of measures. 1 The balanced scorecard approach is summarized in Robert S. Kaplan and David P. Norton, The Balanced Scorecard: Translating Strategy Into Action, Harvard Business School Press, Boston, 1996. The scorecard emphasizes the use of a balanced set of measures across the four categories of financial performance, customer knowledge, internal business processes, and learning and growth. Page 55 GAO/GGD-00-10 Managing for Results Appendix V Practice Category 5: Continue Improving Performance-Based Management In many cases, the organizations cautioned that they were just at the beginning of a journey to successfully implement performance-based management. For example: • MDES said that agencies must be willing to change many management and operational systems to better align them for performance. Changing those systems might take up to 10 years to accomplish. • MDOT said the agency spent 5 years developing a process for establishing meaningful measures and then continuously improving them. In the agency’s early performance management efforts, it began focusing on what the agency could achieve when it had direct control over events. Later, MDOT said it adopted more extensive in-depth analysis of cause-and-effect linkages and the influence of external factors. Overall, the organizations’ experiences indicated that a strong performance-based management approach was under constant review and refinement. The end result was not just intended to be a written strategic or performance plan, but a results-oriented culture within the organization and among stakeholders. Page 56 GAO/GGD-00-10 Managing for Results Bibliography Aristigueta, Maria P. “Fine-Tuning Performance Measurement as a Management Strategy: Insights from the States.” 1998 National Public Sector Productivity On-Line Conference, Nov. 1998. Atkinson, Anthony A., Waterhouse, John H., and Wells, Robert B. “A Stakeholder Approach to Strategic Performance Measurement,” Sloan Management Review. Spring 1997, pp. 25-37. Australian National Audit Office. Better Practice Principles for Performance Information. Internet address: http://www.anao.gov.au. Berman, Evan M. “Measuring Productivity,” Productivity in Public and Nonprofit Organizations. Sage, Thousand Oaks, CA, pp. 51-76, 1998. Berman, Evan M. and West, Jonathan P. “Productivity Enhancement Efforts in Public and Nonprofit Organizations,” Public Productivity & Management Review. Vol. 22, No. 2, Dec. 1998, pp. 207-219. Brizius, Jack A. and Campbell, Michael D. Getting Results. Council of Governors’ Policy Advisors, Washington, D.C., 1991. Buckman, James F. and Holter, Suzanne W. “Alignment: The Path to Performance Improvement,” Journal of Strategic Performance Measurement. Vol. 2, No. 6, Dec. 1998, pp. 40-46. Campbell, Wilson and Fountain, Jay. “Service Efforts and Accomplishments—Lessons Learned.” 1998 National Public Sector Productivity On-Line Conference, Nov. 1998. Center for Accountability and Performance. Performance Measurement: Concepts and Techniques, American Society for Public Administration. Washington, D.C., undated. Crookall, Paul and Ingstrup, Ole. “The Citizen as Customer: An International Survey of Well-Performing Public Service Organizations.” 1998 National Public Sector Productivity On-Line Conference, Nov. 1998. Ernst & Young Center for Business Innovation, Perspectives on Business Innovation. Issue 2, Measuring Business Performance, Ernst & Young, Center for Business Innovation, Cambridge, MA, 1998. Page 57 GAO/GGD-00-10 Managing for Results Bibliography Fischer, Richard J. “An Overview of Performance Measurement,” Benchmarks of Performance, Special Section, reprint of Sept. 1994 issue of Public Management, pp. S2-S8. Frigo, Mark L. and Krumwiede, Kip R. “Balanced Scorecards: A Rising Trend in Strategic Performance Measurement,” Journal of Strategic Performance Measurement. Vol. 3, No. 1, Feb.-Mar. 1999, pp. 42-48. Glover, Mark. A Practical Guide for Measuring Program Efficiency and Effectiveness in Local Government, The Innovation Groups. Tampa, FL, 1994. Grifel, Stuart S. “Organizational Culture: Its Importance in Performance Measurement,” Benchmarks of Performance. Special Section, reprint of Sept. 1994 issue of Public Management, pp. S19-S20. Hacker, Maria E. and Brotherton, Paul A. “Designing and Installing Effective Performance Measurement Systems.” Internet address: http:/www.balancedscorecard.com. Hatry, Harry P. “Where the Rubber Meets the Road: Performance Measurement for State and Local Public Agencies,” Using Performance Measurement to Improve Public and Nonprofit Programs. Kathryn E. Newcomer, Editor, New Directions for Evaluation, Number 75, Fall 1997, Jossey-Bass Publishers, San Francisco, pp. 31-44. Hatry, Harry P. Draft manuscript, Performance Measurement for Results- Focused Agencies, Aug. 1998. Hatry, Harry, Gerhart, Craig, and Marshall, Martha. “Eleven Ways to Make Performance Measurement More Useful to Public Managers,” Benchmarks of Performance. Special Section, reprint of Sept. 1994 issue of Public Management, pp. S15-S18. Julnes, Patria de Lancer. “Lessons Learned About Performance Measurement: Rebuilding Procrustes’ Bed.” 1998 National Public Sector Productivity On-Line Conference, Nov. 1998. Kaplan, Robert S. and Norton, David P. The Balanced Scorecard. Harvard Business School Press, Boston, 1996. Page 58 GAO/GGD-00-10 Managing for Results Bibliography Madigan, James M. “Measures Matrix Chart: A Holistic Approach to Understanding Operations,” Quality Management Journal. Oct. 1993, pp. 77-86. National Academy of Public Administration. Center for Improving Government Performance, Helpful Practices in Implementing Government Performance. Internet address: http://www.napawash.org/NAPA/NewNAPAHome. National Academy of Public Administration. Center for Improving Government Performance, Improving Performance Across Programs: Thinking About the Issue—Taking the First Steps. National Academy of Public Administration,Washington, D.C., undated. National Academy of Public Administration. Center for Improving Government Performance, Planning for Results. National Academy of Public Administration, Washington, D.C., undated. Newcomer, Kathryn E. “Using Performance Measurement to Improve Programs,” Using Performance Measurement to Improve Public and Nonprofit Programs. Kathryn E. Newcomer, Editor, New Directions for Evaluation, Number 75, Fall 1997, Jossey-Bass Publishers, San Francisco, pp. 5-14. Organisation for Economic Co-Operation and Development. Performance Measurement in Government: Issues and Illustrations. Organisation for Economic Co-Operation and Development, Paris, 1994. Organisation for Economic Co-Operation and Development. Performance Management in Government: Performance Measurement and Results- Oriented Management. Organisation for Economic Co-Operation and Development, Paris, 1994. Organisation for Economic Co-Operation and Development. In Search of Results: Performance Management Practices. Organisation for Economic Co-Operation and Development, Paris, 1997. Osborne, David and Gaebler, Ted. “The Art of Performance Measurement,” Accountability for Performance. David Ammons, Editor, 1995, ICMA, Washington, D.C., pp. 33-43. Patton, Michael Q. Utilization-Focused Evaluation. 3rd Edition, Sage Publications, Thousand Oaks, CA, 1997. Page 59 GAO/GGD-00-10 Managing for Results Bibliography Pautke, Robert W. and Redman, Thomas C. “Creating and Aligning High- Quality Data Resources: What Senior Leaders Need to Know,” Journal of Strategic Performance Measurement. Vol. 2, No. 6, Dec. 1998, pp. 33-38. Plantz, Margaret C., Greenway, Martha T., and Hendricks, Michael. “Outcome Measurement: Showing Results in the Nonprofit Sector,” Using Performance Measurement to Improve Public and Nonprofit Programs. Kathryn E. Newcomer, Editor, New Directions for Evaluation, Number 75, Fall 1997, Jossey-Bass Publishers, San Francisco, pp. 15-30. Poister, Theodore H. “Productivity Monitoring: Systems, Indicators, and Analysis,” Accountability for Performance. David Ammons, Editor, 1995, ICMA, Washington, D.C., pp. 98-120. Poister, Theodore H. and Strieb, Gregory D. “Strategic Management in the Public Sector,” Public Productivity and Management Review. Vol. 22, No. 3, Mar. 1999, pp. 308-325. Rossi, Peter H. and Freeman, Howard E. Evaluation: A Systematic Approach. 5th Edition, Sage Publications, Newbury Park, CA, 1993. Scheirer, Mary Ann. “Designing and Using Process Evaluation,” Handbook of Practical Program Evaluation. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer, Editors, 1994, Jossey-Bass Publishers, San Francisco, pp. 40-68. Schneiderman, Arthur M. “Why Balanced Scorecards Fail,” Journal of Strategic Performance Measurement. Special Edition, Jan. 1999, pp. 6-11. Sparrow, Malcolm K. Imposing Duties. Praeger Publishers, Westport, CT, 1994. State of Arizona, 1998 Strategic Planning and Performance Measurement Handbook, 1998. Internet address: http://www.state.az.us/ospb/StatPlan.html. Stenzel, Catherine and Stenzel, Joe. “From the Editors,” Journal of Strategic Performance Measurement. Special Edition, Jan. 1999, pp. 3-5. Swiss, James E. “Performance Monitoring Systems,” Accountability for Performance. David Ammons, Editor, 1995, ICMA, Washington, D.C., pp. 67-97. Page 60 GAO/GGD-00-10 Managing for Results Bibliography Texas State Auditor’s Office, Legislative Budget Board, Governor’s Office of Budget and Planning. Guide to Performance Measurement for State Agencies, Universities, and Health-Related Institutions. Austin, TX, 1995. Truitt, Bruce. “Measuring the Unmeasurable.” Paper presented at the Managing for Results conference, Austin, TX, Nov. 1-3, 1995. Van Wart, Montgomery and Berman, Evan. “Contemporary Public Sector Productivity Values,” Public Productivity and Management Review. Vol. 22, No. 3, Mar. 1999, pp. 326-347. Waller, Carl. “Emerging Issues in Performance Management,” Journal of Strategic Performance Measurement. Special Edition, Jan. 1999, pp. 37-40. Walters, Jonathan. Measuring Up. Governing Books, Washington, D.C., 1998. Weiss, Carol H. Evaluation Research. Prentice-Hall, Englewood Cliffs, NJ, 1972. Wholey, Joseph S. “Assessing the Feasibility and Likely Usefulness of Evaluation,” Handbook of Practical Program Evaluation. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer, Editors, 1994, Jossey- Bass Publishers, San Francisco, pp. 15-39. Wholey, Joseph S. “Trends in Performance Measurement: Challenges for Evaluators,” Evaluation for the 21st Century. Eleanor Chelimsky and William R. Shadish, Editors, 1997, Sage: Thousand Oaks, CA, pp. 124-133. Wholey, Joseph S. “Performance-Based Management: Responding to the Challenges,” Public Productivity and Management Review. Vol. 22, No. 3, Mar. 1999, pp. 288-307. Page 61 GAO/GGD-00-10 Managing for Results Related GAO Products Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information (GAO/GGD-99-139, July 30, 1999). Managing for Results: Opportunities for Continued Improvements in Agencies’ Performance Plans (GAO/GGD/AIMD-99-215, July 30, 1999). Agency Performance Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers (GAO/GGD/AIMD-99-69, Feb. 26, 1999). Managing for Results: Measuring Program Results That Are Under Limited Federal Control (GAO/GGD-99-16, Dec. 11, 1998). Managing for Results: An Agenda to Improve the Usefulness of Agencies’ Annual Performance Plans (GAO/AIMD/GGD-98-228, Sept. 8, 1998). The Results Act: Assessment of the Governmentwide Performance Plan for Fiscal Year 1999 (GAO/AIMD/GGD-98-159, Sept. 8, 1998). Performance Management: Aligning Employee Performance With Agency Goals at Six Results Act Pilots (GAO/GGD-98-162, Sept. 4, 1998). Program Evaluation: Agencies Challenged by New Demand for Information on Program Results (GAO/GGD-98-53, Apr. 24, 1998). The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans, Version 1 (GAO/GGD-10.1.20, Apr. 1998). Executive Guide: Measuring Performance and Demonstrating Results of Information Technology Investments (GAO/AIMD-98-89, Mar. 1998). Agencies’ Annual Performance Plans Under the Results Act: An Assessment Guide to Facilitate Congressional Decisionmaking, Version 1 (GAO/GGD/AIMD-10.1.18, Feb. 1998). Managing for Results: Critical Issues for Improving Federal Agencies’ Strategic Plans (GAO/GGD-97-180, Sept. 16, 1997). Managing for Results: Using the Results Act to Address Mission Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug. 29, 1997). Managing for Results: Regulatory Agencies Identified Significant Barriers to Focusing on Results (GAO/GGD-97-83, June 24, 1997). Page 62 GAO/GGD-00-10 Managing for Results Related GAO Products The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997). Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997). Performance Budgeting: Past Initiatives Offer Insights for GPRA Implementation (GAO/AIMD-97-46, Mar. 27, 1997). Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21, 1997). Executive Guide: Effectively Implementing the Government Performance and Results Act (GAO/GGD-96-118, June 1996). Page 63 GAO/GGD-00-10 Managing for Results Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Order by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touch- tone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested (233584)
Managing for Results: Strengthening Regulatory Agencies' Performance Management Practices
Published by the Government Accountability Office on 1999-10-28.
Below is a raw (and likely hideous) rendition of the original report. (PDF)