oversight

The Government Performance and Results Act: 1997 Governmentwide Implementation Will be Uneven

Published by the Government Accountability Office on 1997-06-02.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States General Accounting Office

GAO              Report to Congressional Committees




June 1997
                 THE GOVERNMENT
                 PERFORMANCE AND
                 RESULTS ACT
                 1997 Governmentwide
                 Implementation Will Be
                 Uneven




GAO/GGD-97-109
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      General Government Division

      B-277071

      June 2, 1997

      The Honorable Fred Thompson
      Chairman
      The Honorable John Glenn
      Ranking Minority Member
      Committee on Governmental Affairs
      United States Senate

      The Honorable Ted Stevens
      Chairman
      The Honorable Robert C. Byrd
      Ranking Minority Member
      Committee on Appropriations
      United States Senate

      The Honorable Pete V. Domenici
      Chairman
      The Honorable Frank R. Lautenberg
      Ranking Minority Member
      Committee on Budget
      United States Senate

      The Honorable Dan Burton
      Chairman
      The Honorable Henry A. Waxman
      Ranking Minority Member
      Committee on Government Reform
        and Oversight
      House of Representatives

      The Honorable Bob Livingston
      Chairman
      The Honorable David Obey
      Ranking Minority Member
      Committee on Appropriations
      House of Representatives
B-277071

The Honorable John R. Kasich
Chairman
The Honorable John M. Spratt, Jr.
Ranking Minority Member
Committee on Budget
House of Representatives

This report responds to the requirement in the Government Performance and Results Act of
1993 that we report to Congress on the implementation of the Act, including prospects for
compliance by federal agencies beyond those participating as pilots under the Act. Our specific
objectives for this report were to (1) assess the status of the Act’s implementation efforts;
(2) identify significant challenges confronting executive agencies in their efforts to become
more results-oriented; and (3) describe ongoing efforts to integrate program, cost, and budget
information into a reporting framework that allows for fuller consideration of resource
allocations, operational costs, and performance results.

We are sending copies of this report to other appropriate congressional committees and
executive branch agencies, including the Director of the Office of Management and Budget and
other interested parties. We also will make copies available to others on request.

If you or your staff have any questions concerning this report, I can be reached at
(202) 512-2637. The major contributors to this report are listed in appendix II.




L. Nye Stevens
Director, Federal Management
   and Workforce Issues




                     Page 2                                               GAO/GGD-97-109 Results Act
Executive Summary


             A consensus that the nation’s persistent federal deficit must be addressed
Purpose      and that the effectiveness of federal programs must improve substantially
             has spurred widespread efforts in the executive branch and Congress to
             dramatically change the way the federal government is managed. During
             the 1990s, Congress enacted a broad statutory framework, with the
             Government Performance and Results Act of 1993 as its centerpiece, to
             improve the management and accountability of federal agencies. In
             essence, the Act, which is referred to as “GPRA” or “the Results Act,” seeks
             to shift the focus of federal management and decisionmaking away from a
             preoccupation with the activities that are undertaken to a focus on the
             results of those activities as reflected in citizens’ lives.

             Congress understood that the management changes required to effectively
             implement the Results Act would not come quickly or easily. The Act
             therefore included a phased implementation approach that began in fiscal
             year 1994 with pilot projects on the Act’s performance planning and
             reporting requirements. Under the Results Act, GAO is to report to
             Congress on the implementation of the Act, including the prospects for
             compliance by executive agencies beyond those that participated in the
             pilot phase. This report culminates GAO’s efforts to meet that mandate.
             GAO’s specific objectives for this report were to (1) assess the status of the
             Results Act’s implementation efforts; (2) identify significant challenges
             confronting executive agencies in their efforts to become more results
             oriented; and (3) describe ongoing efforts to integrate program, cost, and
             budget information into a reporting framework that allows for fuller
             consideration of resource allocations, operational costs, and performance
             results.


             The Results Act is intended to improve the efficiency and effectiveness of
Background   federal programs by establishing a system to set goals for program
             performance and to measure results. Specifically, the Act requires
             executive agencies to prepare multiyear strategic plans, annual
             performance plans, and annual performance reports. As a starting point,
             the Act requires virtually every executive agency to develop a strategic
             plan, covering a period of at least 5 years forward from the fiscal year in
             which it is submitted. These strategic plans are to include an agency’s
             mission statement, general goals and objectives, and the strategies that the
             agency will use to achieve those goals and objectives. Agencies are to
             consult with Congress and solicit the views of other stakeholders in the
             development of those plans and to submit the first strategic plans to the




             Page 3                                               GAO/GGD-97-109 Results Act
Executive Summary




Office of Management and Budget (OMB) and Congress by September 30,
1997.

Next, the Results Act requires executive agencies to develop annual
performance plans covering each program activity set forth in the
agencies’ budgets. The first annual performance plans, covering fiscal year
1999, are to be provided to Congress after the President’s budget is
submitted to Congress in 1998. The annual performance plan is to contain
an agency’s annual goals, the measures that the agency will use to gauge
its performance toward meeting those goals, and the resources the agency
will need to meet its goals. Finally, the Results Act requires executive
agencies to prepare annual reports on program performance for the
previous fiscal year. The performance reports are to be issued by March 31
each year, with the first (for fiscal year 1999) to be issued by March 31,
2000. In each report, an agency is to compare its performance against its
goals, summarize the findings of program evaluations completed during
the year, and describe the actions needed to address any unmet goals.

The Results Act established requirements for pilot projects so that
agencies could gain experience in using key provisions of the Act and
provide lessons for pilots and for other agencies. The Act required OMB to
designate at least 10 agencies to participate in the first set of pilot projects
during fiscal years 1994 through 1996, which focused on the preparation of
annual performance plans and reports. A second set of pilot projects in
managerial accountability and flexibility were to propose waivers of
administrative procedural requirements and controls for one or more of
the major functions and operations of the agency to determine the effects
of providing federal managers with increased managerial flexibility in
exchange for greater accountability for performance. OMB was to designate
at least five agencies from the first set of pilot projects to test managerial
accountability and flexibility during fiscal years 1995 and 1996. A third set
of pilot projects, which are scheduled in the Act for fiscal years 1998 and
1999, are to test performance budgeting—i.e., the presentation of the
varying levels of performance that would result from different budget
levels.

To prepare this report, GAO drew on a large body of work it had done in
recent years on the Results Act and on related goal-setting, performance
measurement, and accountability concepts. GAO also surveyed a sample of
civilian managers at the general schedule (GS) and general management
(GM) levels, GS/GM-13, through Senior Executive Service (SES) levels in 24
major agencies. These 24 agencies accounted for over 99 percent of the



Page 4                                                 GAO/GGD-97-109 Results Act
                   Executive Summary




                   federal government’s net outlay for fiscal year 1996. The sample was
                   stratified by whether the manager was SES or non-SES and by whether the
                   manager was working in a selected agency or agency component that was
                   designated as a Results Act pilot. Of the approximately 1,300 managers GAO
                   surveyed, it received usable responses from about 72 percent.

                   The survey results are statistically generalizable to the 24 agencies
                   included in the survey. The survey data reported throughout this report
                   are the estimated percentages of how officials would have responded had
                   the entire universe of eligible officials been surveyed. In general,
                   percentages reported for the entire sample have confidence intervals
                   ranging from + 5 percentage points to + 12 percentage points. In other
                   words, if all managers and supervisors in the 24 agencies included in GAO’s
                   population had been surveyed, the chances are 95 out of 100 that the
                   results obtained would not differ from the sample estimate, in the most
                   extreme case, by more than + 12 percentage points.


                   GAO’s work shows that the Results Act’s implementation to this point has
Results in Brief   achieved mixed results, which will lead to highly uneven governmentwide
                   implementation in the fall of 1997. While agencies are likely to meet the
                   upcoming statutory deadlines for producing initial strategic plans and
                   annual performance plans, GAO found that those documents will not be of
                   a consistently high quality or as useful for congressional and agency
                   decisionmaking as they could be. On a more promising note, OMB selected
                   over 70 performance planning and reporting pilots that far exceeded the
                   number required by the Results Act and that should provide a rich body of
                   experience for agencies to draw on in the future.

                   GAO  also found that the experiences of some of the Results Act pilot
                   agencies, such as the Social Security Administration (SSA), and related
                   efforts by nonpilot agencies, such as the Veterans Health Administration,
                   showed that significant performance improvements were possible—even
                   in the short term—when an agency adopted a disciplined approach to
                   setting results-oriented goals, measuring its performance, and using
                   performance information to improve effectiveness. However, the reported
                   examples of substantial performance improvements were relatively few,
                   and many agencies did not appear to be well positioned to provide in 1997
                   a results-oriented answer to the fundamental Results Act question: What
                   are we accomplishing?




                   Page 5                                              GAO/GGD-97-109 Results Act
    Executive Summary




    GAO’s review of efforts to date under the Results Act have shown that to
    effectively implement the Act, agencies face a variety of significant
    challenges, some of which will not be resolved quickly. One set of
    challenges arises from the complications of government structure and
    from program proliferation. Others involve methodological difficulties in
    identifying performance measures or the lack of data needed to establish
    goals and assess performance. The following are among the challenges
    that GAO observed.

•   Overlapping and fragmented crosscutting program efforts, such as student
    loan or economic development programs, present the logical need to
    coordinate efforts to ensure that goals are consistent and, as appropriate,
    that program efforts are mutually reinforcing. GAO found that overlapping
    and fragmented program efforts can undermine efforts to establish clear
    missions and goals. In addition to the problem of overlapping and
    fragmented programs, agencies are challenged in setting goals because
    those goals often must reflect a balance of competing policy priorities. For
    example, GAO has reported that the effectiveness of the Forest Service had
    been compromised because it had not reconciled conflicts concerning its
    role in regulating the use of its lands (e.g., promoting timber sales versus
    protecting wildlife). Striking the right balance is a continuing and difficult
    challenge because forging the political consensus needed to create and
    sustain a program often results in that program having competing and/or
    broadly stated goals.
•   The often limited or indirect influence that the federal government has in
    determining whether a desired result is achieved complicates the effort to
    identify and measure the discrete contribution of the federal initiative to a
    specific program result. GAO work has shown that measuring the federal
    contribution is particularly challenging for regulatory programs; scientific
    research programs; and programs that deliver services to taxpayers
    through third parties, such as state and local governments.
•   The lack of results-oriented performance information in many agencies
    hampers efforts to identify appropriate goals and confidently assess
    performance. Even when data exist, GAO has consistently found that the
    quality of agencies’ performance data is often questionable due to several
    factors, including the need to rely on third parties to provide data.
•   Instilling within agencies an organizational culture that focuses on results
    remains a work in progress across the federal government. According to
    GAO’s survey, about 57 percent of managers said that their agencies’ top
    leadership demonstrated a strong commitment to achieving results to a
    great or very great extent. In contrast, however, only about 16 percent said
    that program changes by management above their levels were based on



    Page 6                                               GAO/GGD-97-109 Results Act
    Executive Summary




    results-oriented performance information to a great or very great extent.
    This suggests that despite the strength of managers’ perceptions about top
    leadership commitment, the extent to which leaders are demonstrating
    their commitment by using performance information for decisionmaking is
    not as great.
•   Linking agencies’ performance plans directly to the budget process,
    through the Results Act requirement to base the annual program
    performance goals on the budget’s program activity structure, may present
    significant difficulties. GAO’s analysis has shown that the extent to which
    the budget’s program activity structure can be directly linked to a
    results-oriented performance framework varies widely among activities,
    and GAO’s work has suggested that adjustments and accommodations in
    the program activity structure may be needed. GAO’s work also has
    suggested that reaching agreement on such changes between Congress
    and the executive branch will be a time-consuming and difficult process
    that will take more than one budget cycle to resolve.

    The Results Act is the cornerstone of a series of initiatives that are
    intended to provide a comprehensive framework for integrating program,
    cost, and budget information. GAO notes that improved financial reporting
    and auditing required by the Chief Financial Officers Act (CFO) Act should
    strengthen the reliability of cost and performance information. GAO also
    notes that the Federal Accounting Standards Advisory Board (FASAB) has
    developed a new set of reporting concepts and accounting standards that
    underpin OMB’s guidance to agencies on the form and content of their
    agencywide financial statements. These standards include a new reporting
    model for federal agencies geared to providing users with consolidated
    performance, cost, and other information. These standards also include
    cost accounting standards that became effective beginning with fiscal year
    1997 and are the first set of standards that are to account for the full costs
    of federal programs. GAO work has shown that developing the necessary
    information systems to gather, process, and analyze the needed program
    and cost information will be a substantial undertaking for most federal
    agencies because of agencies’ long-standing weaknesses in the use of
    information technology. To help address problems in agencies’ use of
    information technology, Congress enacted the Paperwork Reduction Act
    of 1995 and the Clinger-Cohen Act of 1996.

    Addressing some of the challenges highlighted by GAO, such as
    crosscutting program efforts and balances among competing priorities,
    will raise significant policy issues for Congress and the administration to
    consider, some of which will likely be very difficult to resolve. In this



    Page 7                                               GAO/GGD-97-109 Results Act
                      Executive Summary




                      regard, GAO believes that the Results Act’s success or failure should not be
                      judged on whether contentious policy issues are fully resolved; rather,
                      judgment of the success or failure of the Results Act should turn on the
                      extent to which the information produced through the Results Act’s
                      goal-setting and performance measurement practices—once those
                      practices are successfully implemented—helps inform policy decisions
                      and improve program management.



Principal Findings

                      Executive agencies, OMB, and Congress are undertaking efforts to
Performance           implement the Results Act. Over 70 federal organizations—in some cases,
Improvements Have     entire agencies such as SSA and the Internal Revenue Service—participated
Been Made, but        in the performance planning and reporting pilot phase. GAO’s work
                      indicated that in addition to overseeing those pilots, OMB has achieved
Overall Results Act   some success working with agencies to seek agreement on strategic and
Implementation        annual goals and performance measures and to increase the prominence
                      of performance information in the executive branch budget process. For
Efforts Are Mixed     example, when GAO reviewed fiscal year 1996 OMB budget documents, it
                      found that more attention had been placed on performance-related issues
                      than in the previous fiscal year. Congress, too, has shown a growing
                      interest in and support for the governmentwide implementation of the
                      Results Act. For example, the House Majority has established teams
                      consisting of staff from various committees to lead its strategic plan
                      consultation efforts. These teams have been reaching out to agencies to
                      review and comment on agencies’ strategic plans.

                      Most important, GAO found some agencies have reported improved results
                      due to the Results Act or related efforts. For example, GAO has reported
                      that the Veterans Health Administration improved services to veterans by
                      more rigorously assessing the results of the medical care it provides to the
                      nation’s veterans. In particular, the Veterans Health Administration
                      reported that it used performance information to target the most
                      important improvement opportunities and thereby lowered the mortality
                      rate for cardiac procedures by an average of 13 percent over the last 8
                      years. In another example, involving SSA’s national toll-free 800 telephone
                      number to handle citizen inquiries, SSA used customer satisfaction and
                      other performance information to identify and make program changes,
                      including providing additional staff to handle phone calls. As a result, the




                      Page 8                                              GAO/GGD-97-109 Results Act
Executive Summary




busy rate decreased from 49 to 34 percent, and the percentage of calls
answered within 5 minutes increased from 74 to 83 percent from fiscal
year 1995 to fiscal year 1996. GAO notes that these and other performance
improvements strongly suggest that the basic goal-setting and
performance measurement model used by the Results Act, if successfully
implemented, will be an important tool for improving federal management
and performance.

Although some agencies reported that significant performance
improvements have been made, GAO noted that, in general, few agencies
have shown substantial performance improvements thus far. These limited
reports of improvements may be understandable at this early stage of the
Results Act’s implementation. However, GAO’s work also indicates that
agencies do not appear to have consistently in place the critical
performance measurement processes that would allow them to make
future improvements. For example, GAO reported in January 1997 that the
Department of Housing and Urban Development’s Public Housing
Management Assessment Program did not collect important information
needed to manage and assess its results. The program is to assess the
performance of local housing authorities by measuring factors such as the
numbers of outstanding work orders and uncollected rents. However, the
system does not measure other factors, such as housing quality, that are
essential for assessing the results that housing authorities are achieving, as
well as for determining which housing authorities are performing well or
poorly.

The situation at the Department of Housing and Urban Development
appears to be typical. According to GAO’s survey, fewer than one-third of
managers in the agencies included in its survey reported that
results-oriented performance measures exist for their programs to a great
or very great extent. The existence of other types of performance
measures needed to successfully manage programs also was reported as
low. For example, of the total managers reporting the existence of such
measures to a great or very great extent, 38 percent of the managers
reported the existence of measures of outputs, 32 percent reported the
existence of customer satisfaction measures, 31 percent reported the
existence of measures of product or service quality, and 26 percent
reported the existence of measures of efficiency.

On a more positive note, while still viewed as low, significantly more
managers reported in GAO’s survey that results-oriented and other
performance measures existed to a greater extent currently than 3 years



Page 9                                               GAO/GGD-97-109 Results Act
                            Executive Summary




                            ago. For example, GAO asked managers about the current situation as
                            compared with what they recollected 3 years ago concerning the existence
                            of results-oriented performance measures and measures of customer
                            satisfaction. The percentage of managers who reported the existence of
                            these measures to a great or very great extent was higher currently by 13
                            and 21 percentage points, respectively. Although this increase is a very
                            favorable development, GAO notes that the overall low level still is a matter
                            for concern.

                            Obviously, it is not sufficient merely to measure current performance. The
                            Results Act envisions that performance information will be used to make
                            decisions and better manage programs. In that regard, about one-fifth of
                            federal managers from agencies included in GAO’s survey reported that
                            results-oriented performance information was used to a great or very great
                            extent to develop their agencies’ budgets. Managers reported even lower
                            levels in the use of results-oriented performance information to help make
                            funding decisions, legislative changes, or program changes. However,
                            managers rated the extent of use of results-oriented performance
                            information currently as somewhat higher than they rated the extent of
                            use 3 years ago. The reported limited availability of results-oriented
                            performance measures, combined with the still infrequent use of such
                            information to help make key decisions, suggests to GAO that many
                            agencies may have a difficult time developing meaningful results-oriented
                            measures in time for fiscal year 1999 performance plans that are due to
                            OMB this fall.



                            GAO  found that agencies are confronting a variety of difficult challenges
Key Challenges              that have limited the implementation of the Results Act. These challenges
Remain to Effective         include establishing clear agency missions and strategic goals, especially
Implementation of the       when program efforts are overlapping or fragmented; measuring
                            performance, particularly when the federal contribution to a result is
Results Act                 difficult to determine; generating the results-oriented performance
                            information needed to set goals and assess progress; instilling a
                            results-oriented organizational culture within agencies; and linking
                            performance plans to the budget process. The experiences of pilot
                            agencies and related efforts by nonpilot agencies suggest that these
                            challenges will not be quickly or easily resolved.


Establishing Clear Agency   Attempting to reach a reasonable degree of consensus on agencies’
Missions and Strategic      missions and establishing clear and precise strategic goals are likely to
Goals Remains a Challenge   raise contentious policy issues that may not be easily resolved. Past GAO


                            Page 10                                              GAO/GGD-97-109 Results Act
Executive Summary




work has shown that as Congress and the executive branch have
responded to new national needs and problems, many different agencies
have been given responsibility for addressing the same or similar national
issues. Of the 18 national mission areas displayed in the federal budget, 14
were addressed by more than 1 executive branch department or major
agency in fiscal year 1996. In some cases, this shared responsibility
demonstrates that addressing a national need requires the efforts of more
than one agency.

However, past GAO work has suggested that some important federal
program areas have suffered from overlap and fragmentation. Overlapping
and fragmented program efforts can frustrate program customers, waste
scarce resources, and limit the overall effectiveness of the federal effort.
For example, GAO reported in 1995 on the Department of Education
programs that provided loans and grants to students to help finance their
higher education. GAO found that although the student loan and Pell grant
programs provided the majority of federal financial aid to students for
postsecondary education, another 22 smaller programs were targeted to
specific segments of the postsecondary school population, such as
prospective students from disadvantaged families or women and
minorities who are underrepresented in graduate education. These 22
programs were collectively funded at $1.1 billion for fiscal year 1995. GAO
concluded that these smaller grant programs could be considered
candidates for consolidation—with other larger programs or among
themselves—with no adverse impact on students’ access to postsecondary
education. GAO also found that the federal government could anticipate
administrative savings of 10 percent each year, or a total of $550 million in
budget authority (adjusted for inflation) over 5 years.

In addition, GAO notes that many agencies confront competing policy
objectives, which are a natural by-product of the complex social and
political environment in which programs are created, funded, and
managed. For example, GAO reported in April 1997 that the Forest Service
had increasingly shifted the emphasis of its efforts from producing timber
to sustaining wildlife. This shift was taking place in reaction to
requirements in planning and environmental laws and their judicial
interpretation—reflecting changing public values and concerns—together
with social, ecological, and other factors. However, GAO noted that the
demand for recreation was also expected to grow and may increasingly
conflict with efforts aimed at sustaining wildlife and producing timber. GAO
found that the disagreement both within the Forest Service and among key
external stakeholders, including Congress, on how the Forest Service is to



Page 11                                              GAO/GGD-97-109 Results Act
                           Executive Summary




                           resolve conflicts or make choices among competing uses on its lands had
                           seriously undermined its efforts to establish the goals and performance
                           measures needed to ensure accountability. GAO concluded that until
                           general agreement is reached, the Forest Service’s decisionmaking will
                           continue to be inefficient and ineffective.


Lack of Federal Control    GAO’s past work shows that agencies must grapple with some difficult
Over Results Complicates   analytic and technical challenges in developing results-oriented
Performance Measurement    performance measures. The variety of influences beyond direct federal
                           control that may determine whether, and the degree to which, a federal
Efforts                    effort is successful is a major challenge to agencies’ efforts to measure
                           performance. Moreover, the end result of a program may occur long after
                           the federal intervention, which makes it difficult to measure progress
                           annually. Because of issues like these, performance measurement efforts,
                           such as those for scientific research, regulatory, and intergovernmental
                           programs, are particularly challenging.

                           For example, GAO found that determining the impact of economic
                           development programs has been a daunting task because of the numerous
                           external forces—including broad national economic trends and the
                           assistance that communities may receive from state and local
                           governments and the private sector—that may contribute to local
                           economic development. Separating out the effects of federal program
                           efforts can be extremely difficult, as GAO observed in a 1996 review on
                           economic development programs, because it would require, first,
                           documentation that there had been some improvement in a targeted area;
                           second, linkage of specific program elements to actual economic changes;
                           and third, measurement of the growth stemming from other influences on
                           the economy of the targeted area in order to isolate the impact that could
                           be attributed to the economic development program.

                           GAO found that although such performance measurement efforts constitute
                           a substantial undertaking, they have the potential to provide important
                           information for decisionmakers. For example, one study of the
                           effectiveness of the Department of Commerce’s Economic Development
                           Administration (EDA) programs found that income in the counties that
                           received EDA funding grew significantly faster than income in the counties
                           that received no aid. However, when EDA’s programs and factors unrelated
                           to EDA were considered simultaneously, the study found that EDA’s
                           programs had a very small effect on income growth rates during the period
                           that the aid was received and had no significant effect in the 3 years after



                           Page 12                                             GAO/GGD-97-109 Results Act
                          Executive Summary




                          the aid ceased. The study found that EDA’s programs could explain only a
                          small part of the difference in these growth rates between the two groups
                          of counties.

                          GAO  found that agencies are exploring a number of approaches for
                          addressing difficulties in developing results-oriented performance
                          measures. These approaches include undertaking program evaluations to
                          try to isolate program impact, such as in the EDA case already discussed;
                          using measures of intermediate results; employing a range of measures to
                          assess whether programs are making progress; and working with
                          stakeholders to seek agreement on appropriate measures.


Absence of Quality        GAO has found that a lack of results-oriented performance information to
Results-Oriented          use as a baseline complicates agencies’ efforts to set appropriate
Performance Information   improvement targets. Department of Veterans Affairs officials told GAO
                          that some of their results-oriented measures for a Loan Guaranty program
Hampers Efforts to Set    were new and that baseline data were not available on those measures.
Goals and Assess          Consequently, they did not have data on past performance to use in setting
Performance               some of the program’s fiscal year 1998 goals. In some of these cases, the
                          Department indicated in its fiscal year 1998 budget submission that those
                          goals were “to be determined.” In another example, Department of
                          Agriculture officials told GAO that they eliminated some performance
                          measures that had been part of their Results Act pilot’s annual
                          performance plan because they did not have a way to collect data on those
                          measures. Lacking these data, they did not have an informed basis on
                          which to set goals.

                          Past GAO work also has shown that agencies face a variety of challenges to
                          collecting valid and reliable performance information, including ensuring
                          that standard definitions are used when measuring performance. For
                          example, GAO reported in 1996 on management challenges that the
                          Department of Health and Human Service’s Office of Child Support
                          Enforcement (OCSE) faced because of a lack of comparable performance
                          data across state and local jurisdictions. OCSE officials said that
                          discrepancies resulting from differences in the way the states and local
                          jurisdictions defined what constitutes a child support enforcement case
                          had contributed to the difficulty of uniformly measuring state
                          performance. To address these discrepancies, OCSE worked with state and
                          local authorities to develop standard data definitions for key child support
                          enforcement terms, including a definition for what constitutes a child
                          support enforcement case, and incorporated the use of standardized



                          Page 13                                             GAO/GGD-97-109 Results Act
                           Executive Summary




                           definitions for measuring state performance. Together, they also
                           developed measures to assess state performance in obtaining support
                           orders and agreed that states would use the OCSE-established definition of
                           a child support enforcement case to report these data.


Making Needed Cultural     According to GAO’s survey, federal managers rated the commitments of top
Changes Is a Work in       leadership to achieving results as higher currently than they did for 3 years
Progress Across Agencies   ago. However, federal managers’ responses to GAO’s survey also suggested
                           that more progress is needed for agencies to develop and sustain cultures
                           that focus on results. For example, when GAO asked federal managers
                           about the extent to which they or supervisors at their levels had the
                           authority they needed to help their agencies accomplish their strategic
                           goals, the managers did not perceive that they had more such authority
                           currently than they recalled having 3 years ago.

                           Significantly, for managers from the Results Act pilots that GAO was able to
                           isolate for its sample, GAO’s survey found these managers’ perception of
                           the extent of their authority currently was much lower than their
                           perception of the situation 3 years ago. For example, 56 percent of SES
                           managers from selected Results Act pilots reported that managers at their
                           level had authority to help the agency accomplish its strategic goals to a
                           great or very great extent 3 years ago. Fewer SES managers, 40 percent,
                           reported having authority to a comparable extent currently—a difference
                           of 16 percentage points.

                           These survey results suggest to GAO that as agencies implement the Results
                           Act and strive to become more results oriented, they need to pay special
                           attention to ensuring that key managers have the authority they need to
                           achieve intended results. In passing the Results Act, Congress recognized
                           that if federal managers were to be held accountable for program results,
                           they would need the authority and flexibility to achieve those results.
                           Congress also understood the importance of affording federal program
                           managers the freedom to be innovative and creative and to marshal
                           resources to achieve results. Indeed, Congress has provided agencies with
                           additional authority in the key area of procurement that allows for more
                           flexibility in managing their programs. In addition, the administration’s
                           National Performance Review has sought the empowerment of employees
                           to achieve results as one of its major initiatives.




                           Page 14                                              GAO/GGD-97-109 Results Act
                           Executive Summary




Integrating Performance    The annual performance plans that agencies are to develop under the
Information Into the       Results Act provide the opportunity, if successfully implemented, to assist
Budget Will Likely Prove   congressional and executive branch resource allocation decisionmaking
                           by providing direct linkages between the performance information
Difficult                  contained in performance plans and agency budget requests. Under the
                           Results Act, the annual performance goals are to be based on the program
                           activities listed in the president’s budget. GAO has observed that although
                           budgeting is inherently an exercise in political choice, the Results Act is
                           based on the premise that budget decisions should be more clearly linked
                           to expectations about program performance.

                           In a 1997 report on past federal initiatives in performance budgeting, GAO
                           noted that an agency’s program activities generally result from
                           negotiations between the agency, OMB, and the relevant congressional
                           appropriations subcommittees. Program activity structures, which
                           represent programmatic, process, organizational, or other orientations,
                           therefore depend on the needs, interests, and experiences of agencies,
                           OMB, and Congress. Because of this wide variability, the suitability of
                           current program activity structures for the Results Act performance
                           planning and measurement may vary.

                           GAO’s  discussions with agency officials and congressional staff as part of
                           its review of past performance budgeting initiatives highlighted the
                           differing expectations held by the two branches of government on the use
                           of the budget’s program activity structure as a basis for planning and
                           measuring agencies’ performance. Congressional staff were generally
                           comfortable with existing activity structures and questioned whether
                           changes would frustrate congressional oversight. The staff generally
                           viewed these structures as fundamental to congressional oversight of
                           agency activities, and thus they viewed changes with apprehension and
                           concern. However, some agency officials saw program activity structures
                           as secondary to planning. Therefore, where current program activity
                           structures proved unsuitable for planning purposes, these officials viewed
                           change in program activity structures as inevitable and appropriate.
                           However, they noted that negotiating changes with Congress could prove
                           difficult and time-consuming.




                           Page 15                                             GAO/GGD-97-109 Results Act
                        Executive Summary




                        GAO’s past work indicates that integrated accountability reports have the
An Augmented            potential of further strengthening congressional and executive branch
Reporting Framework     decisionmaking and agencies’ accountability. The FASAB accountability
Holds Promise for       reporting concepts and accounting standards—building on CFO Act
                        requirements—are to bring together program performance information
Strengthening Federal   with audited financial information to provide congressional and other
Decisionmaking and      decisionmakers with a more complete picture of the results, operational
                        performance, and the costs of agencies’ operations. For the first time,
Accountability          decisionmakers are to be provided with annual “report cards” on the costs,
                        management, and effectiveness of federal agencies. The FASAB cost
                        accounting standards hold a similar promise to improve decisionmaking, if
                        successfully implemented. These standards are to provide decisionmakers
                        with information on the costs of all resources used, including the costs of
                        services provided by others to support activities or programs. Such
                        information would allow for comparisons of the costs of various programs
                        and activities with their performance outputs and results.

                        Congress, in enacting the Results Act, the CFO Act, and information
                        technology reform legislation, sought to create a more focused,
                        results-oriented management and decisionmaking process within both
                        Congress and the executive branch. GAO has noted that these laws were
                        intended to respond to a need for accurate, reliable information for
                        congressional and executive branch decisionmaking, information that had
                        been badly lacking. GAO observed that implemented together, these laws
                        provide a powerful framework for developing fully integrated information
                        about agencies’ missions and strategic priorities, results-oriented
                        performance goals that flow from those priorities, performance data to
                        show the achievement (or not) of those goals, the relationship of
                        information technology investments to the achievement of performance
                        goals, and accurate and audited financial information about the costs of
                        achieving mission results.


                        GAO has long supported the goal-setting, performance measurement, and
Recommendations         results-based accountability concepts embodied in the Results Act. As part
                        of GAO’s statutory mandate under the Results Act, and at the request of
                        congressional committees, GAO monitored the performance planning and
                        reporting pilot phase of the Results Act, highlighted the key
                        implementation issues needing additional attention as those issues were
                        identified, and made suggestions to Congress and agencies as appropriate.
                        Moreover, the GAO reports that examined management issues in individual
                        agencies contained recommendations, as appropriate, to improve the



                        Page 16                                            GAO/GGD-97-109 Results Act
                  Executive Summary




                  management of those agencies. Overall, GAO found nothing in the work
                  that it did specifically for this report that would lead it to alter its position
                  in support of the concepts in the Results Act or the recommendations it
                  has made to specific agencies. Therefore, GAO is making no new
                  recommendations in this report.


                  GAO sent a draft of this report to the Director of OMB for comments. On
Agency Comments   May 21 and 22, 1997, senior OMB officials provided technical comments
                  that were incorporated where appropriate.




                  Page 17                                                 GAO/GGD-97-109 Results Act
Contents



Executive Summary                                                                                  3


Chapter 1                                                                                         22
                         Congress Has Enacted a Statutory Framework for Improving                 22
Introduction               Federal Management and Decisionmaking
                         The Results Act Establishes Specific Requirements                        25
                         The Results Act Incorporates Lessons From Past Initiatives to            30
                           Improve Public Sector Performance
                         Leading Organizations Show a Way                                         31
                         Objectives, Scope, and Methodology                                       34

Chapter 2                                                                                         40
                         The Number of Performance Planning and Reporting Pilots                  40
Progress of the            Exceeded the Results Act Requirements
Results Act              Managerial Accountability and Flexibility Pilot Effort Did Not           41
                           Work as Intended
Implementation Has       Performance Budgeting Pilots Likely to Be Delayed                        42
Been Mixed               OMB Actions to Prepare the Executive Branch for the Results Act          42
                           Implementation
                         Congressional Commitment to the Results Act Is Reflected in Its          44
                           Increased Involvement and Activities
                         Some Agencies Reported Improved Performance                              44
                         Managers Reported Results-Oriented Performance Measures and              46
                           Their Use Was Limited
                         Conclusions                                                              50

Chapter 3                                                                                         51
                         Crosscutting Federal Programs Pose Challenge for Results Act             51
Unclear Agency             Implementation
Missions and Goals       Balancing Goals Is Complicated by Competing Demands                      55
                         Congressional Consultations Provide Mechanism for Identifying            57
Will Challenge Results     and Addressing Crosscutting Issues and Seeking to Balance
Act Implementation         Priorities
                         Conclusions                                                              59




                         Page 18                                           GAO/GGD-97-109 Results Act
                         Contents




Chapter 4                                                                                         61
                         Several Factors Have Complicated Agencies’ Ability to Set Goals          61
Developing Useful          and Identify Results-Oriented Performance Measures
Results-Oriented         Agencies Have Faced Challenges Collecting Reliable and Valid             67
                           Results-Oriented Performance Data
Performance              Approaches to Improve the Usefulness of Results-Oriented                 69
Information Will Be an     Performance Information Are Beginning to Emerge
Ongoing Difficulty       Conclusions                                                              74

Chapter 5                                                                                         76
                         Top Leadership Has Not Fully Demonstrated Commitment to                  76
Building an                Results-Oriented Management
Organizational           Managers Reported Needing Additional Training                            79
                         Managers Reported Reduced Authority to Achieve Results                   82
Culture to Help Create   Less Than Half of Federal Managers Reported Positive                     83
and Sustain a Focus        Recognition for Achieving Results
on Results Remains a     Involvement of Federal Managers in Results Act-Related                   84
                           Activities Is Varied
Work in Progress         Federal Managers’ Views About GPRA’s Past and Future Effects             86
                           on Agency Programs Are Mixed
                         Conclusions                                                              89

Chapter 6                                                                                         90
                         The Results Act Seeks to Better Align Desired Results With               91
Efforts to Integrate        Budget Decisions
Performance              Integrated Performance, Cost, and Management Reports Provide             93
                            Basis for Supporting Key Decisions
Information With         Conclusions                                                              96
Budget Decisions and
to Develop a New
Reporting Framework
Are Promising
Appendixes               Appendix I: Survey on Results-Oriented Management Practices              98
                         Appendix II: Major Contributors and Acknowledgements                    111

Related GAO Products                                                                             115




                         Page 19                                           GAO/GGD-97-109 Results Act
          Contents




Tables    Table 1.1: Analysis of Sample Dispositions                              37
          Table 2.1: Percent of Federal Managers Who Reported Their               48
            Programs Have Various Performance Measures and Recollect
            They Had Such Measures 3 Years Ago
          Table 2.2: Percent of Federal Managers Reporting Extent to              49
            Which Results-Oriented Performance Information Is Used
            Currently and Compared to 3 Years Ago
          Table 2.3: Percent of Federal Managers Reporting Use of                 50
            Performance Information to a Great or Very Great Extent Among
            Managers With Performance Measures Demonstrating
            Achievement of Results to a Great or Very Great Extent
          Table 5.1: Percent of Federal Managers Who Reported That Their          77
            Agencies’ Top Leadership Demonstrated a Strong Commitment to
            Achieving Results
          Table 5.2: Percent of Federal Managers Who Reported That                80
            During the Past 3 Years Their Agency Provided, Arranged, or Paid
            for Training That Would Help Them to Accomplish Specific Tasks
          Table 5.3: Percent of Federal Managers Who Reported Needing             80
            Training or Additional Training to Accomplish Specific Tasks
          Table 5.4: Percent of Federal Managers Reporting That Agency            83
            Managers/Supervisors at Their Levels Had the Decisionmaking
            Authority They Needed to Help the Agency Accomplish Its
            Strategic Goals
          Table 5.5: Percent of Federal Managers Reporting That They Had          85
            Been Involved in GPRA-Related Activities During the Past 3 Years

Figures   Figure 1.1: Key Steps and Critical Practices for Effectively            33
            Implementing GPRA
          Figure 5.1: Distribution of Federal Managers Expressing an              87
            Opinion on the Extent to Which Implementing GPRA to Date Had
            Improved Their Agencies’ Programs
          Figure 5.2: Distribution of Federal Managers Expressing an              88
            Opinion on the Extent to Which Implementing GPRA Could
            Improve Their Agencies’ Programs in the Future




          Page 20                                          GAO/GGD-97-109 Results Act
Contents




Abbreviations

ARC        Appalachian Regional Commission
ARL        Army Research Laboratory
BEP        Bureau of Engraving and Printing
CFO        Chief Financial Officers
CPDF       Civilian Personnel Data File
DLA        Defense Logistics Agency
DOD        Department of Defense
EDA        Economic Development Administration
EPA        Environmental Protection Agency
FAA        Federal Aviation Administration
FASAB      Federal Accounting Standards Advisory Board
FDA        Food and Drug Administration
GM         General Management
GMRA       Government Management Reform Act
GPRA       Government Performance and Results Act
GS         General Schedule
HHS        Department of Health and Human Services
HUD        Department of Housing and Urban Development
IRS        Internal Revenue Service
JFMIP      Joint Financial Management Improvement Program
MBO        Management by Objectives
NCS        National Cemetery System
NHTSA      National Highway and Traffic Safety Administration
NPR        National Performance Review
OCSE       Office of Child Support Enforcement
OMB        Office of Management and Budget
ONDCP      Office of National Drug Control Policy
OPM        Office of Personnel Management
OSHA       Occupational Safety and Health Administration
PPBS       Planning-Programming-Budgeting System
R&D        Research and Development
SES        Senior Executive Service
SSA        Social Security Administration
USDA       Department of Agriculture
VA         Department of Veterans Affairs
ZBB        Zero-Based Budgeting


Page 21                                          GAO/GGD-97-109 Results Act
Chapter 1

Introduction


                       A consensus that the nation’s persistent federal deficit must be addressed
                       and that the effectiveness of federal programs must improve substantially
                       has spurred widespread efforts under way in federal agencies and
                       Congress to dramatically change the way the federal government is
                       managed. Although proposals to reduce the deficit are prompting difficult
                       policy debates over the role of the federal government, better management
                       of federal programs and activities can provide part of the solution to the
                       government’s fiscal problems. But the hard work of managing downsized
                       federal agencies is made more difficult when those agencies lack goals
                       that focus on results; cost-effective strategies to achieve those goals;
                       sound measures of performance; and reliable, accurate, and timely
                       information needed for decisionmaking.

                       Our work over many years repeatedly has found that agencies lack these
                       basic underpinnings of well-run organizations. In enacting the Government
                       Performance and Results Act (GPRA) of 1993, which is referred to as “GPRA”
                       or “the Results Act,” Congress expressed frustration that congressional
                       and executive branch decisionmaking was often hampered by the lack of
                       good information on the results of federal program efforts. However,
                       Congress also recognized that improving management in the federal sector
                       will not be easy or come quickly.


                       In the 1990s, Congress put in place a statutory framework to address the
Congress Has Enacted   long-standing weaknesses in federal operations, improve federal
a Statutory            management practices, and provide greater accountability for achieving
Framework for          results. This framework included as its essential elements financial
                       management reform legislation, information technology reform legislation,
Improving Federal      and the Results Act.
Management and
                       In enacting this framework, Congress sought to create a more focused,
Decisionmaking         results-oriented management and decisionmaking process within both
                       Congress and the executive branch. These laws seek to improve federal
                       management by responding to a need for accurate, reliable information for
                       congressional and executive branch decisionmaking, information that has
                       been badly lacking in the past, as much of our work has demonstrated.
                       Implemented together, these laws provide a powerful framework for
                       developing fully integrated information about agencies’ missions and
                       strategic priorities, the results-oriented performance goals that flow from
                       those priorities, performance data to show the achievement (or not) of
                       those goals, the relationship of information technology investments to the




                       Page 22                                            GAO/GGD-97-109 Results Act
                         Chapter 1
                         Introduction




                         achievement of performance goals, and accurate and audited financial
                         information about the costs of achieving mission results.


Financial Management     The primary financial management reform legislation Congress enacted is
Reform Legislation       the Chief Financial Officers Act of 1990 (CFO) Act, as expanded by the
                         Government Management Reform Act of 1994 (GMRA). This law provides
                         the basis for identifying and correcting financial management weaknesses
                         that have cost the federal government billions of dollars and leave it
                         vulnerable to waste, fraud, and mismanagement. The expanded CFO Act
                         spelled out a long overdue and ambitious agenda to help the government
                         resolve its lack of timely, reliable, useful, and consistent financial
                         information. First, 24 major executive agencies, covering over 99 percent
                         of the federal government’s net outlay for fiscal year 1996, are required to
                         prepare and have audited financial statements for their entire operations,
                         beginning with fiscal year 1996. Second, every year starting with fiscal year
                         1997, the Secretary of the Treasury is required to produce a
                         governmentwide financial statement in coordination with the Director of
                         the Office of Management and Budget (OMB), and the statement is to be
                         audited by the Comptroller General of the United States. With successful
                         implementation, the audited financial statements required by the CFO Act,
                         as expanded by GMRA, will provide congressional and executive branch
                         decisionmakers with the financial and program cost information that they
                         have not previously had.


Information Technology   Information technology reform legislation, including the Paperwork
Reform Legislation       Reduction Act of 1995 and the Clinger-Cohen Act of 1996, is based on the
                         best practices used by leading public and private organizations to more
                         effectively manage information technology.1 Even though agencies have
                         obligated billions of dollars—$145 billion in the last 6 years alone—to
                         build up and maintain their information technology resources, the benefits
                         frequently have been disappointing. Federal technology projects often
                         experience schedule slippages, incur extensive cost overruns, and fail to
                         provide promised performance improvements.

                         We have noted that the sound application and management of information
                         technology to support strategic program goals must be an important part
                         of any serious attempt to improve agency mission performance, cut costs,



                         1
                          Executive Guide: Improving Mission Performance Through Strategic Information Management and
                         Technology—Learning from Leading Organizations (GAO/AIMD-94-115, May 1994).



                         Page 23                                                          GAO/GGD-97-109 Results Act
                         Chapter 1
                         Introduction




                         and enhance responsiveness to the public.2 Under the information
                         technology reform laws, agencies are to better link their technology plans
                         and information technology use to their programs’ missions and goals. To
                         do this, agencies are to, among other things, (1) involve senior executives
                         in information management decisions; (2) establish senior-level Chief
                         Information Officers who are to, among other things, evaluate information
                         technology programs on the basis of applicable performance
                         measurements; (3) impose much-needed discipline on technology
                         spending; (4) redesign inefficient work processes; and (5) use
                         performance measures to assess technology’s contribution to achieving
                         mission results.


Government Performance   The landmark Results Act is the centerpiece of an integrated statutory
and Results Act          framework, which includes reform legislation in the areas of financial
                         management and information technology. The Act focuses on clarifying
                         missions, setting program goals, and measuring performance toward
                         achieving those goals. It emphasizes managing for results and pinpointing
                         opportunities for improved performance and increased accountability. The
                         purposes of the Act are to:

                         “(1) improve the confidence of the American people in the capability of
                         the Federal Government, by systematically holding Federal agencies
                         accountable for achieving program results;

                         “(2) initiate program performance reform with a series of pilot projects in
                         setting program goals, measuring program performance against those
                         goals, and reporting publicly on their progress;

                         “(3) improve Federal program effectiveness and public accountability by
                         promoting a new focus on results, service quality, and customer
                         satisfaction;

                         “(4) help Federal managers improve service delivery, by requiring that they
                         plan for meeting program objectives and by providing them with
                         information about program results and service quality;

                         “(5) improve congressional decisionmaking by providing more objective
                         information on achieving statutory objectives, and on the relative
                         effectiveness and efficiency of Federal programs and spending; and


                         2
                         High-Risk Areas: Actions Needed to Solve Pressing Management Problems (GAO/T-AIMD/GGD-97-60,
                         Mar. 5, 1997).



                         Page 24                                                        GAO/GGD-97-109 Results Act
                           Chapter 1
                           Introduction




                           “(6) improve internal management of the Federal Government.”3

                           Congress intended for the Act to improve the effectiveness of federal
                           programs by fundamentally shifting the focus of management and
                           decisionmaking away from a preoccupation with staffing and activity
                           levels to a broader focus on the results of federal programs. As measures
                           of their performance, agencies have traditionally used the amount of
                           money spent to support their programs, the number of staff employed, or
                           the number of tasks completed. But at a time when the value of many
                           federal programs is undergoing intense public scrutiny, an agency that
                           uses and reports only these measures has not answered the defining
                           question of whether these programs have produced real results. Under the
                           Results Act, virtually every executive agency is required to ask itself some
                           basic questions: What is our mission? What are our goals, and how can we
                           achieve them? How can we measure our performance? How will we use
                           performance information to make improvements?


                           The Results Act requires executive agencies to set goals, measure
The Results Act            performance, and report on the degree to which goals were met. Because
Establishes Specific       Congress understood that most agencies would need to make significant
Requirements               management changes to effectively implement the Act, requirements were
                           to be phased in over several years. Thus, the implementation of the Act
                           began with a set of pilot projects to be designated by OMB so that selected
                           agencies could gain experience in using key provisions of the Act and
                           provide lessons for other agencies. These pilots were to focus on

                       •   the development of agency performance plans, which provide the direct
                           linkage between an agency’s long-term goals (commonly referred to as
                           strategic goals) and what its managers and employees do day-to-day; and
                           performance reports, which are the feedback to managers, policymakers,
                           and the public as to what was actually accomplished for the resources
                           expended;
                       •   managerial accountability and flexibility, which is the granting of waivers
                           from certain nonstatutory administrative procedural requirements and
                           controls in return for greater management accountability for performance;
                           and
                       •   performance budgeting, which is intended to provide Congress with
                           information on the direct relationship between proposed program
                           spending and expected program results and the anticipated effects of
                           varying spending levels on results.

                           3
                            P.L. 103-62, sec. 2.



                           Page 25                                             GAO/GGD-97-109 Results Act
                            Chapter 1
                            Introduction




                            The Act requires the Director of OMB to report to Congress and the
                            president on the results of the pilot projects. The first report was to cover
                            pilot projects in performance planning and reporting and managerial
                            accountability and flexibility and was due no later than May 1, 1997. This
                            report was to include, among other things, an assessment of the benefits
                            and costs of pilot performance plans and reports and any significant
                            difficulties experienced by the pilot agencies in preparing proposed
                            waivers.4 OMB is to prepare a second report to Congress and the president
                            on the performance budgeting pilots. This report is due no later than
                            March 31, 2001.



Agencies Are to Set         The Results Act is designed to improve the efficiency and effectiveness of
Strategic Direction,        federal programs by establishing a system to set goals for program
Identify Annual             performance and to measure results. Specifically, the Act requires
                            executive agencies to prepare multiyear strategic plans, annual
Performance Goals, and      performance plans, and annual performance reports.
Report on Performance



Multiyear Strategic Plans   The Results Act requires virtually every executive agency to develop
                            strategic plans covering a period of at least 5 years forward from the fiscal
                            year in which it is submitted and to update those plans at least every 3
                            years.5 Agencies’ first strategic plans are to be submitted to Congress and
                            the Director of OMB by September 30, 1997. The strategic plans are to
                            (1) include the agencies’ mission statements; (2) identify long-term general
                            goals and objectives, also known as strategic goals; (3) describe how the
                            agencies intend to achieve those goals through their activities and through
                            their human, capital, information, and other resources; and (4) explain the
                            key external factors that could significantly affect the achievement of
                            those goals. Under the Act, strategic plans are the starting point for
                            agencies to set annual performance goals and to measure program
                            performance in achieving those goals. Consequently, strategic plans are
                            also to include a description of how long-term general goals will be related
                            to annual performance goals as well as a description of the program
                            evaluations that agencies used to establish their long-term general goals

                            4
                            The Government Performance and Results Act: Report to the President and the Congress from the
                            Director of the Office of Management and Budget, May 19, 1997.
                            5
                             The Results Act applies to agencies as defined in 5 U.S.C. 306(f), which generally covers executive
                            departments, government corporations, and independent establishments.



                            Page 26                                                                GAO/GGD-97-109 Results Act
                           Chapter 1
                           Introduction




                           and a schedule for subsequent evaluations. As part of the strategic
                           planning process, agencies are required to consult with Congress and
                           solicit the views of other stakeholders—those governmental and
                           nongovernmental entities potentially affected by, or interested in, the
                           agencies’ activities.

Annual Performance Plans   Building on the decisions made as part of the strategic planning process,
                           the Results Act requires executive agencies to develop annual
                           performance plans covering each program activity set forth in the
                           agencies’ budgets.6 The first annual performance plans, covering fiscal
                           year 1999, are to be submitted to OMB this fall and to Congress after the
                           President’s budget in 1998. The plan is to contain an agency’s annual
                           performance goals, which the agency is to use in order to gauge its
                           progress toward accomplishing its strategic goals. The plan also is to
                           include the performance measures that the agency will use to gauge
                           progress toward the achievement of its annual performance goals and the
                           resources the agency will need to meet its goals. Finally, the plan is to
                           discuss how the agency will verify the resulting performance data.

                           The Act authorizes agencies to apply for managerial flexibility waivers in
                           their annual performance plans. Agencies’ authority to request waivers of
                           nonstatutory administrative procedural requirements and controls is
                           intended to provide federal managers with more flexibility to structure
                           agency systems to better support performance goals. An example of
                           increased flexibility would be to allow an organization to recapture
                           unspent operating funds because of increased efficiencies and then to use
                           these funds to purchase new equipment or expand employee training.
                           Another example might involve delegating more authority to line managers
                           to make procurement decisions.

                           OMB  is to use the performance plans that agencies submit to develop an
                           overall federal government performance plan. OMB is to submit this
                           governmentwide plan each year to Congress with the president’s budget.
                           According to the Senate Committee on Governmental Affairs report on the
                           Act, the overall federal government performance plan is to present to
                           Congress a single cohesive picture of the federal government’s annual
                           performance goals for the fiscal year.7 The first overall plan is due with the
                           President’s fiscal year 1999 budget.

                           6
                            The term “program activity” refers to the listings of projects and activities in the Appendix portion of
                           the Budget of the United States Government. Program activity structures are intended to provide a
                           meaningful representation of the operations financed by a specific budget account.
                           7
                            S. Rep. No. 58, 103d Cong. 1st Sess. (1993).



                           Page 27                                                                  GAO/GGD-97-109 Results Act
                             Chapter 1
                             Introduction




Annual Performance Reports   Finally, the Results Act requires each executive agency to prepare annual
                             reports on program performance for the previous fiscal year. The first
                             performance reports for fiscal year 1999 are due to Congress and the
                             president no later than March 31, 2000, and subsequent reports are due by
                             March 31 for the years that follow. In each report, an agency is to review
                             and discuss its performance compared with the performance goals it
                             established in its annual performance plan. When a goal is not met, the
                             agency is to explain in the report the reasons the goal was not met; plans
                             and schedules for meeting the goal; and, if the goal was impractical or not
                             feasible, the reasons for that and the actions recommended. According to
                             the Senate committee report on the Act, actions needed to accomplish a
                             goal could include legislative, regulatory, or other actions. If an agency
                             finds a goal to be impractical or not feasible, it is to include a discussion of
                             whether the goal should be modified.

                             In addition to evaluating the progress made toward achieving its annual
                             goals, an agency’s program performance report is to evaluate the agency’s
                             performance plan for the fiscal year in which the performance report was
                             submitted. Thus, in their fiscal year 1999 performance reports that are due
                             by March 31, 2000, agencies are required to evaluate their performance
                             plans for fiscal year 2000 on the basis of their reported performance in
                             fiscal year 1999. This evaluation is to help show how an agency’s actual
                             performance is influencing its performance plan. The report also is to
                             include (1) the summary findings of program evaluations completed
                             during the fiscal year covered by the report and (2) the use and
                             effectiveness of any of the Results Act managerial flexibility waivers that
                             an agency received.

                             Agencies also are to include baseline and trend data in annual
                             performance reports to help ensure that their reports are complete and
                             that performance is viewed in context. Such data can show whether
                             performance goals are realistic given the past performance of an agency.
                             Such data can also assist users of reports to draw more informed
                             conclusions than they would by comparing only a single year’s
                             performance against an annual goal, because users of reports can see
                             improvements or declines in an agency’s performance over prior years.8
                             For fiscal years 2000 and 2001, agencies’ reports are to include data on the
                             extent to which their performance achieved their goals, beginning with
                             fiscal year 1999. For each subsequent year, agencies are to include
                             performance data for the year covered by the report and 3 prior years.
                             Congress recognized that in some cases not all the performance data will

                             8
                              GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).



                             Page 28                                                      GAO/GGD-97-109 Results Act
                         Chapter 1
                         Introduction




                         be available in time for the required reporting date. In such cases, agencies
                         are to provide whatever data are available with a notation as to their
                         incomplete status. Subsequent annual performance reports are to include
                         the complete data as part of the trend information.


The Results Act Used a   The Results Act required OMB to designate at least 10 agencies to
Phased Implementation    participate in the first set of pilot projects, which were to focus on the
Approach to Build        preparation of program performance plans and reports. The selected
                         agencies were to reflect a representative range of government functions
Experience and Provide   and capabilities in measuring and reporting program performance. Also,
Lessons                  the Act required these pilot projects to use strategic plans during 1 or
                         more years of the pilot period, which ran from fiscal years 1994 through
                         1996.

                         Congress recognized that managerial accountability for results is linked to
                         managers having sufficient flexibility, discretion, and authority to
                         accomplish desired results. Thus, the Act specified a second set of pilot
                         projects in managerial accountability and flexibility. These pilot projects
                         were to propose waivers of administrative procedural requirements and
                         controls for one or more of the major functions and operations of the
                         agency to determine the effects of providing federal managers with
                         increased managerial flexibility in exchange for the greater accountability
                         for performance. At least five agencies from the first set of pilot projects
                         were to be designated by OMB to test managerial accountability and
                         flexibility during fiscal years 1995 and 1996.

                         Finally, the Results Act called for a third set of pilot projects for fiscal
                         years 1998 and 1999 to test performance budgeting—the presentation of
                         varying levels of performance that would result from different budget
                         levels—before requiring it statutorily. OMB is to select at least five
                         agencies, at least three of which had experience developing performance
                         plans during the first pilot phase, to test performance budgeting. By
                         March 31, 2001, OMB is to report to Congress and the president on the pilot
                         test of performance budgeting. OMB’s report is to assess the feasibility of
                         performance budgeting as part of the president’s budget, recommend
                         whether legislation requiring performance budgets should be proposed,
                         and identify any other recommended changes to the requirements of the
                         Act.




                         Page 29                                              GAO/GGD-97-109 Results Act
                            Chapter 1
                            Introduction




                            The Results Act is the most recent federal government reform initiative
The Results Act             designed to better align spending decisions with expected results. Through
Incorporates Lessons        acquiring better information on the relative efficiency and effectiveness of
From Past Initiatives       federal programs and spending, the Act seeks to help federal managers
                            improve program performance. We recently reported on the key design
to Improve Public           elements and approaches of the Act and compared them with those of past
Sector Performance          federal initiatives that sought to link resources with results—broadly
                            known as performance budgeting.9 Since 1950, the federal government has
                            attempted four governmentwide performance budgeting initiatives:
                            (1) reforms flowing from the first Hoover Commission in its efforts to
                            downsize the post-World War II government; (2) the
                            Planning-Programming-Budgeting-System (PPBS) that President Johnson
                            began in 1965, which introduced a decisionmaking framework to the
                            executive branch budget formulation process by presenting and analyzing
                            choices among long-term policy objectives and alternative ways of
                            achieving them; (3) Management by Objectives (MBO) that President Nixon
                            initiated in 1973, which sought to link agencies’ stated objectives to their
                            budget requests and put in place a process to hold agency managers
                            responsible for achieving agreed-upon outputs and results; and
                            (4) Zero-Based Budgeting (ZBB) that President Carter initiated in 1977,
                            which required agencies to set priorities on the basis of the program
                            results that could be achieved at alternative spending levels, one of which
                            was to be below current funding. Even though these reforms were
                            generally perceived as having fallen short of their stated goals, they
                            contributed to the evolution of performance-based measurement and
                            budgeting in the federal government.

                            The Results Act incorporates critical lessons learned from these previous
                            efforts.

                        •   Where past efforts failed to link executive branch performance planning
                            and measurement with congressional resource allocation processes, the
                            Act requires explicit consultation between the executive and legislative
                            branches on agency strategic plans.
                        •   Where past initiatives devised unique performance information formats
                            often unconnected to the structures used in congressional budget
                            presentations, the Act requires agencies to plan and measure performance
                            using the program activities listed in their budget submissions.
                        •   Where past initiatives were generally unprepared for the difficulties
                            associated with measuring the results of federal programs and often

                            9
                            Performance Budgeting: Past Initiatives Offer Insights for GPRA Implementation (GAO/AIMD-97-46,
                            Mar. 27, 1997).



                            Page 30                                                           GAO/GGD-97-109 Results Act
                        Chapter 1
                        Introduction




                        retreated to simple output or workload measures, the Act states a
                        preference for results-oriented measures while recognizing the need to
                        develop a range of measures, including output and, in some cases,
                        nonquantitative measures.


                        The federal government’s efforts to become more results oriented are
Leading Organizations   consistent with a broad national and international trend among public
Show a Way              organizations. According to the Senate committee report on the Results
                        Act, use of performance measurement was a growing trend in state
                        governments. The report further noted that work done by the Organization
                        for Economic Cooperation and Development suggested that several
                        foreign governments may have been 5 to 10 years ahead of the United
                        States in the development of performance measurement, which was a
                        basic part of their broader efforts to better manage for results. More
                        specifically, these foreign governments set out broad themes to (1) define
                        goals clearly, develop measures, and report on progress; and (2) give
                        managers the flexibility to manage for results by providing them the tools
                        and incentives to act.

                        At the request of the Senate Committee on Governmental Affairs and the
                        House Committee on Government Reform and Oversight, we reported on a
                        number of state and foreign governments that were successfully pursuing
                        management reform and becoming more results-oriented.10 We studied the
                        state governments of Florida, Oregon, Minnesota, North Carolina, Texas,
                        and Virginia; and the foreign governments of Australia, Canada, New
                        Zealand, and the United Kingdom. Much like the federal government, a
                        number of these governments were dissatisfied with their public service.
                        Many of them were called upon to improve performance while
                        simultaneously reducing costs. Some faced economic challenges, such as
                        economies dominated by government spending and high budget deficits.
                        Each of these organizations set its agenda for management reform
                        according to its own environment, needs, and capabilities. Yet, despite
                        their differing approaches to reform, these organizations were seeking to
                        become more results oriented.

                        We were asked by the Senate Committee on Governmental Affairs and the
                        House Committee on Government Reform and Oversight whether the
                        experiences of these and other public organizations could yield
                        worthwhile lessons for executive agencies as they attempt to implement

                        10
                         Managing for Results: State Experiences Provide Insights for Federal Management Reforms
                        (GAO/GGD-95-22, Dec. 21, 1994); and Managing for Results: Experiences Abroad Suggest Insights for
                        Federal Management Reforms (GAO/GGD-95-120, May 2, 1995).



                        Page 31                                                             GAO/GGD-97-109 Results Act
Chapter 1
Introduction




the Results Act. In response to that request, we issued the Executive
Guide: Effectively Implementing the Government Performance and Results
Act.11 The Executive Guide identifies a set of key steps and associated
practices that leading federal and other public sector organizations have
used to successfully implement reform efforts that are consistent with the
Results Act. Accompanying the discussion of each practice is a case
illustration of a federal agency that has made progress in incorporating
that practice into its operations.

Taken together, the key steps and practices drawn from the organizations
we studied provide a useful framework to assist Congress and the
executive branch as they work to implement the Act. These steps were to
(1) define mission and desired outcomes, or results; (2) measure
performance to gauge progress; and (3) use performance information as a
basis for decisionmaking. In taking these steps, leading organizations also
found that certain leadership practices, such as devolving operational
authority and creating incentives for managers and staff to focus on
results, were central to making the changes needed for the organizations
to become more results oriented. Figure 1.1 illustrates the key steps and
associated practices.




11
  GAO/GGD-96-118, June 1996.



Page 32                                             GAO/GGD-97-109 Results Act
                                            Chapter 1
                                            Introduction




Figure 1.1: Key Steps and Critical Practices for Effectively Implementing GPRA




                                                   Step 1:
                                                   Define Mission and
                                                   Desired Outcomes
                                                   Practices:
                                                   1. Involve stakeholders
                                                   2. Assess environment
                                                   3. Align activities,
                                                      core processes,
                                                      and resources




                  Step 3:                                                             Step 2:
                  Use Performance                     Reinforce GPRA Implementation   Measure Performance
                  Information
                                                      Practices:                      Practices:
                  Practices:                          9. Devolve decisionmaking       4. Produce measures at
                  6. Identify performance                 with accountability            each organizational
                                                      10. Create incentives
                     gaps                             11. Build expertise                level that
                  7. Report information               12. Integrate management           demonstrate results,
                  8. Use information                      reforms                        are limited to the vital
                                                                                         few,
                                                                                         respond to multiple
                                                                                         priorities, and
                                                                                         link to responsible
                                                                                         programs
                                                                                      5. Collect data




                                            Source: Executive Guide.




                                            Page 33                                                     GAO/GGD-97-109 Results Act
                     Chapter 1
                     Introduction




                     As required by the Act, our overall objective was to report to Congress on
Objectives, Scope,   the implementation of the Act, including the prospects for compliance by
and Methodology      executive agencies beyond those participating as pilot projects. More
                     specifically, our objectives for this report were to (1) assess the status of
                     the Results Act implementation efforts; (2) identify significant challenges
                     confronting executive agencies in their efforts to become more results
                     oriented; and (3) describe ongoing efforts to integrate program, cost, and
                     budget information into a reporting framework that allows for fuller
                     consideration of resource allocations, operational costs, and performance
                     results. As agreed with the Senate Committee on Government Affairs and
                     the House Committee on Government Reform and Oversight, our strategy
                     was to partially fulfill our legislative requirement by reporting regularly on
                     the Act and related initiatives throughout the pilot phase of performance
                     planning, measurement, and reporting. This report culminates our efforts
                     to meet this mandate.

                     To meet each of our objectives, we relied on a wide range of management
                     work we have done for the reports, testimonies, and other products we
                     issued to congressional committees, OMB, and executive agencies on the
                     progress of agencies’ efforts to become results oriented. These products
                     include, most recently, the Executive Guide: Effectively Implementing the
                     Government Performance and Results Act (GAO/GGD-96-118, June 1996); GPRA
                     Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996); GPRA: Managerial
                     Accountability and Flexibility Pilot Did Not Work As Intended
                     (GAO/GGD-97-36, Apr. 10, 1997); Managing for Results: Analytic Challenges in
                     Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997); Performance
                     Budgeting: Past Initiatives Offer Insights for GPRA Implementation
                     (GAO/AIMD-97-46, Mar. 27, 1997); and numerous congressional testimonies.
                     This work also included detailed examinations of related federal
                     management improvement efforts, in particular the National Performance
                     Review (NPR). In addition, we used products we have issued in recent
                     years on the critical management issues confronting specific agencies,
                     such as the Forest Service and the Department of Energy; program efforts
                     that cut across several agencies, such as federal land management, food
                     safety, and early childhood development programs; and, most
                     prominently, our examination of 25 high-risk areas—those federal areas
                     most vulnerable to waste, fraud, and mismanagement.12 The Related GAO
                     Products section at the end of this report lists selected reports and
                     congressional testimonies that we issued on the Results Act
                     implementation and related results-oriented management initiatives in
                     pilot and nonpilot agencies.

                     12
                       See, for example, High-Risk Series: An Overview (GAO/HR-97-1, Feb. 1997).



                     Page 34                                                              GAO/GGD-97-109 Results Act
Chapter 1
Introduction




We reviewed OMB circulars, memoranda, and general guidance to
executive agencies on a wide range of the Results Act implementation
activities, such as the preparation and submission of strategic plans and
examples of performance measurement. For selected pilot agencies, we
reviewed performance plans and performance reports. For these and
nonpilot agencies that we selected to reflect a representative range of
federal functions, we also reviewed draft strategic plans and other
applicable management documents. To supplement our existing
information, we reviewed the American Society of Public Administration’s
(ASPA) case studies on performance management in the federal
government and the National Academy of Public Administration’s (NAPA)
assessment of the initial pilot performance plans.13

To further meet our first two objectives, we sent a questionnaire to a
random sample of 1,300 mid-level and upper level civilian managers and
supervisors working in the 24 executive branch agencies covered by the
CFO Act. These agencies represent about 97 percent of the executive
branch full-time workforce and cover over 99 percent of the federal
government’s net outlay for fiscal year 1996. In reporting this
questionnaire data, when we use the term governmentwide, we are
referring to these 24 CFO Act agencies; and when we use the term federal
managers, we are referring to both managers and supervisors. We drew
our sample from the Office of Personnel Management’s (OPM) Civilian
Personnel Data File (CPDF) as of June 30, 1996, using file designators to
identify individuals who perform management and supervisory functions.
We restricted our sample to individuals who were in the United States due
to the difficulty and time associated with forwarding and receiving mail in
some foreign countries.

This sample was stratified by whether the manager was SES or non-SES and
by whether the manager was working in an agency or agency component
that was designated as a Results Act pilot. The management levels covered
general schedule (GS) and general management (GM) levels GS/GM-13
through the career Senior Executive Service (SES). The identification of
managers working in pilots was limited to selection from the following 13
Results Act pilots: Defense Logistics Agency, Army Research Laboratory,
Social Security Administration, Small Business Administration,
Occupational Safety and Health Administration, National Highway Traffic
Safety Administration, Forest Service, National Technical Information

13
 See the federal case studies developed at the initiative of ASPA on selected agencies’ strategic
planning and performance measurement efforts and Toward Useful Performance Measurement:
Lessons Learned from Initial Pilot Performance Plans Prepared Under the Government Performance
and Results Act, NAPA, Nov. 1994.



Page 35                                                             GAO/GGD-97-109 Results Act
Chapter 1
Introduction




Service, National Oceanographic and Atmospheric Administration,
Internal Revenue Service, U.S. Mint, Bureau of Engraving and Printing, and
the U.S. Customs Service’s Office of Enforcement.

Our selection of pilots from which to sample managers was guided by the
following requirements: (1) unique CPDF agency component identifier
codes that corresponded with the scope or bounds of the Results Act pilot,
(2) the pilot designation was made before fiscal year 1996, and (3) the pilot
covered 350 or more employees. The first restriction was needed to ensure
that any managers who were randomly selected for the pilot strata would
be from an agency or agency component that was encompassed in its
entirety by the pilot. Because many pilots were aligned according to
agency functions or processes and cut across multiple agency
components, the CPDF identifier codes could not be reliably used to ensure
that a randomly selected manager was working within the scope of the
pilot. We also excluded pilots that were designated in fiscal year 1996,
because any significant initiatives would have been fairly recent and may
not have been sufficiently implemented for any effects to be reflected in
questionnaire responses. Small-scale pilots covering fewer than 350
employees were excluded in order to allocate the sample more efficiently,
given the resources and time available to do the survey. In general, our
selected pilots, 12 of which were designated as pilots in fiscal year 1994,
contained a preponderance of pilots that encompassed the whole agency
or a major agency component.

Our questionnaire asked managers about their perceptions on such topics
as (1) agency mission and goals, (2) ways of measuring performance,
(3) the use of performance information now and 3 years ago,
(4) hindrances to measuring and using performance information,
(5) agency climate, (6) Results Act-related training, (7) knowledge of the
Act, (8) involvement in the Act, and (9) improvements seen or expected as
a result of implementation of the Act. Most of the items on the
questionnaire were closed-ended, meaning that depending on the
particular item, respondents could choose one or more response
categories or rate the strength of their perception on a 5-point extent
scale. After designing our questionnaire, we conducted pretests at four
federal agencies: the Department of Defense, the Department of
Commerce, the Department of Agriculture’s Forest Service, and the
Department of Justice. In all, 32 federal managers took part in these
pretests. The questionnaire was revised after each pretest on the basis of
the feedback received.




Page 36                                             GAO/GGD-97-109 Results Act
                                Chapter 1
                                Introduction




                                The survey was initially sent out between November 27, 1996, and
                                January 3, 1997. Managers who did not respond to the initial questionnaire
                                were sent one follow-up questionnaire. Table 1.1 summarizes the
                                disposition of the 1,300 surveys sent to managers originally sampled.

Table 1.1: Analysis of Sample
Dispositions                                                                                                                Number
                                Total managers sampled                                                                         1,300
                                                                      a
                                    Selectees deleted from sample                                                                  47
                                Eligible sample                                                                                1,253
                                    Unable to identify or locate sample selectee                                                   22
                                    Refused to participate                                                                         23
                                    Questionnaires not returned                                                                    299
                                    Questionnaires returned, not usable                                                             4
                                    Usable questionnaires returned                                                                 905
                                a
                                 Includes individuals who were outside the United States, retired, separated, died, or otherwise
                                left the agency or had some other reason that excluded them from the population of interest.



                                We received usable responses from 72 percent of the eligible sample. The
                                response rate across strata ranged from 61 percent to 88 percent. Although
                                we did not test the validity of the respondents’ answers or the comments
                                they made, we took several steps to check the quality of our survey data.
                                We reviewed and edited the completed questionnaires, made internal
                                consistency checks on selected items, and checked the accuracy of data
                                entry on a sample of surveys.

                                We followed up on a sample of the nonrespondents to assess whether the
                                views of the nonrespondent group differed from the views of those who
                                returned the survey. We selected a stratified, random sample of 101
                                nonrespondents to telephone and urge them to return their questionnaires.
                                If necessary, they were sent new questionnaires. Nine of these
                                nonresponse sample members returned completed questionnaires before
                                we contacted them. Of the remaining 92 individuals, we determined that 2
                                were no longer in the population of interest and that various
                                circumstances (e.g., maternity leave) precluded participation by 4 others.
                                Of the 86 individuals we did contact, 51 returned completed, usable
                                questionnaires; 16 refused; 1 returned the questionnaire too late for the
                                data to be included in the analysis; and 18 failed to return the
                                questionnaire. We analyzed the responses of the 51 individuals returning
                                the questionnaire and compared their responses on selected items to those
                                received from all other respondents. Our assessment indicated that across



                                Page 37                                                              GAO/GGD-97-109 Results Act
Chapter 1
Introduction




the selected items and types of analyses performed, the responses of the
follow-up group did not demonstrate a sufficient degree of significant and
consistent differences to merit separate treatment in subsequent analyses
of the questionnaire data. The responses of the 51 individuals were
included with all other responses. Appendix I contains a copy of the
questionnaire with weighted percentage of managers responding to each
item. Percents presented in this report have been rounded to the nearest
whole number.

The overall survey results are generalizable to the CFO Act agencies.
Results presented for the selected pilots are generalizable only to those
pilots, but not to all the other Results Act pilots. All results are subject to
some uncertainty or sampling error as well as nonsampling error. Because
a complex sample design was needed to support a diverse set of planned
analyses and sample size was constrained by the resources and time
available to do the survey, it was not possible to employ a sample size that
would consistently limit overall sampling error to 5 percent or less at the
95 percent confidence level. In general, percentages reported for the entire
sample have confidence intervals ranging from +5 percentage points to
+12 percentage points. In other words, if all CFO Act managers in our
population had been surveyed, the chances are 95 out of 100 that the
results obtained would not differ from our sample estimate, in the most
extreme case, by more than +12 percentage points. The magnitude of
sampling error will also vary in relation to the particular groups or items
being compared. As a result, the magnitude of difference needed to attain
statistical significance between groups or items will also vary. In some
instances, a difference of a certain magnitude may be statistically
significant; in other instances, depending on the nature of the comparison
being made, a difference of equal or even greater magnitude may not be
statistically significant. We note throughout the report when differences
between groups are significant.

The objectives, scopes, and methodologies of the specific products upon
which this report is based are detailed in those products, and, as
appropriate, those products contained comments from the relevant
agencies. That work was done over the last several years. Our work for
this report was conducted between September 1996 and April 1997 in
accordance with generally accepted government auditing standards. We
requested comments from the Director of OMB because of OMB’s
responsibilities for designating pilots, overseeing the pilot process, and
generally leading the executive branch’s Results Act efforts. On May 21




Page 38                                               GAO/GGD-97-109 Results Act
Chapter 1
Introduction




and 22, 1997, senior OMB officials provided technical comments. We have
incorporated OMB’s technical commments where appropriate.




Page 39                                           GAO/GGD-97-109 Results Act
Chapter 2

Progress of the Results Act Implementation
Has Been Mixed

                       Progress in implementing the Results Act has been mixed. The first
                       pilot—on performance planning and reporting—had far more participants
                       than required, although no participants were designated for the second
                       pilot on managerial accountability and flexibility, and the third pilot on
                       performance budgeting likely will be delayed by at least a year. Still, OMB
                       has taken a number of actions to prepare for the implementation of the
                       Act, such as expanding the amount and prominence of performance
                       information that is used during the executive branch budget process.
                       Congress has also taken actions to demonstrate its commitment to the Act
                       and reinforce to agencies the importance that it places on the full and
                       complete implementation of the Act.

                       Overall, the experiences of the pilot agencies and related efforts in
                       nonpilot agencies show that substantial improvements in performance are
                       possible—even in the short term—when organizations adopt a disciplined
                       approach to setting goals, measuring performance, and using performance
                       information to improve effectiveness. However, after 3 years of piloting
                       the Act, the reported examples of substantial performance improvements
                       still are fairly isolated, and many agencies do not appear to be
                       well-positioned to provide a results-oriented answer to the fundamental
                       Results Act question: What are we accomplishing?

                       Federal managers report that results-oriented performance measures often
                       do not exist. As a result, it is not surprising that managers also report that
                       results-oriented performance information frequently is not used to make
                       important decisions affecting their agencies and programs, even when
                       performance information is available.


                       The number of the performance planning and reporting pilots exceeded
The Number of          the Act requirements, thereby giving a large number of organizations a
Performance Planning   head start on the Results Act implementation and providing a rich body of
and Reporting Pilots   experience for nonpilot organizations to draw upon.1 For the first set of
                       pilot projects, the Act required OMB to select at least 10 agencies to pilot
Exceeded the Results   the performance planning and reporting requirements for 1 or more of an
Act Requirements       agency’s major functions and operations during fiscal years 1994 through
                       1996. However, a total of 77 pilot projects, including both entire
                       agencies—e.g., the Defense Logistics Agency (DLA), the Internal Revenue
                       Service (IRS), and the Social Security Administration (SSA)—and
                       programs—e.g., the Department of Veterans Affairs (VA) Loan Guaranty

                       1
                        During the pilot phase, GAO and others issued various assessments of pilot agencies. See, for
                       example, GAO/GGD-96-66R, February 14, 1996, and the federal case studies developed at the initiative
                       of ASPA on selected agencies’ strategic planning and performance measurement efforts.



                       Page 40                                                              GAO/GGD-97-109 Results Act
                           Chapter 2
                           Progress of the Results Act Implementation
                           Has Been Mixed




                           Program, participated at some point during this first pilot. When this part
                           of the pilot phase concluded at the end of fiscal year 1996, a total of 68
                           pilots representing 28 agencies were project participants.

                           The pilots represented a wide range of government activities and
                           functions. For example, the first set of pilot projects included military
                           operations, such as those conducted by the Air Force Air Combat
                           Command in the Department of Defense (DOD); regulatory programs, such
                           as those conducted by the Occupational Safety and Health Administration
                           (OSHA) in the Department of Labor; intergovernmental programs, such as
                           those conducted by the Office of Child Support Enforcement (OCSE) in the
                           Department of Health and Human Services (HHS); and businesslike
                           functions, such as those conducted by the U.S. Mint in the Department of
                           the Treasury.


                           The Results Act required OMB to select a second set of pilot projects during
Managerial                 fiscal years 1995 and 1996 in order to determine the effects of providing
Accountability and         managers of federal programs with increased managerial flexibility in
Flexibility Pilot Effort   exchange for the greater accountability for performance. The Act required
                           OMB to designate at least five agencies from the first set of pilot projects on
Did Not Work as            performance planning and reporting to participate in the second pilot on
Intended                   managerial accountability and flexibility. However, we found that the
                           managerial accountability and flexibility pilot did not work as intended.2
                           OMB did not designate as pilot projects any of the 7 departments and 1
                           independent agency that submitted a total of 61 waiver proposals.

                           We found that three major factors contributed to the failure of the
                           managerial accountability and flexibility pilot to work as intended. First,
                           changes in federal management practices and laws that occurred after the
                           Act was enacted affected agencies’ need for the managerial flexibility
                           waivers. For example, the Federal Workforce Restructuring Act of 1994
                           established a new personnel ceiling for all of the executive branch, which
                           effectively limited OMB’s ability to waive agency personnel ceilings
                           established in the budget. Second, agencies could use other, less rigorous,
                           means to obtain waivers from administrative requirements, such as the
                           administration’s NPR reinvention labs, which were designed to test ways
                           that agencies could improve their performance and customer service by
                           reengineering work processes and eliminating unnecessary regulations.
                           Third, unlike its proactive approach to the first set of pilots, OMB did not

                           2
                           GPRA: Managerial Accountability and Flexibility Pilot Did Not Work As Intended (GAO/GGD-97-36,
                           Apr. 10, 1997).



                           Page 41                                                           GAO/GGD-97-109 Results Act
                         Chapter 2
                         Progress of the Results Act Implementation
                         Has Been Mixed




                         work actively with agencies that were seeking to take part in the
                         managerial accountability and flexibility pilot. For example, OMB’s
                         feedback to agencies concerning their waiver proposals for consideration
                         as pilots was very limited. This contrasted directly with the approach that
                         OMB took with the performance planning and reporting pilots, for which it
                         issued a summary assessment of the 1994 performance plans, including
                         the strengths, weaknesses, and additional actions agencies needed to take
                         to improve these plans.


                         OMB  is also to select a third set of pilot projects to test performance
Performance              budgeting for fiscal years 1998 and 1999. Under the test, pilot agencies are
Budgeting Pilots         to prepare performance budgets, which are to provide Congress with
Likely to Be Delayed     information on the direct relationship between proposed program
                         spending and expected program results. The Senate committee report on
                         the Results Act said that the performance budgeting pilots are to begin
                         “only after agencies had sufficient experience in preparing strategic and
                         performance plans, and several years of collecting performance data.”3
                         Because of this and because it recognizes the importance of concentrating
                         on governmentwide Results Act implementation in fiscal year 1998, OMB
                         has indicated that these pilots will be delayed for at least a year. As
                         envisioned under the Act, performance budgets must show the effects on
                         performance of marginal changes in funding. According to OMB, few
                         agencies currently have either sufficient baseline performance or financial
                         information or the ability to use sophisticated analytic techniques to
                         calculate the effects on performance of marginal changes in funding.
                         According to OMB, delaying the performance budgeting pilots will provide
                         agencies time to develop needed information and abilities.


                         In the nearly 4 years since the passage of the Results Act, OMB has sought
OMB Actions to           to expand the amount and prominence of performance information within
Prepare the Executive    the executive branch budget process. For example, our 1995 review of
Branch for the Results   budget documents and interviews with OMB staff 1 year after OMB 2000—a
                         major April 1994 reorganization and process change in OMB—indicated that
Act Implementation       there was greater attention paid to agency management issues and the use
                         of performance information in the fiscal year 1996 budget process than
                         there had been in the fiscal year 1995 process.4 Whereas the fiscal year
                         1995 documents discussed streamlining primarily in terms of the number

                         3
                          S. Rep. No. 58, 103d Cong. 1st Sess. p. 38. (1993).
                         4
                          Office of Management and Budget: Changes Resulting From the OMB 2000 Reorganization
                         (GAO/GGD/AIMD-96-50, Dec. 29, 1995).



                         Page 42                                                          GAO/GGD-97-109 Results Act
Chapter 2
Progress of the Results Act Implementation
Has Been Mixed




of positions to be eliminated, the fiscal year 1996 documents also included
discussions about how proposed staff reductions could affect the agency’s
performance.

We reported that OMB continued to expand its focus on performance
measures and information during the fiscal year 1997 budget preparation
process. Between March and June 1995, OMB held a “Spring Review” as a
prelude for development of the fiscal year 1997 budget that was intended
to help OMB and agencies work together to identify useful performance
information. OMB conducted a “Summer Review” in July 1996 that
examined the status of agencies’ strategic planning efforts. Guidance from
OMB to executive agencies stated that the review would focus on the
adequacy, relevance, and appropriateness of their mission statements and
their strategic goals.

The 1996 “Fall Review,” according to OMB, was intended to produce
agreement between OMB and the agencies on the performance goals and
measures that agencies would include in their fiscal year 1999 annual
performance plans. OMB’s most recent effort, which was under way in the
spring of 1997, was its strategic assessment initiative. The purpose of this
initiative, which focused on agencies’ goals, commitments, and main
priorities, is to assist agencies in the preparation of their strategic plans
and annual performance plans that are due in September 1997.

To help ensure the success of the Results Act, the CFO Council, chaired by
OMB’s Deputy Director for Management, established a GPRA Implementation
Committee.5 The Committee provides guidance and information to CFOs
and managers in the 24 agencies covered by the CFO Act. The Committee,
concluding that uncertainty or fear of failure may immobilize an agency’s
efforts to implement the Act and that its implementation is evolutionary in
that proficiency comes with time and experience, has been actively
working to assist federal managers with the implementation of the Act. As
part of that effort, the Committee published a set of guiding principles and
key issues for implementing the Act.6 More recently, the Committee issued
a user guide to help federal managers meet the Results Act requirements
and, among other things, develop a framework for using performance



5
 The CFO Council was created by the CFO Act to provide the leadership foundation necessary to
effectively carry out the responsibilities of executive agency CFOs.
6
 Implementation of the Government Performance and Results Act (GPRA), A Report on the Chief
Financial Officer’s Role and Other Issues Critical to the Governmentwide Success of GPRA, Chief
Financial Officers Council, GPRA Implementation Committee, May 1995.



Page 43                                                             GAO/GGD-97-109 Results Act
                    Chapter 2
                    Progress of the Results Act Implementation
                    Has Been Mixed




                    information in making budget decisions.7 The Committee also has
                    undertaken a number of training efforts for federal managers, including
                    sponsoring a conference on the Act in the fall of 1996 and developing a
                    training package. The purpose of the training package, which was
                    provided to executive agencies, was to build awareness and support for
                    the Act among political appointees and senior career staff.


                    Congressional involvement is critical to sustain the momentum of the
Congressional       Results Act implementation and to reinforce to agencies the importance
Commitment to the   that Congress places on the successful and thorough implementation of
Results Act Is      the Act. We testified in March 1996 that officials in some pilot agencies
                    believed Congress’ interest in the Act was limited since, for example,
Reflected in Its    Congress seldom asked about the implementation of the Act in their
Increased           agencies.8 However, as the date for governmentwide implementation has
                    approached, Congress has signaled its strong commitment to the Act
Involvement and     through periodic hearings and other actions. Most prominently, on
Activities          February 25, 1997, the Speaker of the House, the House Majority Leader,
                    the Senate Majority Leader, and key committee chairmen from the House
                    and the Senate sent a letter to the Director of OMB. The letter underscored
                    the importance that the congressional Majority places on the
                    implementation of the Results Act, noted a willingness on the part of
                    Congress to work cooperatively with the administration, and established
                    expectations for consultations. Building on its commitment to the Act,
                    Congress has begun to consult with agencies on their strategic plans. In
                    the House, the Majority’s consultation effort is being led by teams
                    consisting of staff from various committees that are focusing on specific
                    agencies. These teams have been reaching out to agencies to review and
                    comment on agencies’ strategic plans.


                    The Results Act pilot phase has shown that short-term, substantial
Some Agencies       program improvements are possible when organizations adopt a
Reported Improved   disciplined approach to setting goals, measuring performance, and using
Performance         performance information to improve effectiveness, although the reported
                    examples of substantial performance improvements still are limited, as is
                    to be expected at this early stage of implementation. For example, as
                    highlighted in our Executive Guide, the Coast Guard reexamined its

                    7
                     Integrating Performance Measurement into the Budget Process: A User Guide to Multiple Ideas,
                    Practices, Formats, Processes, Steps, and One Model Tying Everything Together, Chief Financial
                    Officers Council, GPRA Implementation Committee, Jan. 21, 1997.
                    8
                     Managing for Results: Achieving GPRA’s Objectives Requires Strong Congressional Role
                    (GAO/T-GGD-96-79, Mar. 6, 1996).



                    Page 44                                                             GAO/GGD-97-109 Results Act
Chapter 2
Progress of the Results Act Implementation
Has Been Mixed




mission and shifted the focus of its marine safety program from regulation
to one that includes education.9 This change contributed to a significant
decline in the towing industry fatality rate—from 91 per 100,000 industry
employees in 1990 to 27 per 100,000 in 1995. Similarly, the Veterans Health
Administration improved services to veterans by more rigorously
assessing the results of the medical care it provides to the nation’s
veterans. In particular, the Veterans Health Administration reported that it
used performance information to target the most important improvement
opportunities and thereby lowered the mortality rate for cardiac
procedures by an average of 13 percent over the last 8 years.

We also have identified other examples of agencies that have reported
improvements to their operations. SSA has had a single, national toll-free
number as an alternative to field office contact for conducting simple
business since 1989. On the basis of customer feedback indicating that this
convenient telephone service was important to the public, yet had
persistently high busy rates, SSA took several steps to increase public
access, including doubling the trained and available workforce to answer
800-number calls. Primarily as a result of the workforce increase, the busy
rate decreased from 49 to 34 percent, and the percentage of calls that were
answered within 5 minutes increased from 74 to 83 percent from fiscal
years 1995 to 1996.10

DLA shows how an agency can make substantial improvements in its
service to its customers and inventory management. DLA’s reengineering
efforts for its delivery and supply processes focused not only on saving
time but also on lowering costs, two results valued by its customers. DLA
focused on improved partnerships between suppliers and its customers,
primarily DOD facilities. According to DLA, this overall focus enabled it to
reduce the delivery time of nearly all medical supplies from 30 days in
1993 to within the current 24 hours; at the same time, it reduced prices by
35 percent. DLA also reported that it is providing Internet ordering of
primarily electronics parts, which took from 35 to 40 days from purchase
to receipt prior to 1996 and now takes from 3 to 5 days, with an estimated
savings of $500 million over 10 years. DLA further reported that it avoided
inventory holding and storage costs of $95.7 million in fiscal year 1995,
with a projection of an additional $353 million over the next 5 years. DLA’s
initiatives have had departmentwide impact. We estimated in March 1997
that between September 1991 and September 1996, DOD reduced its

9
 GAO/GGD-96-118, June 1996.
10
 Social Security Administration: Significant Challenges Await New Commissioner (GAO/HEHS-97-53,
Feb. 20, 1997).



Page 45                                                           GAO/GGD-97-109 Results Act
                     Chapter 2
                     Progress of the Results Act Implementation
                     Has Been Mixed




                     pharmaceutical, medical, and surgical inventories and associated
                     management costs by about $714 million.11

                     The Food and Drug Administration (FDA) is allowed to collect user fees
                     from drug companies seeking approval to market drugs under the
                     Prescription Drug User Fee Act of 1992. The law dedicates the revenues to
                     expediting FDA’s reviews of human drug applications and incorporated
                     time-specific performance goals that FDA is to meet by the end of fiscal
                     year 1997. To satisfy these objectives, FDA consulted with its stakeholders
                     to determine appropriate performance measures and target levels of
                     performance and developed output-oriented performance goals. FDA
                     reported that the program had exceeded its performance goals, improving
                     the speed and efficiency of the drug review process.12 Specifically, the
                     median time taken to approve 53 applications for human drugs that have
                     not been marketed in the United States before was 14.3 months in fiscal
                     year 1996. In comparison, in 1993, which was the beginning of the
                     program, FDA approved only 25 applications, and the median approval time
                     was about 23 months.

                     The U.S. Mint, as one part of its mission, manufactures and markets
                     numismatic and commemorative coins and products. Before 1994, the
                     Mint estimated it took from 8 weeks to about 27 weeks to fill customers’
                     orders for coins. Because of the number of customer inquiries about the
                     status of their orders, the Mint learned that the time it took to fill orders
                     was a problem. Subsequently, the Mint set goals and reduced the time to
                     fill orders. As of January 1997, 97 percent of the orders were filled within 4
                     weeks. Also, the time for customers to get replacements has
                     declined—from 47 to 7 days—from October 1995 to October 1996.


                     Although agencies have reported selected instances where they have used
Managers Reported    results-oriented performance information to make important
Results-Oriented     improvements in performance, federal managers reported to us that
Performance          overall, results-oriented performance measures often were not available,
                     and such information was not frequently used to help make important
Measures and Their   decisions affecting their programs and agencies. According to our survey,
Use Was Limited      although 76 percent of federal managers governmentwide reported that
                     the programs, operations, or projects they were involved in had


                     11
                      Defense Inventory Management: Problems, Progress, and Additional Actions Needed
                     (GAO/T-NSIAD-97-109, Mar. 20, 1997).
                     12
                      Department of Health and Human Services: Management Challenges and Opportunities
                     (GAO/T-HEHS-97-98, Mar. 18, 1997).



                     Page 46                                                          GAO/GGD-97-109 Results Act
                           Chapter 2
                           Progress of the Results Act Implementation
                           Has Been Mixed




                           performance measures, such measures typically were not results oriented.
                           That is, the measures failed to help answer an essential question, “Is this
                           program accomplishing its intended results and contributing to the
                           achievement of the agency’s strategic goals and mission?”


Managers Reported          In crafting the Results Act, Congress recognized that congressional and
Limited Results-Oriented   agency decisionmaking often had been hampered by the absence of sound
Performance Measurement    information on the results of federal programs. In the past, to the extent
                           performance information was collected, agencies generally collected
                           information primarily on such things as the amount of dollars received and
                           spent, staffing levels, and the number of activities performed. Although
                           such information is valuable for program management and accountability,
                           it does not present a complete picture of the results of the federal effort.
                           Our more recent work has confirmed that results-oriented performance
                           information still does not exist as often as it should. For example, the
                           Department of Housing and Urban Development’s (HUD) Public Housing
                           Management Assessment Program has a system that collects data, such as
                           outstanding work orders and uncollected rents, to measure the
                           performance of housing authorities. However, the system does not collect
                           data on housing quality or the quality of maintenance, which are essential
                           for assessing the results that housing authorities are achieving and for
                           determining which housing authorities are performing well or poorly.13

                           Our survey of federal managers suggests that these are not isolated
                           occurrences of a lack of performance information in federal agencies.
                           Table 2.1 shows that, according to our survey, 32 percent of federal
                           managers said that, to a great or very great extent, they have the types of
                           performance measures that would demonstrate whether their programs or
                           operations were achieving their intended result. The table also shows that
                           38 percent or less of federal managers reported having, to a great or very
                           great extent, other important performance-related information, such as
                           efficiency and quality measures.




                           13
                            Public Housing: HUD Should Improve the Usefulness and Accuracy of its Management Assessment
                           Program (GAO/RCED-97-27, Jan. 29, 1997).



                           Page 47                                                         GAO/GGD-97-109 Results Act
                                         Chapter 2
                                         Progress of the Results Act Implementation
                                         Has Been Mixed




Table 2.1: Percent of Federal Managers
Who Reported Their Programs Have                                                          Percent of federal managers reporting to a
Various Performance Measures and                                                          great or very great extent they had or have
Recollect They Had Such Measures 3                                                                  these types of measures
Years Ago                                                                                    3 years
                                         Measures that                                           ago        Currently      Differencea
                                         would demonstrate to someone                             19               32              +13
                                         outside their agencies whether or not
                                         they are achieving their intended
                                         results
                                         tell whether or not they are satisfying                  11               32              +21
                                         their customers
                                         tell about the quality of the products or                19               31              +12
                                         services they provide
                                         tell how many things they produce or                     27               38              +11
                                         services they provide
                                         tell if they are operating efficiently                   17               26              +9
                                         a
                                         All differences are statistically significant.

                                         Source: GAO survey data.



                                         Despite the lower percentage of federal managers reporting that their
                                         programs or operations have performance-related measures to a great or
                                         very great extent, there where statistically significant increases in the
                                         percent of federal managers reporting the existence of these types of
                                         measures over what they recollected the situation to have been 3 years
                                         ago. For example, when asked to recollect what the situation was 3 years
                                         ago, 19 percent of federal managers reported that, to a great or very great
                                         extent, they had results-oriented measures, compared to 32 percent who
                                         reported that they had such measures today. This represents a
                                         13 percentage point change over what federal managers perceived the
                                         situation to have been 3 years ago, suggesting that results-oriented
                                         performance information, which is essential to the success of the Results
                                         Act, is becoming more widely available.

                                         According to federal managers’ responses to our survey, it appears that
                                         agencies are making the greatest change in developing measures for
                                         assessing customer satisfaction. When asked to recollect what the
                                         situation was 3 years ago, 11 percent of federal managers governmentwide
                                         reported that, to a great or very great extent, they had measures for
                                         assessing customer satisfaction. However, 32 percent of federal managers
                                         reported that, to a great or very great extent, they currently had such
                                         measures. This represents a 21 percentage point change. This reported




                                         Page 48                                                            GAO/GGD-97-109 Results Act
                                         Chapter 2
                                         Progress of the Results Act Implementation
                                         Has Been Mixed




                                         change in agencies’ efforts to measure customer satisfaction may be a
                                         reflection of agencies’ efforts to comply with Executive Order No. 12862,
                                         September 1993, which required agencies to establish and implement
                                         customer service standards to guide their operations.


Managers Reported                        Given the limited reported availability of results-oriented information, it is
Limited Use of                           not surprising that federal managers also reported limited use of such
Results-Oriented                         information to make decisions affecting their programs and agencies, as
                                         shown in table 2.2.
Performance Information
Table 2.2: Percent of Federal Managers
Reporting Extent to Which                                                             Percent of federal managers reporting use
Results-Oriented Performance                                                               to a great or very great extent
Information Is Used Currently and        Results-oriented performance                   3 years
Compared to 3 Years Ago                  information                                        ago       Currently       Difference
                                         used by the agency to help develop its              16              21              +5a
                                         budget
                                         used as the basis for funding decisions             14              20              +6a
                                         used as the basis for legislative changes            9              13              +4b
                                         used as the basis for program changes               12              16              +4a
                                         by management above respondent
                                         a
                                         Statistically significant.
                                         b
                                             Approaches statistical significance.

                                         Source: GAO survey data.



                                         Table 2.2 shows that federal managers reported some improvements over
                                         what they recollected the situation to have been 3 years ago. Although
                                         these improvements are statistically significant for most of the uses shown
                                         in the table, these improvements have been modest.

                                         The reported use of results-oriented performance information to help
                                         make key decisions was not high even among federal managers who
                                         reported that they had performance measures that demonstrate their
                                         programs are achieving intended results, to a great or very great extent. At
                                         most, 37 percent these federal managers reported that results-oriented
                                         performance information was used to a great or very great extent. Table
                                         2.3 shows these results.




                                         Page 49                                                      GAO/GGD-97-109 Results Act
                                         Chapter 2
                                         Progress of the Results Act Implementation
                                         Has Been Mixed




Table 2.3: Percent of Federal Managers
Reporting Use of Performance                                                                                           Percent of federal
Information to a Great or Very Great                                                                           managers reporting use to
Extent Among Managers With                                                                                          a great or very great
Performance Measures Demonstrating       Results-oriented performance information                                                 extenta
Achievement of Results to a Great or     used by the agency to help develop its budget                                                37
Very Great Extent                        used as the basis for funding decisions                                                      34
                                         used as the basis for legislative changes                                                    21
                                         used as the basis for program changes by management                                          31
                                         above respondent
                                         a
                                          All percentages of use are statistically significant and higher compared to respective
                                         percentages of use for all other managers.

                                         Source: GAO survey data.




                                         The Results Act performance planning and reporting pilot phase provided
Conclusions                              many agencies with important experience in meeting key requirements of
                                         the Act. These experiences, along with OMB’s efforts to work with agencies
                                         on the Act, suggest that most agencies are likely to meet the upcoming
                                         statutory deadlines for producing initial strategic plans and annual
                                         performance plans. More importantly, the experiences of the pilot
                                         agencies and related efforts by nonpilot agencies show that substantial
                                         improvements in performance are possible when organizations adopt a
                                         disciplined approach to setting goals, measuring performance, and using
                                         performance information to improve effectiveness.

                                         However, the pilot phase also underscored how far organizations still have
                                         to progress in the development and use of results-oriented performance
                                         information. Although there has been progress over the last 3 years,
                                         agencies still have not developed the information necessary to determine
                                         whether their programs are accomplishing their intended results. Even in
                                         those instances where agencies have results-oriented information,
                                         agencies are generally not using the information to a great extent to make
                                         decisions affecting their programs. The subsequent chapters will detail
                                         that agencies face a number of formidable challenges in establishing clear
                                         missions and results-oriented goals, measuring their performance, building
                                         the organizational cultures needed to sustain a focus on results, and
                                         linking performance plans to the budget process.




                                         Page 50                                                               GAO/GGD-97-109 Results Act
Chapter 3

Unclear Agency Missions and Goals Will
Challenge Results Act Implementation

                        Reaching a reasonable degree of consensus on coordinating the missions
                        and goals of individual agencies that are involved in crosscutting program
                        areas and balancing individual agencies’ multiple strategic goals can raise
                        contentious policy issues that may not be easily resolved. The Results Act,
                        with its focus on results and requirement for consultation, can help
                        Congress and the executive branch clarify over the long term these
                        management issues confronting federal agencies.

                        Our work has shown that Congress’ and the executive branch’s approach
                        to addressing national issues has, in many cases, resulted in overlap and
                        fragmentation in federal program efforts. Coordinating these program
                        efforts will take time and may, in some instances, require legislative
                        action. Moreover, many agencies confront competing demands that are a
                        natural by-product of the complex social and political environment in
                        which programs are created, funded, and managed. Balancing these
                        demands will continue to be an ongoing process for federal agencies.

                        Efforts under the Results Act will not answer questions about whether the
                        crosscutting program efforts should be consolidated or reorganized or
                        about the best balance for competing priorities. Those are policy decisions
                        that should appropriately consider many factors, including program
                        performance. However, congressional consultations on agencies’ strategic
                        plans provide an ongoing opportunity for Congress and the executive
                        branch to work together to identify the extent and potential consequences
                        of overlap and fragmentation in federal program efforts and to identify the
                        performance trade-offs associated with striking different balances of
                        competing demands.


                        Although federal programs have been designed for different purposes or
Crosscutting Federal    targeted for different population groups, coordination among federal
Programs Pose           programs with related responsibilities is essential to efficiently and
Challenge for Results   effectively meet national concerns. Uncoordinated program efforts can
                        waste scarce funds, confuse and frustrate program customers, and limit
Act Implementation      the overall effectiveness of the federal effort. A focus on results, as
                        envisioned by the Results Act, implies that federal programs contributing
                        to the same or similar results should be closely coordinated to ensure that
                        goals are consistent, and, as appropriate, program efforts are mutually
                        reinforcing. This means that federal agencies are to look beyond their
                        organizational boundaries and coordinate with other agencies to ensure
                        that their efforts are aligned. Although some initiatives are under way in




                        Page 51                                             GAO/GGD-97-109 Results Act
Chapter 3
Unclear Agency Missions and Goals Will
Challenge Results Act Implementation




this regard, fully coordinating federal programs will take time and require
sustained attention.

Our work has shown that as the federal government has responded over
time to new needs and problems, many federal agencies have been given
responsibilities for addressing the same or similar national issues.1 Of the
18 federal missions or areas of national need described as budget
functions, 14 were addressed by more than 1 executive branch department
or major agency in fiscal year 1996, such as health, international affairs,
and justice. Six of the national missions, including education, income
security, and commerce and housing credit were addressed by six or more
executive branch departments and major agencies. Some of this shared
responsibility was intended to recognize that addressing some issues from
a national perspective would necessarily involve more than one federal
agency or more than one approach. For example, we have reported that
multiagency initiatives to address environmental concerns recognize that
many programs can have an impact on the environment.2

However, in many program areas—such as in food safety, employment
training, early childhood development, at-risk and delinquent youth
programs, federal land management, and national laboratories—our work
suggested that significant overlap and fragmentation existed in the federal
response to national needs and problems.3 For example, in 1995 we
testified on the Department of Education’s programs that provided loans
and grants to students to help finance their higher education.4 We found
that although the student loan and Pell grant programs provided the
majority of federal financial aid to students for postsecondary education,
another 22 smaller programs were targeted to specific segments of the
postsecondary school population, such as prospective students from
disadvantaged families or women and minorities who are
underrepresented in graduate education. These 22 programs were

1
 Budget Issues: Fiscal Year 1996 Agency Spending by Budget Function (GAO/AIMD-97-95, May 13,
1997); and Government Restructuring: Identifying Potential Duplication in Federal Missions and
Approaches (GAO/T-AIMD-95-161, June 7, 1995).
2
 Restoring the Everglades: Public Participation in Federal Efforts (GAO/RCED-96-5, Oct. 24, 1995); and
Ecosystem Management: Additional Actions Needed to Adequately Test a Promising Approach
(GAO/RCED-94-111, Aug. 16, 1994).
3
 Government Reorganization: Issues and Principles (GAO/T-GGD-AIMD-95-166, May 17, 1995); At-Risk
and Delinquent Youth: Multiple Federal Programs Raise Efficiency Questions (GAO/HEHS-96-34, Mar.
6, 1996); Federal Land Management: Streamlining and Reorganization Issues (GAO/T-RCED-96-209,
June 27, 1996); and Federal R&D Laboratories (GAO/RCED/NSIAD-96-78R, Feb. 29, 1996).
4
 Department of Education: Information on Consolidation Opportunities and Student Aid
(GAO/T-HEHS-95-130, Apr. 6, 1995); and Department of Education: Opportunities to Realize Savings
(GAO/T-HEHS-95-56, Jan. 18, 1995).



Page 52                                                               GAO/GGD-97-109 Results Act
                           Chapter 3
                           Unclear Agency Missions and Goals Will
                           Challenge Results Act Implementation




                           collectively funded at $1.1 billion for fiscal year 1995. We concluded that
                           these smaller grant programs may be considered candidates for
                           consolidation—with other larger programs or among themselves—with no
                           adverse impact on students’ access to postsecondary education and that
                           the federal government could anticipate administrative savings of 10
                           percent each year, or a total of $550 million in budget authority (adjusted
                           for inflation) over 5 years.5

                           Similarly, we reported in 1995 on the federal programs that assisted urban
                           communities and their residents through a complex system involving
                           multiple federal agencies.6 At that time, the Catalog of Federal Domestic
                           Assistance listed 342 economic development-related programs
                           administered by 13 agencies.7 These programs were in the areas of
                           housing, economic development, and social services. We reported that
                           considered individually, many of these programs may have made sense.
                           However, the proliferation of federal programs and the lack of
                           coordination among agencies could impose a burden on local
                           organizations that attempted to piece together programs to serve their
                           communities. In particular, the neighborhood organizations we studied
                           found it burdensome to manage multiple programs with individual funding
                           streams, application requirements, and reporting expectations.


Efforts Are Under Way to   In recent years, a number of efforts have been undertaken to coordinate
Coordinate Crosscutting    federal programs to help ensure that national needs are being effectively
Programs                   targeted. These efforts—some of which were started before the Results
                           Act—have shown that coordinating crosscutting programs takes time and
                           requires sustained attention. Moreover, because of the statutory bases of
                           crosscutting programs, congressional involvement is often needed to
                           integrate the federal response to national needs.

                           Within the executive branch, federal agencies have initiated several efforts
                           to address crosscutting issues. As of March 1997, about 25 interagency
                           groups had been formed to discuss common concerns in crosscutting
                           issues, including the Federal Credit Policy Working Group, the Research

                           5
                            Addressing The Deficit: Budgetary Implications of Selected GAO Work for Fiscal Year 1998
                           (GAO/OCG-97-2, Mar. 14, 1997).
                           6
                            Economic Development Programs (GAO/RCED-95-251R, July 28, 1995); and Community
                           Development: Comprehensive Approaches and Local Flexibility Issues (GAO/T-RCED-96-53, Dec. 5,
                           1995).
                           7
                             The Catalog of Federal Domestic Assistance is a governmentwide compendium of federal programs,
                           projects, services, and activities coordinated by OMB and compiled by the General Services
                           Administration.



                           Page 53                                                             GAO/GGD-97-109 Results Act
Chapter 3
Unclear Agency Missions and Goals Will
Challenge Results Act Implementation




Roundtable, and the Interagency Regulatory Reinvention Forum, among
others.8 Typically, these groups were sponsored by a lead agency and
provided a forum for agency officials to discuss informally a wide range of
concerns, such as goal-setting and performance measures, but not full
coordination of program efforts. The Federal Credit Policy Working Group
worked on appropriate goals and performance measures for credit
programs. The Research Roundtable produced a paper on the challenges
of developing and reporting performance measures for research programs.
The Interagency Regulatory Forum discussed common concerns among
regulatory agencies, including identifying customers and conducting
customer service surveys. Although these agency forums have provided an
important opportunity for agencies to work together to address common
concerns in goal-setting and performance measurement, they have not
generally attempted to coordinate crosscutting program efforts. Thus,
despite these initiatives, during its review of major portions of agencies’
strategic plans in the summer of 1996, OMB concluded that on the whole,
there was little sign of significant coordination among agencies.

Addressing the overlap and fragmentation in federal program efforts often
requires congressional involvement. In particular, legislative action can
ensure that the goals and measures of existing programs are coordinated
to meet federal needs. For example, Congress incorporated Healthy
People objectives for the year 2000 into national legislation. Started in
1979, Healthy People is a series of outcome-based public health objectives
and measures developed and updated each decade by the U.S. Public
Health Service in consultation with other federal agencies, state
governments, and national organizations. The year 2000 objectives
succeeded the 1990 health objectives that were set in 1980. The Public
Health Service has started planning for the 2010 health objectives. Over
time, Congress has required three federal programs to incorporate Healthy
People objectives for the year 2000 into their goals; and in one of those
programs, it has linked resource allocations to Healthy People objectives.
The first, in 1989, was the requirement that the Maternal and Child Health
Block Grant Program, Title V of the Social Security Act, report on the
progress made by states toward accomplishing Healthy People objectives
on maternal and child health. The Indian Health Care Improvement Act, as
amended in 1992, created an annual reporting requirement for the Indian
Health Service to inform Congress on the health status of American
Indians and Alaska Natives. In addition, the 1992 authorization of the



8
Information provided by Carl J. Metzger, Management Systems International, Inc., Washington, D.C.,
March 1997.



Page 54                                                             GAO/GGD-97-109 Results Act
                     Chapter 3
                     Unclear Agency Missions and Goals Will
                     Challenge Results Act Implementation




                     Preventive Health and Health Services Block Grant program linked the
                     state grant-funded activities to Healthy People objectives.

                     However, the mere existence of a legislated coordinating mechanism does
                     not ensure success, and sustained effort is required to build consensus on
                     common goals and how best to coordinate approaches. For example, the
                     Office of National Drug Control Policy (ONDCP), established in 1988 by law,
                     is responsible for producing the national drug control strategy and
                     coordinating its implementation with other federal agencies. Given the
                     complexity of drug control efforts and the fragmentation of the approach
                     to the national drug control strategy among more than 50 federal agencies,
                     we have endorsed the need for a central planning agency, such as ONDCP,
                     to coordinate the nation’s efforts and recommended its reauthorization for
                     a finite period of time.9 However, our work has shown that despite some
                     successes, international drug control efforts have not materially reduced
                     the availability of drugs in the United States.10 One reason was a lack of
                     ways to tell whether or how well counternarcotics efforts were
                     contributing to the goals of the national strategy. Consistent with the
                     intent of the Results Act, we recommended that ONDCP complete a
                     long-term plan with meaningful performance measures and multiyear
                     funding needs that were linked to the goals and objectives of the
                     international drug control strategy. Such a plan would provide managers
                     and policymakers with the information to make more informed decisions
                     on prioritizing funding levels based on performance and results. ONDCP
                     concurred with our recommendation. In February 1997, ONDCP proposed a
                     10-year strategy with five major goals—including “break foreign and
                     domestic drug sources of supply”—as the basis for a long-term national
                     effort and stated that it will continue to work on a 5-year drug control
                     budget concept. ONDCP is making progress toward developing performance
                     targets and measures for each of its goals.


                     Federal agencies often face a variety of competing priorities that force
Balancing Goals Is   congressional and executive branch decisionmakers to balance public
Complicated by       expectations, cost, quality, and other factors. These multiple priorities,
Competing Demands    which in many cases are built into the intent and design of an agency or
                     program, constitute one of the primary challenges to public sector


                     9
                      Drug Control: Reauthorization of the Office of National Drug Control Policy (GAO/T-GGD-97-97, May
                     1, 1997); and Drug Control: Observations on Elements of the Federal Drug Control Strategy
                     (GAO/GGD-97-42, Mar. 14, 1997).
                     10
                      Drug Control: Long-Standing Problems Hinder U.S. International Efforts (GAO/NSIAD-97-75, Feb. 27,
                     1997).



                     Page 55                                                             GAO/GGD-97-109 Results Act
Chapter 3
Unclear Agency Missions and Goals Will
Challenge Results Act Implementation




goal-setting because priorities continually need to be balanced; otherwise,
one or two priorities may be inappropriately overemphasized at the
expense of others.

According to our survey, nearly half of the federal managers reported that
reconciling differing congressional views and the views of other parties on
an agency’s mission and strategic goals has been, or will be, somewhat to
very difficult. A greater proportion of federal managers from the Results
Act pilots that we were able to isolate from our survey reported difficulty,
suggesting that as agencies gain experience with the Results Act they will
come to better appreciate the need for, and difficulty of, reconciling
competing views. Fifty-eight percent of managers in selected Act pilots
reported that reconciling differing congressional views has been or will be
somewhat to very difficult, compared to 46 percent of federal managers in
all other agencies. Similarly, 59 percent of managers in selected Results
Act pilots reported that reconciling the views of other parties has been or
will be somewhat to very difficult, compared to 48 percent of federal
managers in all other agencies. The differences between the managers in
selected Results Act pilots and federal managers in all other agencies for
reconciling differing congressional views and the views of other parties
are statistically significant.

We have identified numerous examples of agencies that can benefit from
using the Results Act to help them better balance competing priorities and
thereby improve their effectiveness. For example, we reported in
April 1997 that the Forest Service had been increasingly shifting the
emphasis of its efforts from producing timber to sustaining wildlife.11 This
shift was taking place in reaction to requirements in planning and to
environmental laws and their judicial interpretation—reflecting changing
public values and concerns—together with social, ecological, and other
factors. However, we noted the demand for recreation is also expected to
grow and may increasingly conflict with sustaining wildlife and producing
timber. We found that the disagreement both within the Forest Service and
among key external stakeholders, including Congress, on how the Forest
Service is to resolve conflicts or make choices among competing uses on
its lands had seriously undermined its ability to establish the goals and
performance measures needed to ensure its accountability. Until general
agreement is reached, we believe the Forest Service’s decisionmaking will
continue to be inefficient and ineffective.



11
 Forest Service Decision-Making: A Framework for Improving Performance (GAO/RCED-97-71,
Apr. 29, 1997).



Page 56                                                         GAO/GGD-97-109 Results Act
                        Chapter 3
                        Unclear Agency Missions and Goals Will
                        Challenge Results Act Implementation




                        Our ongoing work examining federal credit agencies identified multiple
                        competing priorities as a challenge in setting results-oriented goals for
                        those programs. Agency officials told us that on the one hand, they are
                        expected to increase program service while also reducing program costs
                        and minimizing default rates. However, these programs were designed to
                        offer credit to a population that the private sector would consider
                        high-risk. For example, to be eligible for a rural housing loan, the borrower
                        must be ineligible to get credit from any other source. Therefore, the target
                        population may be more likely to default on a loan.

                        The need to identify and balance competing demands can be especially
                        complicated in cases where an agency lacks an integrated legislative
                        mission. For example, the Environmental Protection Agency (EPA) was
                        established in 1970 under a presidential reorganization plan in response to
                        concerns over protecting public health as well as air, water, and land
                        resources. Since then, as environmental threats were identified, Congress
                        gave EPA responsibility for implementing over a dozen environmental laws.
                        We reported in May 1995 that because it did not have an overarching
                        legislative mission and its environmental responsibilities had not been
                        integrated, EPA had not been able to target its resources to reduce the
                        greatest risks to human health and the environment.12 EPA has been
                        working with state governments, industry, environmental groups, and
                        other interested parties since 1992 to select the most appropriate national
                        environmental goals. Building on these goals, EPA is also ranking the
                        relative risk of environmental problems to help identify the most
                        appropriate goals for the agency.


                        The Results Act requirement that agencies consult with Congress in
Congressional           developing their strategic plans presents an important opportunity for
Consultations Provide   congressional committees and the executive branch to work together to
Mechanism for           identify the extent and the potential consequences of overlap and
                        fragmentation of federal program efforts.13 As agencies work with
Identifying and         congressional committees on developing, and subsequently updating, their
Addressing              strategic plans, they can identify legislative changes that may be needed to
                        clarify congressional intent as well as legislative adjustments to better
Crosscutting Issues     ensure an effective, coordinated response to national issues. The
and Seeking to          consultation process should also be helpful to agencies in addressing key
Balance Priorities      management issues, such as identifying multiple priorities, reaching a

                        12
                         Environmental Protection: Current Environmental Challenges Require New Approaches
                        (GAO/T-RCED-95-190, May 17, 1995).
                        13
                          GAO/T-GGD-96-79, March 6, 1996.



                        Page 57                                                         GAO/GGD-97-109 Results Act
Chapter 3
Unclear Agency Missions and Goals Will
Challenge Results Act Implementation




workable agreement on priorities, and then establishing results-oriented
goals to reflect the balance struck among those priorities. The requirement
for consultations is also perhaps the Results Act’s most significant
challenge, because consultations will likely raise contentious policy issues
that are inherent in the political process.

We testified in February 1997 that although no single set of best practices
has yet emerged for the Results Act consultation process, making
consultations most useful would require that both Congress and agencies
be actively engaged in the effort.14 On the whole, the House committee
staff we spoke with stressed the very limited nature of the meetings with
agencies that had taken place through February 1997. Rather than
consultations, House staff characterized the meetings as briefings,
preconsultations, or preliminary consultations. As of April 30, 1997,
however, some House Majority committee staff we spoke with had
expressed concern that in their view, not enough agencies had provided
draft strategic plans for consultations. In particular, a number of agencies
have told congressional staff that draft plans will not be available until
June or early July. Congressional staff have said that an important
opportunity for Congress and the administration to clarify and seek
agreement on agencies’ missions, goals, and strategies will be lost if draft
plans are not provided early enough to permit meaningful reviews and
consultations during the consultation process. For example, although EPA
provided a detailed outline of its strategic plan, including a mission
statement, goals, and strategic “principles,” completing a draft plan in
early July may not leave adequate time for EPA to obtain congressional
input and fully consider and incorporate congressional views before the
final plans are due in September 1997.

Although the Results Act’s required consultation can help improve the
management of federal agencies, it will have a less direct role in resolving
difficult policy choices, because such choices should appropriately
consider many factors, including program performance. For example, we
have observed that transferring weapons production from the Department
of Energy to DOD, as was proposed by some, would require careful
consideration of many policy and management issues.15 In a survey of
former Department of Energy executives and experts on energy policy,
some argued that because of the declining strategic role of nuclear

14
 Managing For Results: Enhancing the Usefulness of GPRA Consultations Between the Executive
Branch and Congress (GAO/T-GGD-97-56, Mar. 10, 1997).
15
 Department of Energy: Observations on the Future of the Department (GAO/T-RCED-96-224, Sept. 4,
1996).



Page 58                                                           GAO/GGD-97-109 Results Act
              Chapter 3
              Unclear Agency Missions and Goals Will
              Challenge Results Act Implementation




              weapons, DOD might be better able to balance resource allocations among
              nuclear and other types of weapons if the weapons complex were
              completely under its control. Others argued, however, that the need to
              maintain civilian control over nuclear weapons outweighs any other
              advantages.

              The Results Act can assist decisionmakers in making policy choices by
              providing data on the management and performance implications of
              different options. For example, data generated as part of the Results Act
              can identify potential performance improvements that might result from
              consolidating programs or shifting responsibilities among agencies.
              Congress and the executive branch can then weigh this information
              against policy considerations. Our work suggests that ultimately a
              successful strategic planning effort will often reflect hard choices by the
              agency, and there may be disagreement between the agency and key
              stakeholders about particular goals or the means by which the goals will
              be achieved. In this regard, an OMB official has noted that a strategic plan
              that has achieved complete agreement among all interested parties is
              likely to be at such a high level of generality that the usefulness of the plan
              as a decisionmaking tool would be fairly limited.


              The Results Act, with its focus on results and requirement for
Conclusions   consultation, can help Congress and the executive branch address the
              management implications of two of the seemingly intractable issues
              confronting federal agencies: overlap and fragmentation in program efforts
              that cut across agencies and the need to balance multiple priorities within
              individual agencies. However, although the Results Act provides a
              potentially effective vehicle for addressing these issues, their existence
              also makes the prospects for the effective governmentwide
              implementation of the Act uncertain in the near term. Efforts under the
              Results Act can help identify the extent and consequences of the overlap
              and fragmentation, but addressing such problems will likely be a lengthy
              process, because Congress and the executive branch will have to consider
              policy trade-offs to reach a reasonable degree of consensus on the
              appropriate federal response to national needs. Similarly, balancing
              multiple priorities will need to be an ongoing process, because priorities
              change as new needs arise.

              The strategic planning consultation process is a starting point for Congress
              and the executive branch to work through these different policy issues. If
              successfully implemented, the Results Act will be most helpful to



              Page 59                                               GAO/GGD-97-109 Results Act
Chapter 3
Unclear Agency Missions and Goals Will
Challenge Results Act Implementation




decisionmakers in providing performance information that identifies the
management implications of different structures and policy choices among
competing demands.




Page 60                                          GAO/GGD-97-109 Results Act
Chapter 4

Developing Useful Results-Oriented
Performance Information Will Be an
Ongoing Difficulty
                             Agencies have encountered some difficult analytic and technical
                             challenges in developing useful results-oriented performance information
                             to use in planning their efforts and gauging progress toward the
                             achievement of their goals and missions. There are two primary
                             considerations for agencies in their development of such information. The
                             first is to identify those performance measures that will be meaningful for
                             the agency to use in gauging progress in achieving its goals. The other is to
                             collect reliable and valid data on those measures so that program
                             managers and other decisionmakers have the data they need to effectively
                             manage federal programs.

                             However, several factors—including a lack of data on results-oriented
                             performance measures; the influence of external forces, such as emerging
                             economic, social, and technological trends and the role that third parties,
                             such as state and local governments, may have in determining whether
                             program results are achieved; and the long time frames sometimes needed
                             before the results of agency or program actions can be seen—have
                             complicated agencies’ ability to set goals and identify results-oriented
                             performance measures. Agencies also are challenged in their collection of
                             valid and reliable data. Reasons for this challenge include different parties
                             using different definitions to measure performance and agencies’ reliance
                             on outside parties to provide results-oriented performance data. Some
                             agencies are exploring approaches for addressing the difficulty they have
                             had in developing useful results-oriented performance information.


                             Several factors have complicated the degree to which agencies have faced
Several Factors Have         challenges in setting goals and identifying results-oriented performance
Complicated                  measures. These factors include a lack of data on results-oriented
Agencies’ Ability to         performance measures; the influence of external forces, such as emerging
                             economic, social, and technological trends and the role that third parties,
Set Goals and Identify       such as state and local governments, may have in determining whether
Results-Oriented             program results are achieved; and the long time frames sometimes needed
                             before the results of agency or program actions can be seen.
Performance
Measures
Agencies’ Lack of Data       Our work has shown that baseline and trend data on past performance can
Complicates Efforts to Set   help agencies set realistic goals for their programs given the past
Goals                        performance of those programs. We have previously reported that prior
                             efforts to link resources to results were hampered by the absence of




                             Page 61                                              GAO/GGD-97-109 Results Act
Chapter 4
Developing Useful Results-Oriented
Performance Information Will Be an
Ongoing Difficulty




systems to collect needed data.1 Under PPBS, agencies that attempted to
gather results-oriented performance data found the process to be far more
difficult than expected. Agency officials reported that developing the
information and collection systems envisioned by PPBS would take several
years.

Because agencies often did not focus on having results-oriented
performance information in the past, they generally have not collected
such data. Therefore, they do not have all of the baseline and trend data
needed to set goals. For example, our reviews of the National Park
Service—whose mission is, in part, to preserve and protect park
resources—showed that baseline information about natural and cultural
resources was frequently incomplete or nonexistent.2 Consequently, it was
difficult for Park Service officials to determine whether the best
management decisions about those resources were being made. For
example, for 70 years, the Service has stocked nonnative fish in various
lakes and waterways in Yosemite National Park. However, officials said
they knew little about the types or numbers of fish in the lakes and
waterways, as well as other species inhabiting the park, including birds,
badgers, river otters, wolverines, and red foxes. By not collecting data on
those resources, officials realized that some of their decisions about those
resources were not effective in helping the Park Service achieve intended
results. At the time of our review, nonnative fish outnumbered native
rainbow trout by a 4-to-1 margin, and the stocking had reduced the
numbers of at least one federally protected species.

VA officials said that some of the results-oriented measures for their
agency’s Loan Guaranty program were new and baseline data were not
available on those measures. Consequently, VA did not have data on past
performance to use in setting some of the program’s fiscal year 1998 goals.
In some of these cases, VA indicated in its fiscal year 1998 budget
submission to OMB that those goals were “to be determined.” For example,
according to VA, a key result of the program is providing veterans with
timely service in obtaining a VA guaranteed loan. Performance measures
include the percentage of veterans satisfied with the time it took to
process their loans and the average time taken to process loans. Although
VA set fiscal year 1998 goals for customer satisfaction with timeliness, VA




1
 GAO/AIMD-97-46, March 27, 1997.
2
 National Parks: Park Service Needs Better Information to Preserve and Protect Resources
(GAO/T-RCED-97-76, Feb. 27, 1997); and Park Service: Managing for Results Could Strengthen
Accountability (GAO/RCED-97-125, Apr. 10, 1997).



Page 62                                                            GAO/GGD-97-109 Results Act
                           Chapter 4
                           Developing Useful Results-Oriented
                           Performance Information Will Be an
                           Ongoing Difficulty




                           reported that goals for average processing time would need to be
                           determined.

                           In another example, U.S. Department of Agriculture (USDA) officials said
                           they dropped some performance measures that had been part of their
                           Single Family Housing Results Act pilot’s annual performance plan
                           because they did not have information systems to collect data on those
                           measures.3 Such measures included those that would have provided
                           performance data on the number of loans made in targeted geographic
                           areas and the number of houses upgraded from substandard to safe and
                           sanitary in targeted areas. Lacking these data, USDA did not have an
                           informed basis on which to set goals.

                           In recent years, Congress has enacted landmark information technology
                           legislation—the Paperwork Reduction Act of 1995 and the Clinger-Cohen
                           Act of 1996—that, if successfully implemented, holds a key to ensuring
                           that agencies put in place the systems needed to collect results-oriented
                           performance data. These acts establish an investment framework for
                           having agencies better plan and manage their technology efforts and link
                           those efforts directly to the achievement of agency program goals and
                           mission.4


Agencies’ Results Can Be   The efforts of federal agencies often are but one factor among many
Influenced by External     external forces that may influence whether, and the degree to which, their
Forces                     programs achieve their intended results. Many agencies have been
                           challenged to separate out the influence that program activities have had
                           on the achievement of program results when those results also could have
                           been influenced by external forces. This challenge has complicated
                           agencies’ efforts to identify those performance measures that will be
                           meaningful for the agency to use in gauging progress in achieving its goals.

                           Identifying meaningful performance measures has been a long-standing
                           problem for federal agencies. We have reported that agency officials
                           implementing the PPBS initiative, which was mandated governmentwide by
                           President Johnson in 1965, found it far more difficult than they expected

                           3
                            Information systems are a discrete set of information resources and processes, automated or manual,
                           organized for the collection, processing, maintenance, use, sharing, or dissemination of information.
                           See GAO/AIMD-94-115, May 1994.
                           4
                            For further information see: Information Technology Investment: Agencies Can Improve
                           Performance, Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, Sept. 30, 1996); Information
                           Management Reform: Effective Implementation Is Essential for Improving Federal Performance
                           (GAO/T-AIMD-96-132, July 17, 1996); and GAO/AIMD-94-115, May 1994.



                           Page 63                                                              GAO/GGD-97-109 Results Act
Chapter 4
Developing Useful Results-Oriented
Performance Information Will Be an
Ongoing Difficulty




to relate program activities to a stated result or to separate out other
influences that might affect intended results.5 For example, under the PPBS
initiative, the Upward Bound program was designed to increase skills and
motivation for low-income high school students. Agency officials had no
way to isolate the program’s effect from other environmental influences
that might also have contributed to the success or failure of program
participants.

The influence that external forces can have on the achievement of
program results has continued to challenge agencies’ ability to identify
meaningful measures. Our work looking at the efforts of economic
development and our forthcoming report on regulatory programs
illustrates this challenge. For each of these types of programs, isolating the
federal contribution to the achievement of an intended result has been
exceedingly difficult and accordingly has hindered agency efforts to
identify meaningful performance measures. In situations where the federal
program effort is but one factor among many external forces that may
determine the degree to which an intended result is achieved, determining
the level of federal influence on such achievement requires agencies to
understand and measure the nonfederal influence.

For example, in 1995 we reported on the existence of 342 economic
development-related programs.6 Numerous external forces, including
broad national economic trends, contribute to local economic
development, and communities also may receive assistance from state and
local governments and the private sector. Therefore, determining the
impact of one of these programs on economic development has been a
daunting task. A persuasive study of a program’s impact, as we observed in
our 1996 report,7 would require three elements. First, it would have to
document that there had been some improvement in a targeted area;
second, it would have to link specific program elements to actual
economic changes; and third, it would have to measure the growth
stemming from other influences on the economy of the targeted area in
order to isolate the impact that could be attributed to the economic
development program.

Although such analysis entails a substantial undertaking, it can potentially
provide important information for decisionmakers. As part of that 1996

5
 GAO/AIMD-97-46, March 27, 1997.
6
 GAO/RCED-95-251R, July 28, 1995.
7
Economic Development: Limited Information Exists on the Impact of Assistance Provided by Three
Agencies (GAO/RCED-96-103, Apr. 3, 1996).



Page 64                                                           GAO/GGD-97-109 Results Act
Chapter 4
Developing Useful Results-Oriented
Performance Information Will Be an
Ongoing Difficulty




review, we reported that one study of the effectiveness of the Department
of Commerce’s Economic Development Administration (EDA) programs
found that income in the counties that received EDA funding grew
significantly faster than income in the counties that received no aid.
However, when the researchers simultaneously considered EDA’s programs
and factors unrelated to EDA, they found that EDA’s programs had a very
small effect on income growth rates during the period that the aid was
received and had no significant effects in the 3 years after the aid ceased.
EDA’s programs could explain only a small part of the difference in the
growth rates between the two groups of counties.

Regulatory agencies also have had difficulty sorting out the interaction
that external forces have had on the results that those agencies were
trying to achieve and accordingly have had difficulty identifying
meaningful performance measures. For example, OSHA officials said they
were having difficulty identifying measures to assess program results
because many other factors, such as business cycles and the development
of safer machines, affect the number of workplace injuries each year more
than OSHA’s actions do. In addition, a senior OSHA official said that worker
safety data are sometimes influenced by catastrophic events, such as
bombings or airplane crashes over which OSHA has no control.

Conversely, the less that program results are affected by external forces,
the greater will be the agency’s influence in accomplishing its intended
results. Such agencies also will tend to have an easier time identifying
results-oriented performance measures that demonstrate the extent to
which the agency achieved its goals. For example, the intended goals of
VA’s National Cemetery System (NCS) include burying eligible veterans and
their family members in national cemeteries and maintaining the graves
and their surroundings as national shrines. Because NCS exerts a relatively
high degree of influence over program results, identifying results-oriented
performance measures is a fairly straightforward effort. Its fiscal year 1996
results-oriented performance measures included the percentage of
veterans who chose a burial option provided by NCS and/or who chose a VA
headstone or marker. Because of NCS’ high degree of influence over
program results, it also can more readily pinpoint opportunities for
managing its efforts and improving its performance. To help it do so, NCS
(1) conducted 14 focus groups at 5 locations throughout the country with
specific customer groups who had recent direct contact with the cemetery
and (2) placed 20,000 visitor comment cards at 76 national cemeteries for
visitors to comment on their satisfaction with various aspects of cemetery
appearance and maintenance and what their priorities were.



Page 65                                             GAO/GGD-97-109 Results Act
                         Chapter 4
                         Developing Useful Results-Oriented
                         Performance Information Will Be an
                         Ongoing Difficulty




                         Similarly, the Bureau of Engraving and Printing (BEP) primarily produces
                         currency and stamps in the volume and with the quality required by two
                         primary, immediate customers: the Federal Reserve Board and the U.S.
                         Postal Service. BEP receives no appropriations from Congress and is
                         self-financed by a revolving fund whereby the Reserve Board and Postal
                         Service reimburse BEP for its costs. Because of its direct relationship with
                         its customers, BEP has a relatively high degree of influence over the
                         achievement of its intended results and was able to more readily identify
                         results-oriented performance measures. These measures, which BEP has
                         used for years, focus on customer service and include currency notes
                         delivered (in billions) and postage stamps delivered (in billions), currency
                         and postage stamp productivity measures, and customer satisfaction
                         measures.


Long Time Frames to      Often it can take years before agencies see the results of their programs’
Achieve Results Hinder   activities. This factor has made it difficult for agencies to identify
Annual Performance       performance measures that will provide them with information on the
                         annual progress they are making toward achieving program results. For
Measurement              example, OSHA officials said it was difficult to track their agency’s yearly
                         progress in eliminating workplace hazards when the impact of some
                         regulatory actions could not be seen for years. They said that the latency
                         periods between exposure to a hazardous substance, such as asbestos, and
                         a resulting illness can be 20 years or more. Therefore, OSHA officials said,
                         the results of any actions OSHA takes to reduce exposures to those
                         substances may be equally long in coming. Officials at EPA also said it was
                         difficult to track the yearly progress of some of their agency’s regulatory
                         actions because, for example, a significant lag usually existed between the
                         elimination of a chemical hazard and any corresponding change in illness
                         rates.

                         Our work on research and development (R&D) programs exemplified the
                         significance of the challenge agencies face in identifying meaningful
                         performance measures to use in assessing program results when
                         achievement of those results can take several years and the results also
                         can be affected by external forces.8 Over $71 billion was appropriated for
                         federal R&D efforts in 1996. However, as we and others have reported for
                         almost 20 years, agencies with such programs have encountered difficulty
                         in identifying meaningful performance measures, particularly quantitative
                         measures, that can be used to assess the results that the technological

                         8
                          Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21,
                         1997).



                         Page 66                                                            GAO/GGD-97-109 Results Act
                            Chapter 4
                            Developing Useful Results-Oriented
                            Performance Information Will Be an
                            Ongoing Difficulty




                            advancements of these programs have on economic growth and the
                            overall standard of living. Because of the difficulties in identifying the
                            impacts of research, decisionmakers in the public and private sectors
                            typically have chosen to measure results using a variety of proxies. Three
                            of the most frequently cited quantitative measures are return on
                            investment, patents issued, and bibliometrics—i.e., the study of published
                            data, in particular, counts of citations. These measures imply a degree of
                            precision. However, they generally were not designed to measure the
                            long-term results of R&D programs and, therefore, are not easily adaptable
                            to such a purpose.

                            For example, return on investment is aimed at assessing the sales and
                            profits resulting from investments in R&D. However, long time periods and
                            the multiple research investments involved make the task of calculating
                            the return on basic research especially difficult. Productivity growth may
                            lag 20 years behind the first appearance of research in the scientific
                            community. A more serious impediment, however, is the fact that results
                            are often not directly traceable to a specific research investment or may
                            result from a combination of such investments.


                            In passing the Results Act, Congress emphasized that the usefulness of
Agencies Have Faced         agencies’ performance data depends, to a large degree, on the reliability
Challenges Collecting       and validity of those data. However, the reliability and validity of
Reliable and Valid          performance data are often questionable. From previous reviews, we
                            found that agencies have faced challenges in collecting reliable and valid
Results-Oriented            performance data. These challenges include different parties using
Performance Data            different definitions to measure performance and agencies’ reliance on
                            outside parties to provide results-oriented performance data.


Use of Different            The lack of standard definitions for performance measurement data can
Definitions for Measuring   significantly hinder agencies’ ability to use such data in planning and
Performance                 reporting. For example, we reported challenges that OCSE faced because of
                            a lack of comparable performance data across state and local
                            jurisdictions.9 OCSE officials said that discrepancies resulting from
                            differences in the way the states and local jurisdictions defined what
                            constitutes a child support enforcement case had contributed to the
                            difficulty of uniformly measuring state performance. To address these
                            discrepancies, OCSE worked with state and local authorities to develop

                            9
                             Child Support Enforcement: Reorienting Management Toward Achieving Better Program Results
                            (GAO/HEHS/GGD-97-14, Oct. 25, 1996).



                            Page 67                                                          GAO/GGD-97-109 Results Act
                      Chapter 4
                      Developing Useful Results-Oriented
                      Performance Information Will Be an
                      Ongoing Difficulty




                      standard data definitions for key child support enforcement terms,
                      including a definition for what constitutes a child support enforcement
                      case, and incorporated the use of standardized definitions for measuring
                      state performance. Together, they developed measures to assess state
                      performance in establishing paternities, obtaining support orders, and
                      collecting child support payments, using the established definition of a
                      child support enforcement case.

                      We also found that three economic development agencies we
                      reviewed—EDA, the Appalachian Regional Commission, and the Tennessee
                      Valley Authority’s nonpower programs—computed a key performance
                      measure differently.10 Specifically, each of these three agencies uses
                      “performance ratios” to measure the extent to which other federal, state,
                      local, or private investment in communities is attracted to an economic
                      development project as a result of the agency’s investment. The ratio is a
                      comparison of total dollars invested in or planned for an economic
                      development project—including the funding from other federal
                      programs—with the dollars contributed by the agency itself. However, we
                      found that each agency, in computing its ratio, defined “total dollars”
                      differently and calculated the ratio for only a portion of its programs.
                      Inconsistent definitions for what will be measured among crosscutting
                      programs can hamper efforts to compare the relative effectiveness of
                      individual programs.


Reliance on Outside   Relying on parties outside of agencies for performance data has created
Parties for           challenges for agencies to ensure the accuracy of such data. For example,
Results-Oriented      the Department of Education had faced difficulty in obtaining accurate
                      data from adult education programs administered by states.11 We reported
Performance Data      that states are required to submit to the Department of Education annual
                      statistical performance reports giving basic program information, such as
                      number of students served, student progress over the years, and various
                      types of student achievement. However, Education and state officials
                      acknowledged serious problems with the quality of the data contained in
                      those reports. These problems were based in part on double-counting or
                      under-counting of students in adult education programs.

                      As noted earlier, OCSE has had difficulty in obtaining comparable
                      state-reported data. OCSE also has faced challenges in ensuring that states

                      10
                        GAO/RCED-96-103, April 3, 1996.
                      11
                       Adult Education: Measuring Program Results Has Been Challenging (GAO/HEHS-95-153, Sept. 8,
                      1995).



                      Page 68                                                          GAO/GGD-97-109 Results Act
                           Chapter 4
                           Developing Useful Results-Oriented
                           Performance Information Will Be an
                           Ongoing Difficulty




                           provide accurate data. In 1994, we recommended that OCSE implement
                           additional oversight tools for ensuring that the data states submit are
                           accurate and comparable.12 In a subsequent report, we found that OCSE has
                           placed greater emphasis on its reviews of state reporting systems, which
                           analyze the procedures and systems states use to accumulate, record, and
                           report data.13 However, in its reviews of 20 state reporting systems, OCSE
                           found that most of those states did not have reliable systems for reporting
                           data accurately and that improvements would be needed as it moves to
                           results-oriented management. In 1996, the Personal Responsibility and
                           Work Opportunity Reconciliation Act required OCSE to review and validate
                           the data states report on progress toward achieving program goals.


                           Some agencies are exploring approaches that begin to address the
Approaches to              difficulty they are having in developing useful results-oriented
Improve the                performance information. Five approaches we identified include: (1) using
Usefulness of              impact evaluations; (2) using intermediate performance measures;
                           (3) using a range of measures; (4) providing decisionmakers with
Results-Oriented           information on the reliability and validity of performance data; and
Performance                (5) working with stakeholders to identify and reach consensus on the most
                           meaningful measures for the program, data sources for those measures,
Information Are            and data collection strategies.
Beginning to Emerge
Using Impact Evaluations   During our recent review of analytic challenges that agencies have faced in
                           measuring their performance, agency officials described using a variety of
                           approaches, including techniques that are employed in program
                           evaluations, specifically those evaluating program impact, to address the
                           challenge of isolating the impact of a program on its intended results.14
                           The Results Act defines program evaluation as “an assessment, through
                           objective measurement and systematic analysis, of the manner and extent
                           to which federal programs achieve intended objectives.” Such assessments
                           are to provide evaluative information about whether, and in what
                           important respects, a program is working well or poorly and may address
                           questions of program implementation and impact. Notably, the impact
                           evaluations described by the agency officials were often employed at state
                           or local levels where the influence of other variables was either reduced or
                           easier to observe and control for. For example, because they were well

                           12
                            Child Support Enforcement: Families Could Benefit From Stronger Enforcement Program
                           (GAO/HEHS-95-24, Dec. 27, 1994).
                           13
                             GAO/HEHS/GGD-97-14, October 25, 1996.
                           14
                            Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138,
                           May 30, 1997).


                           Page 69                                                          GAO/GGD-97-109 Results Act
                     Chapter 4
                     Developing Useful Results-Oriented
                     Performance Information Will Be an
                     Ongoing Difficulty




                     aware that the economy has a strong effect on a loan program’s
                     performance, officials from one such program monitored changes in the
                     economy very closely at the regional level. By breaking down the data to
                     follow one regional economy at a time, these officials were able to
                     determine whether an increase in loan defaults in a given region reflected
                     a faltering economy or indicated some problem in the program that
                     needed to be followed up.

                     Although the Results Act does not require agencies to conduct impact
                     evaluations, it does require them to measure progress toward their goals,
                     identify how external forces might affect such progress, and explain why a
                     goal was not met. We previously reported that agencies often need
                     information on a program’s impact on its results relative to the impact of
                     external factors to confidently attribute the achievement of intended
                     results to the program.15 Further, congressional and other decisionmakers
                     will likely request such information to help them make informed decisions.

                     We also have reported on how findings of program evaluations can be a
                     potentially critical source of information on the reasons an agency’s goals
                     were not met and actions that could be taken to improve performance.16
                     However, we have reported that because of their complexity, such
                     evaluation studies may be costly, and many agencies may not have staff
                     with the skills to conduct program evaluations.17


Using Intermediate   As noted earlier, an agency’s ability to identify meaningful performance
Measures             measures to use in planning its efforts and gauging progress—especially
                     annual progress—toward achieving its goals is sometimes hindered
                     because the intended results of the agency’s programs can be affected by
                     various factors or require many years to come to fruition. Our work has
                     found that in such cases, agencies may be able to use intermediate
                     performance measures that provide information on interim results to
                     gauge their progress in meeting their goals.18

                     EPA’s
                         Office of Water has identified intermediate results, with
                     accompanying measures, for local estuary protection programs to use in

                     15
                      Program Evaluation: Improving the Flow of Information to the Congress (GAO/PEMD-95-1, Jan. 30,
                     1995).
                     16
                       GAO/GGD-96-66R, February 14, 1996.
                     17
                       Program Evaluation Issues (GAO/OCG-93-6TR, Dec. 1992).
                     18
                       GAO/GGD-95-22, December 21, 1994.



                     Page 70                                                           GAO/GGD-97-109 Results Act
                            Chapter 4
                            Developing Useful Results-Oriented
                            Performance Information Will Be an
                            Ongoing Difficulty




                            assessing their progress in meeting intended results. According to EPA’s
                            Coastal Management Branch Chief, those programs are to develop
                            partnerships among all government agencies that oversee estuary
                            resources and the people who depend on those resources for their
                            livelihood and quality of life. The estuary programs generally try to affect
                            the behavior of government agencies, businesses, households, and boaters
                            toward achieving improvements in the quality of estuaries and the health
                            of organisms living in those estuaries. According to EPA, because it may
                            take years to see such improvements, intermediate measures can provide
                            information on the performance of those programs on a more timely basis
                            and are, therefore, important for managing estuary protection efforts. EPA
                            has developed procedures for managers of local estuary programs to
                            monitor and report on program results and the progress in improving and
                            maintaining the quality of their estuary waters, according to the official.
                            Further, to the extent that many estuary programs use similar procedures
                            for assessing their progress, such information also can provide a national
                            perspective on progress in estuary protection. EPA’s Coastal Management
                            Branch Chief said that most of the 28 estuary programs have adopted or
                            are considering this approach.

                            EPA identified three sequential intermediate results—which are referred to
                            as first-, second-, and third-order results—that are expected to affect
                            changes in bay quality and the health of organisms living in
                            estuaries—which are referred to as the fourth-order, or end, results of the
                            program. Measures for first-order intermediate results provide information
                            on the extent to which desirable estuary protection actions have been
                            adequately implemented by governments, businesses, or households.
                            Measures for second-order intermediate results provide information on
                            the extent to which those actions have led to reductions in pollutant
                            discharges. Measures for third-order intermediate results provide
                            information on the extent to which water or sediment quality has changed.
                            Finally, measures for assessing the fourth-order results of the program
                            provide information on the extent to which the health of fish, shellfish,
                            other wildlife, habitat and vegetation, and the region’s economy has
                            changed.


Using a Range of Measures   We found that in cases where agencies faced significant challenges
                            identifying results-oriented performance measures, they sometimes used a
                            range of measures to provide a more complete picture of agency
                            performance. For example, the Army Research Laboratory (ARL) found
                            that it could not predict the results or ultimate value of its research.



                            Page 71                                             GAO/GGD-97-109 Results Act
                              Chapter 4
                              Developing Useful Results-Oriented
                              Performance Information Will Be an
                              Ongoing Difficulty




                              Therefore, it could not objectively and quantitatively measure whether its
                              research was achieving a specific intended result. Instead, ARL devised a
                              performance measurement approach that is to make use of “three pillars”:
                              metrics, customer feedback, and peer review. For example, its metrics
                              included about 60 quantitative measures that according to ARL provided
                              useful information for understanding the functional health of the
                              organization and the management of ARL. However, ARL cautioned that the
                              resulting performance information would not enable it to determine the
                              real quality and impact of its programs. Because of its lack of
                              results-oriented performance measures, ARL is relying more heavily on
                              customer surveys and peer review for information about the quality of its
                              efforts.

                              Our work on R&D programs found that although there are strengths to
                              using peer reviews, there are also limitations.19 Peer review uses technical
                              experts to judge R&D results on the basis of the experts’ evaluation of the
                              quality of research. We reported that although peer review has been used
                              extensively in the selection of proposed research projects, it is subject to
                              two serious shortcomings. First, peer review is based on individuals’
                              perceptions of quality, which depend largely on the expertise of the
                              selected experts. Thus, the subjective nature of peer reviews makes
                              performance results vulnerable to bias. Frequently, a numerical rating
                              scale, such as 1 for poor through 5 for excellent, is used to judge the
                              quality of research projects and the selection of proposed projects.
                              However, despite the appearance of precision conferred by a specific
                              number, the numbers represent the best of sometimes widely differing
                              judgments. Second, peer review can be expensive. For example, ARL
                              contracted for a peer review of its activities, which calls for National
                              Research Council technical assessment board reviews of one-third to
                              one-half of ARL’s programs each year at a cost of approximately $650,000
                              per year.


Providing Decisionmakers      As noted earlier, in passing the Results Act, Congress emphasized that the
Information on the            usefulness of agencies’ performance data depends, to a large degree, on
Reliability and Validity of   the reliability and validity of those data. Consequently, the Act requires
                              that agencies describe in their annual performance plans the means to be
Performance Data              used to verify and validate performance data. We found that including
                              such information in performance reports could be equally important in
                              providing assurance to report users of the quality of the data.20 The

                              19
                                GAO/RCED-97-91, March 21, 1997.
                              20
                                GAO/GGD-96-66R, February 14, 1996.



                              Page 72                                             GAO/GGD-97-109 Results Act
                            Chapter 4
                            Developing Useful Results-Oriented
                            Performance Information Will Be an
                            Ongoing Difficulty




                            National Highway and Traffic Safety Administration (NHTSA) included an
                            appendix to its fiscal year 1994 Results Act pilot performance report that
                            discussed the sources, and in some cases the limitations, of the data it
                            used to report on performance. For example, NHTSA’s accident data are
                            based on police reports, but various sources suggested that about half of
                            the motor vehicle crashes in the country were not reported to police.
                            NHTSA believed that the majority of these unreported crashes involved only
                            minor property damage and no significant injury. However, NHTSA’s report
                            stated that it planned “to assess the unreported injury problem.” By
                            incorporating a discussion of the limitations of its accident data, NHTSA
                            provided a context to assist the users of the data in assessing NHTSA’s
                            performance.


Working With Stakeholders   Stakeholders can help agencies identify results-oriented performance
to Reach Consensus on       measures and the ways to collect needed performance data. For example,
Measures and Data           in 1994, we reported that OCSE lacked essential management tools to
                            improve its responsiveness to the child support needs of children and
Collection                  families and recommended actions OCSE could take in this regard.21 Since
                            that time, OCSE has made progress in reorienting its management of the
                            program toward achieving intended program results through greater
                            stakeholder involvement and other means.22 OCSE’s effort to involve states
                            and other stakeholders took time and concerted effort.

                            One of the first steps OCSE took was specifying the goals that it expected
                            states to achieve in such areas as paternities established and collections
                            received. However, state program officials strongly objected to this
                            mandate, because they did not have an opportunity to participate in the
                            planning process. OCSE then sought to obtain wider participation from
                            program officials at the federal, state, and local levels of government
                            through a joint planning process. During the planning process, participants
                            agreed that intended national results would be based on the collective
                            suggestions of the states and consensus among participants. OCSE also
                            established a Performance Measures Work Group to identify statistical
                            measures for assessing state progress toward achieving national results.
                            The work group, which consisted of officials from the Department of
                            Health and Human Services’ (HHS) Administration for Children and
                            Families, OCSE, and state and local child support enforcement programs,
                            met several times in 1995 and 1996 to discuss mutually acceptable
                            performance measures.

                            21
                              GAO/HEHS-95-24, December 27, 1994.
                            22
                              GAO/HEHS/GGD-97-14, October 25, 1996.



                            Page 73                                            GAO/GGD-97-109 Results Act
              Chapter 4
              Developing Useful Results-Oriented
              Performance Information Will Be an
              Ongoing Difficulty




              Officials in VA’s Loan Guaranty program also discussed the importance of
              involving stakeholders in identifying meaningful measures. According to
              those officials, they had difficulty linking field office activities to the
              intended results of the program as established by headquarters staff. To
              address this, VA used input from OMB and a performance measures work
              group that included both headquarters and field office staff to reach
              consensus on the most meaningful performance measures for the program
              and how data would be collected on those measures. Also, key
              headquarters and field managers were brought together to reach
              agreement on the goals to be included in the program’s fiscal year 1997
              business plan.

              Perhaps most important, congressional consultations on agencies’
              strategic plans also can assist agencies in identifying the most meaningful
              measures for the agencies’ programs. A VA official reported that most of
              their discussions with congressional staff have included useful exchanges
              about the most meaningful performance measures for assessing the results
              of VA’s benefit programs. In several cases, congressional staff proposed
              additional performance measures for VA programs. For example, the staff
              suggested that VA’s Vocational Rehabilitation and Counseling program add
              measures to assess the socioeconomic impact of the program on veterans,
              such as the extent to which veterans have to rely on other government
              assistance programs.


              The challenges confronting agencies as they seek to develop useful
Conclusions   results-oriented performance information are substantial, long-standing,
              and will not be quickly or easily resolved. In cases where results-oriented
              baseline and trend data on past performance do not exist, agencies’ will
              have difficulty setting realistic goals. Many agencies also are grappling
              with how best to assess their intended results in an environment where
              federal program efforts are but one factor among many external forces
              that may influence whether program results are achieved. Identifying
              results-oriented performance measures will often require an agency to
              separate out the influence on the achievement of its intended results from
              those of forces external to it in order to isolate the agency’s contribution
              to the achievement of those results—an analytically daunting challenge.
              Moreover, identifying such measures also is challenged by the long lead
              times that may occur between a federal agency’s actions and the
              achievement of intended results.




              Page 74                                             GAO/GGD-97-109 Results Act
Chapter 4
Developing Useful Results-Oriented
Performance Information Will Be an
Ongoing Difficulty




Finally, once agencies have identified results-oriented performance
measures, collecting reliable and valid data on those measures poses
challenges for agencies. Reasons for these challenges include different
parties using different definitions to measure performance and agencies
relying on outside parties to provide results-oriented performance data.
Some agencies have begun exploring approaches to address the difficulty
in developing useful results-oriented performance information. However,
the nature of the challenges agencies need to address and the time and
effort needed to overcome those challenges suggest that agencies will be
hard-pressed to consistently identify results-oriented measures in time for
the first annual performance plans to be submitted to OMB this fall. Those
agencies with the most direct influence on their results generally will
make the most progress in identifying such measures, while those
agencies with less influence will continue to struggle to identify the most
meaningful measures for informing decisionmakers in Congress and the
executive branch.




Page 75                                             GAO/GGD-97-109 Results Act
Chapter 5

Building an Organizational Culture to Help
Create and Sustain a Focus on Results
Remains a Work in Progress
                     Successful implementation of the Results Act in federal agencies will
                     depend on the degree to which those agencies create organizational
                     cultures that foster and maintain a focus on results at all levels. The
                     committed support of top agency leadership is critical to building a
                     results-oriented culture. Yet, according to the opinions of the federal
                     managers who responded to our survey, top agency leadership is not fully
                     committed to results-oriented management. In addition, successful
                     implementation of the Results Act will depend, in part, on managers
                     having the knowledge, skills, and abilities to set goals, measure
                     performance, and use performance information to improve effectiveness.
                     However, managers’ responses to our survey indicated that federal
                     managers needed additional training on critical results-oriented
                     management-related tasks. A results-oriented organization also strives to
                     ensure that its managers at all levels have the authority needed to
                     accomplish goals and provides those managers with the incentives and
                     opportunities for doing so. However, federal managers’ responses to our
                     survey also suggest that more progress is needed in these areas as well.


                     For the Results Act to become part of an agency’s culture, the agency
Top Leadership Has   needs the sustained, demonstrated commitment of its top leadership to
Not Fully            initiate and insist on the use of results-oriented management practices, to
Demonstrated         keep the agency focused on results, and to embed related principles in the
                     agency’s approach to doing business.1 By showing managers and staff its
Commitment to        commitment to achieving an agency’s goals, top agency leadership can
Results-Oriented     encourage a focus on results. When asked about the extent to which their
                     agencies’ top leadership demonstrated a strong commitment to achieving
Management           results, federal managers in our survey reported a significantly greater
                     extent of current top leadership commitment than they recalled existing 3
                     years ago. According to our survey, about 44 percent of managers reported
                     that their agencies’ top leadership demonstrated a strong commitment to
                     achieving results to a great or very great extent 3 years ago, and 57 percent
                     of managers reported such commitment to a great or very great extent
                     currently, which is a 13 percentage point difference.

                     The percentage of federal managers reporting that top leadership
                     demonstrated strong commitment to a great or very great extent both
                     currently and 3 years ago was significantly higher for SES managers than
                     for non-SES managers. This suggests that the perception of top leadership
                     commitment to results is stronger for federal managers at higher levels


                     1
                      GAO/GGD-96-118, June 1996.



                     Page 76                                             GAO/GGD-97-109 Results Act
                                         Chapter 5
                                         Building an Organizational Culture to Help
                                         Create and Sustain a Focus on Results
                                         Remains a Work in Progress




                                         and that this perception has not reached managers at all agency levels to
                                         the same extent. (See table 5.1.)

Table 5.1: Percent of Federal Managers
Who Reported That Their Agencies’                                                     Percent of federal managers who reported
Top Leadership Demonstrated a                                                             that their agencies’ top leadership
Strong Commitment to Achieving                                                          demonstrated a strong commitment to
Results                                                                                achieving results to a great or very great
                                                                                                         extent
                                         Level of managers                                     3 years ago              Currently
                                         SES                                                            60%                    76%
                                         Non-SES                                                        42                     56
                                         Source: GAO survey data.



                                         Without top leadership making its commitment to results-oriented
                                         management clear, the Results Act risks the danger that all management
                                         reforms face: becoming a hollow, paper-driven exercise. As we have said
                                         in our Executive Guide, leaders who integrate results-oriented
                                         management into the culture and day-to-day activities of their
                                         organizations will help avoid that danger.2

                                         One of the most meaningful demonstrations of leaders’ commitment to
                                         results-oriented management is the use of performance information to
                                         make decisions. However, federal managers’ perceptions about the extent
                                         to which such use is occurring are not as strong as their perceptions about
                                         top leadership’s demonstrated commitment in general. This suggests that
                                         despite the strength of managers’ perceptions about top leadership
                                         commitment, the extent to which leaders are demonstrating their
                                         commitment by using performance information for decisionmaking about
                                         their agencies’ day-to-day activities is not as great.

                                         Compared to the 57 percent of managers who reported strong top
                                         leadership commitment to a great or very great extent, only about
                                         16 percent reported that changes by management above their levels to the
                                         programs for which they were responsible were based on results or
                                         outcome-oriented performance information to a great or very great extent.
                                         When we compared managers who reported a great or very great extent of
                                         strong top leadership commitment to managers who rated leadership
                                         commitment from moderate to none, we found that 25 percent of the
                                         managers who rated leadership commitment as great or very great also
                                         reported that managers above their level made program changes on the

                                         2
                                          GAO/GGD-96-118, June 1996.



                                         Page 77                                                      GAO/GGD-97-109 Results Act
Chapter 5
Building an Organizational Culture to Help
Create and Sustain a Focus on Results
Remains a Work in Progress




basis of performance information to a great or very great extent. Only
5 percent of managers who rated the extent of leadership commitment
from moderate to none rated the extent of managerial program changes
based on performance information as great or very great. Although the
difference between these two groups is statistically significant, in either
case, only a minority of managers perceived program changes based on
performance information occurring to a great or very great extent even
when top leadership commitment was perceived to be high.

Although still a minority, more federal managers said that the individuals
to whom they reported periodically reviewed with them the results or
outcomes of the programs for which they were responsible than said that
program changes were made based on performance information. Both of
these activities are indicators of the use of performance information for
management decisionmaking. About 42 percent of managers said that the
individuals to whom they reported periodically reviewed with them the
results or outcomes of the programs for which they were responsible to a
great or very great extent. As before, a comparison of managers who
reported a great or very great extent of strong top leadership commitment
to managers who rated leadership commitment from moderate to none
showed a significant difference. We found that 57 percent of the managers
who rated leadership commitment as great or very great also rated the
extent of periodic reviews of their programs’ results or outcomes as great
or very great. Only 21 percent of managers who rated the extent of
leadership commitment from moderate to none rated the extent of
periodic reviews of their programs’ results or outcomes as great or very
great.

Managers’ responses regarding the extent to which changes to programs
were based on results and the extent to which individuals to whom they
reported reviewed program results with them indicate that as the
perception of strong top leadership commitment increased, the perception
of the use of performance information for decisionmaking also increased.
In addition, as their responses indicate, many managers did not perceive
that performance information was being used to make such decisions to a
great or very great extent.




Page 78                                              GAO/GGD-97-109 Results Act
                     Chapter 5
                     Building an Organizational Culture to Help
                     Create and Sustain a Focus on Results
                     Remains a Work in Progress




                     In our work on identifying techniques that are needed to change an
Managers Reported    organization’s culture, we found that training was one of the most
Needing Additional   important techniques used.3 From our symposium on transforming the
Training             civil service, we discerned that treating continuous learning as an
                     investment in success rather than as a cost to be minimized was a key
                     principle for managing people.4 Investing in training and professional
                     development could help federal agencies meet changing customer needs,
                     keep skills up to date, and develop new personal and organizational
                     competencies. In work we did to identify insights for Results Act
                     implementation from past initiatives on performance budgeting, we spoke
                     with current and former executive branch officials and legislative branch
                     staff as well as individuals with expertise in the Results Act, budgeting,
                     and public administration.5 These experts said that participants in
                     implementing the Act needed to acquire the skills to develop and use
                     performance information. One of the critical practices for implementing
                     the Results Act that we identified in our Executive Guide was the need to
                     build expertise for staff at all levels of the organization so that they are
                     skilled in strategic planning, performance measurement, and the use of
                     performance information in decisionmaking. 6

                     When asked whether, during the past 3 years, their agencies had provided,
                     arranged, or paid for training that would help them accomplish several
                     critical results-oriented management-related tasks, the percentage of SES
                     managers who responded “yes” was significantly greater than that of
                     non-SES managers. According to our survey, 43 to 61 percent of SES
                     managers and 17 to 38 percent of non-SES managers responded yes to
                     questions on training that would help them accomplish each of several
                     critical results-oriented management-related tasks. The task for which the
                     highest percentage in general responded yes was for training on
                     conducting strategic planning. For this task, 61 percent of SES managers
                     and 38 percent of non-SES managers responded yes. However, for the
                     remaining five of the six tasks, fewer than 38 percent of non-SES managers
                     responded yes. Only 46 percent of SES managers and 17 percent of non-SES
                     managers responded yes regarding training on implementing the



                     3
                      Organizational Culture: Techniques Companies Use To Perpetuate or Change Beliefs and Values
                     (GAO/NSIAD-92-105, Feb. 27, 1992).
                     4
                      Transforming the Civil Service: Building the Workforce of the Future—Results of a GAO-Sponsored
                     Symposium (GAO/GGD-96-35, Dec. 20, 1995).
                     5
                      GAO/AIMD-97-46, March 27, 1997.
                     6
                      GAO/GGD-96-118, June 1996.



                     Page 79                                                            GAO/GGD-97-109 Results Act
                                         Chapter 5
                                         Building an Organizational Culture to Help
                                         Create and Sustain a Focus on Results
                                         Remains a Work in Progress




                                         requirements of GPRA. The results for training on each task for SES and
                                         non-SES managers appear in table 5.2.

Table 5.2: Percent of Federal Managers
Who Reported That During the Past 3                                                                        Percent of federal
Years Their Agency Provided,                                                                                  managers
Arranged, or Paid for Training That      Tasks                                                                 SES      Non-SES
Would Help Them to Accomplish
                                         Conduct strategic planning                                              61%            38%
Specific Tasks
                                         Set program performance goals                                           56             33
                                         Develop program performance measures                                    52             35
                                         Use program performance information to make decisions                   43             30
                                         Link the performance of program(s)/ operation(s)/project(s) to          46             27
                                         the achievement of agency strategic goals
                                         Implement the requirements of GPRA                                      46             17
                                         Source: GAO survey data.



                                         In addition, we asked federal managers to what extent they thought they
                                         needed training or additional training to help them accomplish each of
                                         these tasks. For each task, the percentage of non-SES managers was
                                         significantly higher than the percentage of SES managers reporting a need
                                         for training to a great or very great extent. Table 5.3 shows the results for
                                         training or additional training needed to a great or very great extent by SES
                                         and non-SES managers for each task.

Table 5.3: Percent of Federal Managers
Who Reported Needing Training or                                                                            Percent of federal
Additional Training to Accomplish                                                                             managers who
Specific Tasks                                                                                              reported needing
                                                                                                          training or additional
                                                                                                          training to a great or
                                                                                                           very great extent to
                                                                                                           accomplish specific
                                                                                                                  tasks
                                         Tasks                                                                 SES      Non-SES
                                         Conduct strategic planning                                              14%            28%
                                         Set program performance goals                                           15             32
                                         Develop program performance measures                                    20             40
                                         Use program performance information to make decisions                   14             32
                                         Link the performance of program(s)/operation(s)/                        17             33
                                         project(s) to the achievement of agency strategic goals
                                         Implement the requirements of GPRA                                      18             43
                                         Source: GAO survey data.




                                         Page 80                                                      GAO/GGD-97-109 Results Act
Chapter 5
Building an Organizational Culture to Help
Create and Sustain a Focus on Results
Remains a Work in Progress




Implementing the requirements of GPRA was the only task for which a
significant difference existed between the ratings of managers who
reported training that their agencies provided, arranged, or paid for and
managers who reported no such training. About 29 percent of managers
who reported the training in implementing the requirements of GPRA also
reported a great or very great need for additional training for this task.
However, more than half of managers who reported no such training in
implementing the requirements of the Act reported a great or very great
need for it. A need for training on this task received the highest proportion
of great and very great responses from managers who reported no such
training for any given task.

Although responses to the need for training varied, many managers
believed that they needed training to a great or very great extent for at
least some of the tasks. For example, based on our survey, 43 percent of
managers reported that they needed training to a great or very great extent
in at least two of the tasks listed.

Providing the skills to develop and use performance information may well
be a significant challenge to implementing the Results Act because of
declining resources and serious budgetary constraints. Executive branch
officials we spoke with as part of our review of past initiatives on
performance budgeting tended to think that the Results Act was “the right
thing to do.” Thus, they believed that the resources for needed training
related to implementing the Act would be found.7 In an April 2, 1997,
OPM-sponsored interactive broadcast on the Results Act, OMB’s Deputy
Director for Management said he recognized the difficult problem that
federal agencies faced, given the training and other capacity-building
activities needed to measure and evaluate program performance during
this time of significant budgetary constraint.8 But he stressed that federal
agencies do have to devote some resources to these important activities to
increase the effectiveness of their programs, for which they use most of
their resources. Because of this environment, agencies will need to seek
new, creative, and less costly ways to build their capacities to implement
the Act in order to improve their programs. In our past work, we found
that one approach—which has been used at HHS and the Department of the
Interior—was to limit training costs by having in-house coordinators lead



7
 GAO/AIMD-97-46, March 27, 1997.
8
Office of Personnel Management Federal Human Resources Forum: Interactive Broadcast on the
Government Performance and Results Act and How It Affects Every Federal Government Agency
Manager and Worker, April 2, 1997.



Page 81                                                          GAO/GGD-97-109 Results Act
                       Chapter 5
                       Building an Organizational Culture to Help
                       Create and Sustain a Focus on Results
                       Remains a Work in Progress




                       training efforts and serve as mentors on the Results Act.9 Managers’
                       responses to our survey questions on training suggest that federal
                       agencies’ efforts to train managers on results-oriented
                       management-related tasks may best be focused on non-SES managers. The
                       responses also suggest that such training especially include training on
                       implementing the requirements of the Results Act.


                       In crafting the Results Act, Congress recognized that if federal managers
Managers Reported      were to be held accountable for program results, they would need the
Reduced Authority to   authority and flexibility to achieve those results. Congress also understood
Achieve Results        the importance of affording federal program managers the freedom to be
                       innovative and creative and to marshal resources to achieve results.
                       Congress has provided agencies with additional authority in the key area
                       of procurement that allows agencies more flexibility in managing their
                       programs. In addition, one of the major initiatives of NPR has been the
                       empowerment of federal employees to get results.

                       Overall, when asked about the extent to which managers or supervisors at
                       their levels had the authority they needed to help their agencies
                       accomplish their strategic goals, federal managers did not perceive that
                       they had a greater extent of such authority than they recalled having 3
                       years ago. Interestingly, for managers from selected Results Act pilots, the
                       perception of the extent of this authority currently was significantly lower
                       than the perception for 3 years ago. In contrast, for managers who were
                       not in selected pilots, the perception of the extent of authority currently
                       was essentially unchanged from the perception for both 3 years ago. In
                       addition, for both 3 years ago and currently, the percentage of SES
                       managers reporting a great or very great extent of authority was
                       significantly higher than the percentage of non-SES managers. (See table
                       5.4.)




                       9
                        Managing for Results: Status of the Government Performance and Results Act (GAO/T-GGD-95-193,
                       June 27, 1995).



                       Page 82                                                           GAO/GGD-97-109 Results Act
                                         Chapter 5
                                         Building an Organizational Culture to Help
                                         Create and Sustain a Focus on Results
                                         Remains a Work in Progress




Table 5.4: Percent of Federal Managers
Reporting That Agency                                                                 Percent of federal managers reporting that
Managers/Supervisors at Their Levels                                                        managers/supervisors had the
Had the Decisionmaking Authority                                                      decisionmaking authority they needed to a
They Needed to Help the Agency                                                                great or very great extent
Accomplish Its Strategic Goals                                                           3 years
                                         Level of managers                                   ago        Currently      Difference
                                         SES
                                           Selected Results Act pilots                        56%              40%            –16%
                                           All other agencies                                 53               52              –1
                                         Non-SES
                                           Selected Results Act pilots                        36               24             –12
                                           All other agencies                                 29               31              +2
                                         Source: GAO survey data.



                                         When asked about the extent to which managers or supervisors at their
                                         levels were held accountable for the results of the programs for which
                                         they were responsible, no such differences between managers from
                                         selected Results Act pilots and managers from all other agencies or
                                         between SES and non-SES managers were observed. However, more
                                         managers overall rated the extent of accountability for results as higher
                                         currently than for 3 years ago. For example, about 55 percent of managers
                                         reported that managers or supervisors at their levels were currently being
                                         held accountable to a great or very great extent for the results of the
                                         programs for which they were responsible. About 48 percent of managers
                                         reported having this perception for 3 years ago.


                                         In crafting the Results Act, Congress recognized the need to create
Less Than Half of                        incentives for managers to use results-oriented performance information.
Federal Managers                         The commitment demonstrated by Congress and top leadership when they
Reported Positive                        use results-oriented performance information to make decisions is one of
                                         the greatest incentives for changing agencies’ cultures and managers’
Recognition for                          behavior to focus on achieving intended results. In addition, agencies have
Achieving Results                        traditionally rewarded their employees formally through pay increases and
                                         other monetary and nonmonetary awards for performance that helped
                                         agencies achieve their goals. However, the current environment of
                                         constrained resources challenges agency leaders to think creatively about
                                         informal incentives, which need not be costly, such as positive
                                         recognition, that they can use to encourage results-oriented management.




                                         Page 83                                                       GAO/GGD-97-109 Results Act
                       Chapter 5
                       Building an Organizational Culture to Help
                       Create and Sustain a Focus on Results
                       Remains a Work in Progress




                       When asked about the extent to which employees in their agencies
                       currently received positive recognition and received such recognition 3
                       years ago for helping the agencies accomplish their strategic goals, federal
                       managers reported essentially no difference. However, statistically
                       significant differences existed between the perceptions of SES and non-SES
                       managers, both currently and for 3 years ago. According to our survey,
                       48 percent of SES managers and 25 percent of non-SES managers reported
                       that, to a great or very great extent, employees in their agencies currently
                       received positive recognition for helping the agencies accomplish their
                       strategic goals. When asked to recollect the situation 3 years ago,
                       43 percent of SES managers and 27 percent of non-SES managers reported
                       such recognition.

                       In work we did on state governments’ experiences in using
                       results-oriented management, we found that in Oregon and Minnesota,
                       managers and staff in some state agencies were reluctant to commit to
                       achieving results that they did not totally control because of concerns that
                       performance information would be used against them.10 In our survey of
                       federal managers, we listed 16 factors and asked managers to rate each
                       factor on the extent to which it hindered measuring performance or using
                       performance information for the programs with which they were involved.
                       One of these factors was concern that performance information could be
                       used against their programs or agencies. When asked about this factor,
                       only about 9 percent of managers reported it as a hindrance to a great or
                       very great extent—least often of the 16 factors included in the survey.


                       Congress expected that under the Results Act, managers throughout
Involvement of         federal agencies would have active roles in making their organizations
Federal Managers in    more results oriented. One of the stated purposes of the Act was to help
Results Act-Related    federal managers improve service delivery by requiring that they plan for
                       meeting program objectives and by providing them with information about
Activities Is Varied   program results and service quality. More recently, during OPM’s April 1997
                       interactive broadcast on the Results Act, the OMB Deputy Director for
                       Management underscored OMB’s commitment to the effective
                       implementation of the Results Act.11 He told federal managers that if they
                       had not been involved in strategic planning, they should be asking their



                       10
                         GAO/GGD-95-22, December 21, 1994.
                       11
                        Office of Personnel Management Federal Human Resources Forum: Interactive Broadcast on the
                       Government Performance and Results Act and How It Affects Every Federal Government Agency
                       Manager and Worker, April 2, 1997.



                       Page 84                                                          GAO/GGD-97-109 Results Act
                                         Chapter 5
                                         Building an Organizational Culture to Help
                                         Create and Sustain a Focus on Results
                                         Remains a Work in Progress




                                         agencies why they had not been involved and saying that OMB wanted them
                                         to be involved.

                                         However, when we asked federal managers whether, during the past 3
                                         years, they had been involved in a variety of GPRA-related activities that
                                         incorporated a number of key practices needed to implement the Act, they
                                         reported varied levels of involvement, depending on the activity.
                                         Significant differences were especially evident for each activity between
                                         SES managers’ and non-SES managers’ reported levels of involvement. For
                                         example, according to our survey, 72 percent of SES managers reported
                                         involvement in establishing long-term strategic goals for their
                                         agencies—which is a key step in strategic planning—while only 35 percent
                                         of non-SES managers reported involvement in this activity. Table 5.5 shows
                                         the percentages of both SES and non-SES managers as well as the overall
                                         percentages of federal managers who reported that they had been involved
                                         in these activities during the past 3 years.

Table 5.5: Percent of Federal Managers
Reporting That They Had Been                                                                    Percent of federal managers
Involved in GPRA-Related Activities                                                             reporting that they had been
During the Past 3 Years                                                                      involved in these activities during
                                                                                                      the past 3 years
                                         Activities                                                SES     Non-SES        Overall
                                         Establishing long-term strategic goals for the             72%           35%          38%
                                         agency
                                         Reconciling differing congressional views on the           34             8           10
                                         agency’s mission and strategic goals
                                         Reconciling the views of other interested parties          53            29           31
                                         on the agency’s mission and strategic goals
                                         Developing ways to measure whether program                 72            47           49
                                         performance goals are being achieved
                                         Gathering and analyzing data to measure                    65            54           55
                                         whether programs are meeting their specific
                                         performance goals
                                         Using measures for program performance goals               55            34           35
                                         to determine if the agency’s strategic goals are
                                         being achieved
                                         Publicly reporting on how well the agency’s                44            18           20
                                         programs are meeting their specific performance
                                         goals
                                         Source: GAO survey data.



                                         Of activities that showed significantly lower percentages of non-SES
                                         managers reporting involvement as compared to SES managers, some are



                                         Page 85                                                      GAO/GGD-97-109 Results Act
                     Chapter 5
                     Building an Organizational Culture to Help
                     Create and Sustain a Focus on Results
                     Remains a Work in Progress




                     activities where such differences might be expected—for example,
                     reconciling differing congressional views on the agency’s mission and
                     strategic goals and reconciling the views of other interested parties on the
                     agency’s mission and strategic goals. However, such significant differences
                     between SES and non-SES managers’ reported involvement were also
                     evident for activities where involvement of managers at all levels could
                     reasonably be expected—for example, developing ways to measure
                     whether program performance goals are being achieved and gathering and
                     analyzing data to measure whether programs are meeting their specific
                     performance goals. However, gathering and analyzing data to measure
                     whether programs are meeting their specific performance goals was the
                     only one of these GPRA-related activities for which at least half of managers
                     overall, including both SES and non-SES managers, reported being involved.


                     When we asked federal managers about the extent to which their agencies’
Federal Managers’    efforts to implement GPRA to date had improved their agencies’ programs,
Views About GPRA’s   57 percent overall responded that they had not been sufficiently involved
Past and Future      in the Act in their agencies to have an opinion, comprising 22 percent of
                     SES managers and 60 percent of non-SES managers. Of the 41 percent of
Effects on Agency    managers who did express an opinion on the extent to which their
Programs Are Mixed   agencies’ efforts to implement GPRA to date had improved their agencies’
                     programs, 42 percent responded that the Act had improved programs to a
                     moderate or greater extent. Figure 5.1 shows the percentages of the
                     federal managers who did express an opinion for each extent level.




                     Page 86                                             GAO/GGD-97-109 Results Act
                                      Chapter 5
                                      Building an Organizational Culture to Help
                                      Create and Sustain a Focus on Results
                                      Remains a Work in Progress




Figure 5.1: Distribution of Federal
Managers Expressing an Opinion on     Percent
the Extent to Which Implementing
GPRA to Date Had Improved Their
                                       40
Agencies’ Programs
                                                                                            36

                                                                                   32

                                       30




                                                                                                           22

                                       20




                                       10                          9




                                                   1
                                           0
                                                Very great    Great extent   Moderate   Small extent    No extent
                                                  extent                      extent


                                      Source: GAO survey data.




                                      Governmentwide implementation of the Results Act is just beginning for
                                      many federal agencies and programs and, as our earlier work has shown,
                                      changing to a results-oriented culture can take many years.12 Thus, the
                                      57 percent of managers overall reporting that they had not been
                                      sufficiently involved in GPRA to have an opinion on the extent to which
                                      efforts to implement the Act to date had improved their agencies’
                                      programs and the much greater proportion of non-SES managers
                                      (60 percent) than SES managers (22 percent) reporting that they had not
                                      been sufficiently involved to have an opinion are not surprising. In
                                      addition, given the long-term challenges that a move toward a
                                      results-oriented culture involves, it is reasonable to expect that many
                                      federal managers are not yet seeing their agencies’ results improving as an
                                      effect of the Results Act.


                                      12
                                        GAO/T-GGD-95-193, June 27, 1995.



                                      Page 87                                                    GAO/GGD-97-109 Results Act
                                      Chapter 5
                                      Building an Organizational Culture to Help
                                      Create and Sustain a Focus on Results
                                      Remains a Work in Progress




                                      When asked about the extent to which implementing GPRA could improve
                                      their agencies’ programs in the future, federal managers’ opinions were
                                      more optimistic. About 36 percent of managers overall did not express an
                                      opinion, responding that they had no basis to judge. Of the 62 percent of
                                      managers who did express an opinion, about 71 percent responded that
                                      implementing the Act could improve programs in the future to a moderate
                                      or greater extent. Figure 5.2 shows the percentages of the federal
                                      managers who did express an opinion for each extent level.


Figure 5.2: Distribution of Federal
Managers Expressing an Opinion on     Percent
the Extent to Which Implementing
GPRA Could Improve Their Agencies’    40
Programs in the Future
                                                                                   35



                                      30

                                                                 25
                                                                                           23

                                      20




                                                11
                                      10

                                                                                                            6



                                       0
                                             Very great      Great extent    Moderate   Small extent     No extent
                                               extent                         extent


                                      Source: GAO survey data.




                                      Still, many federal managers remained unconvinced or uncertain about
                                      whether the Results Act could help their agencies’ programs even
                                      moderately.




                                      Page 88                                                   GAO/GGD-97-109 Results Act
              Chapter 5
              Building an Organizational Culture to Help
              Create and Sustain a Focus on Results
              Remains a Work in Progress




              According to the opinions of federal managers, building the organizational
Conclusions   cultures necessary to create and sustain a results-orientation in the federal
              government appears to be a work in progress. In some ways, changes
              toward a culture that will allow a focus on results appear to be occurring.
              For example, the perceptions of both SES and non-SES managers about the
              extent to which top leadership is demonstrating a strong commitment to
              achieving results were higher currently than they were for 3 years ago.
              However, we would expect to see federal managers’ experiences with and
              positive perceptions about results-oriented management practices become
              more prevalent if the Results Act is to succeed.

              Over the next several years, we would expect to see more federal
              managers reporting positively on indicators, such as top management’s use
              of results to make decisions, agencies providing training on key
              results-oriented management-related tasks, and managers’ involvement in
              GPRA-related activities. But just as importantly, we would expect to see the
              gap between SES and non-SES managers regarding positive experiences and
              perceptions about results-oriented management begin to narrow if the
              Results Act is to work as intended. Agencies can especially concentrate
              their efforts on areas where managers are not perceiving or experiencing
              progress, such as on devolving decisionmaking authority to managers
              throughout the agencies.

              When both mid-level and upper level managers support and are involved in
              the changes that the Results Act entails, the probability of sustaining those
              changes will increase substantially. Providing federal managers at all
              levels with greater authority and greater opportunities to participate in
              results-oriented activities can give them the experiences needed to
              increase their confidence in the benefits of managing for results. When
              such indicators are more widespread and pervasive among federal
              managers, agencies’ commitment to achieving results and their ability to
              do so should be enhanced governmentwide.




              Page 89                                              GAO/GGD-97-109 Results Act
Chapter 6

Efforts to Integrate Performance
Information With Budget Decisions and to
Develop a New Reporting Framework Are
Promising
              Addressing some of the challenges discussed earlier in this report, such as
              crosscutting program efforts and balances among competing priorities,
              raises significant policy issues for Congress and the administration to
              consider, and some issues will likely be very difficult to resolve. Our work
              has indicated that the program performance information that is to be
              generated under the Results Act will become most useful to congressional
              and executive branch decisionmakers when that information is
              systematically linked with resource allocation decisions and reported
              within a framework that allows for fuller assessments of results,
              operational performance, and costs.

              Although budgeting is inherently an exercise in political choice, the
              Results Act is based on the premise that budget decisions should take into
              consideration an agency’s expectations about program performance.
              Therefore, the Results Act’s success or failure should not be judged on
              whether contentious budget and other policy issues are fully resolved;
              rather, the success or failure of the Results Act will likely turn on the
              extent to which the information produced through the Act’s goal-setting
              and performance measurement practices, once those practices are
              successfully implemented, helps Congress and the executive branch make
              informed policy decisions and improve program management.

              Agencies are to provide the first annual performance plans, covering fiscal
              year 1999, to OMB in the fall of 1997 and to Congress after the President’s
              budget is submitted in 1998. These plans are to directly link executive
              branch performance planning with the budget process. Agencies are to
              accomplish this link by basing their annual program performance goals on
              their program activity structures as listed in the President’s budget.
              However, our recent report on previous governmentwide initiatives to
              better align spending decisions with expected performance—commonly
              referred to as performance budgeting—found that the extent to which the
              budget’s program activity structures can be directly linked to a
              results-oriented performance framework varies widely among program
              activities. For Congress and the executive branch to reach agreement on
              changes to these structures will be a time-consuming and difficult process
              that will require more than one budget cycle to resolve.1

              Over the longer term, improved financial reporting and auditing as
              required by the CFO Act is to strengthen the cost basis and reliability of
              performance information. In this regard, reporting concepts and standards
              developed by the Federal Accounting Standards Advisory Board (FASAB)

              1
               GAO/AIMD-97-46, March 27, 1997.



              Page 90                                             GAO/GGD-97-109 Results Act
                        Chapter 6
                        Efforts to Integrate Performance
                        Information With Budget Decisions and to
                        Develop a New Reporting Framework Are
                        Promising




                        are intended to provide congressional and other decisionmakers for the
                        first time with annual “report cards” on the costs, management, and
                        effectiveness of federal agencies in order to help strengthen federal
                        decisionmaking and agencies’ accountability.2 In addition, FASAB’s new
                        cost accounting standards are intended to strengthen federal
                        decisionmaking by requiring agencies to provide relevant and reliable cost
                        information that can be used, for example, to relate the costs of various
                        programs and activities with their performance outputs and results.


                        Congress intended for the Results Act’s annual performance plans to help
The Results Act Seeks   Congress and the executive branch make informed decisions by providing
to Better Align         a simple, straightforward linkage among plans, budgets, and performance
Desired Results With    results—a linkage that we have found did not exist in the past. Our report
                        on performance budgeting initiatives demonstrated the importance of
Budget Decisions        making this linkage if the Results Act’s goal of better aligning expected
                        results with budget decisions is to be met. We reported that past
                        performance budgeting initiatives, such as PPBS, which was initiated
                        governmentwide in 1965 by President Johnson, generated unique
                        performance information that was often unconnected to the structures
                        used in congressional budget presentations. We found that consequently,
                        congressional interest in these initiatives quickly waned as plans and
                        performance results could not be directly linked to familiar oversight and
                        budget structures. In the end, we concluded that structural
                        incompatibilities meant that resource decisions were not linked to the
                        information generated by the past performance budgeting efforts.

                        The Results Act attempts to establish this linkage by requiring agencies to
                        base the goals in their annual performance plans on the program activity
                        structures used in their budget submissions and to provide these
                        performance plans to Congress each year after the president’s budget is
                        submitted. Program activities are listings of projects and activities
                        contained in the president’s budget and are intended to provide a
                        meaningful representation of agency operations. The Results Act allows
                        agencies to aggregate, disaggregate, or consolidate program activities for
                        performance planning purposes. We noted in our performance budgeting
                        report that program activity structures are used to provide a relatively
                        consistent framework for OMB and congressional budget decisions.
                        Congress also often uses the program activity structures to form the basic



                        2
                        FASAB was created in October 1990 by the Secretary of the Treasury, the Director of OMB, and the
                        Comptroller General to consider and recommend accounting principles for the federal government.



                        Page 91                                                             GAO/GGD-97-109 Results Act
Chapter 6
Efforts to Integrate Performance
Information With Budget Decisions and to
Develop a New Reporting Framework Are
Promising




unit of congressional oversight for determining reprogramming
thresholds.3

We reported that an agency’s program activities generally result from
negotiations between the agency, OMB, and the relevant congressional
appropriations subcommittees. Program activity structures, therefore,
typically are tailored to meet the specific needs of an agency, OMB, and
Congress. Thus, program activity structures represent programmatic,
process, organizational, or other orientations depending on the needs,
interests, and experiences of agencies, OMB, and Congress. Moreover,
program activities differ from agency to agency; and within agencies, they
differ from budget account to budget account. For example, the “space
station” program activity in the National Aeronautics and Space
Administration’s budget is a discrete program. In contrast, the “park
management” program activity under the National Park Service’s budget
represents process and/or functional efforts.

Because of the wide variability across program activities, we noted that
the suitability of current program activity structures for the Results Act’s
performance planning and measurement also will vary. Agency officials
we spoke to as part of our review of previous performance budgeting
initiatives highlighted the varying suitability of their program activity
structures for the Results Act’s purposes. One agency successfully worked
through the performance planning process using its existing program
activities. A second agency had program activities that reflected its
organizational units—a structural approach that is useful for traditional
accountability purposes, such as monitoring outputs and staff levels, but
less useful for results-oriented planning. This agency therefore found it
necessary to devise a separate planning structure and then crosswalk
performance goals back to the budget’s program activities. Still other
agencies separated performance planning from program activity
structures, believing it necessary to first establish appropriate program
goals, objectives, and measures before considering the link to the budget.

Our discussions with agency officials and congressional staff found that
they tended to view the need for and benefits of adjustments to program
activities from very different perspectives. Congressional staff were
generally comfortable with existing activity structures and questioned


3
 Reprogramming is the shifting of funds within an appropriation to purposes different from those
contemplated at the time the appropriation was requested and provided. Several appropriations
subcommittees use program activity structures to establish reprogramming thresholds. If an agency
needs to shift funds from one activity to another above the threshold, it is expected to notify the
appropriate subcommittee.



Page 92                                                               GAO/GGD-97-109 Results Act
                        Chapter 6
                        Efforts to Integrate Performance
                        Information With Budget Decisions and to
                        Develop a New Reporting Framework Are
                        Promising




                        whether changes would frustrate congressional oversight. The staff
                        generally viewed these structures as fundamental to congressional
                        oversight of agency activities; thus, they viewed changes with
                        apprehension and concern. However, some agency officials saw program
                        activity structures as secondary to planning; thus, where current program
                        activity structures proved unsuitable for planning purposes, these officials
                        viewed change in program activity structures as inevitable and
                        appropriate. However, they noted that negotiating changes with Congress
                        could prove difficult and time-consuming.


                        Our work has shown that congressional and executive branch
Integrated              decisionmakers have been handicapped by the absence of reliable and
Performance, Cost,      valid performance, program cost, and management information needed to
and Management          make well-informed decisions.4 If successfully implemented, the
                        performance measurement and reporting requirements under the Results
Reports Provide Basis   Act will provide congressional and executive branch decisionmakers with
for Supporting Key      a wealth of information on the results of agencies’ efforts. We have
                        testified that such information will be most useful to congressional and
Decisions               executive branch decisionmakers if it is consolidated with critical
                        financial and program cost data in financial statements, which agencies
                        are to produce and have audited under the CFO Act.5 The CFO Act also sets
                        expectations for agencies to deploy modern systems to replace existing
                        antiquated, often manual, processes; develop better performance and cost
                        measures; and design results-oriented reports on the government’s
                        financial condition and operating performance by integrating budget,
                        accounting, and program information.

                        The results-oriented reports will be most effective if they support a wide
                        range of decisions—including budget, policy, program management, and
                        accountability—that are routinely made by Congress and the executive
                        branch. The challenge of developing reports that are specifically tailored
                        to meet decisionmakers’ needs and that are based on appropriate
                        accounting standards is being met by FASAB. FASAB has put together a new
                        set of accounting concepts and standards based on a framework that

                        4
                         See Information Resources: Summary of Federal Agencies’ Information Resources Management
                        Programs (GAO/IMTEC-92-13FS, Feb. 13, 1992); Financial Management: CFO Act Is Achieving
                        Meaningful Progress (GAO/T-AIMD-94-149, June 21, 1994); Government Reform: Using Reengineering
                        and Technology to Improve Government Performance (GAO/T-OCG-95-2, Feb. 2, 1995); Managing for
                        Results: Critical Actions for Measuring Performance (GAO/T-GGD/AIMD-95-187, June 20, 1995); and
                        High-Risk Areas: Actions Need to Solve Pressing Management Problems (GAO/T-AIMD/GGD-97-60,
                        Mar. 5, 1997).
                        5
                         Financial Management: Continued Momentum Essential to Achieve CFO Act Goals
                        (GAO/T-AIMD-96-10, Dec. 14, 1995).



                        Page 93                                                           GAO/GGD-97-109 Results Act
    Chapter 6
    Efforts to Integrate Performance
    Information With Budget Decisions and to
    Develop a New Reporting Framework Are
    Promising




    financial reporting should demonstrate accountability and provide
    information useful for planning, controlling, and conducting the federal
    government’s functions. These concepts and standards underpin OMB’s
    guidance to agencies on the form and content of their agencywide
    financial statements.

    The new accounting concept statements—the Objectives of Federal
    Financial Reporting and Entity and Display—establish a new reporting
    model for federal agencies that is geared to providing users with
    information about budgetary integrity, operating performance,
    stewardship, and systems and controls. Through this new reporting model,
    Congress, agency decisionmakers, and the American public will receive
    annual audited financial reports that are to

•   present a top-level overview discussion of the significance of the financial
    information in accounting reports and commentary on performance goals
    and results that are consistent with measures an agency uses under the
    Results Act;
•   report, and thus make auditable, (1) uses of budgetary resources that are
    made available by Congress, (2) obligations incurred, (3) the balance of
    budgetary resources, and (4) outlays;
•   show total cost, earned revenues (if any), and net cost of agency
    operations and programs in a manner that enables users to relate costs to
    outputs and results;
•   report (1) unspent funds; (2) other operating assets available for use in
    providing government goods, services, and benefits; and (3) all liabilities
    incurred, with separate displays of those that would require future funding
    by Congress; and
•   reflect the nation’s assets and investments for which the federal
    government has a stewardship responsibility.

    As authorized by GMRA—which expanded the requirements of the CFO
    Act—OMB is piloting accountability reports that are consistent with the
    FASAB accountability reporting concept. OMB worked with six agencies to
    pilot the development of accountability reports for fiscal year 1995 and
    added an additional eight agencies to the pilot test in fiscal year 1996.
    According to OMB, additional agencies will produce accountability reports
    for fiscal year 1997. By seeking to consolidate and integrate the separate
    reporting requirements of the Results Act, the CFO Act, and other specified




    Page 94                                             GAO/GGD-97-109 Results Act
Chapter 6
Efforts to Integrate Performance
Information With Budget Decisions and to
Develop a New Reporting Framework Are
Promising




acts, the accountability reports are to show the degree to which an agency
met its goals, at what cost, and whether the agency was well run.6

Along with the new reporting model, FASAB developed cost accounting
standards that are to be effective beginning with fiscal year 1997—the first
set of standards that are to account for the costs of federal programs.
These standards require agencies to develop measures of the full costs of
carrying out a mission or producing products or services. Thus,
decisionmakers are to have information on the costs of all resources used,
including the cost of services provided by others to support activities or
programs. Such information would allow for comparisons of the costs of
various programs and activities with their performance outputs and
results. To help agencies implement the cost accounting standards and
related requirements set forth in the CFO Act and the Result Act, the Joint
Financial Management Improvement Program (JFMIP)—which develops the
requirements that agencies’ financial management systems are to
meet—has issued an exposure draft, entitled “Managerial Cost Accounting
System Requirements.”7 This document is intended to facilitate the
acquisition and development of managerial cost accounting systems by
specifying the minimum information and functional processing
requirements agencies needed to accumulate and assign cost data
consistent with governmentwide guidance.

In addition, fiscal year 1997 will usher in the new requirements of the
Federal Financial Management Improvement Act of 1996, which will help
ensure greater attention to making much needed improvements in
underlying financial management systems. This act requires agencies’
financial management systems to comply with FASAB standards and JFMIP
system requirements.

Our work has shown that developing the necessary information systems to
gather, process, and analyze the needed program and cost information will
be a substantial undertaking for most federal agencies because of
agencies’ persistent weaknesses in the use of information technology.8 To
help address problems in agencies’ use of information technology,
Congress enacted the Paperwork Reduction Act of 1995 and the

6
 Managing For Results: Using GPRA to Assist Congressional and Executive Branch Decisionmaking
(GAO/T-GGD-97-43, Feb. 12, 1997).
7
 JFMIP is a joint initiative of OMB, GAO, the Department of the Treasury and the Office of Personnel
Management, which are working in cooperation to improve financial management practices
throughout the federal government.
8
 See, most recently, High-Risk Series: Information Management and Technology (GAO/HR-97-9, Feb.
1997).



Page 95                                                               GAO/GGD-97-109 Results Act
              Chapter 6
              Efforts to Integrate Performance
              Information With Budget Decisions and to
              Develop a New Reporting Framework Are
              Promising




              Clinger-Cohen Act of 1996. This information technology reform legislation
              introduced requirements, in large part based on GAO’s research, that are
              intended to significantly improve agencies’ management, including how
              they select and manage information technology resources. We have
              reported that together with GPRA and the CFO Act, this reform legislation
              directs agencies to implement a framework of modern technology
              management—one based on practices followed by leading public sector
              and private sector organizations that have successfully used technology to
              dramatically improve performance and meet strategic goals.9


              The Results Act’s annual performance plans are to establish a direct
Conclusions   linkage between the goals in those plans and the congressional and
              executive branch budget processes. This linkage between the Results Act
              and budget decisions is key to Congress and the executive branch
              ultimately being able to connect resources to results. Our work reviewing
              the Results Act and previous performance budgeting efforts suggests that
              due to the great variability among agencies’ program activity structures,
              Congress and the executive branch will likely be challenged as they
              attempt to link performance goals with the budget’s program activity
              structures.

              Therefore, the successful integration of performance goals developed
              under the Results Act into budget decisionmaking may take several annual
              budget cycles before Congress, OMB, and executive agencies are able to
              make significant progress. We have noted that in the short term, the
              challenges in reaching agreements on whether and how to realign program
              activities may require agencies to develop crosswalks between their
              performance goals and current program activity structures. However, if
              long-term progress in this integration is not achieved, the overall
              usefulness of the Results Act’s performance plans as a tool for improved
              congressional and executive branch decisionmaking may be
              compromised.

              Reports that present an agency’s financial condition and the results of its
              operations in an integrated way also hold promise for enhancing the
              usefulness of performance information. We have noted that such reports,
              which are independently audited, will help correct the lack of complete
              and reliable information that has been a source of concern for
              congressional and executive branch decisionmakers for decades. Under


              9
              GAO/AIMD-94-115, May 1994 and Assessing Risks and Returns: A Guide for Evaluating Federal
              Agencies’ IT Investment Decision-making (GAO/AIMD-10.1.13, Feb. 1997).



              Page 96                                                           GAO/GGD-97-109 Results Act
Chapter 6
Efforts to Integrate Performance
Information With Budget Decisions and to
Develop a New Reporting Framework Are
Promising




FASAB’s  new cost accounting standards, agencies are also required, for the
first time, to provide decisionmakers with measures of the full costs of
carrying out a mission or providing a service. If fully and effectively
implemented, integrated accountability reports can facilitate
congressional and executive branch decisionmaking by consolidating
performance, cost, and financial data in a single, user-friendly document.




Page 97                                             GAO/GGD-97-109 Results Act
Appendix I

Survey on Results-Oriented Management
Practices




              Page 98            GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 99                                 GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 100                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 101                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 102                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 103                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 104                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 105                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 106                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 107                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 108                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 109                                GAO/GGD-97-109 Results Act
Appendix I
Survey on Results-Oriented Management
Practices




Page 110                                GAO/GGD-97-109 Results Act
Appendix II

Major Contributors and Acknowledgements


                     Michael Brostek, Associate Director, (202) 512-9039
General Government   J. Christopher Mihm, Acting Associate Director, (202) 512-3236
Division             Joseph S. Wholey, Senior Advisor for Evaluation Methodology
                     Thomas M. Beall, Senior Social Science Analyst
                     Donna J. Byers, Senior Evaluator
                     William R. Chatlos, Senior Social Science Analyst
                     Joyce D. Corry, Senior Evaluator
                     Victoria M. O’Dea, Senior Evaluator
                     Kathleen M. Peyman, Senior Evaluator
                     Dorothy L. Self, Evaluator
                     Lisa R. Shames, Senior Evaluator
                     Kiki Theodoropoulos, Senior Evaluator


                     In addition to those named above, the following individuals made
Acknowledgements     important contributions to this report: From the Accounting and
                     Information Management Division: Linda F. Baker, Senior Evaluator;
                     Michael J. Curro, Assistant Director; and Carolyn Yocum, Senior
                     Evaluator. From the General Government Division: James M. Fields,
                     Senior Social Science Analyst; Tracy J. Harris, Report Production
                     Assistant; Catherine M. Hurley, Computer Programmer Analyst; Stuart M.
                     Kaufman, Senior Social Science Analyst; Donna M. Leiss, Communications
                     Analyst; Susan Ragland, Senior Evaluator; Theodore H. Saks, Senior
                     Evaluator; Stephanie Shipman, Assistant Director; Gregory H. Wilmoth,
                     Senior Social Science-Analyst; and Ellen Wineholt, Senior Evaluator. From
                     the Office of Information Management and Technology: Delaney Branch,
                     Customer Service Representative; Kenneth L. Cooper, Electronic
                     Publishing Specialist; Gail M. Traynham, Lead Electronic Publishing
                     Specialist; and Wanda L. Okoro, Customer Service Representative.




                     Page 111                                          GAO/GGD-97-109 Results Act
Appendix II
Major Contributors and Acknowledgements




Page 112                                  GAO/GGD-97-109 Results Act
Appendix II
Major Contributors and Acknowledgements




Page 113                                  GAO/GGD-97-109 Results Act
Appendix II
Major Contributors and Acknowledgements




Page 114                                  GAO/GGD-97-109 Results Act
Related GAO Products


              Managing for Results: Analytic Challenges in Measuring Performance
              (GAO/HEHS/GGD-97-138, May 30, 1997).

              Agencies’ Strategic Plans Under GPRA: Key Questions to Facilitate
              Congressional Review (GAO/GGD-10.1.16, May 1997).

              GPRA:Managerial Accountability and Flexibility Pilot Did Not Work As
              Intended (GAO/GGD-97-36, Apr. 10, 1997).

              Performance Budgeting: Past Initiatives Offer Insights for GPRA
              Implementation (GAO/AIMD-97-46, Mar. 27, 1997).

              Measuring Performance: Strengths and Limitations of Research Indicators
              (GAO/RCED-97-91, Mar. 21, 1997).

              Managing for Results: Enhancing the Usefulness of GPRA Consultations
              Between the Executive Branch and Congress (GAO/T-GGD-97-56, Mar. 10,
              1997).

              Managing for Results: Using GPRA to Assist Congressional and Executive
              Branch Decisionmaking (GAO/T-GGD-97-43, Feb. 12, 1997).

              DOD’s GPRA   Implementation (GAO/NSIAD/GGD-97-65R, Jan. 31, 1997).

              Child Support Enforcement: Reorienting Management Toward Achieving
              Better Program Results (GAO/HEHS/GGD-97-14, Oct. 25, 1996).

              Information Technology Investment: Agencies Can Improve Performance,
              Reduce Costs, and Minimize Risks (GAO/AIMD-96-64, Sept. 30, 1996).

              Federal Downsizing: Better Workforce and Strategic Planning Could Have
              Made Buyouts More Effective (GAO/GGD-96-62, Aug. 26, 1996).

              Information Management Reform: Effective Implementation Is Essential
              for Improving Federal Performance (GAO/T-AIMD-96-132, July 17, 1996).

              Managing for Results: Key Steps and Challenges in Implementing GPRA in
              Science Agencies (GAO/T-GGD/RCED-96-214, July 10, 1996).

              Executive Guide: Effectively Implementing the Government Performance
              and Results Act (GAO/GGD-96-118, June 1996).




              Page 115                                              GAO/GGD-97-109 Results Act
           Related GAO Products




           Budget and Financial Management: Progress and Agenda for the Future
           (GAO/T-AIMD-96-80, Apr. 23, 1996).

           Budget Issues: Deficit Reduction and the Long Term (GAO/T-AIMD-96-66,
           Mar. 13, 1996).

           Managing for Results: Achieving GPRA’s Objectives Requires Strong
           Congressional Role (GAO/T-GGD-96-79, Mar. 6, 1996).

           GPRA   Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).

           Transforming the Civil Service: Building The Workforce of the Future,
           Results of A GAO-Sponsored Symposium (GAO/GGD-96-35, Dec. 20, 1995).

           Financial Management: Continued Momentum Essential to Achieve CFO
           Act Goals (GAO/T-AIMD-96-10, Dec. 14, 1995).

           Financial Management: Momentum Must Be Sustained to Achieve the
           Reform Goals of the Chief Financial Officers Act (GAO/T-AIMD-95-204, July 25,
           1995).

           Managing for Results: Status of the Government Performance and Results
           Act (GAO/T-GGD-95-193, June 27, 1995).

           Government Restructuring: Identifying Potential Duplication in Federal
           Missions and Approaches (GAO/T-AIMD-95-161, June 7, 1995).

           Government Reorganization: Issues and Principles (GAO/T-GGD/AIMD-95-166,
           May 17, 1995).

           Executive Guide: Improving Mission Performance Through Strategic
           Information Management and Technology (GAO/AIMD-94-115, May 1994).




(410135)   Page 116                                             GAO/GGD-97-109 Results Act
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested