oversight

Managing for Results: Regulatory Agencies Identified Significant Barriers to Focusing on Results

Published by the Government Accountability Office on 1997-06-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to Committee on Governmental
                Affairs, U.S. Senate, and Committee on
                Government Reform and Oversight, U.S.
                House of Representatives

June 1997
                MANAGING FOR
                RESULTS
                Regulatory Agencies
                Identified Significant
                Barriers to Focusing on
                Results




GAO/GGD-97-83
      United States
GAO   General Accounting Office
      Washington, D.C. 20548

      General Government Division

      B-275122

      June 24, 1997

      The Honorable Fred D. Thompson
      Chairman
      The Honorable John Glenn
      Ranking Minority Member
      Committee on Governmental Affairs
      United States Senate

      The Honorable Dan Burton
      Chairman
      The Honorable Henry A. Waxman
      Ranking Minority Member
      Committee on Government Reform and Oversight
      House of Representatives

      The Government Performance and Results Act of 1993 was enacted to
      improve the efficiency and effectiveness of federal programs by
      establishing a system to set goals for program performance and to
      measure results. In recognition of the potential difficulties inherent in
      establishing results-oriented goals and performance measures, the Act,
      which is referred to as “GPRA” or “the Results Act,” provided for a phased
      implementation period. Beginning in fiscal year 1994 and extending over
      several years, agencies are to develop strategic goals tied to agency
      missions, identify performance measures associated with those goals, and
      by fiscal year 1999 implement annual results-oriented performance reports
      linked to budget requests. In addition, in a March 4, 1995, memorandum,
      the President specifically directed the heads of regulatory agencies to
      change the way they measure both their agencies’ performance and the
      performance of their regulatory staff so as to focus on results.

      The Results Act required us to report to Congress on the prospects for the
      Act’s implementation governmentwide, and our recent report—The
      Government Performance and Results Act: 1997 Governmentwide
      Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997)—was
      developed in response to that requirement. In support of that broader
      assessment, we initiated an examination of the early progress toward
      results-oriented goal setting and performance measurement in selected
      federal regulatory agencies as they prepared to meet the first statutorily
      required reporting date of September 30, 1997. We considered “regulatory
      agencies” to include any agency (as defined in 5 U.S.C. 551[1]) that issues
      rules (as defined in 5 U.S.C. 551[4]). We examined regulatory agencies in
      part because of the administration’s initiative to better focus the



      Page 1                             GAO/GGD-97-83 Barriers to Managing for Results
B-275122




performance of regulatory agencies on achieving results through more
results-oriented identification of organizational objectives and individual
performance standards. The administration’s initiative differs from the
Results Act by adding the requirement that agencies change the way they
measure the performance of frontline regulators to focus on results. In
addition, because the President required regulatory agencies to start
focusing on results in March 1995, these agencies were expected to adopt
a results orientation sooner than the rest of the federal government.

The regulatory agencies we focused on were the Occupational Safety and
Health Administration (OSHA), the Federal Aviation Administration (FAA),
the Food and Drug Administration (FDA), the Internal Revenue Service
(IRS), and the Environmental Protection Agency (EPA).

We developed this report to describe the status of the five agencies’
attempts as of January 1, 1997, to focus on results and to ascertain what
had assisted or impeded these attempts. Our specific objectives were to
describe (1) the five agencies’ strategic goals and related program
performance measures as well as selected employee performance
standards as of January 1, 1997; (2) the extent to which agency officials
and we believed that these goals, program performance measures, and
employee performance standards focused on results; and (3) the aids and
barriers that officials in the agencies said they faced in attempting to focus
on results.

Focusing on results represents a new approach for both federal agencies
and Congress. Both agency management and congressional oversight have
traditionally focused on inputs (e.g., the number of dollars or staff allotted
to an agency); compliance with legislative or agency-developed rules; and
the accomplishment of outputs (e.g., the number of inspections
completed). Although a results orientation does not ignore these other
factors, it focuses attention on what outcomes agencies—including
regulatory agencies—are trying to accomplish.

Because a focus on results is new for federal agencies, and criteria for
definitively characterizing goals, measures, and standards as “results
oriented” were not yet developed, we adopted an approach that was
intended to stimulate discussion of the characteristics of results-oriented
goals, measures, and standards. We hypothesized that, if there were
differences between our characterizations and those of the agencies, these
differences and the explanations for them might be useful to the agencies
by illustrating how, for example, a goal that appears to be clearly results



Page 2                              GAO/GGD-97-83 Barriers to Managing for Results
                   B-275122




                   oriented and linked to an agency’s mission in the agency’s view may not
                   appear the same way to someone from outside the agency. By comparing
                   our characterizations with those of agency officials, we intended to foster
                   discussion and contribute to agencies’ efforts to develop results-oriented
                   goals, measures, and standards.


                   As would be expected in the early stages of implementing new and
Results in Brief   difficult initiatives, and consistent with our broader assessment of the
                   prospects for implementing the Results Act, we observed that some of the
                   five regulatory agencies were further along in the development of strategic
                   goals, program performance measures, and employee performance
                   standards than others. The agencies also varied in the degree to which
                   their goals, associated sets of program performance measures, and
                   employee performance standards that were in use as of January 1, 1997,
                   focused on results as judged by both agency officials and by us. In this
                   regard, it is important to note that although the Results Act was intended
                   to encourage agencies to focus their goals and measures on results, the
                   Act does not require that all of an agency’s goals or performance measures
                   be explicitly results oriented. Moreover, agencies were not required by the
                   Act to actually have strategic goals and related performance measures at
                   that point in time.1 Similarly, the President’s directive to orient frontline
                   regulators’ performance standards toward results does not explicitly
                   require that every standard be results oriented.

                   There were differences in the extent to which agency officials
                   characterized their goals, program performance measures, and employee
                   performance standards as “results oriented,” and the extent to which we
                   did. In general, agencies frequently concluded that their goals, measures,
                   and standards were more results oriented than we did. At the broader and
                   more conceptual level of strategic goals, there were relatively few
                   differences between agency officials’ assessments of the extent of results
                   orientation and ours. As we and they considered the more specific
                   program performance measures, differences between our views and the
                   agencies were greater, although both we and agency officials
                   characterized the agencies’ performance measures as less results oriented
                   than the strategic goals. The differences between the agencies’
                   assessments of the extent of results orientation and ours were greatest in
                   considering the more detailed performance standards for regulatory
                   personnel. (See apps. I through V for a detailed description of the extent to

                   1
                    Under GPRA, agencies are to submit strategic plans identifying strategic goals tied to agency missions
                   to Congress by September 30, 1997, and annual performance plans related to the strategic plans are to
                   be submitted to Congress after the President’s budget is submitted in February 1998.



                   Page 3                                          GAO/GGD-97-83 Barriers to Managing for Results
B-275122




which we and agency officials characterized the results orientation of
agencies’ goals, measures, and standards.) Since January 1997, the
agencies have continued their efforts to develop strategic goals and
performance measures to meet the requirements of the Results Act. In
commenting on a draft of this report, Office of Management and Budget
(OMB) officials noted that most of the agencies’ goals had changed since
our review and some of the revised goals and measures were more results
oriented. As we stated previously, it was our intent for this work to
contribute to such an outcome.

In enacting the Results Act, Congress realized that the transition to
results-oriented management would not be easy. For that reason, the Act
provided for a phased approach to implementation, during which time
agencies have been able to identify obstacles that need to be overcome
and some factors they found helpful. The factor that agency officials most
commonly said aided the establishment of a results orientation in their
agencies was the enactment of the Results Act. Specifically, they said that
GPRA-related requirements for outcome-oriented strategic goals and
performance measures by September 1997 forced them to focus their
attention on results more than they had prior to the passage of the Act.
Other factors that agency officials said aided their focus on results
included issue-specific legislation that cited results-oriented goals or
measures, feedback from stakeholders including Congress and other
interested parties, and top management leadership focused on achieving
results in agency performance.

While agency officials identified some aids to focusing their agencies on
results, they cited numerous barriers to their efforts to establish
results-oriented goals and measures. These barriers included significant
problems in identifying and collecting the data they needed to
demonstrate their agencies’ results. For example, IRS officials said that
there are significant difficulties involved in measuring the fair amount of
total taxes due, so it is hard to determine how much IRS should be
collecting. Other data-related barriers included the difficulty of
determining the causes of illnesses with long latency periods and
problems with data collection efforts, such as under- or inconsistent
reporting. Agencies also cited as a barrier the fact that diverse and
complex factors affect agencies’ results (e.g., business cycles,
technological innovations, or the need to deliver federal program
initiatives and thus achieve results through third parties), and their lack of
control or influence over those factors. For example, although FDA
regulations attempt to ensure that food is safe, FDA cannot control whether



Page 4                              GAO/GGD-97-83 Barriers to Managing for Results
B-275122




consumers ultimately ensure that food is stored and prepared properly.
Finally, agency officials observed that the long time period needed to see
results in some areas of federal regulation was a barrier to identifying and
managing toward those results in the framework of annual performance
plans and budgets. The impact of some agencies’ regulatory actions, such
as limiting exposure to a hazardous chemical, may not be evident for
years. These barriers suggest that the implementation of the Results Act in
a regulatory environment may prove more difficult in some cases than in
others.

Although we did not determine the validity of their claims or the extent to
which the relative lack of results orientation in the performance measures
was directly traceable to these barriers, the issues the agency officials
raised should not be summarily discounted. Yet, while solutions may not
be readily apparent, the potential long-term benefits merit efforts to
develop better measures. For example, although long-term results may be
difficult to use in an annual process, interim measures indicating the
degree of progress toward the achievement of those results can sometimes
be used to determine whether current policies are having the desired
effects. Likewise, output measures such as the number of inspections
conducted can be used when studies exist to demonstrate their
relationship to the results that agencies are attempting to accomplish.
Finally, agencies may be able to develop analytic techniques to enable
them to control for extraneous factors and isolate the impact the agencies
are having on such outcomes as injury rates or clean water.2

For GPRA to work in a regulatory environment, and for regulatory agencies
to meet their September 1997 deadlines, it is important that those agencies
begin to develop solutions to these measurement problems. Some agencies
have already begun to do so. For example, IRS has been working for some
time on improving its estimates of taxes owed, which might ultimately be
used to determine the effectiveness of the agency’s compliance programs.
Agencies will also need to learn from the experiences of other agencies
that have previously confronted and addressed similar issues. Our
Executive Guide, for example, identified key steps that agencies need to
take to implement the Results Act and provided case illustrations where
federal agencies had put these practices to use.3


2
 Although Congress will probably require agencies to provide information on their unique
contributions to outcomes as part of the oversight process, GPRA does not technically require
agencies’ performance measures to provide this kind of information.
3
 Executive Guide: Effectively Implementing the Government Performance and Results Act
(GAO/GGD-96-118, June 1996).



Page 5                                         GAO/GGD-97-83 Barriers to Managing for Results
             B-275122




             However, while we conceptually agree with the Act’s emphasis on
             managing for results, we also recognize that the development and practical
             application of results-oriented program performance measures in
             regulatory agencies will not always be done quickly or easily. For example,
             in order for an agency to isolate and measure its unique contribution to a
             result that is affected by a variety of factors outside of the agency, the
             agency must control for the effects of all of those nonagency factors. It is
             yet unclear whether that degree of analytical precision will always be
             possible.


             Federal regulations affect the lives of virtually every American, and have
Background   requirements that are designed to ensure, among other things, that
             workplaces, air travel, foods, and drugs are safe; that the nation’s air,
             water, and land are not polluted; and that the appropriate amount of taxes
             are collected. They are the means by which statutory public policy goals
             are put into effect to achieve desired economic and social benefits.
             Regulations also have billions of dollars of impact on businesses, state and
             local governments, and the economy as a whole. Therefore, it is important
             that decisionmakers and taxpayers understand what specific results
             federal regulatory agencies are attempting to achieve and how those
             agencies are measuring their progress toward achieving those results.

             Two efforts—one by Congress and the other by the Clinton
             administration—have required regulatory agencies to become more results
             oriented. Congress passed the Results Act in July 1993 with the intent of
             improving the efficiency and effectiveness of federal programs. Under the
             Act, federal agencies are to develop a strategic plan by September 1997,
             including general goals for the major functions and operations of the
             agency and a description of how the goals are to be achieved. Agencies are
             to submit annual performance plans to Congress after the President’s
             budget submission in early 1998. Beginning in fiscal year 1999, the Act
             requires that each agency report annually to the President and to Congress
             on its performance and its progress toward meeting its goals. Agencies’
             plans are to be used to support the President’s budget, and agencies’
             annual performance reports are intended to provide important information
             to agency managers, policymakers, and the public on what each agency
             accomplished with the resources it was given.

             Agencies’ strategic plans are to cover a period of at least 5 years and, as
             shown in figure 1, should contain the agency’s mission statement, a set of
             strategic goals (including outcome-related goals), a description of how the



             Page 6                             GAO/GGD-97-83 Barriers to Managing for Results
                                        B-275122




                                        agency intends to achieve these goals, and a description of how the
                                        agency’s performance goals will be related to its strategic goals. Strategic
                                        plans are intended to be the starting point for each agency’s performance
                                        measurement efforts. The annual performance plans are to use program
                                        performance measurement to reinforce the connection between the
                                        long-term strategic goals outlined in their strategic plans and the
                                        day-to-day activities of their managers and staff. The annual plans are to
                                        include performance goals for an agency’s program activities as listed in
                                        the budget, a summary of the necessary resources to achieve these goals,
                                        and the performance indicators that will be used to measure performance.
                                        Although these requirements are intended to encourage agencies to move
                                        toward results-oriented management, the Act recognizes that not all goals
                                        and measures will necessarily be results oriented.


Figure 1: Linking Results Act
Requirements to Day-to-Day Activities
                                            Strategic plans                 Annual performance                Day-to-day activities
                                            Mission statement               plans                             of managers and
                                            Strategic goals                 Performance goals                 staff
                                            How to achieve goals            Resource needs
                                                                            Performance
                                                                               indicators


                                        Source: GPRA.




                                        Getting regulatory agencies to focus on achieving desired results was also
                                        one aspect of the administration’s regulatory reinvention initiative. In
                                        Executive Order 12866, which was issued on September 30, 1993, the
                                        President set forth his philosophy for reforming the regulatory process.
                                        One of the “principles of regulation” in that order is that each regulatory
                                        agency should, to the extent feasible, “specify performance objectives,
                                        rather than the behavior or manner of compliance that regulated entities
                                        must adopt.” In a March 4, 1995, memorandum, the President followed up
                                        on this executive order by directing the heads of all federal regulatory
                                        agencies to, among other things, “change the way you measure the
                                        performance of both your agency and your frontline regulators so as to
                                        focus on results, not process and punishment.”4 Although the President’s
                                        requirement that agencies measure their performance in terms of results
                                        achieved is consistent with GPRA’s emphasis, measuring employees’

                                        4
                                         The President’s memorandum did not define “frontline regulators.” We defined “frontline regulators”
                                        as regulatory staff who work directly with the regulated community by enforcing regulations.



                                        Page 7                                         GAO/GGD-97-83 Barriers to Managing for Results
              B-275122




              performance in that manner is unique to the administration’s regulatory
              reinvention initiative. As previously noted, the President’s initiative does
              not explicitly require that all performance standards be results oriented.

              The five regulatory agencies have each been attempting to develop a
              results orientation for some time. EPA launched a National Goals Project in
              1992 to develop measurable long-range environmental goals and, more
              recently, attempted to increase the agency’s focus on results in response
              to an April 1995 National Academy of Public Administration report.5 OSHA
              and IRS began agencywide GPRA pilot projects in 1993.6 Although FAA and
              FDA started their agencywide efforts more recently, they had GPRA pilot
              efforts under way in some of their units. All five of the agencies said they
              are currently consulting with Congress and preparing for agencywide
              compliance as part of the Act’s governmentwide implementation
              beginning in September 1997. In addition, these five agencies are subject
              to the President’s March 4, 1995, memorandum directing them to focus on
              results.

              These agencies also have implemented various reforms in response to the
              administration’s regulatory reinvention initiatives. In June 1995, each
              agency reported on its actions regarding four tasks the President assigned
              to regulatory agencies in March 1995: eliminating or improving regulations,
              rewarding results, creating grassroots partnerships, and negotiating rules.
              The “rewarding results” sections of the reports focused on the status of
              agency program performance measures and steps taken to ensure that
              frontline regulators were evaluated on results rather than process or
              punishment measures. For example, EPA noted in its report that it had
              shifted its employee performance standards from specific process-driven
              numeric goals to definitions of performance in terms of the quality of the
              output.


              We selected the five regulatory agencies in this review—OSHA, FAA, FDA, IRS,
Scope and     and EPA—on the basis of their size (i.e., budget and personnel); the number
Methodology   of significant regulations they planned to finalize or complete from
              October 1, 1995, to October 1, 1996; and the opinions and




              5
              Setting Priorities, Getting Results: A New Direction for the Environmental Protection Agency, a
              National Academy of Public Administration Report to Congress, April 1995.
              6
                GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996) summarized our initial observations on
              selected agencies’ first attempts to report on their performance in the pilot phase of GPRA.



              Page 8                                         GAO/GGD-97-83 Barriers to Managing for Results
B-275122




recommendations of experts in the field of federal regulation.7 The
agencies selected were those that appeared most frequently across all of
these factors, although each agency selected may not have been at the top
of any particular factor. For example, OSHA is a relatively small agency that
typically issues few rules each year, but it was listed by most of the
experts as a major regulatory agency.

To describe the agencies’ strategic goals and program performance
measures, we first obtained copies of the goals and measures the agencies
had developed as of January 1, 1997. To describe the employee
performance standards for selected regulator positions at these agencies,
we first asked officials in each agency to identify frontline regulator
positions that they felt would fairly represent the actions of their agencies’
regulators. Using our definition of “frontline regulator”—regulatory staff
who work directly with the regulated community by enforcing
regulations—the agencies designated which positions they considered
“frontline regulator” positions. We then obtained copies of the
performance standards being used as of January 1, 1997, for the selected
positions. Because the Results Act does not cover employee performance
standards, we judged each set of performance standards according to the
expectation set by the President in his March 4, 1995, memorandum that
these standards should “focus on results, not process and punishment.”

To describe the extent to which we and agency officials believed that the
goals, measures, and performance standards were results oriented, we
developed and provided to each agency guidelines for agency officials to
use if they chose in characterizing the agencies’ goals, program
performance measures, and employee performance standards as results
oriented to a great extent, to some extent, or to little or no extent. We used
a similar framework to characterize the agencies’ goals, performance
measures, and employee performance standards. (See app. VI for a more
extensive discussion of the methodology we used to determine the extent
to which their goals, measures, and standards were results oriented.)

To respond to our third objective, we asked officials in each agency to
describe the aids and barriers that their agencies faced in attempting to
focus on results. We also reviewed related documents that described how
agencies developed results-oriented goals and program performance


7
 We contacted experts in the field of federal regulation from the following organizations and asked
them to identify which agencies they considered to be the major federal regulatory agencies: the
American Enterprise Institute, the U.S. Chamber of Commerce, the National Federation of
Independent Businesses, the Center for the Study of American Business, Citizens for a Sound
Economy, the National Performance Review, and the Small Business Administration.



Page 9                                          GAO/GGD-97-83 Barriers to Managing for Results
                       B-275122




                       measures and any problems or successes agencies had encountered. We
                       did not attempt to determine the validity of the factors that agency
                       officials said aided or hindered their efforts to becoming results oriented.

                       We did our work between April 1996 and March 1997 in Washington, D.C.,
                       in accordance with generally accepted government auditing standards. We
                       sent a draft of this report to the heads of each of the five agencies included
                       in this review and to the Director of OMB. Officials from each of these
                       agencies provided oral comments, which are summarized at the end of this
                       report and discussed in more detail in the appendix relating to that agency.
                       In addition, two of the agencies (OSHA and IRS) commented in writing.
                       These comments are reprinted in their entirety in appendix VII.


                       There were differences in the extent to which agency officials
Views on the Results   characterized their goals, program performance measures, and employee
Orientation of         performance standards as “results oriented,” and the extent to which we
Strategic Goals,       did. At the broader and more conceptual level of strategic goals, there
                       were relatively few differences between agency officials’ assessments of
Performance            the extent of results orientation and ours. However, as we and they
Measures, and          considered the more specific program performance measures, greater
                       differences existed between our views and those of the agencies. The
Performance            differences between the agencies’ assessments of the extent of results
Standards              orientation and ours were greatest in considering the more detailed
                       performance standards for selected frontline regulator positions. (App. VI
                       contains a detailed description of our scope and the methods that we used.
                       Apps. I through V contain detailed information regarding each of the five
                       regulatory agencies’ strategic goals, program performance measures, and
                       employee performance standards.)

                       We and agency officials most frequently agreed in our characterizations of
                       strategic goals. We both said that most of the agencies’ goals were at least
                       somewhat results oriented. Agency officials considered almost 70 percent
                       of the strategic goals to be greatly results oriented, and considered the
                       remaining goals to be results oriented to some extent. We characterized
                       about one-half of the agencies’ goals as greatly results oriented and about
                       one-third as somewhat results oriented. We considered the remaining
                       goals to be results oriented to little or no extent. An example of one of IRS’
                       strategic goals was as follows:

                       “Increase compliance: Our objective is to encourage and assist taxpayers to voluntarily file
                       timely and accurate returns and to pay on time, and, if taxpayers do not comply, to take




                       Page 10                                    GAO/GGD-97-83 Barriers to Managing for Results
B-275122




appropriate compliance actions. We do this to ensure we collect the proper amount of tax
at the least cost.”


Both we and IRS characterized this goal as results oriented to a great
extent because it was directly related to IRS’ mission of collecting tax
revenue, provided a clear definition of the results expected, and permitted
assessment of the agency’s programs.

As we and agency officials considered the more specific program
performance measures, differences between our views and the agencies’
were greater, although both we and agency officials characterized the
agencies’ performance measures as less results oriented than the strategic
goals. However, our characterizations of the performance measures were
not as positive as those of the agencies. Agency officials said they believed
that most of their sets of program performance measures were somewhat
results oriented, and characterized almost 40 percent of the measures as
greatly results oriented. However, we did not consider any of the sets of
program performance measures to be greatly results oriented. We agreed
with agency officials that most of the sets of measures were somewhat
results oriented, but we viewed over 40 percent of the measures as results
oriented to little or no extent. GPRA requires agencies to establish
performance indicators to be used in measuring or assessing the relevant
outputs, service levels, and outcomes of each program activity as part of
agencies’ performance plans.

For example, OSHA’s sole performance measure for its goal to “implement a
common sense strategy developed in partnership with stakeholders, for
rulemaking and alternative approaches to emerging and priority safety and
health issues” was “the percentage of rulemaking activity on the regulatory
agenda undertaken as negotiated rulemaking.” OSHA officials considered
this measure somewhat results oriented because they believed that using
the negotiated rulemaking process would improve the regulated entities’
acceptance of rules, which would lead to improved compliance and fewer
hazards, thereby leading to safer workplaces. However, we said this
measure was results oriented to little or no extent because (1) it was not
clear how use of the negotiated rulemaking process would make
workplaces safer; (2) the measure simply tracked the percentage of cases
in which this rulemaking process was used and, therefore, OSHA did not
have a set of measures that provided relevant information to gauge
progress toward the goal/mission; and (3) this measure alone did not
address other aspects of the goal, such as the use of alternatives to direct
regulation.



Page 11                                   GAO/GGD-97-83 Barriers to Managing for Results
                       B-275122




                       We and agency officials did not often agree about how results oriented the
                       agencies’ employee performance standards were. Agency officials
                       characterized over 80 percent of their sets of performance standards as
                       greatly results oriented, and considered the rest somewhat results
                       oriented. In contrast, we viewed 40 percent of the sets of standards as
                       results oriented to little or no extent. We characterized over one-third of
                       the standards as somewhat results oriented, and thought that less than
                       one-fourth of the performance standards were greatly results oriented.

                       For example, we generally disagreed with EPA officials’ characterization of
                       the agency’s performance standards for the criminal investigator and
                       environmental scientist positions. They said that all but two of the nine
                       sets of performance standards for these positions were greatly results
                       oriented and that those two were somewhat results oriented. However, we
                       characterized eight of these sets of performance standards as results
                       oriented to little or no extent because they were not linked with any
                       expected results. For example, EPA officials said that the “professional
                       attributes” standards for EPA’s criminal investigator position were results
                       oriented to a great extent because “presenting a positive well-balanced
                       presence when conducting investigations” resulted in good internal and
                       external working relationships. EPA officials said these professional
                       attributes were especially important in risky situations where the
                       investigator must be tactful in enforcing the law so that the situation does
                       not get out of hand. However, we characterized this set of standards as
                       results oriented to little or no extent because it was unclear how
                       maintaining “a good level of physical conditioning” or displaying “an
                       appearance and demeanor which is consistent with the position” were
                       linked to any of EPA’s strategic goals or its overall mission of
                       environmental protection.


                       Agency officials identified relatively few aids to focusing their agencies on
Agency Officials       results, but said they faced significant barriers in these efforts. Although a
Identified Some Aids   few of the officials’ comments described aids to establishing
but Numerous           results-oriented goals and employee performance standards, most of their
                       comments focused on barriers to developing results-oriented program
Barriers to Focusing   performance measures.
on Results
Aids to Focusing on    The factor that agency officials most commonly said aided the
Results                establishment of a results orientation in their agencies was the enactment
                       of GPRA. Specifically, they said that GPRA-related requirements for



                       Page 12                              GAO/GGD-97-83 Barriers to Managing for Results
                          B-275122




                          outcome-oriented strategic goals and performance measures by
                          September 1997 forced their agencies to pay more attention to results than
                          they had prior to passage of the Act. The officials also said that
                          congressional interest in GPRA’s implementation was an impetus for a
                          stronger focus on results.

                          Some agency officials cited other issue-specific legislation that they said
                          had also encouraged their agencies to increase their results orientation.
                          For example, agency officials said that the Aviation Security Act of 1990
                          and the Prescription Drug User Fee Act of 1992 contained specific
                          results-oriented goals and performance measures for FAA and FDA,
                          respectively.

                          Other aids that some agency officials also mentioned to focusing agencies
                          on results included: training provided to staff on how to develop program
                          performance measures; feedback from stakeholders (including Congress,
                          the public, industry, other federal agencies, state and local governments,
                          and interest groups); and top agency management leadership. For
                          example, both OSHA and FDA officials said that the heads of their agencies
                          strongly emphasized using results-oriented performance measures to
                          focus on results.


Barriers to Focusing on   Agency officials cited numerous barriers that their agencies faced in
Results                   becoming results oriented, some of which reportedly hindered them from
                          establishing results-oriented strategic goals or employee performance
                          standards. For example, some agency officials said that the conflicting
                          views of diverse stakeholders and inconsistent statutory requirements
                          were barriers to reaching agreement on agencies’ goals. In addition,
                          officials from IRS said that the agency is prevented by federal law from
                          using performance standards that evaluate its staff on the basis of
                          enforcement results.8

                          Most of the barriers that the officials mentioned focused on difficulties
                          they experienced in developing results-oriented program performance
                          measures. The barriers that they mentioned generally fell into three broad
                          categories: (1) problems in collecting program performance data, (2) the
                          diverse and complex factors that affect agencies’ results and the lack of
                          control agencies have over some of those factors, and (3) the long time
                          frames sometimes needed to see the results of agencies’ actions.

                          8
                          The Technical and Miscellaneous Revenue Act of 1988 contains the Omnibus Taxpayer Bill of Rights,
                          which prohibits IRS from evaluating employees on the basis of tax enforcement results.



                          Page 13                                       GAO/GGD-97-83 Barriers to Managing for Results
                              B-275122




Agency Officials Identified   Officials from all five agencies said that their agencies found it difficult to
Problems in Collecting        establish program performance measures because of various problems
Program Performance Data      they experienced in collecting necessary data. Some of those problems
                              involved identifying the data they needed to collect in order to
                              demonstrate their agencies’ results. For example, officials at several
                              agencies said that they sometimes were put in the difficult position of
                              trying to gather data to “prove a negative,” such as the number of aircraft
                              accidents that were avoided by FAA’s efforts to improve airport runways or
                              the number of people who did not get sick because FDA approved a new
                              drug or kept an unsafe product off the market. FAA officials said that they
                              may improve runways after a runway condition was identified as
                              contributing to an accident. However, because each aircraft accident
                              results from a unique combination of factors, it is difficult to determine the
                              results from the improvements made. EPA officials said that in some cases
                              they did not know how to measure environmental outcomes. For example,
                              EPA officials said that scientific uncertainty about the effects of exposures
                              to various levels of pollution and pesticides made it difficult to establish
                              appropriate measures for aquatic environments. IRS officials said that there
                              are significant difficulties in attempting to measure the fair amount of tax
                              due, so it is hard to determine how much IRS should be collecting.9 IRS
                              officials also said it was difficult to determine the impact of customer
                              services on tax compliance levels.

                              Officials from FDA and OSHA pointed out that uncertainties regarding the
                              causes of illnesses with long latency periods made it more difficult for
                              their agencies to collect accurate data on mortality and morbidity. OSHA’s
                              Deputy Director of Policy said it was particularly difficult to determine
                              whether an illness resulted from conditions in the workplace or from other
                              factors. For example, he said it is difficult to determine whether a worker
                              developed cancer after inhaling asbestos fibers at the workplace for 20
                              years, or from the cigarettes the worker smoked during that period. FDA
                              officials said that an ongoing study attempting to monitor and define the
                              sources of foodborne diseases is expected to take years to complete.10

                              Agency officials noted other problems that made obtaining
                              results-oriented performance data difficult. For example, OSHA officials
                              said that it was difficult to collect data and track trends in the number of

                              9
                               IRS has been working to improve estimates of taxes owed by conducting research projects and
                              building databases to allow researchers access to current data. Taxpayer Compliance: Analyzing the
                              Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9, 1997).
                              10
                               This study, known as the Foodborne Diseases Active Surveillance Network, or “FoodNet,” is a
                              collaborative project among the Centers for Disease Control and Prevention, seven state health
                              departments, the Department of Agriculture’s Food Safety Inspection Service, and FDA.



                              Page 14                                        GAO/GGD-97-83 Barriers to Managing for Results
                          B-275122




                          accidents and injuries that occurred in the construction industry because
                          construction site locations frequently change. Other agency officials said
                          that some outcome-oriented performance data must be collected through
                          voluntary self-reporting methods, which are often susceptible to
                          underreporting. For example, FDA officials said that people who become ill
                          from food poisoning may not recognize it as such, or even if they do, they
                          may not report it to a health official. Consequently, data on the number of
                          food poisoning incidents may severely undercount such occurrences.11
                          Similarly, one FAA official said that the data the agency collected on flight
                          “incidents” were self-reported, and so were likely to be an understatement
                          of the true number of such incidents.

                          Agency officials cited still other data-related problems. EPA officials said
                          that some states were not measuring certain environmental conditions
                          and, where data were being gathered, the states sometimes differed in the
                          measures they used. As a result, they said, EPA lacked nationwide
                          measures for some environmental conditions. FAA officials said that the
                          small number of aircraft accidents each year made analysis of the data
                          statistically unreliable. Therefore, they said that it was difficult to identify
                          trends in the factors contributing to aviation accidents.


Complexity and Lack of    Officials in four of the five agencies we reviewed said it was often difficult
Control Increased the     to sort out the interaction of diverse factors that affected the results that
Difficulty of Measuring   the agencies were trying to achieve. They said that this complexity,
                          combined with their agencies’ lack of control over many of these factors,
Results                   made it difficult to establish meaningful performance measures that
                          focused solely on their agencies’ contributions toward achieving their
                          goals. For example, one of EPA’s strategic goals was to achieve clean and
                          safe water, which is affected by numerous factors that are outside of EPA’s
                          control (e.g., natural causes including weather conditions). In addition, EPA
                          shares governmental responsibility to regulate water cleanliness and
                          safety with other federal agencies and with state and local governments.
                          As a result, EPA officials said it is difficult to establish performance
                          measures that gauge only the agency’s contribution to “clean and safe
                          water.”

                          Similarly, OSHA officials said that business cycles, the development of new
                          and safer technologies, changes in laws or regulations, and other factors
                          over which OSHA has no control affect the number of workplace accidents

                          11
                            See Food Safety: Information on Foodborne Illnesses (GAO/RCED-96-96, May 8, 1996), where we
                          found that most cases of foodborne illness go unreported and that existing data therefore understate
                          the extent of the problem.



                          Page 15                                         GAO/GGD-97-83 Barriers to Managing for Results
                             B-275122




                             and injuries occurring each year more than OSHA’s actions. For example,
                             OSHA’s Deputy Director of Policy said that recent declines in accident and
                             injury rates in Maine were a function of at least three factors: (1) the
                             partial closure of a major industrial facility, (2) a tightening of the state’s
                             worker compensation laws, and (3) OSHA’s programs in that state. He also
                             said that worker safety data are sometimes influenced by catastrophic
                             events, such as bombings or aircraft crashes, over which OSHA has no
                             control. Finally, OSHA officials pointed out that the 2,000 OSHA and state
                             inspectors can only visit a small fraction of the 6 million workplaces in the
                             nation. Therefore, they said that they did not think it would be appropriate
                             to hold OSHA accountable for increases in nationwide workplace injury or
                             fatality rates.

                             FAA officials said that the diversity of factors in the aviation system that
                             contribute to aircraft accidents presented a strong barrier to establishing
                             results-based performance measures. Although they and academic
                             researchers have studied aircraft safety issues for years, no techniques
                             have been developed that can isolate any one factor or group of factors
                             that can predict an accident. Because a combination of unique factors
                             generally causes aircraft accidents, an FAA official said that it is “virtually
                             impossible” to scientifically measure FAA’s effect on aircraft safety. FAA
                             officials also said that they could not control many of the factors that
                             contribute to aircraft accidents, such as how alert aircraft pilots feel
                             during a flight or the weather conditions. Therefore, they said that FAA can
                             only “influence” aviation safety.

                             Similarly, FDA officials said that many factors affect whether food and
                             drugs are safe, some of which are outside of the agency’s control. For
                             example, although FDA enforces regulations at various stages of the
                             production process to ensure that food is safe and drugs are safe and
                             effective, FDA officials said that consumers must ultimately ensure that
                             food is stored properly and drugs are taken only as prescribed. Although
                             FDA also regulates the provision of consumer information to ensure the
                             public is aware of such considerations, FDA officials said that these and
                             other public health issues are primarily influenced by individual behaviors
                             that FDA cannot control.


Agency Officials Said Long   Officials from several of the agencies said that it was difficult to develop
Time Frames Are Needed       annual performance measures and track their agencies’ yearly progress
to Judge Results             when the impact of some regulatory actions cannot be seen for years. For
                             example, OSHA’s Deputy Director of Policy said that the latency period



                             Page 16                              GAO/GGD-97-83 Barriers to Managing for Results
                              B-275122




                              between exposure to a hazardous substance, such as asbestos, and a
                              resulting illness can be 20 years or more. Therefore, the results of any
                              actions OSHA takes to reduce exposures to those substances may be
                              equally long in coming. EPA also said that a significant lag usually exists
                              between the elimination of a chemical hazard and any corresponding
                              change in illness rates. Similarly, FAA officials said that it takes several
                              years to determine the effect that new equipment and facilities have on the
                              air system’s safety. They also told us that FAA’s impact on safety is only
                              evident when examining data over a number of years. The officials said
                              that an increase in the number of accidents in 1 year does not necessarily
                              mean that FAA’s performance was worse, because the small number of
                              accidents is affected by a variety of factors that the agency does not
                              control, such as weather. Likewise, an IRS official said that focusing on
                              yearly progress may result in solving immediate problems but may divert
                              attention and reduce resources available for accomplishing longer term
                              goals. For example, he said that a focus on current compliance levels and
                              enforcement efforts could lessen the agency’s long-term efforts to increase
                              taxpayer compliance through education and outreach efforts.


Agency Officials Identified   Agency officials cited a variety of other barriers that they said hampered
Various Other Barriers to     their agencies’ efforts to focus on results. For example, some agency
Results Orientation           officials said that other agency priorities, some of which are also
                              presidential initiatives, were inconsistent with a results orientation
                              because they focused on processes rather than results. For example, an
                              EPA official said that the President’s call to reduce reporting burdens on
                              the public appeared to be somewhat contrary to the emphasis on gathering
                              data to measure progress toward results. OSHA officials said that their
                              agency measured the number of times the negotiated rulemaking process
                              was used because the OSHA Administrator wanted to emphasize the
                              importance of responding to a presidential initiative to use this process.

                              In addition, some agency officials said resource constraints were a barrier
                              to a results orientation in their agencies. For example, an OSHA official said
                              that they could only do one or two retrospective studies each year to
                              determine the effects of prior regulatory initiatives on workplace safety
                              and health. OSHA officials planned to use the retrospective studies to
                              provide baseline information on the results of prior regulatory actions.
                              (See app. I for an example of one of OSHA’s retrospective studies.) FDA
                              officials said that resource constraints made it difficult to divert resources
                              from high priority risks to establish a statistically valid baseline for
                              measuring industry compliance.



                              Page 17                             GAO/GGD-97-83 Barriers to Managing for Results
                         B-275122




                         Also, some agency officials said that their agencies found it difficult to
                         focus on results when they did not perceive that Congress shared that
                         orientation. They said some congressional attention continued to focus on
                         regulatory activities unrelated to results or that the congressional
                         committees they work with had not yet shown any interest in GPRA.


                         We obtained comments from each of the five regulatory agencies in this
Agency Comments          review as well as from OMB. The agencies generally agreed with the draft
and Our Evaluation       report’s overall message that regulatory agencies face difficulties in
                         focusing on results. Officials from FDA, for example, said that the report’s
                         discussion of the aids and barriers was fair, essentially complete, and
                         consistent with their perceptions. However, officials from three of the
                         agencies (OSHA, FDA, and IRS) commented that the “snapshot” approach we
                         used did not adequately capture the dynamic nature of the process of
                         establishing results-oriented goals, measures, and standards or the
                         progress the agencies had made in improving their results orientation.

                         We recognized in the draft report that our approach provided an interim
                         assessment at an early stage of implementing new and difficult initiatives
                         (i.e., GPRA and regulatory reinvention) and that the agencies’ goals,
                         program performance measures, and some of their employee performance
                         standards were changing. We did not have historical information that
                         would have allowed us to determine the extent to which goals and
                         performance measures that were in effect as of January 1, 1997,
                         represented advances over past goals and measures. However, we have
                         expanded the discussion of our methodology to further clarify our purpose
                         and to describe the related constraints in our scope.

                         The following section summarizes the comments that we received from
                         OMB. The comments from each of the agencies are summarized, and our
                         response included, at the end of each of the relevant appendixes (apps. I
                         through V).


OMB’s Comments and Our   We met with OMB officials, including the Deputy Administrator of the
Evaluation               Office of Information and Regulatory Affairs, on May 5, 1997, to obtain
                         their comments on the draft report. In general, the OMB officials thought
                         that our characterizations of the extent of results orientation of agencies’
                         strategic goals, program performance measures, and employee
                         performance standards were fair and reasonable. OMB officials also said
                         that they agreed with the draft report’s overall message that these



                         Page 18                             GAO/GGD-97-83 Barriers to Managing for Results
B-275122




regulatory agencies face significant barriers in focusing on results. They
said that the discussion of barriers was important and noted that meeting
GPRA requirements will probably take a number of iterations and
considerable time. The officials said that given the difficulties agencies
face, it was not surprising that there were gaps in agencies’ results
orientation.

The officials said that most of the agencies’ goals had changed since our
review. For example, they said that all of OSHA’s goals have changed and
that OSHA’s new goals are more results oriented than those we examined.
They said that OSHA continues to grapple with the question that OSHA
officials identified in our report about how to measure OSHA’s effects on
national illness, injury, and fatality rates. As stated in our draft report, we
described the agencies’ goals, measures, and standards as of January 1,
1997, to provide an early indication of the extent to which these agencies
were implementing new and difficult initiatives. The draft report
recognized that many of the agencies were making changes to their goals
and measures and some of their performance standards at the time of our
review.

OMB officials recommended that our draft report state more clearly that the
expectation that employee performance standards for frontline regulators
focus on results was directed by the President’s regulatory reinvention
initiative and not by the Results Act. The officials pointed out that there
are many factors interspersed between the two end points we reviewed
(i.e., between the agencies’ goals and measures and the standards that
frontline regulators are held accountable for as they implement the
agencies’ regulatory programs). These factors include statutory provisions,
agencies’ cultures, and the level of individual performance. We agree that
it is important to clearly communicate that the administration’s regulatory
reinvention initiative, not GPRA, set the expectation for results-oriented
employee performance standards. We placed greater emphasis on this
difference throughout the final report by clearly stating in each section on
performance standards that the administration’s regulatory reinvention
initiative set the expectation for results-oriented performance standards.


We are sending copies of this report to the heads of OSHA, FAA, FDA, IRS, and
EPA as well as to the Director of OMB, the Chairmen and Ranking Minority
Members of the House and Senate Appropriations and other interested
congressional committees, and others who may have an interest in this
matter. Copies will also be made available to others upon request.



Page 19                              GAO/GGD-97-83 Barriers to Managing for Results
B-275122




The major contributors to this report are listed in appendix VIII. If you
have any questions, please contact me on (202) 512-8676.




L. Nye Stevens
Director
Federal Management and Workforce Issues




Page 20                             GAO/GGD-97-83 Barriers to Managing for Results
Page 21   GAO/GGD-97-83 Barriers to Managing for Results
Contents



Letter                                                                                              1


Appendix I                                                                                         26
                      Views on the Results Orientation of OSHA’s Strategic Goals                   27
OSHA’s Mission,       Views on the Results Orientation of OSHA’s Program                           29
Strategic Goals,        Performance Measures
                      Views on the Results Orientation of OSHA’s Performance                       33
Performance             Standards for Selected Frontline Regulator Positions
Measures, and         OSHA’s Comments and Our Evaluation                                           34
Performance
Standards for
Selected Frontline
Regulator Positions
Appendix II                                                                                        36
                      Views on the Results Orientation of FAA’s Strategic Goals                    37
FAA’s Mission,        Views on the Results Orientation of FAA’s Program Performance                40
Strategic Goals,        Measures
                      Views on the Results Orientation of FAA’s Performance                        45
Performance             Standards for Selected Frontline Regulator Positions
Measures, and         FAA’s Comments and Our Evaluation                                            46
Performance
Standards for
Selected Frontline
Regulator Positions
Appendix III                                                                                       49
                      Views on the Results Orientation of FDA’s Strategic Goals                    50
FDA’s Mission,        Views on the Results Orientation of FDA’s Performance Goals                  52
Strategic and         FDA Replaced Performance Standards With a Customized                         58
                        Performance Evaluation Plan
Performance Goals,    FDA’s Comments and Our Evaluation                                            59
and Its Performance
Evaluation Plan




                      Page 22                           GAO/GGD-97-83 Barriers to Managing for Results
                          Contents




Appendix IV                                                                                            60
                          Views on the Results Orientation of IRS’ Strategic Objectives                61
IRS’ Mission, Strategic   Views on the Results Orientation of IRS’ Key Performance                     62
Objectives, Key             Indicators
                          Views on the Results Orientation of IRS’ Performance Standards               65
Performance                 for Selected Frontline Regulator Positions
Indicators, and           IRS’ Comments and Our Evaluation                                             68
Performance
Standards for
Selected Frontline
Regulator Positions
Appendix V                                                                                             70
                          Views on the Results Orientation of EPA’s Strategic Goals                    71
EPA’s Mission,            EPA Had No Program Performance Measures in Place at the                      73
Strategic Goals, and        Time of Our Review
                          Views on the Results Orientation of EPA’s Performance                        74
Performance                 Standards for Selected Frontline Regulator Positions
Standards for             EPA’s Comments and Our Evaluation                                            77
Selected Frontline
Regulator Positions
Appendix VI                                                                                            80

Scope and
Methodology
Appendix VII                                                                                           85

Comments From
OSHA and IRS
Appendix VIII                                                                                          89

Major Contributors to
This Report
Tables                    Table I.1: Differences Between Our and OSHA Officials’                       27
                            Characterizations




                          Page 23                           GAO/GGD-97-83 Barriers to Managing for Results
          Contents




          Table II.1: Differences Between Our and FAA Officials’                       37
            Characterizations
          Table III.1: Differences Between Our and FDA Officials’                      50
            Characterizations
          Table IV.1: Differences Between Our and IRS Officials’                       61
            Characterizations
          Table V.1: Differences Between Our and EPA Officials’                        71
            Characterizations
          Table VI.1: Initial Framework Provided to Agency Officials                   81
          Table VI.2: Frontline Regulator Positions and Number of Job                  84
            Elements as of January 1, 1997

Figures   Figure 1: Linking Results Act Requirements to Day-to-Day                      7
            Activities
          Figure I.1: Assumptions Linked Regulatory Approach to Results                28
          Figure III.1: Sets of FDA Performance Goals Varied By Strategic              52
            Goal and FDA Program


          Abbreviations

          CID        Criminal Investigation Division
          CSHO       Compliance Safety and Health Officer
          CSR        customer service representative
          ELA        establishment licensing agreement
          EPA        Environmental Protection Agency
          FAA        Federal Aviation Administration
          FDA        Food and Drug Administration
          FTD        federal tax deposits
          FTE        full time equivalent
          GAO        General Accounting Office
          GPRA       Government Performance and Results Act
          HACCP      Hazard Assessment Critical Control Points
          IDRS       Integrated Data Retrieval System
          IRC        Internal Revenue Code
          IRS        Internal Revenue Service
          MQSA       Mammography Quality Standards Act
          NDA        new drug approval
          OMB        Office of Management and Budget
          OSHA       Occupational Safety and Health Administration
          PDUFA      Prescription Drug User Fee Act
          PEP        performance evaluation plan
          PLA        product licensing agreement


          Page 24                           GAO/GGD-97-83 Barriers to Managing for Results
Page 25   GAO/GGD-97-83 Barriers to Managing for Results
Appendix I

OSHA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions
               OSHA’s mission is “to assure so far as possible every working man and
               woman in the nation safe and healthful working conditions.” OSHA
               administers portions of the Occupational Safety and Health Act of 1970,
               and is responsible for promulgating federal occupational safety and health
               regulations, enforcing occupational safety and health standards, and for
               approving plans for the development and enforcement of standards by
               state governments.

               We assessed the results orientation of OSHA’s strategic goals, performance
               measures, and employee performance standards according to the criteria
               presented in the scope and methodology appendix of this report. Because
               we characterized the extent to which each agency goal and set of
               performance measures was results oriented, the application of these
               criteria represented a higher standard of results orientation than required
               by the Government Performance and Results Act (referred to as “GPRA” or
               “the Results Act”). GPRA does not require that all of an agency’s goals or
               measures be results oriented. Also, the Results Act does not cover
               employee performance standards. We judged each set of performance
               standards according to the expectation set by the President in his March 4,
               1995, memorandum that these standards should “focus on results, not
               process and punishment.” Similar to the way in which GPRA views agencies’
               goals and performance measures, we do not believe that each such
               standard must be results oriented to achieve the President’s objective.

               We applied our criteria to the agencies’ written strategic goals, program
               performance measures, and employee performance standards and did not
               consider any supplementary or contextual information. In some cases, an
               agency’s goals, measures, and standards appeared results oriented on the
               surface, but closer examination and strict application of our criteria
               resulted in a lower characterization. We recognize that other information
               outside of the written statements and outside of our review may have
               existed that could have resulted in our characterizing certain goals,
               measures, or standards differently.

               Both we and OSHA officials characterized the extent of results orientation
               of OSHA’s strategic goals, program performance measures, and employee
               performance standards for selected frontline regulator positions. As
               shown in table I.1, there were differences in how we and OSHA officials
               viewed the agency’s goals, measures, and standards. We did not agree with
               agency officials that most of OSHA’s goals were somewhat results oriented.
               There was more disagreement between us and OSHA regarding the extent
               of results orientation of OSHA’s performance measures, in that OSHA



               Page 26                            GAO/GGD-97-83 Barriers to Managing for Results
                                         Appendix I
                                         OSHA’s Mission, Strategic Goals,
                                         Performance Measures, and Performance
                                         Standards for Selected Frontline Regulator
                                         Positions




                                         characterized them all as somewhat results oriented, but we characterized
                                         them all as results oriented to little or no extent. We agreed with OSHA
                                         officials, however, regarding the results orientation of the performance
                                         standards for Compliance Safety and Health Officers (CSHOs).


Table I.1: Differences Between Our and OSHA Officials’ Characterizations
                                           Number that OSHA officials considered               Number we considered results oriented
                                                       results oriented to                                     to
                                                  A great                      Little or no          A great                       Little or no
                                    Total          extent Some extent                extent           extent Some extent                 extent
Strategic goals                        3                0                 3                0                 0                1                   2
Performance measures                   3                0                 3                0                 0                0                   3
Performance standards                  3                2                 1                0                 2                1                   0
                                         Source: Agency and GAO analysis.




                                         As of January 1, 1997, OSHA had three strategic goals:1
Views on the Results
Orientation of OSHA’s                    1. “Eliminate hazards through offering partnerships or traditional
Strategic Goals                          enforcement;

                                         2. Implement a common sense strategy, developed in partnership with
                                         stakeholders, for rulemaking and alternative approaches to emerging and
                                         priority safety and health issues; and

                                         3. Focus OSHA programs and service delivery systems using internal and
                                         external partnerships to achieve results.”

                                            Although OSHA characterized each of these goals as results oriented to
                                            some extent, we believed that only the first goal met that standard. We
                                            considered the other two goals to be results oriented to little or no extent.

                                         OSHA’s    Deputy Director for Policy said that the agency’s first goal was
                                            somewhat results oriented because hazard elimination should lead to a
                                            reduction in workplace fatalities, injuries, and illnesses—the outcomes or
                                            results OSHA was trying to achieve. He described this goal as a surrogate
                                            for these results, and said that OSHA did not focus directly on reducing
                                            workplace fatalities, injuries, and illnesses because of the difficulty

                                         1
                                          These goals were actually developed for fiscal year 1996. OSHA officials said that they were revising
                                         the agency’s goals and performance measures and that we should use these goals to reflect their
                                         position as of January 1, 1997.



                                         Page 27                                          GAO/GGD-97-83 Barriers to Managing for Results
                                          Appendix I
                                          OSHA’s Mission, Strategic Goals,
                                          Performance Measures, and Performance
                                          Standards for Selected Frontline Regulator
                                          Positions




                                          created by other factors such as business cycles and technological
                                          advances which also affected these outcomes.2 We agreed that this goal
                                          was somewhat results oriented because the goal, although interim in
                                          nature, was plausibly related to the results OSHA was trying to achieve.
                                          However, it was not clear what results OSHA was expecting (i.e., whether
                                          the elimination of a few hazards was sufficient) or how progress would be
                                          assessed.

                                          OSHA  officials said that the agency’s second goal regarding implementation
                                          of a “common sense strategy” and “alternative approaches” was somewhat
                                          results oriented because, as figure I.1 shows, they believed that building
                                          partnerships into the rulemaking process would increase the acceptability
                                          of regulations, which would in turn improve regulatory compliance by
                                          both industry and employee groups, and ultimately make workplaces
                                          safer.3



Figure I.1: Assumptions Linked Regulatory Approach to Results


   Partnership            Improved rule             Improved                    Fewer                         Fewer
   techniques             acceptance by             compliance                  hazards                       injuries,
                          regulated                                                                           illnesses, and
                          entities                                                                            fatalities




                                          Source: Information from agency officials.




                                          However, we considered this goal to be results oriented to little or no
                                          extent because the statement was a strategy rather than an outcome that
                                          the agency was attempting to achieve.4 OSHA has not clearly established
                                          this goal’s link to OSHA’s mission and the goal did not define what results

                                          2
                                           Similarly, OSHA’s fiscal year 1995 report on performance measurement said that increased hazard
                                          identification and abatement was an “intermediate outcome,” capturing the short-term results of
                                          activities that reflect desirable changes in the workplace.
                                          3
                                           OSHA’s Deputy Director for Policy said that this goal may also help OSHA change its regulatory
                                          framework. For example, he said that in some instances regulations might not be the best way to
                                          achieve a desired result, so OSHA might issue guidelines instead of regulations.
                                          4
                                           Similarly, in Managing For Results: Achieving GPRA’s Objectives Requires Strong Congressional Role
                                          (GAO/T-GGD-96-79, Mar. 6, 1996), we said that OSHA’s goal to “focus resources on achieving
                                          workplace hazard abatement through strong enforcement and innovative incentive programs”
                                          represented a strategy rather than a measurable goal.



                                          Page 28                                         GAO/GGD-97-83 Barriers to Managing for Results
                              Appendix I
                              OSHA’s Mission, Strategic Goals,
                              Performance Measures, and Performance
                              Standards for Selected Frontline Regulator
                              Positions




                              OSHAhoped to achieve, and therefore did not permit assessment of the
                              agency’s progress. OSHA could not provide any evidence that these
                              approaches actually result in increased compliance or contribute to the
                              agency’s desired results of reducing fatalities, injuries, and illnesses in the
                              workplace.

                              OSHA  officials said that their third goal—to “focus OSHA programs and
                              service delivery systems using internal and external partnerships to
                              achieve results”—was established in reaction to the administration’s
                              desire for OSHA to improve its customer service by becoming more
                              responsive to its customers. The officials said that they considered the
                              goal somewhat results oriented because they believed that hazards would
                              be addressed more quickly through partnerships and hazard reduction
                              would ultimately result in fewer workplace fatalities, injuries, and
                              illnesses. However, we believed that this goal was results oriented to little
                              or no extent because it was unclear (“focus OSHA programs and ...
                              systems”) and it utilized a strategy (“partnerships”) to achieve unspecified
                              “results.” Therefore, it was also impossible to assess OSHA’s progress
                              toward achieving any results.


                              OSHA officials said that the sets of program performance measures OSHA
Views on the Results          had adopted for the three strategic goals were each results oriented to
Orientation of OSHA’s         some extent. However, we believed that each of these sets of performance
Program Performance           measures was results oriented to little or no extent.

Measures
Views on the Results          As of January 1, 1997, the performance measures that OSHA used to assess
Orientation of Measures       progress toward its first strategic goal, “to eliminate hazards through
Associated With OSHA’s        offering partnerships or traditional enforcement,” were:
Strategic Goal to         •   “Percentage of programmed inspections that result in the identification of
“Eliminate Hazards”           significant hazards;
                          •   Percentage of initial consultation visits that result in the identification of
                              significant hazards;
                          •   Median time for hazard abatement for inspections;
                          •   Median time for hazard abatement for consultation visits;
                          •   Median time to respond to requests from smaller employers in high hazard
                              industries and operations;
                          •   Average cost per person trained through the targeted training grant
                              program; and




                              Page 29                                      GAO/GGD-97-83 Barriers to Managing for Results
    Appendix I
    OSHA’s Mission, Strategic Goals,
    Performance Measures, and Performance
    Standards for Selected Frontline Regulator
    Positions




•   Average evaluation score on courses offered by the OSHA Training Institute
    and Education Centers.”

    OSHA  officials characterized this set of program performance measures as
    somewhat results oriented because they believed that all of the measures
    in the set were related to the goal of eliminating hazards. However, OSHA’s
    Deputy Director for Policy also said that OSHA is working to make the
    measures even more results oriented, and has taken other actions to
    measure the agency’s effects and focus on achieving particular outcomes.
    For example, he said that OSHA has done some retrospective analyses to
    identify the results of their efforts. In one such study OSHA found that
    within 5 years of the promulgation of OSHA’s general industry lead standard
    in 1978, the number of workers in the lead smelting and battery
    manufacturing industries who had high-lead concentrations in their blood
    dropped by 66 percent, from about 19,000 to about 6,500. The Deputy
    Director also told us that OSHA is in the process of redesigning its field
    offices, and that each redesigned office has a local problem-solving
    component with outcome measures. One of these redesigned field offices
    is using data from the state registry on lead poisoning to measure OSHA’s
    impact on the health and safety of construction workers who paint bridges
    (which involves the removal of lead paint). The field office is conducting a
    training program, and next year they will determine whether there are
    significantly fewer reports of lead poisoning in the state registry.

    We believed that the set of program performance measures associated
    with OSHA’s goal of eliminating hazards was results oriented to little or no
    extent because, although they initially appeared to be results oriented, the
    measures were not clearly and unambiguously related either to the goal or
    to the agency’s desired results of reducing workplace fatalities, injuries,
    and illnesses. The set of measures did not address all aspects of the goal
    and also did not provide relevant information for assessing progress
    toward the goal or the agency’s mission.

    For example, the first two measures—the percentage of inspections or
    consultations that resulted in the identification of significant hazards—did
    not reflect the actual number of significant hazards found, only how often
    inspections or consultations revealed at least one such hazard. One
    inspection that identified 50 significant hazards would be recorded the
    same as another inspection that identified only 1 significant hazard. The
    measures did not reflect the number of hazards that were eliminated in
    those workplaces or whether the OSHA inspections and consultations
    resulted in any reduction in fatalities, injuries, or illnesses. Furthermore, it



    Page 30                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix I
OSHA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




was unclear whether changes in these measures would indicate that
workplaces are becoming more or less hazardous. For example, if
30 percent of the inspections conducted in 1995 resulted in the
identification of a significant hazard, and that figure rose to 50 percent in
1996, it would not be clear whether the 20 percentage point increase
reflected increased diligence and effectiveness on the part of OSHA
inspectors, better targeting of OSHA inspections, or that workplaces had
become more hazardous.

The third and fourth measures—“median time for hazard abatement” for
inspections and consultation visits—were also not as results oriented as
they appeared. These measures reflected the time OSHA gave companies to
abate a hazard, not the actual amount of time it took until the hazard was
abated or even whether the hazard ever was abated.5 Similarly, the fifth
measure gauged the time it took OSHA to respond to requests from smaller
employers in high hazard industries and operations and did not measure
whether hazards were eliminated or whether workplace fatalities, injuries,
or illnesses were reduced as a result of their response. Also, OSHA’s 1995
performance report showed that smaller employers in the manufacturing
sector had the lowest percentage of inspections with significant hazards of
any size group. Therefore, this measure appears to have focused on the
group of employers with the fewest significant hazards.

We also did not believe that the training-related measures that OSHA
associated with this goal were clearly related to hazard abatement. For
example, although the average cost per person trained through the
targeted training grant program may be a useful measure in other respects
(e.g., a cost-effectiveness analysis), the resultant dollar figure may bear
little relationship to the elimination of hazards or reduction of workplace
fatalities, injuries, or illnesses. Similarly, average evaluation scores on
OSHA training may have had relevance in terms of selecting one course or
instructor over another, but those scores may be totally unrelated to the
agency’s goal or desired results.6



5
 OSHA’s 1995 performance report indicated that OSHA might move to measure actual time to abate
hazards rather than the amount of time assigned for abatement.
6
 However, the provision of relevant training (not the cost per person trained or average evaluation
scores) can be a legitimate and results-oriented measure for hazard abatement or even the reduction of
fatalities, injuries, and illnesses when the linkage between training and the result is demonstrated. For
example, OSHA’s fiscal year 1995 Performance Measurement Report cited a study examining the
effects of OSHA-funded training on logging safety provided to 250 companies in Idaho and Washington
during fiscal year 1993 and the first quarter of fiscal year 1994 that discovered that accident claims and
lost work days per year declined after the safety training was provided.



Page 31                                          GAO/GGD-97-83 Barriers to Managing for Results
                             Appendix I
                             OSHA’s Mission, Strategic Goals,
                             Performance Measures, and Performance
                             Standards for Selected Frontline Regulator
                             Positions




Views on the Results         OSHA  used one performance measure, “the percentage of rulemaking
Orientation of the Measure   activity on the regulatory agenda undertaken as negotiated rulemaking,” to
for OSHA’s Second Goal on    assess the agency’s progress in meeting its second strategic goal, to
                             “implement a common sense strategy, developed in partnership with
Rulemaking and               stakeholders, for rulemaking and alternative approaches to emerging and
Alternative Approaches       priority safety and health issues.”

                             A negotiated rulemaking brings together interested parties who draft a
                             proposed rule that then goes through notice and comment procedures.7
                             OSHA officials characterized this measure as results oriented to some
                             extent because, as noted previously and depicted in figure I.1, they
                             believed that using partnership techniques such as negotiated rulemaking
                             would ultimately make workplaces safer. However, OSHA officials could
                             provide no evidence to support this assumption. Therefore, we considered
                             this measure to be results oriented to little or no extent because it was not
                             clearly and unambiguously related to OSHA’s mission and would not
                             provide relevant information to permit assessment of the agency’s
                             programs. Also, according to OSHA’s fiscal year 1995 Performance
                             Measurement Report, the negotiated rulemaking process was applicable to
                             only a portion of proposed rules, and therefore this measure could not be
                             used for all of OSHA’s regulatory activities. Finally, the measure did not
                             address all aspects of the goal in that it did not gauge any “alternative
                             approaches” to regulation.


Views on the Results         OSHA   used one performance measure, “the median time to respond to
Orientation of the Measure   complaints,” to measure progress toward its third goal, to “focus OSHA
for OSHA’s Third Goal to     programs and service delivery systems using internal and external
                             partnerships to achieve results.” OSHA officials characterized this measure
Focus OSHA Programs and      as results oriented to some extent because they said it reflected one of
Service Delivery Systems     OSHA’s responses to the administration’s reinvention initiatives to improve
                             its customer service.8 We characterized the measure as results oriented to
                             little or no extent because it did not clearly relate to OSHA’s goal or mission
                             and did not provide relevant information regarding progress toward the
                             goal. The measure focused on how long it took OSHA to respond to a


                             7
                              See the Negotiated Rulemaking Act of 1990, 5 U.S.C. 561-570.
                             8
                              OSHA has changed its policy so that when it receives a complaint, it determines whether to
                             immediately inspect or investigate the alleged hazards. If OSHA determines an investigation is
                             appropriate, the agency will telephone the employer, describe the alleged hazards, and follow up with
                             a fax or a letter. The employer can respond in kind, identifying any problems found and noting
                             corrective actions taken or planned. An adequate response generally negates the need for an
                             inspection. According to OSHA officials, lower priority complaints can be resolved through the
                             phone/fax procedure, therefore allowing a more effective use of OSHA inspection resources.



                             Page 32                                         GAO/GGD-97-83 Barriers to Managing for Results
                        Appendix I
                        OSHA’s Mission, Strategic Goals,
                        Performance Measures, and Performance
                        Standards for Selected Frontline Regulator
                        Positions




                        complaint, not how long it took to abate the hazard that originated the
                        complaint or even whether the hazard was eliminated. Also, there were no
                        other measures regarding the contributions of internal or external
                        partnerships to any results achieved, so the goal was incompletely
                        addressed.


                        In contrast to strategic goals and program performance measures, the
Views on the Results    administration’s regulatory reinvention initiative, not GPRA, set the
Orientation of OSHA’s   expectation for agencies to focus on results when measuring the
Performance             performance of their frontline regulators. Two of OSHA’s frontline regulator
                        positions were safety inspectors and health inspectors.9 Although both
Standards for           positions were covered by the same set CSHO standards, the standards
Selected Frontline      varied somewhat by OSHA regional office. OSHA officials said we could
                        select the standards used in any of its regions, so we used the San
Regulator Positions     Francisco regional office’s CSHO standards in this study.

                        As of January 1, 1997, OSHA had three CSHO job elements, with one
                        descriptive performance standard for each element. They were:

                        1. “Building Partnerships and Improving the Agency’s Effectiveness.
                        Participates in planning and executing safety and health program activity
                        that serves our customer and improves OSHA’s mission of assuring worker
                        protection.

                        2. On Site Inspection. Performs inspections in accordance with agency
                        policy and organizational goals. The scope and focus of the inspection is
                        based on the type of inspection, the hazards found and the effectiveness of
                        the employer’s safety and health program.

                        3. Case Management. Plans, prepares, organizes and documents
                        enforcement and non-enforcement cases in accordance with agency policy
                        and organizational goals.”

                        Each standard also had a set of “performance measurement
                        considerations” that elaborated the standard. For example, the
                        considerations for the first standard included “promotes voluntary
                        compliance efforts” and “maintains and improves safety/health knowledge
                        and skills.”

                        9
                         Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not
                        define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated
                        community by enforcing regulations. OSHA officials identified these two frontline regulator positions
                        for this review.



                        Page 33                                         GAO/GGD-97-83 Barriers to Managing for Results
                     Appendix I
                     OSHA’s Mission, Strategic Goals,
                     Performance Measures, and Performance
                     Standards for Selected Frontline Regulator
                     Positions




                     OSHA officials characterized the performance standard for the “building
                     partnerships” and “on-site inspection” job elements as results oriented to a
                     great extent because they believed they were directly linked with OSHA’s
                     mission to assure worker protection. OSHA officials characterized the
                     performance standard for the “case management” element as somewhat
                     results oriented because it was less directly linked to the agency’s mission.

                     We agreed with OSHA’s characterizations of these three performance
                     standards. We viewed the first two performance standards as greatly
                     results oriented because they explicitly referred to OSHA’s mission and
                     goals, and a majority of the associated performance measurement
                     considerations addressed results. For example, in “building partnerships”
                     the CSHOs were expected to identify and design intervention tools “that will
                     effect the greatest reduction of illness, injury, and death.” We agreed with
                     OSHA officials that the third performance standard regarding case
                     management was somewhat results oriented because the standard and the
                     associated performance measurement considerations were only indirectly
                     related to results. For example, both the standard and the considerations
                     were primarily focused on internal procedures such as documenting cases,
                     demonstrating knowledge of OSHA standards, and applying professional
                     judgment—actions that may or may not lead to safer workplaces.


                     We met with OSHA officials, including the Acting Director of Policy, on
OSHA’s Comments      April 25, 1997, to receive their comments. OSHA officials said that they
and Our Evaluation   generally agreed with the draft report’s overall message regarding the
                     difficulties regulatory agencies have experienced in becoming results
                     oriented. They provided one technical correction, which was included in
                     the final report.

                     Both in our meeting with OSHA officials and in OSHA’s written comments
                     (see app. VII), OSHA emphasized that its approach to focusing on results
                     began with the development of processes and data systems, not the
                     development of results-oriented goals. They said that agencies can
                     legitimately be in different stages of results orientation because of
                     differences in the approaches agencies took to focusing on results.
                     Therefore, the officials said they believed that any comparison of the
                     information presented about the agencies would be misleading and
                     inappropriate. They also recommended that the final report more
                     prominently reflect that not all goals need to be results oriented.




                     Page 34                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix I
OSHA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




We agree that agencies can use different approaches to focus on results
and that they can legitimately vary in the degree to which they are results
oriented at a particular point in time. As reflected in the draft report, we
expected such variance in the early stages of implementing GPRA and the
regulatory reinvention initiative and explicitly recognized the interim
nature of the characterizations we made as agencies continued to change
their goals, measures, and standards. We removed tables in the draft
report that were intended only to summarize information across agencies
because OSHA’s comments led us to realize that the tables could have the
unintended effect of leading a reader to compare the agencies with one
another and to assume that all goals, measures, and standards are required
to be results oriented.

OSHA officials also said that the discussion in the draft report about the
barriers agencies face in focusing on results should be highlighted more
prominently in the results-in-brief section. In response, we included more
information in the results-in-brief section of the final report describing the
barriers agencies identified and also summarized the information about
the agencies’ goals, measures, and standards in a more general manner.

In its comment letter, OSHA said that its fiscal year 1995 Performance
Measurement System Framework had long-term outcome measures of
injuries, illnesses, and fatalities prevented that were not recognized in the
draft report. However, the letter also said that “these long term outcomes
may not be immediately evident and therefore were not included in OSHA’s
fiscal year 1996 goals and objectives.” Therefore, OSHA apologized “for any
misunderstanding or miscommunication of information related to our
measures.” The scope of this review included only OSHA’s fiscal year 1996
goals and measures, not those for fiscal year 1995. When asked to confirm
that our information on OSHA’s fiscal year 1996 goals and program
performance measures was accurate and complete, both before doing our
analysis and again in January 1997, OSHA officials did not identify any
additional measures for our review.




Page 35                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix II

FAA’s Mission, Strategic Goals, Performance
Measures, and Performance Standards for
Selected Frontline Regulator Positions
               FAA’s mission is to provide “a safe, secure, and efficient global aerospace
               system that contributes to the national security and the efficiency of U.S.
               aerospace” and be “responsive to the dynamic nature of customer needs,
               economic conditions, and environmental concerns.”

               We assessed the results orientation of FAA’s strategic goals, performance
               measures, and employee performance standards according to the criteria
               presented in the scope and methodology appendix of this report. Because
               we characterized the extent to which each agency goal and set of
               performance measures was results oriented, the application of these
               criteria represented a higher standard of results orientation than required
               by the Results Act. The Act does not require that all of an agency’s goals or
               measures be results oriented. Also, GPRA does not cover employee
               performance standards. We judged each set of performance standards
               according to the expectation set by the President in his March 4, 1995,
               memorandum that these standards should “focus on results, not process
               and punishment.” Similar to the way in which the Act views agencies’
               goals and performance measures, we do not believe that each such
               standard must be results oriented to achieve the President’s objective.

               We applied our criteria to the agencies’ written strategic goals, program
               performance measures, and employee performance standards and did not
               consider any supplementary or contextual information. In some cases an
               agency’s goals, measures, and standards appeared results oriented on the
               surface, but closer examination and strict application of our criteria
               resulted in a lower characterization. We recognize that other information
               outside of the written statements and outside of our review may have
               existed that could have resulted in our characterizing certain goals or sets
               of measures and standards differently.

               Both we and FAA officials characterized the extent of results orientation of
               FAA’s strategic goals, program performance measures, and employee
               performance standards for selected frontline regulator positions. As
               shown in table II.1, there were differences in how we and FAA officials
               viewed the agency’s goals, measures, and standards. Generally, we agreed
               that most of FAA’s strategic goals were results oriented. There was some
               disagreement between us and FAA regarding the extent of results
               orientation of FAA’s performance measures, although we both
               characterized the measures as less results oriented than the goals. There
               was a sharp disagreement in ours and the officials’ characterizations of
               FAA’s performance standards. FAA officials said that five of the six sets of
               performance standards were greatly results oriented. However, we



               Page 36                             GAO/GGD-97-83 Barriers to Managing for Results
                                          Appendix II
                                          FAA’s Mission, Strategic Goals,
                                          Performance Measures, and Performance
                                          Standards for Selected Frontline Regulator
                                          Positions




                                          characterized one set as greatly results oriented, one set as somewhat
                                          results oriented, and the remaining four sets as results oriented to little or
                                          no extent.


Table II.1: Differences Between Our and FAA Officials’ Characterizations
                                             Number that FAA officials considered               Number we considered results oriented
                                                        results oriented to                                     to
                                                    A great                     Little or no          A great                        Little or no
                                     Total           extent Some extent               extent           extent Some extent                  extent
Strategic goals                         9                  3               6                0                 3                5                    1
                     a
Performance measures                    6                  4               2                0                 0                4                    2
Performance standards                   6                  5               1                0                 1                1                    4
                                          a
                                              Three of FAA’s strategic goals had no program performance measures.

                                         Source: Agency and GAO analysis.




                                         We and FAA officials reviewed 9 of FAA’s 10 strategic goals in place as of
Views on the Results                     January 1, 1997:1
Orientation of FAA’s
Strategic Goals                          1. “System safety—zero accidents. Eliminate accidents2 and incidents3 in
                                         aviation and protect public safety and property in space transportation
                                         systems by targeting the most critical areas.

                                         2. Security—zero incidents. Eliminate security incidents in the aviation
                                         system.

                                         3. Human factors safety—zero accidents. Eliminate human factors as a
                                         causal factor in accidents and incidents.




                                          1
                                           FAA officials said that one of the agency’s strategic goals—“promote active, vital, and efficient
                                          aviation and space transportation industries”—would soon be eliminated because of a change in the
                                          agency’s mission contained in the Federal Aviation Reauthorization Act of 1996. This act changed
                                          FAA’s mission by eliminating the dual mandate of promoting both the “development” and “safety” of
                                          U.S. air commerce. Therefore, we decided to eliminate it from our review.
                                          2
                                           An aircraft “accident” is defined by regulation as “an occurrence associated with the operation of an
                                          aircraft which takes place between the time any person boards the aircraft with the intention of flight
                                          until all such persons have disembarked, and in which any person suffers death or serious injury as a
                                          result of being in or upon the aircraft or by direct contact with the aircraft or anything attached
                                          thereto, or in which the aircraft receives substantial damage.”
                                          3
                                          An aviation “incident” is defined by regulation as “an occurrence other than an accident associated
                                          with the operation of an aircraft which affects or could affect the safety of operations.”



                                          Page 37                                          GAO/GGD-97-83 Barriers to Managing for Results
Appendix II
FAA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




4. System capacity. Meet the system capacity needs for air and space
transportation safely and efficiently through near-term actions targeted at
specific problems and a long-term comprehensive program of research,
planning, and investment matching user needs.

5. Achieve safe, secure, and efficient global air and space transportation
systems by promoting international cooperation and harmonization.

6. Provide strong leadership regarding the environmental impacts of
aviation and commercial space transportation.

7. Manage resources and investments to control costs and increase
productivity, efficiency, and effectiveness, while promoting safety and
customer service.

8. Transform FAA into the model Federal workplace.

9. Establish and maintain mutual lines of communications with the public
and with employees to promote understanding, awareness, and
cooperation, and to serve the interests of the traveling public.”

FAA officials characterized the first three goals as results oriented to a
great extent and the six remaining goals as somewhat results oriented. We
considered the first three goals to be results oriented to a great extent, five
goals somewhat results oriented, and one goal results oriented to little or
no extent.

According to FAA officials, the agency’s first strategic goal was results
oriented to a great extent because it established a clear definition of the
results expected (“zero accidents”). An FAA official said the agency
recognized that, although it was possible to achieve zero accidents for
commercial aviation during 1 year, in the long term an accident would
likely occur. However, the official said that this goal was created in
response to the administration’s desire to ensure aviation safety and
express no tolerance for aviation accidents. He also said that FAA
management did not want to imply that having accidents would be
acceptable or that anything other than achieving zero accidents was
desirable. Finally, the official said that the goal reflected a quality
management perspective of setting high expectations and striving for the
best results that could be achieved. We agreed that the goal was greatly
results oriented because it was directly linked to FAA’s mission of
providing a safe and secure aerospace system, contained a clear definition



Page 38                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix II
FAA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




of expected results, and permitted assessment of the agency’s progress
toward the goal.

FAA officials said the agency’s second strategic goal of eliminating security
incidents (i.e., no hijackings or explosions aboard aircraft) was also
results oriented to a great extent because, like the first goal, FAA
deliberately established a clear outcome-related goal of eliminating
security incidents in the aviation system. We agreed that this goal was
greatly results oriented because it provided a clear definition of what
outcomes were expected, was measurable, and was unambiguously linked
to FAA’s mission.

FAA’s third strategic goal, “human factors safety—zero accidents,” was a
subset of the first goal, but FAA established it as a separate goal to enable
the agency to isolate human factors from other causes of accidents. FAA
officials characterized this goal as results oriented to a great extent. We
agreed with that assessment because the goal was directly linked to FAA’s
mission, was clear in the results expected, and permitted assessment of
the agency’s progress toward the result.

FAA officials characterized the fourth strategic goal (“to meet system
capacity”) as somewhat results oriented because of the difficulties they
experienced in defining aviation system “capacity.” An FAA official said
that defining system capacity solely in terms of the increased number of
aircraft traveling through the nation’s airspace or the number of additional
airport runways would not be sufficiently inclusive. He said that any goal
relating to system capacity would need to reflect other considerations as
well, such as delays, capital expenditures for aviation facilities, and the
needs of the aviation community. We agreed with FAA that the fourth
strategic goal was somewhat results oriented because, although it was
clearly linked to FAA’s mission, it did not clearly identify the results FAA
was trying to achieve regarding system capacity, nor did it provide a
means of assessing the agency’s progress in achieving the results. We also
agreed with FAA officials’ characterization that the fifth strategic goal, to
“achieve safe, secure, and efficient global air and space transportation
systems,” was somewhat results oriented. Although the goal was linked to
the agency’s mission, it did not clearly define the results expected and did
not provide a means of assessing those results.

FAA officials characterized FAA’s sixth strategic goal, to “provide strong
leadership regarding the environmental impacts of aviation and
commercial space transportation,” as results oriented to some extent.



Page 39                                      GAO/GGD-97-83 Barriers to Managing for Results
                       Appendix II
                       FAA’s Mission, Strategic Goals,
                       Performance Measures, and Performance
                       Standards for Selected Frontline Regulator
                       Positions




                       However, they noted that the term “strong leadership” was subject to a
                       variety of interpretations by interest groups and other stakeholders, such
                       as aircraft manufacturers, airline operators, and the general public. We
                       characterized this goal as somewhat results oriented because, although it
                       was linked to FAA’s mission, it did not provide any description of what
                       would result from the “strong leadership” that FAA was to provide or how
                       to assess the effects of this leadership.

                       FAA characterized the seventh (“managing resources”), eighth
                       (“transforming FAA into a model Federal workplace”), and ninth
                       (“establishing and maintaining lines of communications”) strategic goals
                       as somewhat results oriented because they were written in broad terms
                       and not easily measurable. We agreed with FAA officials that all three
                       strategic goals were not measurable, although two of the goals (“managing
                       resources” and “establishing and maintaining lines of communications”)
                       were linked to the agency’s mission. As such, we agreed with FAA officials’
                       characterization that these two goals were somewhat results oriented. We
                       considered the goal of transforming FAA into a model federal workplace as
                       results oriented to little or no extent because it was not linked to FAA’s
                       mission and did not provide a clear definition of what results were
                       expected or permit assessment of the agency’s progress.


                       FAA officials said that their agency’s program performance measures were
Views on the Results   evolving and they planned to modify these measures to better focus on
Orientation of FAA’s   results before submitting them to the Office of Management and Budget
Program Performance    (OMB) for review in September 1997 as part of GPRA implementation.
                       However, as of January 1, 1997, FAA had sets of program performance
Measures               measures for six of its nine strategic goals. FAA officials said that FAA was
                       still developing measures for the agency’s seventh and ninth goals. FAA
                       officials did not identify any performance measures for the agency’s fifth
                       goal.

                       FAA officials said that the sets of program performance measures for the
                       first four strategic goals were results oriented to a great extent and the
                       sets of measures for the sixth and eighth strategic goals were somewhat
                       results oriented. We considered three of the first four sets of measures as
                       somewhat results oriented and the sets of measures for the third and
                       eighth strategic goals to be results oriented to little or no extent. We
                       agreed with FAA officials that the set of measures for the sixth strategic
                       goal was somewhat results oriented.




                       Page 40                                      GAO/GGD-97-83 Barriers to Managing for Results
                               Appendix II
                               FAA’s Mission, Strategic Goals,
                               Performance Measures, and Performance
                               Standards for Selected Frontline Regulator
                               Positions




Views on the Results           The program performance measures FAA used for both its first (“system
Orientation of the             safety—zero accidents”) and third (“human factors safety—zero
Measures for FAA’s First       accidents”) strategic goals were:
and Third Goals            •   “percent reduction in accident rate;
                           •   number of fatalities per million passenger miles;
                           •   percent reduction in operational error4 rates;
                           •   percent reduction of near mid-air collisions;5 and
                           •   percent reduction in runway incursions.”6

                               FAA officials said they characterized this set of program performance
                               measures as greatly results oriented because the measures enabled the
                               agency to track accidents and fatalities and were directly related to the
                               FAA’s first and third strategic goals. We considered this set of measures to
                               be somewhat results oriented with regard to FAA’s first goal because it did
                               not address all aspects of the goal. For example, although part of the goal
                               was the elimination of all aviation incidents, the incident-related measures
                               focused on only certain types of incidents (operational errors, midair
                               collisions, and runway incursions) and not other types (e.g.,
                               weather-related incidents). Also, the accident rate measure (the number of
                               accidents per 100,000 flight hours) was not clearly related to the goal of
                               “zero accidents.” For example, the absolute number of accidents could
                               actually increase while the percentage of accidents per 100,000 flight
                               hours decreased because of a large increase of flight hours. We considered
                               this set of measures to be results oriented to little or no extent with regard
                               to FAA’s third goal—human factors safety. The measures were not clearly
                               linked to the goal of eliminating human factors in accidents and incidents
                               because none of them isolated human factors in the results they were
                               measuring. For example, the number of fatalities could be a function of
                               factors totally unrelated to human factors.




                               4
                                An operational error is an occurrence attributable to an element of the air traffic control system that
                               results in less than the applicable separation minima between two or more aircraft, or between an
                               aircraft and terrain or obstacles and obstructions. Obstacles include vehicles, equipment, and/or
                               personnel on runways.
                               5
                                A near midair collision is an incident associated with the operation of an aircraft in which a possibility
                               of collision occurs as a result of proximity of less than 500 feet to another aircraft, or a report is
                               received from a pilot or flight crew member stating that a collision hazard existed between two or
                               more aircraft.
                               6
                                A runway incursion is any occurrence at an airport involving an aircraft, vehicle, person, or object on
                               the ground that creates a collision hazard or results in loss of separation with an aircraft taking off,
                               intending to take off, landing, or intending to land.



                               Page 41                                           GAO/GGD-97-83 Barriers to Managing for Results
                                Appendix II
                                FAA’s Mission, Strategic Goals,
                                Performance Measures, and Performance
                                Standards for Selected Frontline Regulator
                                Positions




Views on the Results            FAA’sprogram performance measures for FAA’s second strategic goal
Orientation of the              (“security—zero incidents”) were:
Measures for FAA’s
                            •   “Percent improvement in detection of complex improvised explosive
Security Goal                   devices through all threat vectors;7
                            •   percent increase in the system’s ability to sustain 100 percent compliance
                                with contingency measures;8 and
                            •   percent decrease in vulnerability to existing or new threats.”9

                                FAA officials said they characterized this set of performance measures as
                                results oriented to a great extent because the measures were quantitative
                                and reflected improvements made to the security system. Although we
                                agree that this set of measures was related to the agency’s security system
                                goal, we considered it somewhat results oriented because it did not
                                measure all aspects of system security. For example, an FAA official said
                                that the set’s first measure will initially measure only the detection
                                performance of “threat vectors” with screening equipment (e.g., x-ray
                                machines) and may not include other threat sources. Also, the set’s first
                                measure refers only to explosive devices and not other weapons such as
                                handguns.


Views on the Results            The program performance measures for FAA’s fourth strategic goal
Orientation of the              regarding system capacity were:
Measures for FAA’s System
                            •   “Percent increase in Instrument Flight Rule/Visual Flight Rule capacity
Capacity Goal                   ratio;10




                                7
                                 This measure refers to an air carrier or airport security system’s ability to detect explosive devices
                                attempting to reach a target through different approaches, such as within a checked bag, cargo, or with
                                a person.
                                8
                                 “Contingency measures” refers to security countermeasures to be taken by air carriers and aviation
                                industry personnel as directed by FAA through its emergency authority. These countermeasures are
                                previously agreed-upon actions based on assessments made by law enforcement and intelligence
                                agencies. FAA assesses a security system’s implementation of a countermeasure through surveillance,
                                interviews, audits, and testing.
                                9
                                 This measure refers to FAA’s airport-by-airport vulnerability assessments and use of risk assessment
                                methodologies and tools to determine the reduction of risk when mitigating or closing vulnerabilities
                                in airports’ security systems.
                                10
                                  The Instrument Flight Rule/Visual Flight Rule capacity ratio estimates an airport’s capacity to handle
                                air traffic during good and poor weather conditions.



                                Page 42                                          GAO/GGD-97-83 Barriers to Managing for Results
                              Appendix II
                              FAA’s Mission, Strategic Goals,
                              Performance Measures, and Performance
                              Standards for Selected Frontline Regulator
                              Positions




                          •   percent increase in number of Global Positioning System non-precision
                              and precision approaches;11
                          •   percent reduction in ground movement time;12
                          •   percent reduction in system induced delays per number of operations;
                          •   percent reduction in equipment caused delays per number of operations;
                              and
                          •   percent reduction in weather induced delays per number of operations.”

                              FAA officials characterized this set of program performance measures as
                              greatly results oriented because it focused on how the end user actually
                              experiences the aviation system. However, FAA officials also told us that
                              the agency recognized that this set of measures did not include indicators
                              for some capacity issues that were of concern to the aviation system’s
                              users, including the system’s flexibility, accessibility, and predictability.13
                              We concluded that the measures do not provide a means of assessing how
                              FAA will “meet the system capacity needs” in the near and long term as
                              described in the system capacity strategic goal. Therefore, we considered
                              this set of measures to be somewhat results oriented.


Views on the Results          The program performance measures for FAA’s sixth strategic
Orientation of Measures       goal—“provide strong leadership regarding environmental impacts of
for FAA’s Environmental       aviation and commercial space transportation”—were as follows:
Responsibility Goal           For industry operations:

                          •   “Percent reduction in number of residents exposed to significant airport
                              noise levels.”

                              For FAA operations:

                          •   “Percent of facilities which have received safety inspections,
                          •   percent of facilities which have received environmental inspections,
                          •   number of fuel tanks brought into compliance with EPA requirements, and
                          •   number of facilities audited for energy conservation.”


                              11
                               The Global Positioning System is a satellite-based navigation system which can provide aircraft with
                              extremely precise position information.
                              12
                               The “percent reduction in ground movement time” measures the amount of time aircraft taxi in and
                              out of the airport, a means of measuring a delay in the aviation system.
                              13
                               FAA has recently developed a new set of measures for system capacity. An FAA official said that this
                              new set of measures will be included in FAA’s Air Traffic Services’ Fiscal Year 1999 Business
                              Performance Plan.



                              Page 43                                         GAO/GGD-97-83 Barriers to Managing for Results
                               Appendix II
                               FAA’s Mission, Strategic Goals,
                               Performance Measures, and Performance
                               Standards for Selected Frontline Regulator
                               Positions




                               FAA officials characterized this set of program performance measures as
                               somewhat results oriented because they believed that several of the
                               measures assessed FAA’s compliance with environmental laws rather than
                               being focused on results. However, because FAA’s environmental goal
                               lacked a clear definition of the outcomes the agency was trying to achieve,
                               it was difficult for us to determine whether this set of measures was
                               clearly linked to the goal. Although some of the measures may relate to the
                               goal (e.g., “number of fuel tanks brought into compliance with EPA
                               requirements”), we considered this set of measures somewhat results
                               oriented because many of the measures did not permit assessment of
                               progress toward the agency’s goal. For example, the percent of facilities
                               that have received safety or environmental inspections may bear no
                               relation to whether those facilities are in compliance with OSHA or EPA
                               requirements.



Views on the Results           The program performance measures for FAA’s eighth strategic
Orientation of the             goal—“transform FAA into the model Federal workplace”—were:
Measures Associated With
                           •   “Conduct an employee attitude survey of FAA employees and register a
the Goal on the FAA            70 percent positive response on overall job satisfaction;
Workplace                  •   implement national actions to help manage diversity and transform the FAA
                               into a model Federal workplace; and
                           •   through demonstrated actions and accomplishments, the FAA’s workforce
                               demographics will reflect positive trends or increases with regard to
                               workforce representation for minorities, women, and people with
                               disabilities.”

                               FAA officials characterized this set of measures as somewhat results
                               oriented because it did not provide a “measurable outcome” of what FAA
                               was trying to achieve. We agreed with FAA officials that the set did not
                               provide a measurable outcome. We also believe that the set did not
                               address all aspects of the strategic goal, such as the amount of support for
                               innovation or risk taking within the agency. Perhaps most importantly,
                               though, the set of performance measures and its related strategic goal
                               were not linked to the agency’s mission. Therefore, we considered this set
                               of measures to be results oriented to little or no extent.




                               Page 44                                      GAO/GGD-97-83 Barriers to Managing for Results
                            Appendix II
                            FAA’s Mission, Strategic Goals,
                            Performance Measures, and Performance
                            Standards for Selected Frontline Regulator
                            Positions




                            In contrast to strategic goals and program performance measures, the
Views on the Results        administration’s regulatory reinvention initiative, not GPRA, set the
Orientation of FAA’s        expectation for agencies to focus on results when measuring the
Performance                 performance of their frontline regulators. Two of FAA’s frontline regulator
                            positions were air traffic control specialists and aviation safety
Standards for               inspectors.14 Air traffic control specialists are responsible for issuing
Selected Frontline          control instructions and advice to pilots in the vicinity of airports as well
                            as guiding aircraft operations en route along the airways. Aviation safety
Regulator Positions         inspectors administer and enforce regulations concerning the
                            airworthiness of aircraft and aircraft systems and the competence of
                            pilots, mechanics, and other aviation personnel.

                            FAA used six sets of performance standards for these positions. FAA
                            officials characterized five sets as results oriented to a great extent and
                            one set as somewhat results oriented. They said the standards were linked
                            to outcomes specified in FAA’s strategic goals, but that the linkage was
                            through several steps and not always clearly apparent. We characterized
                            only one set of standards as results oriented to a great extent, one set as
                            somewhat results oriented, and four sets as results oriented to little or no
                            extent.


Views on the Results        As of January 1, 1997, the air traffic control specialist position in en route
Orientation of              centers and terminals had three job elements and associated sets of
Performance Standards for   performance standards: “operating methods and procedures,”
                            “communications,” and “training.” An FAA official characterized the first
Air Traffic Control         two sets of performance standards as results oriented to a great extent
Specialists                 because following proper air traffic procedures and effective
                            communication with pilots were essential in providing safe aircraft
                            landings and departures. We agreed with FAA’s characterization of the first
                            set of standards because the standards were clearly related to the agency’s
                            mission, described the position’s purpose (provide for “the safe, orderly
                            and expeditious flow of air traffic”), and provided criteria for evaluating
                            performance (“in accordance with procedures, directives and letters of
                            agreement”). The second set of standards on “communications” required
                            incumbents to be “cooperative, courteous and tactful,” use “prescribed
                            phraseology,” and listen “effectively,” but did not clearly link these
                            standards to the agency’s safety goal or state what results would occur as


                            14
                              Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not
                            define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated
                            community by enforcing regulations. We selected these two positions for review from the frontline
                            regulator positions FAA officials identified.



                            Page 45                                         GAO/GGD-97-83 Barriers to Managing for Results
                             Appendix II
                             FAA’s Mission, Strategic Goals,
                             Performance Measures, and Performance
                             Standards for Selected Frontline Regulator
                             Positions




                             a result of these behaviors. Therefore, we considered this set of standards
                             results oriented to little or no extent.

                             The “training” standards required employees taking instruction to
                             complete classes “in a timely manner,” and said that employees providing
                             instruction should do so “objectively.” FAA officials said that this set of
                             standards was somewhat results oriented because training was essential
                             to support the duties of the position. We considered the set of training
                             standards to be results oriented to little or no extent because it was not
                             clearly related to the agency’s goals or mission and did not provide a
                             description of the expected results from the training or criteria for
                             evaluating results.


Views on the Results         As of January 1, 1997, the aviation safety inspector position (for
Orientation of the           operations/maintenance/avionics with responsibility for general aviation
Performance Standards for    aircraft) had three job elements and associated sets of performance
                             standards: “technical administration,” “certification,” and “surveillance.”
Aviation Safety Inspectors   FAA officials characterized all three sets of performance standards as
                             results oriented to a great extent. However, we considered the first two
                             sets of standards results oriented to little or no extent because they were
                             not clearly linked to FAA’s mission or any of its strategic goals. For
                             example, the technical administration standards required inspectors to
                             conduct investigations “thoroughly” and “in a timely manner,” but did not
                             say what mission-related results would follow from those reviews. We
                             considered the set of performance standards for “surveillance” to be
                             somewhat results oriented because it described actions that were more
                             clearly linked to FAA’s mission. For example, inspectors were required to
                             identify and communicate safety issues that require special emphasis to
                             the appropriate management levels. However, the set of standards was
                             unclear as to what outcomes incumbents should achieve as a result of
                             those actions.


                             We met with FAA officials, including several program managers, on April
FAA’s Comments and           29, 1997, to receive their comments. FAA officials generally agreed with the
Our Evaluation               draft report’s overall message regarding the barriers regulatory agencies
                             face in focusing on results and particularly noted how difficult it was to
                             determine the unique effect that FAA had on aviation system safety. They
                             indicated that the agency’s goals and performance measures had changed
                             since the time of our review. They also said that differences between their
                             and our characterizations of the goals, measures, and standards in use as



                             Page 46                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix II
FAA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




of January 1, 1997, were partially due to the fact that we did not use the
same criteria to determine the extent of result orientation. FAA officials
also provided some technical corrections, which were included in the final
report.

The FAA officials said that FAA had a set of performance goals that
connected the agency’s fiscal year 1996 strategic goals to its fiscal year
1996 program performance measures. They contended that there was a
closer linkage between FAA’s program performance measures and its
performance goals than existed between the measures and FAA’s strategic
goals (the subject of our review). They also said that FAA’s strategic goals
were broad goals for the aviation system as a whole, whereas its
performance goals related more specifically to the agency’s performance.
We reviewed FAA’s strategic goals and the program performance measures
FAA officials identified that related to these goals. In order to be consistent
in reviewing the five regulatory agencies, we reviewed the strategic goals
of each of these agencies.15

FAA officials pointed out that the performance standards for the aviation
safety inspector frontline regulator position were written before the
President’s March 4, 1995, memorandum calling for performance
standards to be results oriented. They also believed that individual
regulators understood the relationship of their performance standards to
FAA’s mission. Finally, the FAA officials said they thought it was unrealistic
to expect the standards, by themselves, to reflect a results orientation
when there were other supporting documents that linked the activities of
the regulators to the agency’s mission.

The President’s memorandum directed the heads of all federal regulatory
agencies to change the way they measure the performance of their
frontline regulators so as to focus on results, not process and punishment.
Therefore, a group of performance standards for a frontline regulator
position that does not in some way focus on results would not comply
with the memorandum, regardless of when the standards were written.
Also, Office of Personnel Management guidance states that goal setting is
the first component of effective performance management.16 According to
this guidance, research has shown that specific, hard goals result in higher
performance than vague or abstract goals, or no goals at all. Therefore, the

15
  The only performance goals included in this report were those of FDA because these goals served as
the agency’s program performance measures.
16
 “Improving Customer Service Through Effective Performance Management,” Office of Personnel
Management Performance Management Practitioner Series PMD-04, May 1996.



Page 47                                        GAO/GGD-97-83 Barriers to Managing for Results
Appendix II
FAA’s Mission, Strategic Goals,
Performance Measures, and Performance
Standards for Selected Frontline Regulator
Positions




President’s initiative focusing standards on results could improve
regulators’ performance in obtaining desired results. Although other
supporting documents may have linked regulators’ performance standards
to the agency’s mission, we focused solely on agencies’ performance
standards to ensure a fair and balanced approach across all of the
agencies.




Page 48                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix III

FDA’s Mission, Strategic and Performance
Goals, and Its Performance Evaluation Plan

                   FDA’s
                       mission is to “protect, promote and enhance the health of the
                   American people.” FDA seeks to achieve this mission by ensuring that:

               •   “Foods are safe, wholesome, and sanitary; human and veterinary drugs,
                   biological products, and medical devices are safe and effective; cosmetics
                   are safe; and electronic products that emit radiation are safe;
               •   regulated products are honestly, accurately, and informatively
                   represented; and
               •   products are in compliance with the law and FDA regulations;
                   noncompliance is identified and corrected; and any unsafe or unlawful
                   products are removed from the marketplace.”

                   According to the fiscal year 1996 FDA Almanac, the agency supports its
                   mission through the enforcement of 13 major laws enacted throughout the
                   century, from the Food and Drugs Act of 1906 to the Dietary Supplement
                   Health and Education Act of 1994.

                   We assessed the results orientation of FDA’s strategic and performance
                   goals according to the criteria presented in the scope and methodology
                   appendix of this report.1 Because we characterized the extent to which
                   each strategic goal and related sets of performance goals were results
                   oriented, the application of these criteria represented a higher standard of
                   results orientation than required by GPRA. The Results Act does not require
                   that all of an agency’s goals or measures be results oriented.

                   We applied our criteria to FDA’s written strategic and performance goals
                   and did not consider any supplementary or contextual information. In
                   some cases, the agency’s goals appeared results oriented on the surface,
                   but closer examination and strict application of our criteria resulted in a
                   lower characterization. We recognize that other information outside of the
                   written statements and outside of our review may have existed that could
                   have resulted in our characterizing certain goals differently.

                   Both we and FDA officials characterized the extent of results orientation of
                   FDA’s strategic and performance goals. We did not assess any employee
                   performance standards because, as of January 1, 1997, FDA was using
                   generic performance standards which varied from employee to employee.
                   Therefore, no group of standards applicable to a frontline regulator
                   position existed. As shown in table III.1, we and FDA officials differed

                   1
                    Instead of using performance measures as the means of measuring their progress on the agency’s
                   strategic goals, FDA used sets of “performance goals.” These “performance goals,” as defined by
                   GPRA, are the target level of performance (either output or outcome) expressed as a tangible,
                   measurable objective, against which actual achievement will be compared.



                   Page 49                                        GAO/GGD-97-83 Barriers to Managing for Results
                                          Appendix III
                                          FDA’s Mission, Strategic and Performance
                                          Goals, and Its Performance Evaluation Plan




                                         somewhat in how we characterized the results orientation of FDA’s
                                         strategic and performance goals. Generally, we thought that FDA’s goals
                                         were less results oriented than the FDA officials did. However, we agreed
                                         with FDA officials that most of their performance goals were somewhat
                                         results oriented.


Table III.1: Differences Between Our and FDA Officials’ Characterizations
                                             Number that FDA officials considered           Number we considered results oriented
                                                        results oriented to                                 to
                                                   A great                   Little or no        A great                     Little or no
                                     Total          extent Some extent             extent         extent Some extent               extent
Strategic goals                         3               3               0               0               0               3                 0
Performance goalsa                      9               2               7               0               0               5                 4
                                          a
                                          FDA had nine sets of performance goals that it used to assess progress toward its three
                                         strategic goals, with different sets of measures for different FDA programs. These performance
                                         goals contained the program performance measure in the goal statement. In addition,
                                         performance standards were not included because, as of January 1, 1997, the agency was using
                                         performance standards that were customized for individual employees.

                                         Source: Agency and GAO analysis.




                                          As of January 1, 1997, FDA had three strategic goals:
Views on the Results
Orientation of FDA’s                     1. Pre-market: “Make timely and cost effective pre-market review
Strategic Goals                          decisions while assuring product safety and efficacy;”

                                         2. Post-market: “Strengthen assurance that products on the market or
                                         about to enter the market are safe;” and

                                         3. Capacity: “Strengthen the nation’s capacity to manage health risks
                                         associated with FDA-regulated products.”

                                         Although FDA characterized all three strategic goals as results oriented to a
                                         great extent, we viewed all three goals as somewhat results oriented.

                                          FDA    officials said that the agency’s first strategic goal regarding pre-market
                                             review decisions was results oriented to a great extent. By establishing a
                                             review process, they said that FDA can ensure the safety of FDA-regulated
                                             products and, therefore, promote and protect the health of the American
                                             public. Although we agreed that “assuring product safety and efficacy” is
                                             related to the agency’s mission to promote and protect public health, we




                                         Page 50                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix III
FDA’s Mission, Strategic and Performance
Goals, and Its Performance Evaluation Plan




considered the goal to be somewhat results oriented because it did not
establish a clear definition of the results expected through its pre-market
review decisions (e.g., what a “timely” or “cost effective” decision was)
and, therefore, did not permit assessment of the agency’s progress toward
achieving these results.2 Also, the major emphasis in the goal is on making
timely and cost-effective decisions, which is not part of the agency’s stated
mission.3

FDA officials said the agency’s second strategic goal regarding the safety of
products on the market or about to enter the market was greatly results
oriented because the agency’s inspection process provided a safeguard to
prevent the distribution of potentially harmful food, drugs, and other
FDA-regulated products. They also said that by preventing the distribution
of these harmful products, FDA protected the health of the American
public. We again characterized this goal as somewhat results oriented
because it did not provide a clear definition of its intended results or
permit assessment of the agency’s progress toward achieving the results.
For example, the goal did not identify how or the extent to which FDA
expects to “strengthen assurance” that products are safe, or even whose
“assurance” would be strengthened.

Finally, FDA officials said the agency’s third strategic goal—to “strengthen
the Nation’s capacity to manage health risks associated with FDA-regulated
products”—was results oriented to a great extent because it referred to
the agency’s internal processes to maintain the science capacity of its
staff. Through its science capacity, FDA officials said the agency can be
assured that it has valid and reliable science to support its review
processes and inspections—the activities by which FDA protects and
promotes the public’s health. However, we characterized this goal as
somewhat results oriented because, although it was tenuously linked to
FDA’s mission, it was unclear whose “capacity” would be strengthened,
what “capacity” means, or what “manage health risks” means. If, as FDA
officials said, the agency’s capacity was to be strengthened through
performance of this goal, it was unclear how and the degree to which FDA
would strengthen its capacity or the effect that strengthening would have
on the nation’s or the agency’s ability to manage health risks.


2
 In October 1992, Congress passed the Prescription Drug User Fee Act (PDUFA) to provide FDA with
additional resources to expedite the review of human drug applications. PDUFA incorporated by
reference specific goals and time frames for FDA to review and act on drug applications set forth in
letters from the FDA Commissioner to the Chairmen of the House Committee on Energy and
Commerce and the Senate Committee on Labor and Human Resources. However, FDA did not link its
strategic goal with these time frames or any other criteria defining what would be considered timely.
3
 PDUFA did not make any specific alterations to FDA’s mission.


Page 51                                         GAO/GGD-97-83 Barriers to Managing for Results
                                        Appendix III
                                        FDA’s Mission, Strategic and Performance
                                        Goals, and Its Performance Evaluation Plan




                                        As of January 1, 1997, FDA was using nine sets of performance goals for its
Views on the Results                    three strategic goals. As table III.2 shows, different sets of performance
Orientation of FDA’s                    goals were used not only for the different goals but also for many of the
Performance Goals                       different FDA programs. Post-market performance goals were not
                                        applicable to two of the agency’s programs. However, the set of
                                        performance goals for the capacity strategic goal was applicable to all of
                                        the agency’s programs except for toxicological research. FDA officials
                                        characterized two of these sets of performance goals as results oriented to
                                        a great extent and seven sets as somewhat results oriented. However, we
                                        considered five sets of FDA’s performance goals as somewhat results
                                        oriented and four sets as results oriented to little or no extent.


Figure III.1: Sets of FDA Performance
Goals Varied by Strategic Goal and
FDA Program                                                                                                                    Strategic goals




                                                                                                                                       t
                                                                                                                                      rke
                                                                                                                          et
                                                                                                                       ark




                                                                                                                                                 y
                                                                                                                                  ma


                                                                                                                                             cit
                                                                                                                    e-m




                                                                                                                                             pa
                                                                                                                                st-

                                                                                                                                            Ca
                                                                                                                               Po
                                                                                                                  Pr
                                        All FDA programsa                                                        na          na
                                        Biologics and drugs                                                                  na         na

                                        Food safety and applied nutrition                                                               na
                                        Devices and radiological health                                                                 na
                                        Veterinary medicine                                                                             na
                                        Toxicological research                                                               na         na
                                        Note 1: The use of "na" indicates that no performance goals were established specifically for that FDA program.
                                        Note 2: A     indicates that FDA established a set of performance goals applicable to the strategic goal
                                                 and that FDA program or programs.
                                        a Includes all FDA programs except for toxicological research.




Views on the Results                    FDA officials characterized four sets of performance goals related to the
Orientation of the Sets of              agency’s pre-market strategic goal as somewhat results oriented and one
Performance Goals for the               set as results oriented to a great extent. We considered all five sets of
                                        performance goals as somewhat results oriented.
Pre-Market Strategic Goal
                                        FDA officials characterized three sets of pre-market performance goals
                                        (biologics/drugs, food safety and nutrition, and devices and radiological
                                        health) as somewhat results oriented. The officials classified these sets of
                                        performance goals as such because, although the sets measured the



                                        Page 52                                                GAO/GGD-97-83 Barriers to Managing for Results
    Appendix III
    FDA’s Mission, Strategic and Performance
    Goals, and Its Performance Evaluation Plan




    number or percentage of applications FDA completed within a certain
    period of time, these sets did not include measures for the length of time
    needed to approve an application. For example, the performance goals
    related to biologics and drugs were:

•   “Review 90 percent of complete product licensing agreements (PLA),
    establishment licensing agreements (ELA), and new drug approvals (NDA)
    for priority applications within 6 months after the submission date;
•   review 90 percent of complete PLAs, ELAs, and NDAs for standard
    applications, within 12 months after the submission date;
•   review 90 percent of priority supplements to complete PLAs, ELAs, and NDAs
    within 6 months after the submission date;
•   review 90 percent of standard supplements to complete PLAs, ELAs, and
    NDAs that require review of clinical data (efficacy supplements) within 12
    months after the submission date;
•   review 90 percent of standard supplements to PLAs, ELAs, and NDAs that do
    not require review of clinical data (efficacy supplements) within 6 months
    after the submission date;
•   review 90 percent of complete PLA, ELA, and NDA applications resubmitted
    following receipt of a non-approvable letter, within 6 months after the
    resubmission date; and
•   review and act upon 55 percent of fileable generic drug applications
    received during Fiscal Year 1997 within 6 months after submission.”

    According to an FDA official, completed applications include those
    approved by FDA or those requiring additional information from the
    applicant.4 When FDA requests additional information, the applicant may
    respond to FDA’s request or withdraw the application from consideration.
    The FDA official said that because applicants are never denied the
    opportunity to return to the application process, FDA rarely issues a final
    negative response to applications. However, some FDA programs do have a
    “refuse to file” policy that prevents some incomplete applications from
    entering FDA’s review process.

    An FDA official said that if the performance goals had measured the
    timeliness of approvals they would be more results oriented because
    timeliness would translate into new products being made available for
    public use, potentially improving the health of the American public. We
    also considered these sets of performance goals somewhat results

    4
     FDA notifies applicants as to the disposition of their application by using one of three “action letters.”
    Applicants may receive either an “approved letter,” an “approvable letter” (an approval of the
    application noting minor deficiencies that could be corrected by the applicant), or a “not approvable
    letter” (which provides the applicant the option to resubmit a modified application).



    Page 53                                           GAO/GGD-97-83 Barriers to Managing for Results
Appendix III
FDA’s Mission, Strategic and Performance
Goals, and Its Performance Evaluation Plan




oriented because they did not cover all aspects of the strategic goal. None
of the goals addressed the cost-effectiveness of these reviews or whether
product safety and efficacy were being assured.

FDA  officials said the set of performance goals for veterinary medicine was
results oriented to a great extent because, unlike the first three sets of
performance goals, it measured the length of time to approve an
application.5 For example, one performance goal was to “reduce the
average number of months required for approval of a new animal drug
from 41 months in fiscal year 1996 to 40 months in fiscal year 1997.” FDA
officials said that approving applications for animal drugs and feeds
assisted in maintaining the health of livestock and limited the spread of
contaminated foods, and therefore protected public health. However, we
considered this set of performance goals somewhat results oriented
because, like the previous sets of goals, it did not address all aspects of the
strategic goal. None of the performance goals measured the
cost-effectiveness of the reviews or whether product safety and efficacy
were being assured.

FDA   had three performance goals relating to toxicological research:

1. “Complete the inlife portions of the mechanistic and tumor studies on
fumonisin and chloral hydrate, which will provide information on specific
agent toxicity and a new methodology for future assessments;

2. develop and evaluate a prototype predictive screening process to
streamline the review of new compounds and to more accurately predict
human risk; and

3. construct a prototype ‘knowledge base’ (expert systems to evaluate and
predict risk associated with toxicity by estrogenic compounds) for
selected reviewer evaluation.”

FDA officials characterized this set of goals as somewhat results oriented
because it focused on the development of a good “knowledge base,” which
they said enabled FDA to make valid and reliable decisions during the
approval process.

We agreed that developing a knowledge base is important for FDA’s review
process and that the measures were generally related to the strategic goal

5
 Similar to other FDA programs, completed applications for veterinary medicine could be either
approved by FDA or a request could be made to the applicant for additional information.



Page 54                                        GAO/GGD-97-83 Barriers to Managing for Results
                                 Appendix III
                                 FDA’s Mission, Strategic and Performance
                                 Goals, and Its Performance Evaluation Plan




                                 of making “timely and cost-effective pre-market decisions.” However, the
                                 set of goals focused on developing processes (i.e., developing “a new
                                 methodology for future assessments,” “a prototype predictive screening
                                 process,” or “prototype ‘knowledge base”) did not provide a clear means
                                 to assess progress toward achieving the pre-market strategic goal. In
                                 addition, the set of goals did not address the elements of timeliness or
                                 cost-effectiveness of the review process, and did not assess whether
                                 product safety or efficacy were being assured. Therefore, we considered
                                 this set of performance goals to be somewhat results oriented.


Views on the Results             FDA had three sets of performance goals for its post-market strategic goal
Orientation of the Sets of       to “strengthen assurance that products on the market or about to enter the
Post-Market Performance          market are safe.” FDA officials characterized one set of performance goals
                                 as results oriented to a great extent and the remaining two sets as
Goals                            somewhat results oriented. We considered all three sets of performance
                                 goals to be results oriented to little or no extent.

                                 The first set of post-market performance goals, related to food safety and
                                 nutrition, were:

                             •   “By December 31, 1996, develop a training program on how to conduct
                                 Hazard Assessment Critical Control Points (HACCP) inspections;
                             •   by April 1, 1997, develop a model Federal/State partnership agreement for
                                 seafood HACCP to implement HACCP-based inspections of seafood
                                 processors by the states;
                             •   by May 31, 1997, provide training to FDA inspectors who will conduct
                                 verification inspections, and appropriate headquarters staff of core HACCP
                                 curriculum;
                             •   by June 30, 1997, revise the FDA Inspection Operations Manual for
                                 HACCP-based inspections of seafood processors;
                             •   by June 30, 1997, revise compliance program criteria for HACCP-based
                                 inspections of seafood processors;
                             •   by September 30, 1997, determine utility of establishing a
                                 reference/baseline of industry preventive controls based on an assessment
                                 of past data;
                             •   by September 30, 1997, establish a Seafood HACCP database to record
                                 inspection results and facilitate evaluation of the system; and
                             •   increase Threshold of Regulation Policy Actions Completed from 50 in
                                 1996 to 55 in 1997.”




                                 Page 55                                  GAO/GGD-97-83 Barriers to Managing for Results
    Appendix III
    FDA’s Mission, Strategic and Performance
    Goals, and Its Performance Evaluation Plan




    FDA officials characterized the set of performance goals related to food
    safety and nutrition as results oriented to some extent. They said that the
    HACCP inspections indicated a new strategic direction for FDA by
    emphasizing the detection and prevention of microbiological
    contamination of food products and extending the responsibility for food
    safety to the food processing industry. An FDA official said that this set of
    performance goals illustrated how FDA was implementing HACCP (e.g.,
    developing partnership agreements, training inspectors, and revising an
    inspection manual and compliance program criteria). However, we
    characterized this set of performance goals as results oriented to little or
    no extent because none of the goals clearly related to or measured the
    strategic goal or the agency’s mission. It was not clear to us that any of the
    actions contained in these measures (e.g., develop a training program,
    revise a manual, or establish a database) would, by themselves, strengthen
    assurance that products are safe.

    The second set of post-market performance goals, related to devices and
    radiological health, were:

•   “Process 95 percent of the medical device reports received, and maintain a
    zero backlog;6
•   process 5,400 reports under the MedWatch Voluntary Reporting Program;7
•   train and certify approximately 250 State and Federal inspectors to
    conduct annual and follow-up inspections at the approximately 10,000 U.S.
    mammography facilities;
•   conduct 10,000 annual and 200 follow-up inspections of U.S.
    mammography facilities under the Mammography Quality Standards Act
    (MQSA); and
•   conduct 3,000 U.S. mammography facility recertifications under MQSA.”

    FDA officials characterized this set of performance goals as greatly results
    oriented because three of the goals related to the MQSA, which authorized
    FDA to accredit and inspect mammography facilities. An FDA official said
    that FDA enhances the performance of equipment and radiologists through


    6
     Under the 1976 Medical Device Amendments to the Federal Food, Drug, and Cosmetic Act,
    manufacturers of medical devices are required to report to FDA—and maintain records
    on—device-related deaths, serious injuries, and malfunctions that, should they recur, would be likely
    to result in death or serious injury.
    7
     The MedWatch Voluntary Reporting Program (MedWatch) was established by FDA to encourage
    health professionals to report problems associated with medical devices to FDA. MedWatch and
    medical device reports are intended to gather information to enable FDA and the medical device
    industry to take corrective action on device problems and, when appropriate, to alert the public to
    potentially hazardous devices.



    Page 56                                         GAO/GGD-97-83 Barriers to Managing for Results
                            Appendix III
                            FDA’s Mission, Strategic and Performance
                            Goals, and Its Performance Evaluation Plan




                            its MQSA program, thereby producing improved patient care through a
                            better screening process and diagnosis.

                            However, we characterized this set of performance goals as results
                            oriented to little or no extent because it did not clearly relate to or provide
                            relevant information for assessing progress toward the strategic goal of
                            strengthening assurance that products are safe. For example, the fourth
                            goal only measured the number of MQSA inspections conducted and did not
                            measure how many facilities were in compliance with mammography
                            standards or assess the impact or effect of these inspections.8 Similarly,
                            the performance goals regarding medical device reports and the
                            MedWatch Voluntary Reporting Program only measured the processing of
                            reports, and did not gauge FDA’s impact on the safety of medical devices.9

                            The third set of post-market performance goals, related to veterinary
                            medicine, were (1) “conduct 700 establishment inspections” and
                            (2) “conduct, by States under FDA contract, 550 feed mill inspections.” FDA
                            officials said they considered this set of goals somewhat results oriented
                            because inspections can improve the quality of animal drugs and feeds on
                            the market. However, similar to the performance goals on MQSA
                            inspections, this set did not measure how many establishments or feed
                            mills were in compliance with FDA regulations. As such we considered this
                            set of performance goals as results oriented to little or no extent because
                            it did not clearly relate to or provide relevant information for assessing
                            progress toward the strategic goal of strengthening assurance that
                            products are safe.


Views on the Results        FDA had three performance goals related to its strategic goal of
Orientation of the Set of   strengthening the nation’s capacity to manage health risks:
Capacity Performance
                            1. “Complete design on improved version of full Operational and
Goals                       Administrative System for Import Support;


                            8
                             In our report entitled FDA’s Mammography Inspections: While Some Problems Need Attention,
                            Facility Compliance Is Growing (GAO/HEHS-97-25, Jan. 27, 1997), we found that although “FDA has
                            made progress in bringing facilities into compliance with mammography standards, it lacks procedures
                            to enforce timely correction of all deficiencies found during inspections.”
                            9
                             In our report entitled Medical Device Reporting: Improvements Needed in FDA’s System for
                            Monitoring Problems With Approved Devices (GAO/HEHS-97-21, Jan. 29, 1997), we found that FDA
                            had not “routinely documented the final corrective actions taken by manufacturers and FDA to resolve
                            the problems identified in these reports.” The report concluded that “without adequate documentation,
                            FDA has little assurance that the many reports received are, in fact, useful and result in better
                            protection of the public health.”



                            Page 57                                        GAO/GGD-97-83 Barriers to Managing for Results
                   Appendix III
                   FDA’s Mission, Strategic and Performance
                   Goals, and Its Performance Evaluation Plan




                   2. allow for a range of between 62 percent and 65 percent automated ‘may
                   proceeds’ on a monthly basis; and

                   3. maintain a foreign inspection program with a planned total of 1,020
                   foreign inspections in fiscal year 1997.”

                   FDA  officials characterized this set of performance goals as somewhat
                   results oriented because they were all output measures of FDA’s internal
                   processes. An FDA official said that the first goal was a process
                   improvement goal to automate FDA’s system of handling imported foods
                   and other products. The second goal measured the percentage of “may
                   proceed” decisions being automated as part of FDA’s import entry
                   clearance process, indicating a more efficient process. The third goal
                   referred to the number of inspections FDA expected to complete of foreign
                   manufacturers of drugs and devices, which FDA officials said was another
                   example of how FDA protects public health. However, we characterized
                   this set of performance goals as results oriented to little or no extent
                   because the actions contained in these goals (designing systems,
                   automating processes, or maintaining an inspection program) were not
                   clearly and unambiguously related to FDA’s mission of promoting and
                   protecting the health of the American people. Also, the set did not provide
                   relevant information to assess progress toward that mission, such as the
                   number of foreign establishments in compliance with FDA regulations.


                   On September 1, 1996, FDA adopted a new employee performance
FDA Replaced       evaluation program that replaced most of FDA’s employee performance
Performance        standards.10 FDA officials said that the new program includes a
Standards With a   “performance evaluation plan” (PEP)—an evaluation form that lists five
                   generic job elements on “individual work management,” “technical
Customized         competency,” “teamwork,” “innovation,” and “customer service.” PEP also
Performance        contains blank spaces for supervisors to write in new job elements
                   tailored to the employee’s position. FDA documentation stated that each
Evaluation Plan    employee and his or her supervisor should discuss the employee’s official
                   duties and responsibilities and use PEP’s checklist format to customize a
                   set of performance standards for the employee. PEP requires that
                   nonsupervisors be evaluated on at least one job element, but supervisors
                   have the final decision on the content of the PEP form.




                   10
                    FDA officials said that the majority of FDA staff were under the new performance evaluation system
                   with the exception of some union employees remaining under the prior system.



                   Page 58                                        GAO/GGD-97-83 Barriers to Managing for Results
                     Appendix III
                     FDA’s Mission, Strategic and Performance
                     Goals, and Its Performance Evaluation Plan




                     FDA officials said that most of the positions they considered “frontline
                     regulators” were using PEP as of January 1, 1997, and, therefore had unique
                     sets of performance standards. As a result, we did not include any
                     performance standards for FDA in our review because FDA officials told us
                     that the performance standards for any of their frontline regulator
                     positions would vary from employee to employee.


                     We met with FDA officials, including the Director for Planning and
FDA’s Comments and   Management Communications Staff, on April 22, 1997. FDA officials said
Our Evaluation       they generally agreed with the report’s overall message regarding the
                     difficulties that regulatory agencies have experienced in becoming results
                     oriented. They also said that they would have reached conclusions similar
                     to ours regarding the results orientation of their strategic and performance
                     goals had they used the same criteria we used. The FDA officials also
                     provided several technical corrections, which were included in the final
                     report.

                     FDA  officials said that (1) the progress made by FDA’s field offices was not
                     reflected in the report’s focus on FDA’s agencywide goals; (2) the
                     performance goals FDA is developing for the fiscal year 1999 budget cycle
                     will be more results oriented than the January 1, 1997, goals assessed in
                     this report; and (3) many of the interim goals that FDA had (e.g., developing
                     a training program for conducting HACCP verification inspections), while
                     not results oriented themselves, would ultimately achieve
                     outcome-oriented results.11

                     We agree that focusing on agencywide goals did not capture information
                     on progress made in field office pilot programs. In the draft report, we
                     recognized that the goals, measures, and some of the standards were
                     changing at the time of our review. FDA’s efforts to establish
                     results-oriented systems, such as HACCPs, may allow FDA to achieve
                     outcome-oriented results in the future. However, our objective was to
                     describe the agency’s degree of results orientation as of January 1, 1997.




                     11
                      The HACCP system is designed to identify the critical points in food processing and establish
                     controls to prevent adulteration caused by microbiological, chemical, or physical hazards in order to
                     produce safer food products.



                     Page 59                                         GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV

IRS’ Mission, Strategic Objectives, Key
Performance Indicators, and Performance
Standards for Selected Frontline Regulator
Positions
               IRS’
                  mission is to “collect the proper amount of tax revenue at the least
               cost; serve the public by continually improving the quality of our products
               and services; and perform in a manner warranting the highest degree of
               public confidence in our integrity, efficiency, and fairness.” IRS also has
               developed a business vision “to be a state-of-the-art financial services
               company that eliminates most paper in the quest for quick, efficient
               customer service. The new IRS will be handling less paper and collecting
               taxes in new and more efficient ways.”

               We assessed the results orientation of IRS’ strategic objectives,
               performance indicators, and employee performance standards according
               to the criteria presented in the scope and methodology appendix of this
               report.1 Because we characterized the extent to which each agency goal
               and set of performance measures was results oriented, the application of
               these criteria represented a higher standard of results orientation than
               required by the Results Act. The Act does not require that all of an
               agency’s goals or measures be results oriented. Also, GPRA does not cover
               employee performance standards. We judged each set of performance
               standards according to the expectation set by the President in his March 4,
               1995, memorandum that these standards should “focus on results, not
               process and punishment.” Similar to the way in which GPRA views agencies’
               goals and performance measures, we do not believe that each such
               standard must be results oriented to achieve the President’s objective.

               We applied our criteria to the agencies’ written strategic goals, program
               performance measures, and employee performance standards and did not
               consider any supplementary or contextual information. In some cases an
               agency’s goals, measures, and standards appeared results oriented on the
               surface, but closer examination and strict application of our criteria
               resulted in a lower characterization. We recognize that other information
               outside of the written statements and outside of our review may have
               existed that could have resulted in our characterizing certain goals or sets
               of measures and standards differently.

               Both we and IRS officials characterized the extent of results orientation of
               IRS’ strategic objectives, key performance indicators, and employee
               performance standards for selected frontline regulator positions. As
               shown in table IV.1, there were differences in how we and IRS officials
               viewed the agency’s objectives, indicators, and standards. We and IRS
               officials agreed that IRS’ strategic objectives were greatly results oriented.

               1
                IRS used the term “strategic objective” instead of “strategic goal” and the term “key performance
               indicators” instead of “program performance measures.”



               Page 60                                         GAO/GGD-97-83 Barriers to Managing for Results
                                          Appendix IV
                                          IRS’ Mission, Strategic Objectives, Key
                                          Performance Indicators, and Performance
                                          Standards for Selected Frontline Regulator
                                          Positions




                                          There was somewhat more disagreement regarding IRS’ performance
                                          indicators in that we characterized them as somewhat less results oriented
                                          than did the IRS officials. The differences in our characterizations were
                                          greatest with regard to the employee performance standards. IRS officials
                                          said that most of the standards were greatly results oriented, but we
                                          characterized most as somewhat results oriented.


Table IV.1: Differences Between Our and IRS Officials’ Characterizations
                                             Number that IRS officials considered                 Number we considered results oriented
                                                        results oriented to                                       to
                                                    A great                       Little or no          A great                 Little or no
                                     Total           extent Some extent                 extent           extent Some extent           extent
Strategic objectivesa                    3                  3                0                0                 3          0              0
                         b
Key performance indicators               3                  2                1                0                 0          3              0
Performance standards                  10                   9                1                0                 3          7              0
                                          a
                                             IRS used the term “strategic objective” instead of “strategic goal.”
                                          b
                                              IRS used the term “key performance indicators” instead of “program performance measures.”

                                          Source: Agency and GAO analysis.




                                          As of January 1, 1997, IRS had established three “strategic objectives”:
Views on the Results
Orientation of IRS’                          1. “Increase compliance: Our objective is to encourage and assist
Strategic Objectives                         taxpayers to voluntarily file timely and accurate returns and to pay on
                                             time, and, if taxpayers do not comply, to take appropriate compliance
                                             actions. We do this to ensure we collect the proper amount of tax at the
                                             least cost.

                                          2. Improve customer service: Our objective is to reduce the time and
                                          expense experienced by taxpayers, tax professionals and others in
                                          complying with the tax laws, while increasing their satisfaction with tax
                                          administration. We want to ensure that filing materials are readily
                                          available, and to reduce the data required to be reported by taxpayers and
                                          captured by IRS to only the most critical. We do this to help collect the
                                          proper amount of tax at the least cost, serve the public, and increase
                                          public confidence in tax administration.

                                          3. Increase productivity: Our objective is to continually improve our
                                          operations and the quality of products and services we provide to our
                                          customers, through strategic use of reengineering and systems



                                          Page 61                                            GAO/GGD-97-83 Barriers to Managing for Results
                                  Appendix IV
                                  IRS’ Mission, Strategic Objectives, Key
                                  Performance Indicators, and Performance
                                  Standards for Selected Frontline Regulator
                                  Positions




                                  management tools and the development of a highly skilled work force. We
                                  do this to improve compliance, provide improved customer service and
                                  reduce the cost of tax administration.”

                                  IRSofficials said that these three objectives were results oriented to a great
                                  extent because they related directly to IRS’ mission. We agreed with that
                                  assessment with regard to each of the strategic objectives. IRS’ first
                                  objective, to increase taxpayer compliance, was clearly related to IRS’
                                  mission to collect the proper amount of tax revenue, provided a clear
                                  definition of the results expected, and permitted assessment of the
                                  agency’s programs. Similarly, IRS’ second objective, to improve customer
                                  service, directly related to IRS’ mission to “serve the public,” and IRS’ third
                                  objective of increasing productivity was also linked to IRS’ mission to
                                  collect revenue “at the least cost.”


                                  As of January 1, 1997, IRS was using three sets of servicewide “key
Views on the Results              performance indicators” for its strategic objectives. IRS officials said that
Orientation of IRS’               two of the three sets of indicators were results oriented to a great extent,
Key Performance                   and one was results oriented to some extent. We characterized the three
                                  sets of indicators as somewhat results oriented.
Indicators
Views on the Results              The indicators that IRS used to assess progress toward its first strategic
Orientation of the                objective, to “increase compliance,” were:
Indicators Associated With
                              •   “Total collection percentage”: A ratio of IRS’ total collections to the total
IRS’ Strategic Objective to       estimated true tax liability.
“Increase Compliance”         •   “Total net revenue collected”: All revenue collected by the IRS.
                              •   “Servicewide enforcement revenue collected/protected”: All revenue
                                  collected through enforcement actions and all revenue prevented from
                                  being released from the Treasury in response to fraud, disallowed claims,
                                  and other issues.2
                              •   “Servicewide audit coverage”: The percentage of returns audited divided
                                  by the total number of returns filed during the prior calendar year.

                                  IRSofficials said that this set of indicators was results oriented to a great
                                  extent because they believed that the first three indicators were clear
                                  indications of IRS’ collection results. However, we characterized this set of
                                  indicators as somewhat results oriented because, although the “total
                                  collection percentage” was clearly and unambiguously related to the

                                  2
                                   According to IRS, enforcement revenue protected and enforcement revenue collected will be
                                  expressed as separate indicators in fiscal year 1998.



                                  Page 62                                       GAO/GGD-97-83 Barriers to Managing for Results
                              Appendix IV
                              IRS’ Mission, Strategic Objectives, Key
                              Performance Indicators, and Performance
                              Standards for Selected Frontline Regulator
                              Positions




                              strategic objective of increasing compliance, the set did not address all
                              aspects of the strategic objective.3 For example, the set did not contain
                              indicators for IRS efforts to encourage and assist taxpayers in filing
                              accurate and timely returns.

                              We did not view the remaining indicators in this set as clearly linked to the
                              strategic objective to “increase compliance.” The second indicator, “total
                              net revenue collected,” is already contained in the ratio of the “total
                              collection percentage” and is ambiguous in its relation to the strategic
                              objective. An increase or decrease in revenue collected may simply reflect
                              a change in economic activity or tax laws rather than any change in the
                              level of taxpayer compliance. Similarly, the amount collected/protected
                              through enforcement actions and the percentage of returns audited may
                              not provide a valid indication of the level of taxpayer compliance. For
                              example, enforcement revenues collected could increase while voluntary
                              revenues decreased, resulting in no change (or even a decrease) in the
                              overall level of taxpayer compliance.


Views on the Results          IRS used two indicators to assess the agency’s progress toward its second
Orientation of the            strategic objective, to “improve customer service”:
Indicators Associated With
                              1. “Taxpayer burden cost (in dollars) for IRS to collect $100”: A ratio that
IRS’ Strategic Objective to   measured taxpayers’ paperwork burden costs compared to the cost for IRS
“Improve Customer             to collect $100 in net tax revenue.4
Service”
                              2. “Initial contact resolution rate”: The rate at which taxpayer issues were
                              resolved during the first contact with IRS.5

                              IRSofficials characterized this set of indicators as results oriented to some
                              extent, and said they were working to improve IRS’ customer service
                              measures. We also characterized the set of indices as somewhat results
                              oriented because the indicators did not address all aspects of the customer


                              3
                               Although the “total collection percentage” information is relevant, it may not be accurate. Our reports
                              have shown that IRS’ estimate of the true tax liability lacks statistical precision at the national level.
                              Without such an estimate, IRS will not be able to accurately determine the “total collection
                              percentage” nor judge its progress toward increasing compliance. See Taxpayer Compliance:
                              Analyzing the Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9, 1997).
                              4
                               Paperwork burden costs were based on the estimated time individual and business taxpayers spend
                              keeping tax records, learning about tax laws, preparing tax forms, and sending tax forms to the IRS.
                              Taxpayer paperwork burden was converted from time to dollars by multiplying total time by $20 per
                              hour.
                              5
                               As of January 1, 1997, IRS was revising the definition of this rate.



                              Page 63                                            GAO/GGD-97-83 Barriers to Managing for Results
                                Appendix IV
                                IRS’ Mission, Strategic Objectives, Key
                                Performance Indicators, and Performance
                                Standards for Selected Frontline Regulator
                                Positions




                                service objective.6 For example, IRS did not measure “public confidence in
                                tax administration” even though the agency regularly conducts customer
                                service surveys. IRS also did not include any indicators of how they would
                                “ensure that filing materials are readily available.”


Views on the Results            IRS’   indicators for the strategic objective to “increase productivity” were:
Orientation of Indicators
Associated With IRS’        •   “Budget cost to collect $100 in revenue.”
                            •   “Percent of returns filed electronically”: The number of electronically filed
Strategic Objective to          tax returns, divided by the total returns filed.
“Increase Productivity”     •   “Percent of dollars received electronically.”7
                            •   “Percent of dollars received via third party processors”: The dollar amount
                                of federal tax deposits (FTD) processed by commercial banks under
                                contract to the Financial Management Service as a percentage of the total
                                FTD dollars received (paper and electronic).


                                IRS officials said that this set of indicators was results oriented to a great
                                extent mainly because the first indicator provided an overall indicator of
                                how efficiently IRS collects revenues. However, we characterized the set as
                                somewhat results oriented because, although the indicator “budget cost to
                                collect $100 in revenue” is a good overall productivity measure, the set of
                                indicators did not address all aspects of the objective. For example, none
                                of the indicators measured the “quality of the products and services” IRS
                                provides.




                                6
                                 The paperwork burden data upon which the first indicator is based, although relevant to assessing
                                progress toward the strategic objective, is generally considered to be inaccurately measured. Our
                                reviews of tax compliance burden have shown that a reliable estimate of the overall costs of tax
                                compliance is not currently available, and would be costly and burdensome to obtain. See Tax System:
                                Issues in Tax Compliance Burden (GAO/T-GGD-96-100, Apr. 3, 1996); Tax System Burden: Tax
                                Compliance Burden Faced by Business Taxpayers (GAO/T-GGD-95-42, Dec. 9, 1994); and Tax
                                Administration: Opportunities to Reduce the Burden of Filing and Processing Tax Returns
                                (GAO/T-GGD-92-41, May 13, 1992).
                                7
                                 As of January 1, 1997, IRS was revising this indicator.



                                Page 64                                           GAO/GGD-97-83 Barriers to Managing for Results
                        Appendix IV
                        IRS’ Mission, Strategic Objectives, Key
                        Performance Indicators, and Performance
                        Standards for Selected Frontline Regulator
                        Positions




                        In contrast to strategic objectives and performance indicators, the
Views on the Results    administration’s regulatory reinvention initiative, not GPRA, set the
Orientation of IRS’     expectation for agencies to focus on results when measuring the
Performance             performance of their frontline regulators. IRS identified two frontline
                        regulator positions for this review: customer service representatives (CSR)
Standards for           and revenue officers.8 Employees in the CSR position work as technicians
Selected Frontline      in a district office or service center to provide information and direction
                        on tax issues to the public. Revenue officers collect delinquent taxes and
Regulator Positions     secure delinquent returns. Revenue officers’ work requires application of a
                        knowledge of (a) general or specialized business practices; (b) pertinent
                        tax laws, regulations, procedures, and precedents; (c) judicial processes,
                        laws of evidence, and the interrelationship between federal and state laws
                        with respect to collection and assessment processes; and (d) investigative
                        techniques and methods.

                        As of January 1, 1997, the CSR and revenue officer positions had a total of
                        12 job elements and we and IRS officials reviewed 10 of the 12 sets of
                        associated performance standards.9 IRS officials characterized nine of the
                        sets of standards as results oriented to a great extent and one set as
                        somewhat results oriented. We considered three of the sets of standards to
                        be greatly results oriented and the remaining seven sets as somewhat
                        results oriented.


Views on the Results    We and IRS officials reviewed the sets of performance standards related to
Orientation of IRS’     five job elements for the customer service representative position:
Customer Service
                        1. “Customer relations: Achieves quality customer service by
Representatives’        communicating professionally with internal and external customers in all
Performance Standards   oral and written contacts.

                        2. Workload management: Plans and utilizes time based on priorities in
                        accordance with Service policy and managerial direction. Arranges
                        activities to minimize unproductive time and unnecessary delays in
                        servicing the customer.


                        8
                         Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not
                        define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated
                        community by enforcing regulations.
                        9
                         We did not review the sets of standards for the CSR or revenue officer positions’ job elements for
                        “other duties and assignments” because they were not specific and, although they may have related to
                        results in specific instances, reviewing these standards would not have provided any consistent
                        perspective on the extent to which these standards focused on results.



                        Page 65                                         GAO/GGD-97-83 Barriers to Managing for Results
    Appendix IV
    IRS’ Mission, Strategic Objectives, Key
    Performance Indicators, and Performance
    Standards for Selected Frontline Regulator
    Positions




    3. Procedures and notices: Applies procedural instructions issued by
    National, Regional, District and local offices to maximize the efficient flow
    of work in meeting program objectives. Correctly interprets notices to
    resolve inquiries appropriately.

    4. Integrated Data Retrieval System (IDRS) inquiries and on-line resolution:
    Resolves notice inquiries, refund and other account related issues through
    accessing IDRS. Researches and analyzes or performs on-line adjustment to
    accounts by utilizing appropriate reference materials.

    5. Tax law: Encourages voluntary compliance and improves the public’s
    confidence in the Service by correctly interpreting and applying tax laws,
    researching technical reference materials, and analyzing and evaluating
    the customer’s needs; informs customer of options and consequences as
    they relate to tax obligations.”

    The performance standards for each element described what the CSRs
    were expected to do for that element. For example, the performance
    standards for the “tax law” job element were:

•   “Interpretation and application: Generally interprets and applies tax laws
    appropriately resulting in technically correct responses to customer
    inquiries.
•   Analysis and evaluation: Generally evaluates and analyzes the customer’s
    circumstances and advises of tax computation and other benefits, i.e.,
    Earned Income Credit, Child Care Credit, estimated tax payments,
    reasonable cause, etc.
•   Research: Generally researches in appropriate and mandatory reference
    materials such as the Probe and Response Guide, IRS Publications, Tax
    Services, Internal Revenue Code (IRC), etc., to provide correct and
    complete response; generally does not seek assistance from others even
    on complex and non-routine issues. Increases customer confidence by
    citing specific references and offering tax publications.”

    IRS officials characterized all five sets of performance standards as results
    oriented to a great extent because they said each set was related to IRS’
    strategic objectives and/or its mission. We agreed with IRS officials’
    assessment for the sets of standards for the “workload management” and
    “tax law” job elements. For example, the set of standards related to the
    “tax law” job element included the expectation that CSRs would provide
    “technically correct responses to customer inquiries” and “increase
    customer confidence by citing specific references and offering tax



    Page 66                                      GAO/GGD-97-83 Barriers to Managing for Results
                                Appendix IV
                                IRS’ Mission, Strategic Objectives, Key
                                Performance Indicators, and Performance
                                Standards for Selected Frontline Regulator
                                Positions




                                publications.” Therefore, these standards appeared directly related to IRS’
                                mission of encouraging voluntary compliance and warranting public
                                confidence in the agency, described the results expected, and established
                                criteria to evaluate performance. However, we characterized the
                                remaining three sets of performance standards as somewhat results
                                oriented, because they did not clearly describe the results CSRs were
                                expected to work toward and/or did not establish criteria for evaluating
                                performance directly associated with an expected result. For example, the
                                set of standards for the “IDRS inquiries and on-line resolution” job element
                                called for CSRs to “perform adjustments correctly” and “recognize when
                                related account data is pertinent and handle appropriately,” but these
                                actions were not clearly linked to an expected result.


Views on the Results            We and IRS officials reviewed the sets of employee performance standards
Orientation of the              related to five job elements for the revenue officer position: “investigation
Performance Standards for       and analysis,” “case decisions,” “time and workload management,”
                                “customer relations,” and “accounts maintenance.” The standards
IRS’ Revenue Officer            described the types of actions revenue officers were expected to take for
Position                        that element. For example, the performance standards for the “case
                                decisions” job element were:

                            •   “Collectibility: Based on the available facts, determines the proper case
                                direction. Decisions to pursue full payment, immediate or deferred, or
                                suspend collection activity are based on full consideration of the overall
                                benefit to the government and taxpayers’ rights.
                            •   Collection tools: Based on case history, determines when levy, seizure, or
                                summons action is appropriate. Considers other collection tools such as
                                transferee, alter-ego, nominee, and Offer-in-Compromise. May need
                                occasional assistance making final decision on use of transferee, alter-ego,
                                etc.
                            •   Protection of the government’s interest: Based on case history, determines
                                when actions are necessary to protect the government’s interest, including
                                collection and assessment statutes. Decides on the proper action to take,
                                including but not limited to filing and/or refiling liens, recommending
                                suits, 100 percent penalty, prompts, quick, and IRC 6020(b) assessments,
                                fraud, or other appropriate referrals and waivers.”

                                IRS officials characterized four of the sets of performance standards as
                                results oriented to a great extent, and said the set of standards related to
                                “customer relations” was somewhat results oriented. The officials said that
                                all five of the sets of standards were primarily process measures, but said



                                Page 67                                      GAO/GGD-97-83 Barriers to Managing for Results
                    Appendix IV
                    IRS’ Mission, Strategic Objectives, Key
                    Performance Indicators, and Performance
                    Standards for Selected Frontline Regulator
                    Positions




                    the standards were related to IRS’ goals. For example, they said that the
                    standards for the “investigation and analysis” job element allow IRS to
                    collect the proper amount of tax and increase productivity. In addition, the
                    IRS officials pointed out that the Technical and Miscellaneous Revenue Act
                    of 1988 prohibits the IRS from evaluating employees on the basis of tax
                    enforcement results. Section 6231 of this act states that IRS cannot use
                    records of tax enforcement results to evaluate employees directly involved
                    in collection activities and cannot impose production quotas or goals on
                    individual employees. We characterized one set of performance standards
                    as results oriented to a great extent and the remaining four sets as
                    somewhat results oriented.

                    We agreed with IRS that the standards for the “case decisions” job element
                    were greatly results oriented because they were directly related to IRS’
                    mission to collect the proper amount of tax, clearly described the results
                    the revenue officers were expected to work toward, and established
                    criteria for evaluating the officers’ performance directly associated with
                    the desired results. For example, the standards stated that “decisions to
                    pursue full payment, immediate or deferred, or suspend collection activity
                    are based on full consideration of the overall benefit to the government
                    and taxpayers’ rights.” We characterized the remaining four sets of
                    performance standards as somewhat results oriented because they were
                    indirectly linked with IRS’ mission or one of its three strategic objectives,
                    but did not clearly specify the results revenue officers were expected to
                    work toward or establish criteria for evaluating performance in relation to
                    those results. For example, the set of standards for the “time and workload
                    management” job element were unclear about the results expected and did
                    not establish criteria for evaluating performance in relation to those
                    results. Revenue officers were to take “timely” initial, follow-up, and
                    closing actions, but the standards did not define what “timely” meant nor
                    how it was related to results.


                    We met with IRS officials, including the National Director of the Strategic
IRS’ Comments and   Planning Division, on April 29, 1997. IRS also provided a comment letter,
Our Evaluation      which is reprinted in appendix VII. Generally, the IRS officials said they
                    were pleased that we characterized IRS’ goals and measures as results
                    oriented.

                    In both the meeting and in written comments, IRS officials said that we did
                    not give the agency sufficient credit for doing a major overhaul of its
                    measures. However, one of the objectives of our review was to describe



                    Page 68                                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV
IRS’ Mission, Strategic Objectives, Key
Performance Indicators, and Performance
Standards for Selected Frontline Regulator
Positions




each agency’s performance measures as of January 1, 1997, not to describe
the specific efforts each agency had taken in developing these measures.
Furthermore, any efforts IRS had made to overhaul its measures prior to
January 1, 1997, should have been reflected in the measures we reviewed.

IRS’written comments also state that IRS was concerned that we created
our own set of criteria in lieu of using GPRA criteria to evaluate the
agencies’ goals and program performance measures. Other than the fact
that GPRA does not require all of an agency’s goals or program performance
measures to be results oriented, which we emphasize throughout the
report, we do not believe that our criteria differ substantially from GPRA
requirements. For example, the report of the Senate Committee on
Governmental Affairs on GPRA states that strategic or general goals should
elaborate on the mission statement, should constitute a specific set of
objectives for the agency’s programs and operations, and must be
expressed in a manner that allows a future assessment of whether the goal
is being achieved.10 This is very similar to our criteria that greatly
results-oriented strategic goals (1) were linked closely to the agency’s
mission, (2) provided a clear definition of what results were expected, and
(3) permitted assessment of progress in achieving these results.

IRS officials also did not believe that the draft report placed enough
emphasis on the fact that the Technical and Miscellaneous Revenue Act of
1988 prohibits IRS from evaluating employees on the basis of tax
enforcement results. We agree that the Technical and Miscellaneous
Revenue Act of 1988 prohibits IRS from evaluating employees on the basis
of tax enforcement results and included this information in this appendix.




10
  Senate Report No. 103-58, June 16, 1993, page 24.



Page 69                                          GAO/GGD-97-83 Barriers to Managing for Results
Appendix V

EPA’s Mission, Strategic Goals, and
Performance Standards for Selected
Frontline Regulator Positions
                  EPA’s mission is “to protect public health and to safeguard and improve the
                  natural environment—air, water, and land—upon which human life
                  depends.” In doing so, EPA ensures that:

              •   “Federal environmental laws are implemented and enforced fairly,
                  effectively, and cost effectively;
              •   environmental protection is an integral consideration in U.S. policies
                  concerning economic growth, energy, transportation, agriculture, industry,
                  international trade, and natural resources;
              •   national efforts to reduce environmental risk are based on the best
                  available scientific information; and
              •   all parts of society—business, state and local governments, communities,
                  citizens—have full access to information so that they can become full
                  participants in preventing pollution and protecting human health and the
                  environment.”

                  We assessed the results orientation of EPA’s strategic goals and employee
                  performance standards according to the criteria presented in the scope
                  and methodology appendix of this report. Because we characterized the
                  extent to which each agency goal was results oriented, the application of
                  these criteria represented a higher standard of results orientation than
                  required by the Results Act. The Act does not require that all of an
                  agency’s goals be results oriented. Also, GPRA does not cover employee
                  performance standards. We judged each set of performance standards
                  according to the expectation set by the President in his March 4, 1995,
                  memorandum that these standards should “focus on results, not process
                  and punishment.” Similar to the way in which GPRA views agencies’ goals
                  and performance measures, we do not believe that each such standard
                  must be results oriented to achieve the President’s objective.

                  We applied our criteria to the agencies’ written strategic goals and
                  employee performance standards and did not consider any supplementary
                  or contextual information. In some cases an agency’s goals and standards
                  appeared results oriented on the surface, but closer examination and strict
                  application of our criteria resulted in a lower characterization. We
                  recognize that other information outside of the written statements and
                  outside of our review may have existed that could have resulted in our
                  characterizing certain goals or sets of standards differently.

                  Both we and EPA officials characterized the extent of results orientation of
                  EPA’s strategic goals and employee performance standards for selected
                  frontline regulator positions. As shown in table V.1, there were differences



                  Page 70                            GAO/GGD-97-83 Barriers to Managing for Results
                                         Appendix V
                                         EPA’s Mission, Strategic Goals, and
                                         Performance Standards for Selected
                                         Frontline Regulator Positions




                                         in how we and EPA officials viewed the goals and performance standards.
                                         Generally, we agreed that most of EPA’s strategic goals were greatly results
                                         oriented. However, we sharply disagreed regarding the extent to which the
                                         performance standards were results oriented. EPA officials characterized
                                         most of the 11 sets of performance standards as greatly results oriented,
                                         but we characterized most of them as results oriented to little or no extent.


Table V.1: Differences Between Our and EPA Officials’ Characterizations
                                            Number that EPA officials considered            Number we considered results oriented
                                                       results oriented to                                  to
                                                  A great                    Little or no        A great                 Little or no
                                    Total          extent Some extent              extent         extent Some extent           extent
Strategic goalsa                       10               10               0              0               8            1             1
Performance standards                  11                9               2              0               1            2             8
                                         a
                                            EPA did not have program performance measures as of January 1, 1997.

                                         Source: Agency and GAO analysis.




                                         As of January 1, 1997, EPA had 10 strategic goals, which were:
Views on the Results
Orientation of EPA’s                     1. “Clean air: Every American city and community will be free of air
Strategic Goals                          pollutants at levels that cause cancer or respiratory and other health
                                         problems. The air will be clearer in many areas and life in damaged forests
                                         and polluted waters will rebound as acid rain is reduced.

                                         2. Clean and safe water: America’s rivers, lakes and coastal waters will
                                         support healthy communities of fish, plants and other aquatic life and will
                                         support uses such as fishing and swimming. Wetlands will be protected
                                         and rehabilitated to provide wildlife habitat, reduce floods and improve
                                         water quality. Ground waters will be uncontaminated and every American
                                         public water system will provide water that is safe to drink all the time.

                                         3. Safe food: All foods Americans produce or consume will continue to be
                                         safe for all people to eat.

                                         4. Toxic free communities and workplaces: By relying on pollution
                                         prevention in the way we produce, consume, reuse and recycle materials,
                                         all Americans will live and work in safe and healthy communities.




                                         Page 71                                       GAO/GGD-97-83 Barriers to Managing for Results
Appendix V
EPA’s Mission, Strategic Goals, and
Performance Standards for Selected
Frontline Regulator Positions




5. Safe waste management: Wastes produced by every person and business
in America will be stored, treated and disposed of in ways that prevent
harm to people and other living things. Places in America currently
contaminated by hazardous or radioactive materials will not endanger
public health nor the natural environment and will be restored to uses
desired by surrounding communities.

6. Reducing global and transboundary environmental risks: The United
States and other nations will eliminate significant risks to human health
and ecosystems arising from climate change, stratospheric ozone
depletion and other environmental problems of global concern.

7. Empowering people with information and education and expanding
their right to know: Information on the state of the environment will be
user-friendly and widely accessible, increasing opportunities for
Americans to learn about the environment and natural resources and
fostering environmental stewardship through information exchange within
and among communities and across economic sectors.

8. Develop tools for the future: EPA will develop innovative tools and
approaches that will permit environmental protection programs to
transcend the simple additivity of individual media-based programs and
move the Agency towards truly integrated and continually improving
environmental protection programs. The EPA will enhance its science and
research program in order to better anticipate environmental issues
coming over the horizon and to find early solutions.

9. Provide a credible deterrent: The EPA will continue to implement
programs and monitor compliance to ensure that environmental gains that
have been achieved are not lost.

10. Effective management: EPA will establish a management infrastructure
that will set and implement the highest quality standards for effective
internal management and fiscal responsibility.”

EPA officials characterized each of these goals as results oriented to a great
extent because the goals described the results the agency was trying to
achieve. For example, EPA officials said that the goal regarding “clean and
safe water” was greatly results oriented because it addressed the
environmental result of improving or maintaining water quality so that
aquatic communities will be healthy, human uses will be supported,
drinking water will be safe, and ground water will be uncontaminated. EPA



Page 72                               GAO/GGD-97-83 Barriers to Managing for Results
                       Appendix V
                       EPA’s Mission, Strategic Goals, and
                       Performance Standards for Selected
                       Frontline Regulator Positions




                       officials said that the goal to “provide a credible deterrent” was greatly
                       results focused in two ways. First, they said that the enforcement program
                       targets industries and facilities for attention based in part on the expected
                       degree of environmental benefit and risk reduction. Second, they cited the
                       deterrent effect of ensuring that the regulated community perceives that
                       compliance will be strongly pursued by EPA, so that overall environmental
                       results are enhanced due to voluntary compliance. Finally, EPA officials
                       said that the goal for “effective management” was greatly results oriented
                       because it complied with GPRA’s requirement to measure performance in
                       all areas, and that the results of the management area were primarily
                       outputs focused on customer service.

                       We agreed with EPA officials that 8 of the 10 goals were results oriented to
                       a great extent. In addition to the reasons cited by EPA officials, we also
                       viewed the eight goals as relating to EPA’s mission and permitting
                       assessment of the agency’s progress in achieving desired results. However,
                       we characterized the goal “developing tools for the future” as results
                       oriented to some extent because it did not provide a clear definition of the
                       results expected and it was unclear how EPA would assess progress toward
                       achieving this goal. We characterized the remaining goal for “effective
                       management” as results oriented to little or no extent because it was
                       focused internally and therefore did not relate to the agency’s mission.


                       As of January 1, 1997, EPA had not established sets of program
EPA Had No Program     performance measures to assess progress in meeting the agency’s goals.
Performance            According to EPA officials, the agency was establishing quantitative
Measures in Place at   “objectives” that describe what EPA wants to achieve by an agreed-upon
                       target date. EPA plans to have a full set of program performance measures
the Time of Our        relating to these objectives by July 1997.
Review
                       Although EPA had no official program performance measures as of
                       January 1, 1997, the agency has been working since 1992 to develop
                       long-range national environmental goals with results-oriented
                       performance goals and measures. EPA officials said that some of the
                       performance measures for the national goals program may be used as
                       measures for the agency’s strategic goals.




                       Page 73                               GAO/GGD-97-83 Barriers to Managing for Results
                        Appendix V
                        EPA’s Mission, Strategic Goals, and
                        Performance Standards for Selected
                        Frontline Regulator Positions




                        In contrast to strategic goals and program performance measures, the
Views on the Results    administration’s regulatory reinvention initiative, not GPRA, set the
Orientation of EPA’s    expectation for agencies to focus on results when measuring the
Performance             performance of their frontline regulators. EPA identified two frontline
                        regulator positions for this review, criminal investigators and
Standards for           environmental scientists.1 As of January 1, 1997, there were the equivalent
Selected Frontline      of about 225 full-time criminal investigators at EPA, and the equivalent of
                        about 1,225 full-time environmental scientists working out of EPA’s
Regulator Positions     regional offices.2 EPA criminal investigators investigate allegations or
                        suspected violations of criminal environmental laws, and environmental
                        scientists implement and enforce environmental regulations.

                        EPA officials characterized 9 of the 11 sets of performance standards
                        reviewed for these 2 positions as results oriented to a great extent, and 2
                        sets of standards as somewhat results oriented. They believed that most of
                        the sets of standards clearly described the results that should occur
                        through performance of the standards. However, we characterized eight of
                        the sets of employee performance standards we reviewed as results
                        oriented to little or no extent, two as somewhat results oriented, and one
                        as results oriented to a great extent.


Views on the Results    We and agency officials reviewed five job elements for the criminal
Orientation of EPA’s    investigator position:3
Criminal Investigator
                        1. “Professional attributes: In the performance of duties, displays and
Performance Standards   possesses professional attributes and conduct which are a benefit to the
                        agency and which foster good working relationships within and outside
                        the office. Such professional attributes include physical conditioning,
                        appearance and demeanor, proficiency with weapons and equipment,
                        work attitude, and self-improvement measures.

                        2. Communication (oral and written): Prepares and presents written and
                        oral communication and reports as needed to the public; Regional



                        1
                         Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not
                        define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated
                        community by enforcing regulations.
                        2
                         EPA also has environmental scientists working in its headquarters offices, but EPA officials said that
                        these environmental scientists were not frontline regulators.
                        3
                         We did not include the “special projects” job element in our review because of the general nature of
                        these standards and their variability between criminal investigators.



                        Page 74                                          GAO/GGD-97-83 Barriers to Managing for Results
Appendix V
EPA’s Mission, Strategic Goals, and
Performance Standards for Selected
Frontline Regulator Positions




officials; Criminal Investigation Division (CID) Headquarters; supervisors,
the Director, CID; prosecuting attorneys and other officials.

3. Knowledge of directives, laws, regulations, and procedures: Maintains
and demonstrates knowledge of programs, missions, and operations
within EPA; CID policy; and pertinent environmental laws and regulations.

4. Case investigation: Conducts complex investigations within an assigned
area of responsibility.

5. Liaison: Identifies, establishes and maintains contacts with other
federal, state, and local law enforcement and regulatory agencies, EPA
Regional officials, public institutions, the private sector, and other CID
offices.”

EPA officials classified all but one of the sets of performance standards
related to these elements as results oriented to a great extent. For
example, EPA officials said that the set of standards related to the
“communication” job element was greatly results oriented because
presentations that were thorough, easily understood, and appropriately
targeted would result in an accurate understanding of the message being
delivered. EPA officials said the set of standards for the “liaison” job
element was somewhat results oriented because regular communications
with a broad base of external contacts provided the intermediate result of
a highly efficient and timely resource for the agency.

However, we considered four of the sets of performance standards (for the
“professional attributes,” “communication,” “knowledge of directives,” and
“liaison” job elements) as results oriented to little or no extent because
these standards were not tied to any expected results and therefore had no
criteria for evaluating performance associated with expected results. For
example, the standards for the “professional attributes” job element
included maintaining “a good level of physical conditioning,” displaying
“an appearance and demeanor which is consistent with the position,” and
displaying “a positive attitude towards the job.” We did not believe that
these standards clearly described the results that criminal investigators
were expected to work toward or were linked to any environmental
results that EPA was trying to achieve. We viewed the remaining set of
performance standards for the “case investigation” job element as
somewhat results oriented because although it was linked to EPA’s goal to
“provide a credible deterrent,” the set of standards did not clearly describe
the results the investigator was expected to work toward. For example,



Page 75                               GAO/GGD-97-83 Barriers to Managing for Results
                            Appendix V
                            EPA’s Mission, Strategic Goals, and
                            Performance Standards for Selected
                            Frontline Regulator Positions




                            investigators were expected to “identify and preserve physical evidence in
                            accordance with currently acceptable standards,” but the result expected
                            from following the standards for evidence was not specified.


Views on the Results        The environmental scientist position had six job elements:
Orientation of EPA’s
Performance Standards for   1. “Customer service: To meet customer expectations with respect to
                            products or services provided.
the Environmental
Scientist Position          2. Communication: To communicate effectively with internal and external
                            customers in accomplishing the organization’s work.

                            3. Teamwork: To contribute individual talents and energies toward
                            achieving team goals and objectives.

                            4. Grants/cooperative agreements: To coordinate the negotiation and
                            award of grants, cooperative agreements and/or interagency agreements,
                            and/or monitor work plans, progress, and activities performed under these
                            arrangements for acceptability and achievement of objectives.

                            5. Enforcement/compliance: To ensure compliance with asbestos
                            regulations under the Clean Air Act, National Emissions Standards for
                            Hazardous Air Pollutants, Toxic Substances Control Act, and related
                            legislation and regulations through effective management of compliance
                            assistance and enforcement activities while meeting programmatic goals
                            and objectives.

                            6. Inspections: To conduct compliance inspections consistent with Agency
                            priorities and programmatic goals and objectives.”

                            EPA officials classified all but one of the sets of performance standards for
                            these elements as results oriented to a great extent. For example, they said
                            that the set of standards for the “customer service” job element covered all
                            the major indicators of providing highly efficient customer services,
                            including mutual agreement between the customer and service provider
                            on product delivery dates, frequent feedback to customers, and checking
                            to determine whether customer needs were met. EPA officials said the set
                            of standards related to the “communication” job element was somewhat
                            results oriented because the set of standards did not cover whether the
                            communications were helpful to customers or could have been improved,




                            Page 76                               GAO/GGD-97-83 Barriers to Managing for Results
                     Appendix V
                     EPA’s Mission, Strategic Goals, and
                     Performance Standards for Selected
                     Frontline Regulator Positions




                     leaving a somewhat incomplete set regarding whether effective two-way
                     communication had been achieved.

                     We agreed that the set of standards associated with the environmental
                     scientist “enforcement” job element was greatly results oriented because it
                     was directly related to EPA’s mission and strategic goals, clearly described
                     the results scientists were expected to work toward, and established
                     criteria for evaluating their performance directly associated with the
                     result. However, we characterized the sets of standards for four of the job
                     elements (“customer service,” “communication,” “teamwork,” and
                     “grants/cooperative agreements”) as results oriented to little or no extent
                     because they did not establish what results would be achieved through
                     performance of the standards and did not establish results-oriented
                     criteria for evaluating performance. For example, the standards for the
                     “communication” job element included “conducts internal and external
                     communications through appropriate channels;” “writes clearly and
                     concisely, in compliance with established formats, and neat in
                     appearance;” and “promptly handles and responds to phone, local area
                     network, or voice mail messages.” It was not evident what environmental
                     results were anticipated by performing these communications in
                     accordance with the standards. We classified the remaining set of
                     standards for the “inspections” job element as somewhat results oriented.
                     This set was somewhat linked to EPA’s mission of protecting public health,
                     but the results scientists were expected to work toward and the criteria
                     EPA would use to evaluate performance toward achieving these results
                     were unclear.


                     We met with EPA officials, including the Acting Deputy Director of the
EPA’s Comments and   Office of Planning, Accountability, and Analysis and the Acting Director of
Our Evaluation       the Strategic Planning and Policy Services Division, on April 28, 1997, to
                     obtain their comments. The EPA officials agreed with the draft report’s
                     overall message that regulatory agencies face significant barriers in
                     becoming results oriented and noted that EPA faces a challenge in trying to
                     determine what portions of changes in environmental conditions can
                     realistically be considered a result of EPA’s programs. In fact, the officials
                     said they believed the draft report should more strongly reflect the
                     barriers that agencies face in order to emphasize that focusing on results
                     will take time and, in some situations, may be difficult to do. We agree that
                     managing for results may be difficult in some cases and, therefore,
                     included more information in the results-in-brief section of the final report
                     describing the barriers agencies identified.



                     Page 77                               GAO/GGD-97-83 Barriers to Managing for Results
Appendix V
EPA’s Mission, Strategic Goals, and
Performance Standards for Selected
Frontline Regulator Positions




EPA officials said that some EPA programs were upgrading their ability to
report on environmental results. They said that other EPA programs have
long had in place the environmental monitoring and reporting systems
needed to measure results and that the prospect of meeting GPRA reporting
requirements would not be an insurmountable challenge for these
programs. For example, the officials said that the “criteria air pollutants
program” has consistent, quality-assured data on both environmental
quality and emissions from many of the major sources of air pollution.
They said that these emissions data link EPA’s regulatory actions to
environmental results. However, they said that some EPA programs (those
that provide services) will have more difficulty determining
outcome-related results, as opposed to output-related (i.e., activity)
results. The EPA officials also said that the agency has made significant
progress in revising its goals and developing performance measures since
the time of our review in early January 1997.

EPA officials said that although it is appropriate to hold frontline regulators
accountable for organizational results, we should not have expected
agencies to have results-oriented performance standards that met our
criteria. They said that the Office of Personnel Management and the chief
financial officers and human resource communities have not decided how
to translate the mission of an agency into measurable standards that are
results oriented at the individual level. They also said that we based our
characterizations on a definition that could not be found in any law,
regulation, or guidance.

Although there may not yet be a consensus among the relevant
stakeholders or a formal definition regarding how the measurement of
employee performance should be linked to agencies’ results, the
President’s memorandum nevertheless requires that agencies change the
way they measure both agency and frontline regulators’ performance “so
as to focus on results, not process and punishment.” In addition, in a
June 15, 1995, report to the President, EPA stated that it reviewed its
existing employee performance standards and said that it had made some
changes to those standards and had plans for further changes. Nowhere in
the June 1995 report did EPA assert, as it does now, that it is inappropriate
to expect the agency’s employee performance standards to focus on
results. To develop our criteria for employee performance standards, we
considered Office of Personnel Management guidance on establishing
employee performance standards in conjunction with GPRA criteria for
goals and performance measures as a model.




Page 78                               GAO/GGD-97-83 Barriers to Managing for Results
Appendix V
EPA’s Mission, Strategic Goals, and
Performance Standards for Selected
Frontline Regulator Positions




The officials also said that one of the reasons we may have differed with
agencies’ conclusions about the results orientation of their employee
performance standards was the fact that we did not consider additional
contextual information when we reviewed the standards. As stated in the
draft report, we applied our criteria to the agencies’ performance
standards without other contextual information. We believe that in order
to characterize each agency’s performance standards using a fair and
balanced approach, it was necessary to limit our scope to the standards
themselves. Otherwise, we could have had additional information about
one aspect of an agency’s performance standard but not another, which
would have resulted in an inconsistent application of our criteria. In
addition, Office of Personnel Management guidance states that research
has shown that specific, hard goals result in higher performance than
vague or abstract goals, or no goals at all. Therefore, the President’s
initiative focusing standards on results could improve regulators’
performance in obtaining desired results.




Page 79                               GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI

Scope and Methodology


              To describe the agencies’ strategic goals and program performance
              measures, we first obtained copies of the goals and measures the agencies
              were using as of January 1, 1997. To describe the employee performance
              standards for selected regulator positions at these agencies, we first asked
              officials in each agency to identify frontline regulator positions that they
              felt would fairly represent the actions of their agencies’ regulators.1 We
              then obtained copies of the performance standards being used as of
              January 1, 1997, for the selected positions.

              To describe the extent to which agency officials and we believed that
              these goals, program performance measures, and employee performance
              standards focused on results, we developed a standardized data collection
              framework. We provided the initial framework to agency officials and told
              them that the criteria (as shown in table VI.1) were under development
              and would be further refined and revised as we progressed through the
              assignment. We left it up to the agency officials to decide whether they
              wanted to use these criteria, because we were primarily interested in
              obtaining their perspectives about the extent to which and, more
              importantly, why they considered their agencies’ goals, measures, and
              standards to be results oriented. At a minimum, however, we asked
              agencies to characterize their goals, program performance measures, and
              employee performance standards as results oriented to a great extent,
              some extent, or little or no extent.




              1
               Using our definition of “frontline regulators”—regulatory staff who work directly with the regulated
              community by enforcing regulations—the agencies designated which positions they considered
              “frontline regulator” positions.



              Page 80                                         GAO/GGD-97-83 Barriers to Managing for Results
                                             Appendix VI
                                             Scope and Methodology




Table VI.1: Initial Framework Provided
to Agency Officials                                                     To be considered results oriented to a great extent:
                                             Strategic goals            Must provide a clear definition of what results, effects, or
                                                                        impact (i.e., outcomes) are expected from the regulatory
                                                                        program, and permit assessment of progress in meeting
                                                                        the goal.
                                             Program performance        Should (1) assess the actual results, effects, or impact of
                                             measures                   a program activity compared with its intended purpose;
                                                                        (2) be linked to a goal; (3) permit comparison of
                                                                        organizational performance over time; and (4) if
                                                                        appropriate, include a range of measures such as quality,
                                                                        quantity, cost, and timeliness. (A range of measures
                                                                        would be appropriate when the goal is multipart and not
                                                                        self-measuring, or when the agency has set forth several
                                                                        measures and we can specify gaps in the measures
                                                                        used.)
                                             Employee performance       Should describe the objective or overall results which the
                                             standards                  frontline regulators are expected to accomplish, and set
                                                                        forth tangible or demonstrable criteria for evaluating each
                                                                        performance element directly associated with desired
                                                                        results. (It should enable a “cold reader” to clearly
                                                                        understand how an individual would have to perform to
                                                                        meet the standard.)

                                             In our analysis, we also characterized the agencies’ goals, program
                                             performance measures, and employee performance standards as results
                                             oriented to a great extent, some extent, or little or no extent. As we carried
                                             out these judgments, we expanded our initial framework somewhat as we
                                             learned more about how agencies’ goals or measures were structured. As a
                                             result, the framework we used was in the end somewhat more detailed
                                             than the initial framework we provided to agency officials. We did not
                                             believe these differences materially affected the outcome of our analysis.
                                             Specifically,

                                         •   We characterized an agency’s strategic goal as results oriented to a great
                                             extent if it was clearly linked to the agency’s mission, provided a clear
                                             definition of what results were expected, and permitted assessment of
                                             progress in achieving these results. (For example, IRS’ goal to increase
                                             taxpayer compliance was clearly related to IRS’ mission to collect the
                                             proper amount of tax revenue. It provided a clear definition of the
                                             expected results—“to encourage and assist taxpayers to voluntarily file
                                             timely and accurate returns and to pay on time, and, if taxpayers do not
                                             comply, to take appropriate compliance actions.” Finally, it is possible to
                                             assess progress toward increasing compliance.) If a goal was not linked to
                                             an agency’s mission, did not clearly define desired results, and did not
                                             permit assessment of progress toward achieving these results, we




                                             Page 81                               GAO/GGD-97-83 Barriers to Managing for Results
    Appendix VI
    Scope and Methodology




    considered it results oriented to little or no extent. We classified goals that
    fell between these two categories as results oriented to some extent.
•   We characterized an agency’s set of program performance measures as
    results oriented to a great extent if the measures (1) were clearly and
    unambiguously linked to the associated goal (or the agency’s mission, if
    the goal was not results oriented); (2) provided relevant information for
    assessing progress toward achieving the goal; and (3) addressed all
    aspects of the goal. We classified sets of program performance measures
    that were not linked to a goal or mission, that would not provide relevant
    data for an agency to assess progress toward a result, and did not address
    all aspects of the goal as results oriented to little or no extent. We
    considered sets of measures that fell between these categories as results
    oriented to some extent.

•   We considered a position’s performance standards to be greatly results
    oriented if they clearly described the results that the frontline regulators
    were expected to work toward, these results were clearly and
    unambiguously linked to the agency’s mission or goals, and the standards
    established criteria for evaluating performance directly associated with
    the desired results. We classified performance standards as having little or
    no results orientation if they were unclear about the results that the
    frontline regulators were expected to accomplish and did not establish
    criteria for evaluating performance related to expected results. We
    considered performance standards that fell between these two categories
    as results oriented to some extent. We did not include performance
    standards associated with “other duties as assigned” or “special projects”
    in this review because of the general nature of these standards.

    There are several aspects of our work which need to be emphasized to
    clarify the significance of our results. First, our review was not intended to
    determine agencies’ compliance with GPRA; compliance was not required at
    the time of our review. Furthermore, because we characterized the extent
    to which each agency goal and set of performance measures was results
    oriented, the application of these criteria represented a higher standard of
    results orientation than will be required by GPRA. As noted previously, the
    Act does not require that all of an agencies’ goals or measures be results
    oriented and the President’s March 4, 1995, memorandum does not
    explicitly require that each employee performance standard be results
    oriented to achieve the President’s objective.

    In applying our criteria for results orientation to the goals, performance
    measures, and performance standards, we limited our analysis to the



    Page 82                              GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI
Scope and Methodology




documents describing those goals, measures, and standards as of
January 1, 1997, as provided by the agencies. As a result, we did not
consider additional contextual information that may have existed and
could have resulted in our reaching different conclusions about results
orientation if it had been a part of our review. For example, it is possible
that we might have considered the goals or measures for an agency’s
training or inspection program as more results oriented if we had obtained
information about how they ultimately produced mission-oriented results.

We did not judge whether the agencies’ goals and program performance
measures were appropriate or achievable, only whether they appeared to
be results oriented. We also did not assess whether better program
performance measures existed that the agencies could have used. Because
many of these agencies were making changes to those goals and measures
and some of the performance standards at the time of our review, our and
the agencies’ assessments should be viewed as a snapshot at a particular
point in time.

Because we reviewed the performance standards for only a few frontline
regulator positions at each agency, the information we obtained is not
representative of all such positions. Rather, the discussion of these
standards should be viewed as examples, or case studies, of performance
standards used by the agencies as of January 1, 1997. Each of the positions
that were assessed contained several job elements (e.g., oral and written
communication or customer service) that, in combination, constituted the
major responsibilities for the position. Each job element, in turn, had a set
of performance standards that described the level of performance that the
employee needed to meet to be considered successful in that position.
Table VI.2 shows the seven frontline regulator positions that we focused
on in assessing employee performance standards. The positions had a total
of 30 job elements, each accompanied by a set of performance standards
associated with that element.




Page 83                             GAO/GGD-97-83 Barriers to Managing for Results
                                       Appendix VI
                                       Scope and Methodology




Table VI.2: Frontline Regulator
Positions and Number of Job Elements                                                                                                Number
as of January 1, 1997                                        Position (and number                                                     of job
                                       Agencya               of frontline regulators)       Description of position               elementsb
                                       OSHA                  Compliance safety and          Conduct workplace safety and                      3
                                                             health officer                 health inspections.
                                                             (930)c
                                       FAA                   Air traffic control            Regulate air traffic throughout
                                                             specialist (12,900)            the nation.                                       3
                                                             Aviation safety inspector      Inspect and certify aircraft for
                                                             (1,525)                        safe operations.                                  3
                                       IRS                   Customer service               Provide information and
                                                             representative (5,750)         direction on tax issues to the                    5
                                                                                            public.
                                                             Revenue officer (7,360)        Collect delinquent taxes and                      5
                                                                                            secure delinquent returns.
                                       EPA                   Criminal investigator          Investigate allegations or
                                                             (225)d                         suspected violations of                           5
                                                                                            criminal laws.
                                                             Environmental scientist        Implement and enforce
                                                             (1,225)c                       environmental regulations.                        6
                                                             e
                                       Total                                                                                                 30
                                       a
                                        FDA is not included because, as of January 1, 1997, the agency was using performance
                                       standards that were customized for individual employees.
                                       b
                                        We reviewed the sets of performance standards that were associated with each of these job
                                       elements.
                                       c
                                       Includes both safety and health inspector positions.
                                       d
                                        The number of EPA frontline regulators (for the criminal investigator and environmental scientist
                                       positions) were full time equivalent (FTE) staffing levels, not number of frontline regulators. In
                                       addition, the number of FTEs in the environmental scientist position includes only environmental
                                       scientists in EPA’s regional offices. EPA officials said that they would not classify environmental
                                       scientists who work in headquarters as frontline regulators.
                                       e
                                        Not applicable because the numbers listed include both number of staff and FTEs and therefore
                                       these amounts cannot be added to a meaningful total.

                                       Source: Agency information.



                                       To describe the aids and barriers that officials in the agencies said they
                                       faced in attempting to focus on results, we asked officials in each agency
                                       to describe the aids and barriers they had experienced. We also reviewed
                                       related documents that described any problems or successes agencies had
                                       encountered.




                                       Page 84                                         GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII

Comments From OSHA and IRS




               Page 85   GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII
Comments From OSHA and IRS




Page 86                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII
Comments From OSHA and IRS




Page 87                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII
Comments From OSHA and IRS




Page 88                      GAO/GGD-97-83 Barriers to Managing for Results
Appendix VIII

Major Contributors to This Report


                     Curtis Copeland, Assistant Director, Federal Management and Workforce
General Government     Issues
Division             Susan Ragland, Evaluator-in-Charge
                     Steven Lozano, Senior Evaluator
                     Thomas Beall, Technical Analyst
                     Kiki Theodoropoulos, Communications Analyst
                     Joseph S. Wholey, Senior Advisor for Evaluation Methodology




(410043)             Page 89                          GAO/GGD-97-83 Barriers to Managing for Results
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested