oversight

Moving Towards A Results-Oriented Organization --- A Report on the Status of ED's Implementation of the Results Act.

Published by the Department of Education, Office of Inspector General on 1998-09-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

    Moving Towards A Results-Oriented Organization
                  A Report on the Status of ED’s Implementation of the Results Act



                                           TABLE OF CONTENTS


EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

BACKGROUND - THE RESULTS ACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

BACKGROUND - ED’s PLANS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

AUDIT RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

          ED Needs to Take Additional Steps to Effectively Implement the Results Act . . . . . . . . 7

          ED Needs to Finalize and Implement a Process for the Accurate and Timely
                Reporting of GPRA Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

OTHER MATTERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

          Valid, Reliable, and Timely Performance Information . . . . . . . . . . . . . . . . . . . . . . . . . . 20

          Range of Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

OBJECTIVES, SCOPE, METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

STATEMENT ON MANAGEMENT CONTROLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Appendix A - Results Act Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Appendix B - ED’s Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Appendix C - Publications Cited in This Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Appendix D - Report Distribution List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37


ACN 17-70007                                          September 1998
Moving Towards A Results-Oriented Organization                                              FINAL



                                                 EXECUTIVE SUMMARY


                            The objectives of our audit were to assess the status of the U.S.
                            Department of Education’s (ED) implementation of the Government
                            Performance and Results Act of 1993 (GPRA or the Results Act) and
                            the development of systems for the timely and accurate reporting of
                            performance indicators. We conducted our audit between October
                            1997 and March 1998 and our results address what was found during
                            that period. Since that time, ED has engaged in additional activities to
                            implement the Results Act. Moving towards a results-oriented
                            organization is a dynamic, evolving process. Our assessment
                            represents a “snapshot” of where ED was in implementing the Results
                            Act.

                            The Results Act is the centerpiece of a statutory framework that
                            Congress put in place to improve federal management and provide a
                            greater focus on results. GPRA requires that federal agencies prepare
                            a five year strategic plan and annual performance plans beginning with
                            fiscal year 1999. The Results Act also requires that federal agencies
                            prepare performance reports. The first performance report on fiscal
                            year 1999 is due March 2000.

                            ED has prepared a strategic plan and an annual performance plan for
                            fiscal year 1999. ED has distributed those plans and established a
                            reporting system on progress with the objectives in the strategic plan.
                            However, during our audit, we identified issues similar to those raised
                            recently by GAO: the need to establish a results-oriented culture; the
                            importance of senior leadership involvement; the challenging nature of
                            measuring the federal contribution; and the importance of the
                            information being used by the federal agencies and Congress in
                            decision-making. To address these issues and effectively implement the
                            Results Act, ED needs to take additional steps. We recommend that
                            ED reassess its reporting systems to the Secretary and Deputy
                            Secretary, increase senior leadership visibility, and work with Congress
                            to enact any needed changes to program legislation.

                            As required by the Results Act, ED has designed a framework for the


ACN 17-70007                            September 1998                                      Page 1.
Moving Towards A Results-Oriented Organization                                              FINAL


                            verification and validation of its performance indicators. ED now
                            needs to finalize and implement a process for assembling the data and
                            analyzing that data, and preparing the performance report. To ensure
                            accurate and fair reporting, ED needs to establish controls over the
                            analysis and reporting of performance indicators in its performance
                            report. We are recommending actions to address these issues.

                            During our audit, we noted certain other matters which warrant
                            consideration by ED as it continues to implement the Results Act. To
                            effectively implement the Results Act, ED will need valid, reliable, and
                            timely data about program performance. However, in some cases, ED
                            lacks such information. Obtaining quality data will warrant
                            consideration by ED as it continues to implement the Results Act.

                            With 930 performance indicators, ED also faces a significant reporting
                            burden. ED should reassess the number of indicators. In addition, data
                            sources have not been identified for all indicators and baselines have
                            not been established for some indicators. ED should consider requiring
                            program offices to identify the remaining data sources and establish
                            baselines by October 1998.

                            ED faces many challenges in moving towards a results-oriented
                            organization. ED’s strategic plan and annual performance plan, which
                            were rated by Congressional evaluators above most of the federal
                            agencies’ plans, and the activities ED has done and plans to do can
                            provide ED with the foundation to meet those challenges.

                            ED concurred with the findings and recommendations in this report.
                            ED’s comments, including an attachment indicating corrective actions
                            underway or planned, are presented in their entirety in Appendix B.




ACN 17-70007                            September 1998                                      Page 2.
Moving Towards A Results-Oriented Organization                                              FINAL



                                     BACKGROUND - THE RESULTS ACT

                            The Government Performance and Results Act of 1993, commonly
                            referred to as “GPRA” or “the Results Act,” was enacted as the
                            centerpiece of a statutory framework that Congress put in place to
                            improve federal management and provide a greater focus on results.
                            The Results Act seeks to shift the focus of government decision making
                            and accountability away from a preoccupation with the activities that
                            are undertaken by a Federal agency - such as the number of grants
                            awarded - to a focus on the results of those activities - such as real
                            gains in ensuring equal access to education for all individuals.

                            Under the Results Act, agency heads were required to submit a five-
                            year strategic plan to the Congress and the Office of Management and
                            Budget (OMB), no later than September 30, 1997. Updates are
                            required at least every three years thereafter. (Appendix A contains the
                            requirements for the strategic plans.)

                            Beginning with fiscal year 1999 (October 1, 1998 to September 30,
                            1999), and annually thereafter, agencies must submit to OMB
                            performance plans covering each program activity in the agency’s
                            budget. Using the agencies’performance plans, OMB must prepare a
                            government wide performance plan for inclusion in the President’s
                            annual budget submission to Congress. (Appendix A contains the
                            requirements for the annual performance plans.)

                            Beginning with March 31, 2000, and annually thereafter, agency heads
                            must submit program performance reports to the President and
                            Congress, covering performance for the previous fiscal year. Program
                            performance reports beginning in fiscal year 2002 must include actual
                            program performance results for the three preceding fiscal years.
                            (Appendix A contains the requirements for the program performance
                            reports.)




ACN 17-70007                            September 1998                                      Page 3.
Moving Towards A Results-Oriented Organization                                                     FINAL



                                                 BACKGROUND - ED’s PLANS


                                   Strategic Plan

                                   The U.S. Department of Education (ED) has met the statutory
                                   requirements for submitting a strategic plan to Congress and OMB.
                                   ED’s Strategic Plan for 1998-2002 integrated its mission and program
                                   authorities, and described how it will work to improve education.
                                   ED’s Strategic Plan has four goals. Under those four goals, there are
              Time Line
                                   22 objectives.
 Dec 1994 ED prepared its first
          strategic plan.
                                   In January 1998, the General Accounting Office (GAO) issued a report
 Mar 1997 ED provided program      on their review of 24 major federal agencies’(including ED) strategic
          performance plans for
          17 major programs to     plans that were formally submitted to Congress and OMB.1 In that
          Congress for
          consultation.
                                   report, GAO commented favorably on ED’s strategic plan. GAO
                                   concluded that the agencies’ plans appeared to provide a workable
 Sep 1997 ED submitted its
          strategic plan to        foundation for Congress to use in helping to fulfill its appropriation,
          Congress.                budget, authorization and oversight responsibilities and for agencies to
 Feb 1998 ED submitted to          use in setting a general direction for their efforts. Congressional
          Congress with the        evaluations of agency draft and final strategic plans for compliance
          budget its annual
          performance plan for     ranked ED second both times, following the Social Security
          fiscal year 1999.        Administration for the draft and the Department of Transportation for
 Oct 1998 Start of fiscal year     the final.
          1999.

 Mar 2000 Due date for the first
          performance report.
                                   Performance Plan

                                   ED has met the statutory requirement for submitting an annual
                                   performance plan for fiscal year 1999 to Congress and OMB. ED’s
                                   annual performance plan included annual performance plans for each
                                   of the 22 objectives in ED’s Strategic Plan and 99 program
                                   performance plans for each of ED’s programs reported individually or




          1
         GAO, 1998. MANAGING FOR RESULTS: Agencies’Annual Performance Plans Can Help
Address Strategic Planning Challenges.

ACN 17-70007                                   September 1998                                      Page 4.
Moving Towards A Results-Oriented Organization                                                  FINAL


                              grouped by related program purpose.2 Congressional evaluations of
                              the agency annual performance plans ranked ED third, following the
                              Department of Transportation and the Department of Veterans Affairs.

                              On June 8, 1998, subsequent to our audit work, GAO issued its report
                              on ED’s fiscal year 1999 performance plan. In that report, GAO
                              expressed concerns about the description of ED’s intended
                              performance, discussion of strategies and resources, and recognition of
                              data limitations and the plans for addressing those limitations. GAO
                              commented favorably on the plan’s discussion of coordination with
                              other federal agencies and verification and validation of performance
                              information.3


                              Congressional Budget Justification

                              The Results Act required that OMB have agencies prepare annual
                              performance plans beginning with fiscal year 1999. As a result, OMB
                              issued Circular A-11 Part 2 which requires that agencies include
                              performance information, including annual goals and indicators, in their
                              written budget justifications to explain major program issues and
                              financial requirements. Each agency was to work with OMB
                              representatives to determine the form and content. A precise format
                              was not prescribed. ED’s budget justification for fiscal year 1999 was
                              the first one to be prepared under this requirement.




       2
         In this report, annual performance plan refers to ED’s agency performance plan which contains
the 22 objective plans and the 99 program performance plans. Program performance plans refers to the 99
plans.
       3
        GAO, 1998. The Results Act: Observations on the Department of Education’s Fiscal Year 1999
Annual Performance Plan. Pg 9.

ACN 17-70007                               September 1998                                       Page 5.
Moving Towards A Results-Oriented Organization                                              FINAL



                                                   AUDIT RESULTS

                            The objectives of our audit were to assess the status of the U.S.
                            Department of Education’s (ED) implementation of the Government
                            Performance and Results Act of 1993 (GPRA or the Results Act) and
                            the development of systems for the timely and accurate reporting of
                            performance indicators. We conducted our audit between October
                            1997 and March 1998 and our results address what was found during
                            that period. Since that time, ED has engaged in additional activities to
                            implement the Results Act. Moving towards a results-oriented
                            organization is a dynamic, evolving process. Our assessment
                            represents a “snapshot” of where ED was in implementing the Results
                            Act.

                            As required by the Results Act, ED has prepared a strategic plan and
                            an annual performance plan for fiscal year 1999, and designed a
                            framework for the validation and verification of performance
                            indicators. ED distributed those plans and established a reporting
                            system for progress with the objectives in the strategic plan.

                            In order to be prepared to meet the additional requirements of the
                            Results Act in a timely and proper manner, ED now needs to take
                            additional steps, including finalizing and implementing a process for
                            the accurate and timely reporting of GPRA performance indicators.

                            ED faces many challenges in moving towards a results-oriented
                            organization. ED’s strategic plan and annual performance plan, which
                            were rated by Congressional evaluators above most of the federal
                            agencies’ plans, and the activities ED has done and plans to do can
                            provide ED with the foundation to meet those challenges.

                            ED concurred with the findings and recommendations in this report.
                            ED’s comments, including an attachment indicating corrective actions
                            underway or planned, are presented in their entirety in Appendix B.




ACN 17-70007                            September 1998                                      Page 6.
Moving Towards A Results-Oriented Organization                                               FINAL



                                   ED Needs to Take Additional Steps to Effectively
                                             Implement the Results Act


                            As required by the Results Act, ED has prepared a strategic plan and
                            an annual performance plan for fiscal year 1999. ED has distributed
                            those plans and established a reporting system on progress with the
                            objectives in the strategic plan. However, during our audit, we
                            identified issues similar to those raised recently by GAO: the need to
                            establish a results-oriented culture; the importance of senior leadership
                            involvement; the challenging nature of measuring the federal
                            contribution; and the importance of the information being used by the
                            federal agencies and Congress in decision-making. To address these
                            issues and effectively implement the Results Act, ED needs to take
                            additional steps. We recommend that ED reassess its reporting
                            systems to the Secretary and Deputy Secretary, increase senior
                            leadership visibility, and work with Congress to enact any needed
                            changes to program legislation.


                            The Results Act Requires a Results-Oriented Organization

                            The Committee Report accompanying the Results Act states that the
                            goals in the annual performance plans “should drive much of the daily
                            operations of the agency, and they should aim at achieving the long-
                            term general goals of the agency’s strategic plan.”4 The Committee
                            Report also states that “the ultimate objective is to change agency and
                            managerial behavior — not create another bureaucratic system.”


                            ED’s Initial Activities to Implement the Results Act

                            ED has taken initial steps in implementing the Results Act, including
                            the following:

                            Distribution of Plans


       4
        Committee on Governmental Affairs, U.S. Senate, 1993. Report to Accompany the Government
Performance and Results Act of 1993.

ACN 17-70007                            September 1998                                       Page 7.
Moving Towards A Results-Oriented Organization                                                         FINAL


                                 In November 1997, ED distributed its strategic plan desk-to-desk at
                                 headquarters and arranged for similar distribution in the regional
                                 offices. In December 1997, the Deputy Secretary’s message in ED’s
                                 internal newsletter was devoted to encouraging ED employees to
                                 review the strategic plan and identify their role in its implementation.

                                 The Deputy Secretary distributed to the Assistant Secretaries the
                                 Department’s final annual performance plan along with an
                                 individualized transmittal note. That transmittal emphasized the role
                                 that each Assistant Secretary’s office plays in implementing the plan.
                                 The transmittal also recommended that commitment to the strategic
                                 plan should be a key factor in determining bonuses and awards.

                                 Weekly Reports to the Secretary

                                 Each office submits a report weekly to the Secretary structured by the
                                 Seven Priorities.5 The Seven Priorities were developed by the
                                 Secretary and senior department officials from the President’s 1997
                                 State of the Union Address. The Seven Priorities are incorporated into
                                 ED’s strategic plan.

                                 Periodic Reports to the Deputy Secretary

                                 The senior managers responsible for each of the 22 objectives in the
                                 strategic plan submit periodic reports to the Deputy Secretary. In
                                 addition, those managers and their staff meet periodically with the
                                 Deputy Secretary to discuss progress on achieving those objectives.
                                 This system is similar to the system that was used to track progress
                                 with ED’s first strategic plan in 1994.6


        5
          We reviewed the weekly reports between October 4, 1997 and April 4, 1998. During that time,
two program offices reported less than six times on any of the priorities; three staff offices never reported
on any of the priorities; and one staff office reported once.
        6
          15 of the 27 people we interviewed were familiar enough with the reporting system to evaluate it.
Opinions on the usefulness of the reporting system were divided. Strengths: the importance of having a
process in place, the visual nature of the report and the face-to-face meetings with the Deputy Secretary.
Weakness: the report could not convey all the necessary information. One questioned whether the
information reported is only information that people believe the Deputy Secretary prefers to hear. When
asked if written feedback should be provided on the reports, some raised concerns that too much

ACN 17-70007                                  September 1998                                           Page 8.
Moving Towards A Results-Oriented Organization                                                       FINAL


                                Budget Document

                                ED’s fiscal year 1999 budget justification incorporated some of its
                                program performance objectives and indicators. The sections in the
                                budget justification can be linked to the 99 program performance plans.


                                Results of Interviews of Selected Key Staff

                                In order to determine actions ED has taken and plans to take in
                                implementing GPRA, in addition to reviewing relevant documentation,
                                we conducted 27 structured interviews with staff in ED offices. For
                                these interviews, we judgmentally selected key staff who were either
                                involved with the strategic plan or the program performance plans or
                                who were operational or policy managers of ED programs and were
                                in positions where they should have been knowledgeable about the
                                implementation of the Results Act. Of the 27 individuals interviewed,
                                all but four represented themselves as having some involvement with
                                preparation or review of their programs’performance plans. The four
                                who said that they did not have involvement with their programs’
                                performance plans were from OPE. In order to foster candor during
                                the interviews, we advised those interviewed that we would not
                                disclose their individual identities in reporting the results.7

                                OIG Observations from Interviews

                                Nearly all of the interviewees indicated that they believed that the
                                planning, performance measurement, and accountability required by
                                the Results Act are long-term focuses of the federal government.
                                Nearly all of the interviewees were supportive of the accountability and
                                performance measurement required by the Results Act.

                                During our interviews, we noted that:

                                     # All the people we interviewed had a conceptual understanding


formalization would slow the process.
        7
          The methodology for the structured interviews is explained in more detail in the section of the
report entitled “Objectives, Scope, Methodology.”

ACN 17-70007                                 September 1998                                          Page 9.
Moving Towards A Results-Oriented Organization                                                   FINAL


                                        of performance measurement.

                                    # In more than half of the offices, training had been given or
                                      would be given to some but not all employees.

                                    # In about half of the program offices, references to GPRA
                                      related activities were already added to upper management’s
                                      performance agreements or were in the process of being
                                      added.8

                                    # Less than half of the program offices were considering non-
                                      monetary awards related to GPRA activities.

                                    # Except for some within the Office of Postsecondary Education
                                      (OPE), the people we interviewed were able to describe how
                                      the program performance plans were to be used by their
                                      offices. In OPE, the four who stated that they did not have
                                      involvement with their programs’performance plans and some
                                      of the other interviewees stated that they did not know how
                                      the program performance plans were to be used.9

                               Interviewees’Assessment

                               During the interviews, we asked what changes were needed and what
                               barriers there were to implementing the Results Act and what
                               improvements had occurred because of, or since, the Results Act:

                                    # The single most commonly identified theme in the category of
                                      changes needed for the Department to better implement the
                                      Results Act was to have senior officials being more involved
                                      or more visible in providing leadership for the implementation
                                      of the strategic plan.

                                    # Many interviewees recommended that ED put greater focus


        8
          ED’s annual performance plans indicates that performance appraisals of senior management will
be aligned to the strategic plan.
        9
          Subsequent to our interviews, on February 18, 1998, the Assistant Secretary of OPE assigned
responsibility for monitoring each of the OPE program indicators to individual senior managers.

ACN 17-70007                               September 1998                                      Page 10.
Moving Towards A Results-Oriented Organization                                              FINAL


                                     on its GPRA activities and ensure that they are integrated into
                                     the day-to-day activities.

                                 # The three most frequently identified barriers to successful
                                   implementation were:

                                     • ED not having sufficient number of staff qualified in
                                       information processing, evaluation and reporting;

                                     • the difficulty of analyzing and interpreting performance
                                       measurement data; and

                                     • the possibility of Congress not using the information.

                                 # Many of the interviewees identified the need for the GPRA
                                   process to be a continuous system for program improvement.
                                   Some of the recurring themes noted during the interviews
                                   were:

                                     • There has already been an impact from previous
                                       performance-based management efforts and from
                                       preparing to measure program performance for the Results
                                       Act.

                                     • There is anticipation that there will be additional impact
                                       and that programs and budgets will be changed.

                                     • One potential or needed outcome is more alignment across
                                       programs towards common goals.

                                     • Congressional actions after data is available will be a
                                       motivating factor for program managers.

                                   • Conflicts between program legislation and the Results Act
                                      may make it difficult to rapidly implement the changes
                                      suggested by program indicators.
                            Our Assessment of ED’s Implementation of the GPRA

                            ED has distributed its plans, assigned responsibility, and established a
                            reporting system. The key staff we interviewed generally understood


ACN 17-70007                            September 1998                                     Page 11.
Moving Towards A Results-Oriented Organization                                                 FINAL


                               strategic planning and performance measurement and were supportive
                               of efforts to implement planning and performance measurement into the
                               federal government and into ED. Those key staff identified the need for
                               more involvement by senior leadership and greater focus on GPRA
                               activities to ensure integration into day-to-day activities.

                               The issues the interviewees raised on changes that are needed and on
                               barriers are similar to issues that GAO has identified in recent reports
                               and testimony:

                                   # the need to establish a results-oriented culture;10

                                   # the importance of senior leadership involvement;11

                                   # the challenging nature of measuring the federal contribution;
                                     and12

                                   # the importance of the information being used by the federal
                                     agencies and Congress in decision-making.13

                               We are concerned that ED has two reporting systems: one to the
                               Secretary by the Seven Priorities and one to the Deputy Secretary by
                               the Strategic Plan. While we understand that the Seven Priorities are
                               incorporated into the Strategic Plan, we believe that having reports to
                               the Secretary structured by the Seven Priorities dilutes the
                               Department’s focus on the Strategic Plan.

                               We recognize that implementing the Results Act is a long-term,
                               challenging process. As GAO pointed out in a recent report on GPRA,

        10
        GAO, 1997. MANAGING FOR RESULTS: Prospects for Effective Implementation of
Government Performance and Results Act. Pg 12-13.
        11
             GAO, 1997. HIGH RISK AREAS: Actions Needed to Solve Pressing Management Problems.
Pg 6.
        12
             GAO, 1997. MANAGING FOR RESULTS: Analytic Challenges in Measuring Performance.
Pg 3.
        13
          GAO, 1997. MANAGING FOR RESULTS: Using GPRA to Assist Congressional and
Executive Branch Decisionmaking. Pg 12.

ACN 17-70007                               September 1998                                     Page 12.
Moving Towards A Results-Oriented Organization                                                FINAL


                            federal agencies need more progress in developing and sustaining
                            cultures that focus on results.14 We also realize that, in some cases,
                            program legislation may need to be changed in order to achieve the
                            goals of the Results Act. With that in mind, we have made
                            recommendations to help ED develop and sustain a results-oriented
                            culture, in line with the Results Act.


                            OIG Recommendations

                            We recommend that ED:

                                 (1) Reassess its reporting systems to the Secretary and Deputy
                                     Secretary to ensure a consistent, uniform focus;

                                 (2) Increase senior leadership visibility with the strategic plan to
                                     ensure integration of the plan in day-to-day activities, e.g., the
                                     Secretary and Deputy Secretary could hold Department-wide
                                     meetings to formally introduce ED’s strategic plan and its
                                     annual plan and periodically update employees on its status;

                                 (3) Integrate the strategic plan and annual performance plan into
                                     day-to-day activities, e.g., senior managers could regularly
                                     communicate with their employees about the importance of
                                     ED’s strategic plan and the annual performance plan and how
                                     those plans relate to the work;

                                 (4) Review program legislation to identify where it could be more
                                     results-oriented or where data collection could be improved,
                                     and develop and submit recommended changes; and work with
                                     Congress to enact such legislative changes.

                            Department’s Comments

                            ED concurred with this finding and the recommendations. ED’s
                            comments, including an attachment indicating corrective actions
                            underway or planned, are presented in their entirety in Appendix B.


       14
        GAO, 1997. MANAGING FOR RESULTS: Prospects for Effective Implementation of the
Government Performance and Results Act. Pg 12.

ACN 17-70007                            September 1998                                       Page 13.
Moving Towards A Results-Oriented Organization            FINAL




ACN 17-70007                            September 1998   Page 14.
Moving Towards A Results-Oriented Organization                                                         FINAL



                                         ED Needs to Finalize and Implement a Process
                                           for the Accurate and Timely Reporting of
                                                 GPRA Performance Indicators


                                As required by the Results Act, ED has designed a framework for the
                                verification and validation of its performance indicators. ED now
                                needs to finalize and implement a process for assembling the data and
                                analyzing that data, and preparing the performance report. To ensure
                                accurate and fair reporting, ED needs to establish controls over the
                                analysis and reporting of performance indicators in its performance
                                report.


                                Results Act Requirement for a Performance Report

                                Fiscal year 1999, which begins October 1998, is the first year a
                                performance report will be required under the Results Act. The
                                performance report on fiscal year 1999 is due March 2000. ED will
                                have six months to prepare its first program performance report. To do
                                so, ED will need a process to assemble the data needed, analyze that
                                data and prepare the report. In addition, the Results Act requires that
                                each agency have a means to verify and validate the measured values.


                                ED’s Plan for a GPRA Performance Report

                                ED has not yet established a process for assembling and analyzing the
                                data or preparing the report. ED does have a process for receiving
                                periodic information on the status of the 115 performance indicators in
                                the strategic plan.15 As part of the quarterly reporting on progress
                                towards the objectives in the strategic plan, the senior managers
                                responsible for each of the 22 objectives provide information on data
                                sources for and analysis of the performance indicators.



        15
           ED has 930 indicators: 115 indicators for the 22 objectives in the strategic plan and 815
indicators for the 99 program performance plans.


ACN 17-70007                                 September 1998                                        Page 15.
Moving Towards A Results-Oriented Organization                                                      FINAL

                                As required by the Results Act, ED included in its annual performance
                                plan the framework for the validation and verification of its
                                performance indicators.16 ED’s framework will include:

                                     # requiring program managers to provide an assertion on the
                                       reliability and validity of their data sources or identify the
                                       weaknesses and have a plan for improvement;

                                     # establishing data collection standards;

                                     # subjecting studies and evaluations that will provide data for
                                       performance indicators to review by expert panels; and

                                     # monitoring selected data sources and the assertions through
                                       evaluations by the Planning and Evaluation Service (PES)17
                                       and audits conducted by the Office of Inspector General
                                       (OIG).

                                In addition, ED plans to provide training to staff on performance
                                measurement. ED has already issued guidance on performance
                                measurement through the PES sponsored Guide To Program Outcome
                                Measurement.18


                                Barriers to Analyzing and Reporting


        16
          GAO noted that the plan adequately addressed how ED plans to validate and verify performance
information for its postsecondary and to some extent its elementary and secondary education programs.
(GAO, 1998. THE RESULTS ACT: Observations on the Department of Education’s Fiscal Year 1999
Annual Performance Plan, Pg 3.)
        17
           PES is a division within the Office of the Under Secretary (OUS) responsible for evaluating the
effectiveness of ED programs.
        18
           We reviewed that guide and found it, in general, to be useful. During our interviews, we polled
individuals to determine how many were familiar with the guide. Of 27 interviewees, 13 were familiar
enough with the guide to rate it. Most rated it “somewhat to moderately useful” (2 on a 5 point
scale).Although not requested, eleven of the 13 familiar with the guide provided verbal comments on the
guide: five noted that the guide was widely distributed; three noted that the guide may not have been used;
two indicated that the guide would have been more helpful if provided earlier; and one thought that training
should have accompanied the guide.


ACN 17-70007                                 September 1998                                        Page 16.
Moving Towards A Results-Oriented Organization                                                        FINAL

                                 ED’s annual performance plan identifies the lack of formal training of
                                 staff in information processing, evaluating, and reporting as a barrier
                                 to providing performance information. As mentioned above, ED is
                                 planning to provide training on performance measurement to staff.
                                 Based on our audit, we have additional concerns about the use of
                                 secondary statistics and the interpretation of data.

                                 The data sources for some indicators are secondary statistics19, such as
                                 education statistics from the National Center for Education Statistics
                                 (NCES)20. Secondary statistics should be used with care.

                                      “Secondary statistics must not be used indiscriminately. The
                                      user should be thoroughly aware of the inclusions and
                                      exclusions, definitions of terms, scope of the populations
                                      covered, period of time covered by the data, method of
                                      collecting data..., and other points of information that
                                      delineate the nature and limitation of the data.”21

                                 Program managers need to ensure that secondary statistics are usable
                                 for the purposes the program manager intends. One way to achieve
                                 this is through direct communication with the providers of the
                                 secondary statistics.

                                 During our interviews, some of the program managers indicated that
                                 either NCES representatives were not involved with the program
                                 performance plans or that they did not know if NCES representatives
                                 were involved with the program performance plans. The GPRA
                                 coordinators in the program offices were aware that NCES reviewed
                                 and commented on the program performance plans as part of the
                                 review by OUS. We are concerned that program managers are not
                                 actively communicating with providers of secondary statistics about the


        19
          For this report, we define a “secondary statistic” as data that has already been collected for other
purposes and is available in public records.
        20
           NCES is a division within the Office of Educational Research and Improvement. NCES is the
primary federal entity for collecting, analyzing, and reporting data related to education in the U.S. and
other nations.
        21
             Mandel, B.J., 1984. Statistics for Management. Page 52.


ACN 17-70007                                 September 1998                                         Page 17.
Moving Towards A Results-Oriented Organization                                                FINAL

                            nature and limitations of those statistics prior to use in the program
                            performance plans.

                            Some interviewees noted that much of the data is subjective and will
                            need to be interpreted. In addition, some interviewees noted that
                            determining the cause of changes in performance indicators will be
                            difficult. For example,

                                 # How can a trend in a performance indicator be related to a
                                   specific ED program?

                                 # How can the impact of specific programs be determined when
                                   multiple programs (sometimes among more than one federal
                                   agency) are working to achieve similar goals?

                                 # How can the federal contribution be determined when
                                   significant state and local effort is involved?


                            OIG Conclusions

                            ED has designed a framework for the verification and validation of its
                            performance indicators. However, ED has not defined a process for
                            how it will assemble and analyze the data and how it will prepare the
                            required program performance reports. In our opinion, the basic
                            process needs to be defined and communicated prior to the beginning
                            of fiscal year 1999 (the first year for which program performance
                            reports are required) so that each individual understands the basic
                            process and his and her role in it before the beginning of fiscal year
                            1999. This is to help ensure that proper mechanisms are in place to
                            measure performance occurring in fiscal year 1999. In addition,
                            because the credibility of the performance report will depend upon the
                            credibility of the use and interpretations of the data, we believe that ED
                            should establish controls over the analysis of data and the reporting of
                            performance information. We recognize that additional evaluation may
                            be needed to interpret performance data.


                            OIG Recommendations

                            We recommend that OUS:

ACN 17-70007                            September 1998                                      Page 18.
Moving Towards A Results-Oriented Organization                                               FINAL

                                 (1) Establish a process for assembling the data and other
                                     information needed in the performance report, i.e., persons
                                     responsible, due dates, formats, etc.;

                                 (2) Establish controls over analysis and reporting of data including
                                     secondary statistics, e.g., requiring documentation of analysis
                                     (such as the nature and limitations of the data, causality
                                     models or identification of external factors), requiring reviews
                                     of analysis, developing a framework for the performance
                                     report, and establishing standards for reporting performance
                                     information;

                                 (3) Communicate to ED staff the process and their role and
                                     responsibilities in it; and

                                 (4) Establish a formal department-wide system for tracking the
                                     indicators, e.g., an electronic database.

                            Department’s Comments

                            ED concurred with this finding and the recommendations. ED’s
                            comments, including an attachment indicating corrective actions
                            underway or planned, are presented in their entirety in Appendix B.




ACN 17-70007                            September 1998                                     Page 19.
Moving Towards A Results-Oriented Organization                                           FINAL


                                                   OTHER MATTERS


                            Valid, Reliable, and Timely Performance Information

                            The scope of our work did not include a review of the quality of any
                            particular data source or make any determinations on any individual
                            data improvement effort. Therefore, the following observations on the
                            quality of ED’s data was a by-product of our review of ED’s
                            implementation of the Results Act and its development of a reporting
                            system.

                            ED officials, Congress, and others need reliable, timely information
                            about program performance and costs to operate, manage and oversee
                            ED’s programs. However, that information is not free and the goal of
                            having enough information for accountability and program management
                            competes with initiatives to lessen the burden on state and local
                            governments and others and provide for more flexibility.

                            The single most common issue raised during our interviews was the
                            availability of quality data. Specifically, interviewees noted that:

                                 # Some current data collections need to be changed to make
                                   them more useful.

                                 # Some current data collections are general purpose statistics
                                   about the state of Education in the U.S. and thus not the most
                                   useful for managing and evaluating specific programs.

                                 # Some current data collections are known to have data quality
                                   problems.

                                 # Data is not always available in a timely fashion. Lags are
                                   caused both by waiting for data submissions and time to verify
                                   and refine the data submitted.

                                 # Data is not collected often enough. Some data is available
                                   only annually or even less frequently.

                            Some interviewees were also concerned that the data sources cited for

ACN 17-70007                            September 1998                                  Page 20.
Moving Towards A Results-Oriented Organization                                                FINAL

                            GPRA reporting were not permanent. Some of the data sources are
                            multi-year studies which take years to collect data. Then, if not
                            renewed, the data is not available for future comparisons.

                            Availability of quality data has been identified by both ED and GAO as
                            a concern. ED’s annual plan noted that the lack of integration of ED’s
                            SFA systems and its heavy dependence on external systems hampers its
                            ability to provide timely and accurate information. Recent GAO
                            reports have cited data concerns. For example, some programs allow
                            for a wide range of activities and permit states to define the information
                            they collect on program activities and effectiveness. With no
                            requirement that states use consistent measures, ED faces a difficult
                            challenge in assembling reports necessary to develop a nationwide
                            picture of the program’s effectiveness. In addition, GAO noted that
                            inaccurate loan information provided by guaranty agencies impairs
                            ED’s ability to manage the loan programs.

                            These concerns warrant consideration by ED as it continues to
                            implement the Results Act.

                            On June 8, 1998, subsequent to our audit work, GAO issued its report
                            on ED’s fiscal year 1999 performance plan. In that report, GAO noted
                            that the plan does not provide sufficient confidence that its elementary
                            and secondary education performance information will be credible.22


                            Range of Key Performance Indicators

                            ED has 930 indicators: 115 indicators for the 22 objectives in its
                            strategic plan and 815 indicators for the 99 program performance
                            plans. Each program performance plan has between 1 and 28
                            indicators. Data sources have not been identified for all indicators and
                            many baselines have not yet been established. ED should require
                            program offices to identify the remaining data sources and establish the
                            remaining baselines by October 1, 1998.

                            With 930 indicators, ED faces an enormous task in preparing its


       22
         GAO, 1998. THE RESULTS ACT: Observations on the Department of Education’s Fiscal
Year 1999 Annual Performance Plan. Pg 3.


ACN 17-70007                            September 1998                                      Page 21.
Moving Towards A Results-Oriented Organization                                               FINAL

                            performance report. While all the indicators may be necessary for
                            internal management, as part of the first reporting cycle, ED should
                            reassess the indicators to determine if all 930 are necessary for GPRA
                            reporting to OMB and Congress. ED should consult with OMB and
                            Congress about any indicators which it identifies for possible
                            elimination.

                            On May 7, 1998, subsequent to our audit work, the Deputy Secretary
                            requested that for the fiscal year 2000 budget, the Assistant Secretaries
                            should review and update their program performance plans to focus on
                            the critical objectives and indicators. In addition, the Assistant
                            Secretaries were to explain why any targets or baselines were missing
                            from the fiscal year 2000 performance plans and what steps would be
                            taken to obtain the missing targets and baselines.




ACN 17-70007                            September 1998                                     Page 22.
Moving Towards A Results-Oriented Organization                                            FINAL


                                  OBJECTIVES, SCOPE, METHODOLOGY

                            The objectives of our audit were to assess the status of the U.S.
                            Department of Education’s (ED) implementation of the Government
                            Performance and Results Act of 1993 (GPRA or the Results Act) and
                            the development of systems for the timely and accurate reporting of
                            performance indicators. To achieve our objectives, we conducted
                            interviews and reviewed relevant documents. Our conclusions about
                            the implementation of the Results Act at ED are based primarily on the
                            information gathered during the structured interviews. As part of this
                            audit, we did not review the quality of any particular data source or
                            make any determinations on any individual data improvement efforts.
                            We conducted the audit from October 1997 through March 1998 and
                            our results address what was found during that period. We conducted
                            our audit in accordance with generally accepted government auditing
                            standards applicable to the scope of the review.

                            Research on the Results Act and Performance Measurement

                            We reviewed the Results Act to determine the requirements of the law.
                            We also reviewed Office of Management and Budget (OMB) Circular
                            A-11 and material related to the Paperwork Reduction Act. In
                            addition, we reviewed literature about strategic planning and
                            performance measurement to obtain a technical understanding of the
                            concepts and identify current practices. We reviewed recent GAO
                            reports and testimony to identify significant government-wide and ED
                            specific issues related to the Results Act. We also reviewed OIG
                            reports related to these issues.

                            Interviews Using a Standard Questionnaire (Structured Interviews)

                            We conducted 27 structured interviews with staff in the following
                            offices: Office of Bilingual Education and Minority Language Affairs
                            (OBEMLA); Office for Civil Rights (OCR); Office of the Deputy
                            Secretary (ODS); Office of Educational Research and Improvement
                            (OERI); Office of Elementary and Secondary Education (OESE);
                            Office of Postsecondary Education (OPE); Office of the Secretary
                            (OS); Office of Special Education and Rehabilitative Services
                            (OSERS); Office of the Under Secretary (OUS); and Office of

ACN 17-70007                            September 1998                                   Page 23.
Moving Towards A Results-Oriented Organization                                               FINAL

                            Vocational and Adult Education (OVAE). These interviews were
                            conducted in November and December 1997 and January 1998.

                            For these interviews we judgmentally selected staff who were either
                            involved with the strategic plan or program performance plans or who
                            were operational or policy managers of ED programs and as such in
                            positions where we believe they should be knowledgeable about the
                            implementation of the Results Act. Ten of the 27 interviewed were in
                            senior executive service positions; thirteen were in GS-15 positions;
                            and the remaining four were in GS-14 positions. Of the 27 individuals
                            interviewed, all but four represented themselves as having some
                            involvement with the preparation or review of their programs’
                            performance plans. The four who said that they did not have
                            involvement with their programs’performance plans were from OPE.

                            We used a standard questionnaire which contained both closed and
                            open ended questions about the strategic plan, program performance
                            plans, data sources and the Results Act. In order to foster candor
                            during the interviews, we advised those interviewed that we would not
                            disclose their individual identities in reporting the results. Issues
                            presented in the report from these interviews were generally mentioned
                            at least four times (usually more) and by at least three different
                            program offices.

                            Other Interviews

                            In addition to those 27 structured interviews, we interviewed other
                            staff members in Office of the Chief Financial and Chief Information
                            Officer (OCF/CIO), OERI, OUS, and Office of the General Counsel
                            (OGC) about specific issues related to the implementation of the
                            Results Act and the development of reporting systems.

                            Review of ED’s Strategic Plan and Annual Plan

                            We reviewed the 1994 and 1997 versions of ED’s strategic plan. We
                            also reviewed volume 1 of ED’s annual performance plan for FY 1999
                            (which contained the annualized plans for the objectives in the strategic
                            plan and information on validation and verification of performance
                            indicators). Although volume 2 of the annual performance plan (which
                            contained the 99 program performance plans) was completed during
                            the course of our audit, we did not perform a detailed review of each

ACN 17-70007                            September 1998                                     Page 24.
Moving Towards A Results-Oriented Organization                                        FINAL

                            of the 99 program performance plans. However, we did review the
                            program performance plans for the largest ED programs.

                            Review of Other Documents

                            We also reviewed other ED documents related to the implementation
                            of the Results Act.




ACN 17-70007                            September 1998                              Page 25.
Moving Towards A Results-Oriented Organization                                               FINAL


                               STATEMENT ON MANAGEMENT CONTROLS

                             We assessed ED’s management control structure, policies, procedures
                             and practices applicable to our audit. The purpose of the management
                             control review was to determine whether ED had adequate controls in
                             place or under development to ensure the accurate and timely reporting
                             of performance indicators. As noted in the report, the first period for
                             which reporting of performance under the Results Act is required is
                             fiscal year 1999.

                             For this report, we assessed and classified ED’s significant management
                             controls for implementing the Results Act according to the following
                             objectives based on the COSO23 model of control environment, risk
                             assessment, information and communication, control activities, and
                             monitoring:

                                  # Efficient and effective operations;

                                  # Reporting of performance information; and

                                  # Compliance with laws and regulations.

                             Because of the inherent limitations, a study and evaluation made for the
                             limited purposes described above would not necessarily disclose all
                             material weaknesses in the control environment. However, our
                             assessment revealed that the process for assembling and analyzing the
                             data and preparing the report has not been finalized or implemented.
                             We discuss this weakness in this report.




       23
           Committee of Sponsoring Organizations of the Treadway Commission (COSO), 1992. Internal
Control - Integrated Framework.


ACN 17-70007                             September 1998                                    Page 26.
Moving Towards A Results-Oriented Organization                                              FINAL

                         Appendix A - Results Act Requirements
                                  (Public Law 103-62)


The strategic plan, covering not less than 5 years, must contain:

    (1) a comprehensive mission statement for major functions and operations of the agency;

    (2) general and outcome-related goals and objectives of the agency;

    (3) a description of how the agency will achieve the goals and the operational processes and
    resources required;

    (4) a description of how the goals relate to annual performance plan goals;

    (5) an identification of key factors external to and beyond the control of the agency that could
    significantly affect the achievement of goals; and

    (6) a description of program evaluations the agency used in establishing and revising general
    goals, with a schedule for future program evaluations.


The agencies’annual performance plan must:

    (1) establish performance goals that define the level of performance to be achieved by a
    particular program activity;

    (2) express goals in an objective, quantifiable, and measurable form unless an alternative form
    is approved by OMB;

    (3) describe the operational processes, skills and technology, and the human, capital,
    information, or other resources required to achieve performance goals;

    (4) establish performance indicators to be used in measuring or assessing the relevant outputs,
    service levels, and outcomes of each program activity;

    (5) provide a basis for comparing actual program results with the established performance goals;
    and

    (6) describe the means to be used to verify and validate measured values.



ACN 17-70007                             September 1998                                    Page 27.
Moving Towards A Results-Oriented Organization                                                FINAL

                 Appendix A - Results Act Requirements - Continued
                                (Public Law 103-62)

The agencies’annual program performance reports must:

    (1) review how successfully performance goals were achieved;

    (2) evaluate the performance plan for the current fiscal year relative to the performance goals
    achieved during the fiscal year covered by the reports;

    (3) where goals are not met, explain and describe (a) why the goals were not met, (b) plans and
    schedules for achieving the goals, and (c) if the goals are impractical or infeasible, why that is
    the case and what action is recommended;

    (4) describe the use and assess the effectiveness in achieving performance goals of any waiver
    under 31 U.S.C. section 9703; and

    (5) include the summary findings of program evaluations completed during the fiscal year.




ACN 17-70007                             September 1998                                     Page 28.
Moving Towards A Results-Oriented Organization               FINAL

                               Appendix B - ED’s Comments




ACN 17-70007                            September 1998      Page 29.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 30.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 31.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 32.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 33.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 34.
Moving Towards A Results-Oriented Organization                   FINAL

                       Appendix B - ED’s Comments - Continued




ACN 17-70007                            September 1998          Page 35.
Moving Towards A Results-Oriented Organization                                       FINAL

                    Appendix C - Publications Cited in This Report

Committee of Sponsoring Organizations of the Treadway Commission (COSO), 1992. Internal
Control - Integrated Framework.

Committee on Governmental Affairs, U.S. Senate, 1993. Report to Accompany the Government
Performance and Results Act of 1993.

GAO, 1998. THE RESULTS ACT: Observations on the Department of Education’s Fiscal Year
1999 Annual Performance Plan. (GAO/HEHS-98-172R).

GAO, 1998. MANAGING FOR RESULTS: Agencies’Annual Performance Plans Can Help Address
Strategic Planning Challenges. (GAO/GGD-98-44)

GAO, 1997. HIGH RISK AREAS: Actions Needed to Solve Pressing Management Problems.
(GAO/T-AIMD/GGD-97-60)

GAO, 1997. MANAGING FOR RESULTS: Analytic Challenges in Measuring Performance.
(GAO/HEHS/GGD-97-138)

GAO, 1997. MANAGING FOR RESULTS: Prospects for Effective Implementation of the
Government Performance and Results Act. (GAO/T-GGD-97-113)

GAO, 1997. MANAGING FOR RESULTS: Using GPRA to Assist Congressional and Executive
Branch Decisionmaking. (GAO/T-GGD-97-43)

Mandel, B.J., ed 1984. Statistics for Management. Baltimore: Dangary Publishing Company.




ACN 17-70007                            September 1998                              Page 36.
Moving Towards A Results-Oriented Organization                                   FINAL

                          Appendix D - Report Distribution List
                             AUDIT CONTROL NO. 17-70007

Action Official
    Mr. Marshall Smith                                                                4
    Acting Deputy Secretary


Other ED Offices

    Post Audit Group, Office of Chief Financial and Chief Information Officer         1

    Office of Public Affairs                                                          1

    Office for Civil Rights                                                           1

    Office of the Under Secretary                                                     1

    Office of General Counsel                                                         1

    Office of Special Education and Rehabilitative Services                           1

    Office of Legislation and Congressional Affairs                                   1

    Office of the Chief Financial and Chief Information Officer                       1

    Office of Management                                                              1

    Office of Postsecondary Education                                                 1

    Office of Educational Research and Improvement                                    1

    Office of Elementary and Secondary Education                                      1

    Office of Bilingual Education and Minority Languages Affairs                      1

    Office of Vocational and Adult Education                                          1




ACN 17-70007                            September 1998                          Page 37.