INSPECTION MEMORANDUM TO: William Hansen Deputy Secretary FROM: Mary Mitchelson Acting Assistant Inspector General Analysis and Inspection Services SUBJECT: Review of the Department of Education’s Annual Plan 2002-2003 Reveals Strengths and Areas for Improvement (ED/OIG I13C0014) This memorandum provides the results of our review of the U.S. Department of Education Annual Plan 2002–2003 (Plan) published in March 2002. The Plan is the Department’s Annual Performance Plan. We analyzed the Plan to determine (1) how it addresses the management challenges identified by the Office of Inspector General (OIG) and (2) how it links performance plans and budgets under the Government Performance and Results Act of 1993 (GPRA), using General Accounting Office methodology.1 In addition, we compared the results that selected Department programs received from the Office of Management and Budget’s (OMB) Program Assessment Rating Tool (PART) in the areas of performance goals and program budget alignment. Our review showed that the Plan effectively addresses the management challenges identified by OIG, although additional steps could be developed to enhance the Plan. We applied the GAO methodology and found that the Plan does not link program performance measures to program budgets. The results of the PART review also highlighted a need for the Department to strengthen program performance measure and budget linkages. We are providing suggested revisions and additional action steps to enhance the Department’s efforts to resolve its management challenges. We also discuss the importance of the Department undertaking improvement activities to develop program performance measures that are linked to funding levels. 1 GAO-02-236, Managing for Results: Agency Progress in Linking Performance Plans With Budgets and Financial Statements, January 2002. Inspection Results Issue 1. Department Officials Should Consider Adding or Revising Action Steps Related to OIG Identified Management Challenges While the Plan is not required to include a reference to the OIG identified management challenges (February 2002), the Plan’s action steps provide the best evidence of the Department’s commitment to meeting the management challenges, which will also support the Department’s goals of implementing the No Child Left Behind Act and creating a culture of accountability. Our analysis revealed that the Plan generally addresses the management challenges. In addition, we identified areas within the Plan where the Department should consider making additional action steps or revisions to existing steps to make them more effective in meeting the management challenges. Out of a total of 383 action steps listed in the Department’s Plan, we found that 158 or 41 percent are related to meeting the management challenges. Of the 158, we suggested revisions for 8 action steps and proposed adding one additional action step.2 For example, action step 310 directs Department officials to complete remedial actions on all problems identified through security reviews. To better address Management Challenge 2, Strengthen Information Technology Security, Department officials should consider amending this action step to specify that, in addition to addressing remedial actions, the step include completion of corrective action plans implementing recommendations made by the OIG GISRA audit and other security reviews. Improvement Activity: We provided specific OIG suggested revisions for the improvement of the Department’s action steps in Attachment 1. These suggested revisions should be provided to the appropriate management officials for their consideration when preparing the next Department Plan. Issue 2. Department Officials Should Explicitly Link Program Performance Measures and Program Budgets During 2001, GAO reviewed 32 different federal government agencies to determine how their performance plans linked to the agencies’ budgets and financial statements.3 At the time of GAO’s review, the Department had not yet completed its Plan. Therefore, GAO could not conduct a review of the Plan. To determine how the Department’s Plan linked to the Department’s budget and financial statements, we applied the GAO methodology. Our analysis revealed that the Plan does not explicitly link program performance measurement with program budgets. 2 Our comments and specific recommendations for steps requiring revisions are provided in Attachment 1. We are currently conducting audit work in some areas related to specific management challenges, so we chose not to comment on some challenges pending the outcome of this audit work. 3 GAO-02-236, Managing for Results: Agency Progress in Linking Performance Plans With Budgets and Financial Statements, January 2002. ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 2 The Plan is designed to support the achievement of the Department’s Strategic Plan, which has six goals.4 The Plan contains 24 objectives with 166 performance indicators that measure the accomplishment of these objectives. Many of the 24 objectives and their associated measures report national achievement on such essential areas as reading, math, and drug free schools. Other objectives are aimed at achieving management excellence within the Department in such areas as financial integrity and the management of human capital. The Plan’s appendix, Program to Objective Crosswalk (Attachment 2), provides a matrix of programs, budgets, and associated objectives. Using this crosswalk, our analysis showed that multiple programs supported 21 of the 24 objectives. In one instance, Objective 5.1 was associated with 58 programs. An average of 17 programs were related to each objective. Similarly, we noted that many of the programs were associated with multiple objectives. Specifically, we found that 73% of the programs were associated with multiple objectives. The matrix provided in Attachment 2 shows the links identified by the Department in its Plan. The importance of linking funding with accountability for results is recognized in Objectives 1.1 and 6.5 of the Plan. These objectives use the same measures. These measures are: • The percentage of Department programs that demonstrate effectiveness in terms of outcomes, either on performance indicators or through rigorous evaluations. • The percentage of Department program dollars that are in programs that demonstrate effectiveness in terms of outcomes, either on performance indicators or through rigorous evaluations. An asterisk at the bottom of both objectives is followed by the statement: For more detailed information on Department programs, visit the site: www.ed.gov/pubs/annualreport2001. A visitor to this site will find program performance measures for 88 Department programs. There is no indication, however, how these measures support the Plan and how they are tied to the Department’s budget, as there are no specific linkages between the program measures and the Plan objectives. This lack of emphasis on program performance measurement is reflected in the results of our evaluation of the Plan using GAO methodology. GAO employed a methodology that applied three characteristics to agency performance plans and then provided a table reflecting agency status in linking plans and budgets for fiscal year 2002 (Attachment 3). We used the same GAO characteristics and applied them to the Department’s Plan. Our analysis of the Plan showed that it meets minimum GPRA requirements, but it does not link program activities to program performance goals or show funding levels needed to achieve program performance goals. Overall, we concluded GAO would have likely assigned a low ranking in the area of linkages between programs and budgets compared to the majority of the agencies rated by GAO. For each GAO characteristic we found: 4 Goal 1: Create a Culture of Achievement Goal 2: Improve Student Achievement Goal 3: Develop Safe Schools and Strong Character Goal 4: Transform Education into an Evidence-based Field Goal 5: Enhance the Quality of and Access to Postsecondary and Adult Education Goal 6: Establish Management Excellence ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 3 Characteristic 1: Program activities were linked to goals. GAO identified agencies that linked program activities directly, or by aggregation, disaggregation, or consolidation to some level of performance planning structure. The Program to Objective Crosswalk appendix (Attachment 2) in the Plan provides linkages at a general level of performance. That is, performance measurement is at the broad objective level and not at the specific program level. Therefore, while this Plan provides some linkage to a level of performance planning, it does not reflect specific program linkages. This measurement at the objective level may not provide the specific information needed by Congress or Department management. Characteristic 2: Plans associated dollars with goals. GAO identified agencies that associated an amount of funding with some level of their performance planning structure. The Program to Objective Crosswalk reflects the appropriation/budget connection with performance at the general, objective level. As previously noted, Attachment 2 reflects that program budgets support multiple objectives; therefore, there are only general associations with funding and performance planning. While this approach provides a general association, it does not provide a clear linkage between requested resources and expected results. Characteristic 3: Funding was allocated to a discrete set of goals and/or measures. GAO identified agencies that displayed how requested funding for program activities directly, or by aggregation, disaggregation, or consolidation was allocated among specific or a unique set of performance goals or measures. The Program to Objective Crosswalk reflects connections at a general level with individual programs supporting multiple objectives and each objective being supported by an average of 17 programs. The Plan does not link budget amounts with specific or unique sets of performance goals or measures. As previously noted, an asterisked comment at the bottom of Objectives 1.1 and 6.5 provides a web link to a listing of program performance measures. As the Department does not link these measures to funding, this characteristic is not met. Without appropriate linkage between the program performance measures and the budget, Congress may not have the necessary information to make decisions on the relative effectiveness and efficiency of federal programs and spending. In addition, the linkage could help provide federal managers with information about program results, cost, and service quality. ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 4 Improvement Activity: The Department Plan should be modified to include specific program measure and funding allocation linkages to better inform congressional decision makers and program managers. Issue 3. The OMB PART Reviews Identified Needed Improvements in Program Performance Measures and Budget Alignment The results of the OMB PART review highlight the same need for improvement in program performance measures and budget linkages as demonstrated in our review using the GAO methodology. Specifically, in the initial Spring 2002 application and then again in the larger Fall 2002 PART review, the overall results of Section II, Strategic Planning, showed that the program budget was not aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance was readily demonstrated. We determined that the results of PART Section II, Strategic Planning, questions one, two, and six provide the most relevant information. Question one and two relate to performance goals, and question six relates to budget alignment. For our analysis, we reviewed the 6 programs that were part of the OMB Spring PART review and 12 that were part of the Fall review. 5 Under PART Section II, question one asked: Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program? The results showed that only 3 or 17 percent of the 18 programs had appropriate performance goals. A “yes” answer required identifying a limited number (e.g., two or three) of specific, easily understood program outcome goals that directly and meaningfully support the program’s mission and purpose. Question two asked: Does the program have limited performance goals that demonstrate progress toward achieving the long-term goals? The results showed that only 7 or 39 percent of the 18 programs had annual performance goals that demonstrate progress toward achieving the long-term goals. A “yes” answer required annual performance goals that are discrete, quantifiable, and measurable. Question six asked: Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known? The results showed that only 4 or 22 percent of the 18 programs had the budget appropriately aligned with program goals. A “yes” answer required the annual budget request to be clearly derived by estimating what is needed to accomplish annual performance measures and long-term goals. 5 The cumulative 18 programs are: National Assessment, NCES, Safe and Drug Free Schools and Communities State grants, Adult Education State grants, America’s Career Research Network, Tech-Prep Education State Grants, Tribally Controlled Vocational & Technical Institutions, Vocational Education State Grants, Grants for Infants and Families, IDEA Grants to States, Preschool Grants (IDEA, section 619), Even Start, Comprehensive School Reform, Vocational Rehabilitation State Grants, TRIO Upward Bound, TRIO Student Support Services, Federal Pell Grants, and Student Aid Administration. ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 5 The results of the PART Section II review highlight the same lack of program performance measures and budget linkages indicated in our review using the GAO methodology. As a consequence, the effectiveness of many of the Department’s programs cannot be directly measured. Background In the past, the Congress has expressed an interest in having OIG identify the Department’s most important performance measures. In 2001, the Chairman of the Committee on Government Reform asked OIG to identify the Department’s most important performance measures. We selected those measures associated with the Department’s most significant management challenges as identified by the OIG. In anticipation of a similar request, we performed this analysis of the Department’s Plan. Last year GAO published the report, Agency Progress in Linking Performance Plans With Budgets and Financial Statements, (GAO-02-236, January 2002). This report focuses on applying OMB Circular No. A-11 and GPRA to agency performance plans. The Department’s Plan was one of the performance plans selected for review. However, as the new Plan was not finished when the review was conducted, it was not included in the report. Therefore, we applied the methodology GAO used in its review to the now completed Plan (Issue 2). Objectives, Scope, and Methodology The objectives of our analysis were to evaluate the Department’s Annual Plan to determine (1) how it addresses the management challenges identified by the OIG and (2) how it links performance plans and budgets under GPRA, using GAO methodology. In addition, we compared the Plan and the results the Department’s programs received from OMB’s PART analysis in the areas of performance goals and program budget alignment. The references used to conduct this inspection are: • U.S. Department of Education Annual Plan 2002-2003 • OMB Circular No. A-11 (2001) • Government Performance and Results Act • U.S. Department of Education 2001 Performance Plan • The updated management challenges issued by the OIG in February 2002 • Office of Management and Budget Program Assessment Rating Tool • The results of the OMB ”Spring Review” of six Department programs as well as 12 Department PART submissions for the Fall Review • Interview with Hugh Walkup, Director of Strategic Accountability Services, in January 2003 OIG managers in Audit Services and Analysis and Inspection Services were provided a list of the OIG management challenges published in February 2002 and the related action steps from the Department’s Performance Plan. They reviewed the action steps to determine if these steps addressed the management challenges. If not, they were requested to identify additional actions that, in their professional judgment, the Department should take. ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 6 The review was conducted in Washington, D.C. and was done in accordance with the President’s Council on Integrity and Efficiency Inspection Standards. We appreciate the cooperation given to us during the inspection. If you have any questions or wish to discuss the contents of this report, please call me on 202-260-3556, or Brent Weston, Director, A&I, on 202-205-9833. Please refer to the control number in all correspondence relating to this report. Attachments 1. Management Challenges 2. Appendix B Program-to-Objective Crosswalk 3. Table 1: Agency Status in Linking Plans and Budgets, Fiscal Year 2002 cc: Hugh Walkup ED/OIG I13C0014 Review of the Department of Education’s Annual Plan 2002-2003 Page 7
Review of the Department of Education's Annual Plan 2002-2003 Reveals Strengths and Areas for Improvement. I13C0014, Date Issued: 02/07/2004 PDF (120K) Ms Word (58K)
Published by the Department of Education, Office of Inspector General on 2004-02-07.
Below is a raw (and likely hideous) rendition of the original report. (PDF)