UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL 501 I STREET, SUITE 9-200 SACRAMENTO, CALIFORNIA 95814 PHONE (916) 930-2388 • FAX (916) 930-2390 November 14, 2006 Control Number ED-OIG/A09G0009 Dr. Terry Bergeson State Superintendent of Public Instruction State of Washington Office of the Superintendent of Public Instruction Old Capitol Building, P.O. Box 47200 Olympia, WA 98504-7200 Dear Dr. Bergeson: This Final Audit Report, entitled Data Quality Review of Washington Consolidated State Performance Reports, presents the results of our audit. The purpose of the audit was to determine whether graduation and dropout rates reported to the U.S. Department of Education (Department) by the State of Washington’s Office of the Superintendent of Public Instruction (WOSPI) in the Consolidated State Performance Reports for 2003-2004 and 2004-2005 are supported by reliable data and meet the requirements of Section 1111(b)(2) of the Elementary and Secondary Education Act of 1965, as amended, and 34 Code of Federal Regulations § 200.19 (a) and (c). BACKGROUND The Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB), requires states receiving ESEA, Title I, Part A funds to implement a statewide accountability system. The accountability system must include a high school graduation rate indicator – a required factor in determining whether local educational agencies and high schools made adequate yearly progress in students’ achievement of state academic standards. States report graduation rates for all students and by student subgroups in the Consolidated State Performance Report (CSPR) – the required annual state report to the Secretary. The CSPR instructions require the state to report graduation rates computed in accordance with the definition approved as part of the State’s accountability plan (i.e., approved Consolidated State Application Accountability Workbook). The CSPR instructions also require states to report the annual dropout rates for high school students using the National Center for Education Statistics’ definition of a high school dropout. States reported graduation and dropout rates for school year 2002-2003 in the 2003-2004 CSPR. Rates for school year 2003-2004 were reported in the 2004-2005 CSPR. The Department uses information in the CSPR to prepare reports on progress Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations. ED-OIG/A09G0009 Page 2 of 21 in meeting strategic plan goals, individual program performance, and States’ progress in implementing NCLB. In the definition of “adequate yearly progress” at ESEA § 1111(b)(2)(C)(vi), Congress defined graduation rate as “the percentage of students who graduate from secondary school with a regular diploma in the standard number of years.” The implementing Federal regulation at 34 Code of Federal Regulations (C.F.R.) § 200.19(a)(1)(i) further defines the graduation rate to mean— (A) The percentage of students, measured from the beginning of high school, who graduate from high school with a regular diploma (not including an alternative degree that is not fully aligned with the State’s academic standards, such as a certificate or a GED) in the standard number of years; or (B) Another definition, developed by the State and approved by the Secretary in the State plan, that more accurately measures the rate of students who graduate from high school with a regular diploma as defined in paragraph (a)(1)(i)(A) of this section. The cited regulation also states that “[i]n defining [the] graduation rate, the State must avoid counting a dropout as a transfer.” The regulation at 34 C.F.R. § 200.19(c) requires states to ensure that academic indicators, such as the graduation rate, are valid and reliable; consistent with relevant, nationally recognized professional and technical standards, if any; and consistent throughout the State within each grade span. In September 2005, WOSPI received approval from the Department to use an extended graduation rate for adequate yearly progress (AYP) purposes rather than a rate based on students who graduated within a standard number of years (i.e., on-time graduation rate). The extended graduation rate includes students who graduated in more than the standard four years as graduates in the calculation of the rate. WOSPI received about $157 million and $171 million in Title I, Part A funds in fiscal years 2003 and 2004, respectively. In its 2003-2004 CSPR, WOSPI reported an on-time graduation rate for all students of 65.7 percent and an annual dropout rate of 6.7 percent based on student data for school year 2002-2003. In its 2004-2005 CSPR, WOSPI reported an on-time graduation rate for all students of 70.1 percent, an extended graduation rate for all students of 74.0 percent, and an annual dropout rate of 5.8 percent based on student data for school year 2003-2004. Attachment 1 provides the formulas and calculations WOSPI used to compute the on-time and extended graduation rates reported in the CSPRs. ED-OIG/A09G0009 Page 3 of 21 AUDIT RESULTS WOSPI complied with the ESEA requirement to provide an annual report to the Secretary by submitting the 2003-2004 and 2004-2005 CSPRs, which included the State’s graduation and dropout rates. However, student data used by WOSPI to calculate the reported rates was not reliable and the required definition of a “high school dropout” was not used to calculate the reported dropout rates. The OTHER MATTER section of this report presents our concerns with the presentation in the State of Washington’s Consolidated State Application Accountability Workbook of the State’s formulas for computing graduation rates for accountability purposes. In its comments to the draft report, WOSPI concurred with our findings and agreed to implement our recommendations. In response to the other matter, WOSPI submitted an amendment to the State’s Accountability Workbook, but the amendment did not fully address our concerns about the clarity of WOSPI’s description of the graduation rate used for AYP purposes. WOSPI’s comments are summarized at the end of each finding. The full text of WOSPI’s comments on the draft report are included as Attachment 2 to the report. FINDING NO. 1 – W OSPI Used Unreliable Data to Calculate Graduation and Dropout Rates WOSPI did not use reliable data to calculate graduation and dropout rates reported in the 2003-2004 and 2004-2005 CSPRs. WOSPI acknowledged that the data used to calculate rates reported in the 2003-2004 CSPR was of questionable accuracy and subsequently took steps to improve the data. Our review of the student records at three school districts found that data used in calculating the rates reported in the 2004-2005 CSPR was also unreliable. WOSPI needs to take additional actions to improve the accuracy of the data used in calculating the graduation and dropout rates. The ESEA § 1111(b)(2)(D) and the implementing regulations at 34 C.F.R. § 200.19(c) require states to ensure high school graduation rates are reliable and consistent throughout the state within each grade span. The Government Accountability Office’s Assessing the Reliability of Computer Processed Data states data are reliable when they are (1) complete (they contain all of the data elements and records needed for the engagement) and (2) accurate (they reflect the data entered at the source or, if available, in the source documents). A subcategory of accuracy is consistency. Consistency refers to the need to obtain and use data that are clear and well-defined enough to yield similar results in similar analyses. For example, if data are entered at multiple sites, inconsistent interpretation of data rules can lead to data that, taken as a whole, are unreliable. WOSPI used information from a statewide student-level data file to compute graduation and dropout rates reported in the CSPR. The statewide data file for each school year was created from data files submitted in the fall of each year by the State’s 249 school districts with high schools. The individual district data files (referred to as P210 Reports) contained the enrollment ED-OIG/A09G0009 Page 4 of 21 status for students enrolled in the previous school year. To create the P210 report, the districts extracted student data from the district’s student information system and converted the district’s data codes to the State’s data codes. WOSPI staff performed various reasonableness, completeness, and accuracy checks of data in the P210 reports. WOSPI categorizes students in the 9th through 12th grade into the following enrollment categories based on the student’s enrollment status code in the statewide data file: On-Time Graduate: Students who receive a diploma in the expected year of graduation.1 Late Graduate: Students who receive a diploma after their expected year of graduation. Dropout: Students who intentionally leave school for any reason, except death, before receiving a diploma and who do not transfer to another school or educational program. Students whose status is unknown because they are no longer enrolled, but who are not a confirmed transfer or dropout. For purposes of graduation and dropout rate calculations, WOSPI includes students who receive General Education Development (GED) certificates in the dropout category. Transfer (confirmed): Students who depart school and whose transcripts/student records have been requested by a gaining school, or students who file an “Intent to Receive Home-Based Instruction.” Students who move outside the United States before completing high school graduation requirements are also considered confirmed transfers. For purposes of graduation and dropout rate calculations, WOSPI includes deceased students in the transfer category.2 Continuing Student: Students who do not have an enrollment status of graduate or other type of completer (e.g., GED certificate), transfer, dropout, unknown, or deceased. A subgroup of this category is the “extended students,” who are 12th grade students who are enrolled beyond their expected graduation year. The number of students enrolled in each grade (less transfers), the number of students in each grade in the dropout category, the number of students in the extended student subgroup, and the number of students in the on-time graduate and late graduate categories are used to calculate the State’s graduation and dropout rates and, for accountability purposes, the graduation rates for each school and district in the State of Washington. WOSPI issues an annual report on the State’s graduation and dropout statistics that provides schools and districts with the actual student data used to compute individual school and district graduation and dropout rates. The report for school year 2003-2004 was issued in September 2005. 1 “Graduate” includes students who receive a regular high school diploma, students who receive an adult diploma from a community college program, and students with disabilities who complete their Individualized Education Plan (IEP). For school year 2003-2004, the statewide data file included 244 adult diploma recipients and 120 IEP completers. 2 For school year 2003-2004, the statewide data file included 101deceased students. ED-OIG/A09G0009 Page 5 of 21 WOSPI Had Concerns Regarding the Reliability of Data Used to Calculate Rates Reported in the 2003-2004 CSPR WOSPI officials advised us that the data used to calculate the rates for school year 2002-2003 were not accurate, or consistent throughout the state, as required by the ESEA. In its report titled Graduation and Dropout Statistics for Washington’s Counties, Districts, and Schools: School Year 2002-03, dated September 2004, WOSPI stated that “some districts have found errors in the information provided for some of their students” and that district and school “[r]ates that are extremely high or low may also reflect inaccurate reporting.” Due to concerns about data reliability, WOSPI took the following actions to improve the quality of data for school year 2003-2004. • WOSPI published the P210 Collection Data Manual for the 2003-2004 School Year in August 2004. The manual used a new format to present data element definitions that identified the various parameters (e.g., data type, valid codes, code descriptions, business rules, and input examples) for each data field. • WOSPI required districts to report an enrollment status code for every student. If the enrollment status code field was blank for any student, the district’s P210 file was returned for correction. Previously, the enrollment status code field defaulted to a continuing student status when the field was left blank. • WOSPI provided additional training via video teleconference to district staff on the P210 reports. Documentation showed WOSPI held four P210 training sessions (September and October 2004) for the school year 2003-2004 reporting period. Since WOSPI acknowledged that the data for school year 2002-2003 were not accurate and had subsequently taken steps to improve data, we focused our review on the reliability of the data used to calculate the graduation and dropout rates for school year 2003-2004, which were reported in the 2004-2005 CSPR. WOSPI Also Used Unreliable Data to Calculate Rates Reported in the 2004-2005 CSPR Our review of student enrollment data in the statewide student-level data file for three school districts (Seattle Public Schools, Spokane Public Schools, and Edmonds School District) found that a group of students at Spokane Public Schools was not included in the statewide file and student enrollment statuses in the statewide file were not accurate for the three districts. Students on the Spokane Public Schools’ P210 Report Were Not Included in the Statewide Student-level Data File. Spokane Public Schools’ staff provided us with its final P210 report for school year 2003-2004. The P210 report included 94 students in the 12th grade who were in the dropout category (i.e., enrollment statuses of dropouts, unknown, or GED completers). The statewide file contained no 12th grade students in the dropout category for Spokane Public Schools for school year 2003-2004. With the inclusion of the 12th grade dropouts, Spokane ED-OIG/A09G0009 Page 6 of 21 Public Schools’ on-time graduation rate for school year 2003-2004 would decrease from 85.1 percent to 81.0 percent and its extended graduation rate would decrease from 86.4 percent to 82.3 percent. Student Enrollment Statuses in the Statewide Student-level Data File Were Not Accurate. Our review of school year 2003-2004 student data extracted from the statewide file for selected high schools in three school districts identified inaccurate or unsupported enrollment codes for 141 of the 760 students in our sample (18.6 percent). Table 1: Results of Review of Student Enrollment Status for the School Year 2003-2004 for Selected High Schools in Three Districts Enrollment Category Shown in On-time Late Extended Total Statewide File Graduates Graduates Transfers Dropouts Students (a) Students Number of Students in Category 2,979 60 1,462 430 225 5,156 (b) Number of Students Reviewed 303 60 208 (c) 90 99 760 School Documentation Confirmed 289 50 126 80 74 619 Enrollment Codes Inaccurate Enrollment Codes 14 10 82 10 25 141 Error Rate 4.6% 16.7% 39.4% 11.1% 25.3% 18.6% Description of Error Type Enrollment Should Have Been Coded as: Transfer --- --- --- 10 --- 10 Graduate --- --- --- --- 19 19 Dropout --- --- 82 --- 6 (d) 88 Total --- --- 82 10 25 117 Student Had Less than Minimum 14 10 --- --- --- 24 Credit Hours to Graduate (e) Total Errors 14 10 82 10 25 141 (a) Continuing students that were enrolled beyond their expected graduation year. (b) Total does not include continuing students who were enrolled in 9th, 10th, and 11th grade. This group was not included in our sampling universe. We considered this group to be at lowest risk for inaccurate enrollment statuses. (c) Our computerized analysis of student data for the 1,462 transferred students confirmed that 840 of the students had been reported as enrolled by another school. We selected the sampled students from the remaining 622 students. (d) The six errors did not impact graduation rates since extended students are included with dropouts in the calculation of the grade-level dropout rates used to derive the on-time and extended graduation rates. (e) Each district established the minimum number of credits that a student needed for graduation. For students in the on-time and late graduate groups, we reviewed student records to confirm that students met the district’s minimum credits for graduation. We were unable to determine from the records whether the 24 students cited as errors received graduation diplomas. ED-OIG/A09G0009 Page 7 of 21 While we were not able to identify the specific cause of each enrollment status error, the following conditions and weaknesses in District and WOSPI processes may have caused or contributed to the occurrence of the errors: • The P210 Collection Data Manual for the 2003-2004 School Year (P210 Manual) was issued in August 2004 and included new enrollment status coding requirements applicable to the school year that ended in August 2004. • Information technology and administrative staff at the three districts described the extraction and conversion process for creating the P210 Report as difficult. • A reporting module in the Edmonds School District’s student information system, which was used to create the District’s P210 Report, did not show the proper enrollment status for students who took classes at more than one school in the district. Specifically, we noted that Edmonds’ student information system (WESPaC) identified 12th grade students enrolled full-time at one high school (home high school), who took at least one class at another high school, as continuing students at the end of 12th grade even if the student graduated from the home high school. In the State of Washington, 214 school districts use the WESPaC student information system, which was developed by the Washington School Information Processing Cooperative (WSIPC). • Based on our interviews with school staff, we concluded that staff at seven of the nine high schools did not fully understand enrollment status definitions and documentation requirements. For example, staff entered a transfer status for students pursuing a GED and relied on statements from a responsible adult as evidence for a transfer. The P210 Manual states that transfer codes are only used for confirmed transfers to another school (i.e., transcripts/records have been requested by another school or student filed an Intent to Receive Home-Based Instruction). The P210 Manual instructs districts to enter an unknown enrollment status code when the sole evidence is hearsay reports and unsubstantiated statements of intent to transfer. We also found that, for students with an unknown status in the district’s student information system, school staff did not change the enrollment status to transfer after the district received a records request from another school (i.e., confirmed transfer). • School staff in Spokane Public Schools altered the graduation year of students that were not promoted to the next grade. WOPSI uses the graduation year, which is entered at the time the student enters high school, to identify students who graduated in the standard number of years. As a result of altering the graduation year, Spokane did not report any extended students in school year 2003-2004. On March 30, 2006, Spokane sent an email instructing school staff to not change the graduation year in the student record and to check student records and make changes, as necessary, to reflect the student’s original graduation year. • Spokane Public Schools information technology staff inadvertently changed student records in the district’s student information system to show an enrollment status of graduate for students with less than 21 credits and a graduation year of 2004. ED-OIG/A09G0009 Page 8 of 21 • In the districts we reviewed, a school counselor determines whether a student has met the minimum credits for graduation based on information in the student’s school transcript. There was no independent review by a second person or edit check in the district’s student information system to validate the counselor’s determination. • Reviews of student enrollment statuses recorded in districts’ student information systems were limited. WOSPI performed various reasonableness, completeness, and accuracy checks of data submitted in the P210 reports. The Washington State Auditor’s Office (WSAO) analyzed trends in graduation rates for districts subject to the audit requirements of the Single Audit Act. When conducting school district audits, WSAO confirmed that the district submitted the required P210 reports and reviewed school records for selected students with an enrollment status of graduate in the district’s student information system. However, WSAO did not review support for other enrollment statuses and conducted no reviews of school records supporting the student’s enrollment status for districts that were not subject to the audit requirements of the Single Audit Act. WOSPI did not review student enrollment statuses as part of its on-site monitoring reviews nor did the three districts perform such reviews. Our tests of the reliability of student-level data was limited to the 760 sampled students who attended selected high schools in three districts. Thus, the results of our review of the sampled students may not necessarily be representative of data errors statewide. However, if the results of the 760 sampled students are representative of the reliability of data in other districts in the statewide file, the State’s on-time graduation rate for school year 2003-2004 would decline from 70.1 percent to 64.7 percent and its extended graduation rate would decline from to 74.0 percent to 68.0 percent.3 (The impact of enrollment status errors on the reported dropout rate is presented in Finding No.2.) Thus, the Department may have relied on inaccurate rates to assess the progress of the State of Washington’s Title I, Part A program. Additionally, WOSPI used unreliable data to make AYP determinations for districts and high schools. For school year 2004-2005, WOSPI collected data from districts using its new Core Student Record System (CSRS). WOSPI issued the Core Student Record System Monthly Collection Data Manual for the 2004-2005 School Year in May 2004. Districts continue to maintain their own student information systems, but provide updated student data to CSRS on a monthly basis. WOSPI is using student data in CSRS to calculate the graduation and dropout rates that will be reported in the 2005-2006 CSPR. While the conversion to CSRS may address some of the conditions found in our review, others will still need to be addressed since CSRS continues to rely on data extracted from the districts’ student information systems. During our on-site visits at school districts, district staff indicated that the procedures for preparing the monthly CSRS data submissions were difficult and time consuming to perform. 3 The revised rates are a non-statistical estimate of the impact of our audit results on the State’s graduation rates. The revised rates were derived by adjusting the statewide total of students in each enrollment category using the error rates from our sample and recalculating the rates using the State’s formulas and the adjusted statewide totals. ED-OIG/A09G0009 Page 9 of 21 Recommendations We recommend that the Assistant Secretary for Elementary and Secondary Education require WOSPI to— 1.1 Identify the specific difficulties that district information technology staff encountered when extracting student data from district student information systems for transmittal to WOSPI and take action to mitigate the difficulties. 1.2 Advise the WSIPC of the deficiency noted in the WESPaC reporting module and take the necessary actions to ensure that districts using the WESPaC system properly report the graduation status of students who took classes at more than one school. 1.3 Continue to provide annual training to districts and take steps to target the training to district and school personnel who enter enrollment codes to ensure consistent understanding of enrollment status definitions and required supporting documentation. If feasible, the training should be provided online to enable new hires and others to easily access the training. 1.4 Assess whether the standard data checks performed by WOSPI staff to identify incomplete or unusual student numbers in district data should be expanded to include additional data analyses, such as confirming that the P210 reports and the statewide student-level data file include students in each enrollment category for each district and school. 1.5 Issue a notice advising districts of the importance of adequate internal controls for ensuring that enrollment statuses are accurately recorded and that students meet the minimum credits for graduation. 1.6 Develop and implement a review process for ensuring that school staff accurately record student enrollment statuses and retain required documentation. WOSPI could consider reviewing enrollment statuses of selected students during state on-site monitoring reviews and having the WSAO expand its review of graduate status during single audits to include review of support for other enrollment statuses. WOSPI Comments WOSPI concurred with the finding and described the action taken or planned for the recommendations. WOSPI agreed that errors were present in the student data for the three districts, but expressed concern that generalization from these districts to the entire state may overestimate the extent of data errors statewide. WOSPI also noted that the review focused on student data for a time period (school year 2003-2004) that occurred shortly after passage of NCLB, which increased the need for ensuring that accurate data was available for AYP determinations. WOSPI disagreed with information presented in the report in the paragraph titled “Students on the Spokane Public Schools’ P210 Report Were Not Included in the Statewide Student-level Data File.” Specifically, WOSPI asserted that dropouts from Spokane Public Schools were included in the statewide student-level data file as extended students. ED-OIG/A09G0009 Page 10 of 21 OIG Response To address WOSPI’s concern regarding the generalization of the audit results, we added language in the final report to emphasize the limited nature of our review and that the results may not necessarily represent the extent of errors statewide. We made no other changes to the information in the report. The WOSPI report supporting the student data used to calculate the graduation rate for Spokane Public Schools for school year 2003-2004 showed no students in either the dropout or extended student categories. FINDING NO. 2 – W OSPI Reported Dropout Rates That Were Not Computed Using the Required Definition of a Dropout In the 2003-2004 and 2004-2005 CSPRs, WOSPI reported dropout rates that were not computed in adherence with the National Center for Education Statistics (NCES) definition of a dropout. The CSPR instructions state— . . . States should use the annual event school dropout rate for students leaving a school in a single year determined in accordance with the NCES’ Common Core of Data. Consistent with this requirement, States must use NCES’ definition of “high school dropout,” An individual who: 1) was enrolled in school at some time during the previous school year; and 2) was not enrolled at the beginning of the current school year; and 3) has not graduated from high school or completed a state- or district- approved educational program; and 4) does not meet any of the following exclusionary conditions: a) transfer to another public school district, private school, or state- or district-approved educational program …, b) temporary absence due to suspension or school-excused illness; or c) death. [Emphasis added.] The NCES definition of a dropout does not include students who completed a state- or district- approved education program, such as a GED. WOSPI used the following state definition of a dropout rather than the NCES definition when calculating the dropout rate reported in the CSPRs. Dropouts are students who drop out of school for any reason, finish their schooling without a regular diploma, or whose status is “unknown” because they are no longer enrolled but are not confirmed transfers or dropouts. [Emphasis added.] The state definition of a dropout includes students who received a GED (GED completers). ED-OIG/A09G0009 Page 11 of 21 By using the state definition, WOSPI overstated the state’s dropout rate in the CSPRs. WOSPI’s calculation of the dropout rate for school year 2003-2004 included 18,365 dropouts of which 1,441 were GED completers. The exclusion of the GED completers in the calculation would decrease WOSPI’s reported dropout rate from 5.8 percent to 5.4 percent. However, the dropout rate increases when the rate is also adjusted for the enrollment status errors noted in Finding No. 1. If the GED completers were excluded and the results of the sampled students reviewed in our audit are representative of the reliability of data in other districts in the statewide file, WOSPI’s reported dropout rate would increase from 5.8 percent to 7.7 percent.4 WOSPI staff considered the state definition of a dropout as the more appropriate definition for reporting purposes to maintain consistency with the graduation rate defined at ESEA § 1111(b)(2)(C), which does not include GED completers with graduates. Recommendation 2.1 We recommend that the Assistant Secretary for Elementary and Secondary Education require WOSPI to adhere to the CSPR instructions for reporting dropout rates. WOSPI Comments WOSPI concurred with the finding and agreed to implement the recommendation. 4 The revised rates are a non-statistical estimate of the impact of our audit results on the State’s reported dropout rates. ED-OIG/A09G0009 Page 12 of 21 OTHER MATTER WOSPI’s graduation rate formulas were not accurately reflected in the State’s Consolidated State Application Accountability Workbook.5 Thus, the Department and other readers of the Accountability Workbook were unaware of the actual formula used for AYP purposes. We identified two reports that mistakenly concluded, based on the Accountability Workbook, that the State of Washington used a graduation rate derived from data accumulated over a four-year period.6 In fact, the State’s actual formula used student data exclusively from the most recently completed school year to derive the rates for each of the four grade levels. The Accountability Workbook needs to clearly disclose the formulas used to calculate high school graduation rates used for AYP determinations and reported in the CSPRs. In its comments on the draft report, WOSPI referred to an amendment to its Accountability Workbook that was submitted in August 2006 for Departmental approval. The amendment includes the actual formulas used by WOSPI to compute graduation rates. However, further clarification is needed. The amendment does not state that WOSPI uses data solely from one school year to compute the rate and use of the term “cohort group” could lead a reader to incorrectly conclude that the State uses student data obtained over multiple school years. In addition, the amendment incorrectly states that the extended graduation rate “may be used for AYP purposes,” when, in fact, the extended graduation is used for such purposes. The Department had not approved WOSPI’s amendment request as of the issuance of this report. 5 The Washington Accountability Workbook was initially submitted to the Department in January 2003 and, after revision, approved by the Department in June 2004. The Department approved subsequent amendments to the Washington Accountability Workbook on September 1, 2005 and July 19, 2006. 6 The National Institute of Statistical Sciences/Education Statistics Services Institute Task Force’s Final Report on Graduation, Completion, and Dropout Indicators (NCES 2005-105) and the Urban Institute’s, Keeping Count and Losing Count: Calculating Graduation Rates for All Students Under NCLB Accountability. ED-OIG/A09G0009 Page 13 of 21 OBJECTIVES, SCOPE, AND METHODOLOGY The objective of our audit was to determine whether graduation and dropout rates reported to the Department by WOSPI in the 2003-2004 and 2004-2005 CSPRs are supported by reliable data and meet the requirements of the applicable provisions of the ESEA and Federal regulations. Specifically, we determined whether the— • Graduation and dropout rates reported in the CSPRs for the “All Students” category were computed in accordance with the requirements of ESEA § 1111(b)(2) and 34 C.F.R. §200.19 (a) and (c). • Data supporting the graduation and dropout rates were sufficiently valid and reliable for the Department to use the rates to complete reports on its progress in meeting its strategic plan goals, individual program performance, and State’s progress in implementing NCLB. To accomplish our objective, we— • Reviewed applicable laws, regulations, and other guidance; • Reviewed pertinent reports dealing with graduation and dropout rates published by Federal, state, and private organizations; • Identified, assessed, and tested internal controls employed by WOSPI to improve the reliability of data supporting graduation and dropout rates; • Interviewed officials and reviewed written policies and procedures related to graduation rates and dropout rates at WOSPI, three school districts, and nine high schools; • Reviewed student files at nine high schools. From the state’s eleven districts with more than 20,000 enrolled students in school year 2003-2004, we selected three districts for review using a risk-based approach. We selected Edmonds School District and Seattle Public Schools because each met WOSPI’s high school graduation rate goal of 66 percent using the extended graduation rate, but failed to meet the goal using the on-time graduation rate employed in school year 2002-2003.7 We selected Spokane Public Schools for review because of the district’s 19.4 percent increase in on-time graduation rates from 2002-2003 to 2003-2004 (the largest increase for districts with more than 20,000 students). In each district, we selected for review the three high schools having at least 1,300 students that met WOSPI’s extended graduation rate goal of 66 percent by the smallest margin. To select the student sample at each high school, we used the statewide student-level data file that WOSPI utilized to compute the graduation and dropout rates in school year 2003-2004. To assess the reliability of the data files, we compared the total records in the statewide file for 9th through 12th grades to enrollment counts reported by WOSPI. We confirmed that the totals by category for each of nine high schools matched the totals used to compute graduation and 7 For school year 2003-2004, high schools and districts in the State of Washington must have an extended graduation rate of 66 percent or increase their rate by at least two percent from the prior school year to meet AYP. ED-OIG/A09G0009 Page 14 of 21 dropout rates for those schools. We also determined whether the school had records for sampled students that confirmed that the students were enrolled during the school year. Based on these tests, we concluded that the data were sufficiently reliable for use in selecting the sample. However, as noted in Finding No. 1 of the Audit Results section, we found that the student enrollment status information in the statewide file was not reliable. For the nine high schools selected for review, we extracted student-level data from the statewide student universe of 341,996 students in grades 9 through 12. Then, we segregated the students at each school into six groups: graduates, late graduates, transfers, dropouts, extended students, and continuing students (other than extended students). For the transfer group, we separated students into two subgroups: (1) students who had a gaining school enrollment code to support the losing school’s transfer code, and (2) students who did not have a gaining school enrollment code to support the losing school’s transfer code. We attributed the greatest risk of error to the second transfer subgroup and selected the sampled transfer students solely from that subgroup. We randomly selected students from the graduate, transfer subgroup, dropout, and extended student categories. For the selected students and all students in the late graduate category, we compared the student’s enrollment code to the school’s documentation supporting the code. We did not include students from the continuing student group in our sampling universe as we considered this group to be at lowest risk for inaccurate enrollment statuses. Table 2: School Year 2003-2004 Student Group and Sample Sizes by District and School District High School On-time Late Transfers Dropouts Extended Graduates Graduates Students Group Size Tested Group Size Tested Group Size Group Size Unmatched Tested Group Size Tested Group Size Tested Edmonds Edmonds-Woodway 309 31 8 8 114 54 12 98 10 27 10 Lynnwood 241 25 17 17 247 127 25 62 10 47 10 Meadowdale 304 31 7 7 134 71 (a) 71 45 10 49 (a) 49 Seattle Ballard 360 38 13 13 111 28 12 15 10 34 10 Franklin 328 33 0 0 162 29 17 63 10 25 10 Roosevelt 380 38 14 14 92 46 10 13 10 43 10 Spokane Lewis & Clark 426 43 0 0 169 68 17 31 10 0 0 North Central 292 30 0 0 194 73 20 28 10 0 0 Rogers 339 34 1 1 239 126 24 75 10 0 0 Totals 2979 303 60 60 1462 622 208 430 90 225 99 Total Students 5,156 Total Tested 760 (a) Due to the high error rates at Meadowdale High School, we reviewed 100 percent of students in the transfer and extended groups. We performed our fieldwork at WOSPI offices in Olympia, Washington and administrative offices of the three districts and nine schools. We held an exit briefing with WOSPI officials on July 12, 2006. Our audit was performed in accordance with generally accepted government auditing standards appropriate to the scope of the review described above. ED-OIG/A09G0009 Page 15 of 21 ADMINISTRATIVE MATTERS Statements that managerial practices need improvements, as well as other conclusions and recommendations in this report, represent the opinions of the Office of Inspector General. Determinations of corrective action to be taken will be made by the appropriate Department of Education officials. If you have any additional comments or information that you believe may have a bearing on the resolution of this audit, you should send them directly to the following Education Department official, who will consider them before taking final Departmental action on this audit: Henry Johnson Assistant Secretary Office of Elementary and Secondary Education U.S. Department of Education 400 Maryland Ave., SW Washington, DC 20202 It is the policy of the U. S. Department of Education to expedite the resolution of audits by initiating timely action on the findings and recommendations contained therein. Therefore, receipt of your comments within 30 days would be appreciated. In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. Sincerely, /s/ Gloria Pilotti Regional Inspector General for Audit Attachments ED-OIG/A09G0009 Page 16 of 21 Attachment 1: WOSPI Graduation Rate Formulas and Calculations WOSPI uses the below formulas to calculate the on-time graduation rate and extended graduate rate reported in the CSPR. The student numbers used in the formulas are based on the student’s enrollment status at the end of the school year. The on-time graduation rate formula uses the dropout rate in each grade level for that school year as a proxy for the rates in previous years. WOSPI’s On-time Graduation Rate Formula (asterisk denotes multiplication) On-time Graduation Rate: 100*(1 minus grade 9 dropout rate)*(1 minus grade 10 dropout rate)* (1 minus grade 11 dropout rate)* (1 minus grade 12 dropout rate minus grade 12 continuing rate) Formula used to derive grade-level dropout rates for the above: Number of students with a dropout, unknown, GED completer code Total number of students served (minus transfers out) WOSPI’s Extended Graduation Rate Formula Extended Graduation Rate: Number of on-time and late graduates Number of on-time graduates divided by on-time graduation rate ED-OIG/A09G0009 Page 17 of 21 The OIG prepared the following charts to assist the reader in understanding the computation of graduation rates using the actual figures used by WOSPI. SCHOOL YE AR 2003-2004 WOSPI ON- TIME AND EXTENDED GRA DUA TlON RATE CALCULA TlONS Relen/ioll Rate Grade-LeV91 (100%- Dropout Dropout Remaining Grade,oroop Rate!> Rate) Pet'C(!ntaoe 9th Grad. (2OO3_2QO.II 9th Grade Dropouts ("""'~) 4.486 9th Graders Served (}!dual! 87.620 [II = S.l% [1J9H% 1O!h Grad. (20037200() 10th Gr..x o..""",t, (.'.ctutl) 4.561 [D - ,.'" [!jS..4'1O t [!] .!d9.G% 10th Grlde<$ SeMd ("""'tI) ~ 11th Gra" (2003-200() 11th Gr.cIe o..""",t. (Atluan 11th ao-_S Serve<! ( _ ) U371I1 77. ~ • 6.4'10 mS3.5%~ ~ 8% "~ L 12th Grad. (2OO3·20N) D.-Op<),l1$ 12th Grad. (2OO3·ZOo() 12th Gr.cIe o..""",t. (Adual) 12th ao-lde<S SeM(! ("""'tI) 12th Grlde Repeaters ("""'al) ~[!J 59.252 6.995 [D " = 2 10.1'10 lM%~ 70 U ,5'1O-->i 10.'''' on:nm. GiOod ROI. I Continuing Stud..-u 12th Gr_s Serwd ("""'tI) ~ Exten(l'" Grad Rot . On· TIme Gn")Jl!U (ACtuet) On· TIme GraOJlti«l Rate -Tm[!] =8/F 7 ,1'Mo E~;rnat.., Nurntler (II 9th 0,..,.11' that $I ~ed Hiif' Schoo! in the Fan or 2000 Uote ao-I ""ates (ACtuat) On· TIme ao-aOJato. (ACtual) 3.468 57.928 / I Tol<IIGr_t<o. [!] 6t .m 10 = U.l% E>rtended Go-;>duOlI«l ROI. E.M.,ledFIU2O(J() 9th Gr_. 82.SS i d Ulition Rlite Calculation Steps: I C~e Dropcut Rat. lor Each Grade Indthe 12th GradeC«l!inu;ng Student Go-oup (Student. Staying in School More Thin the Standa,d Nurntl..-orYeorsJ 2 Con'Cline 12th Grade o.-q:,out Rate (6,3%) with 12th Gracie CC<>tinuing Stu"...,t Rate (10. 1'Mo), ~ Subtrael Gracle· Lewl Dropout Ratu ~om l00%(Retcntion R.te). ~ Sub/riel Cotri);ned 12th Grade OropoutICon~nuing Studenl Rat •• (16.4'Mo) from 10(I'Mo I=f M uIt~ 9th Gracie Relen~c<> RIte (94.9'101 By 10th Gra cie Relen~c<> RIte (94.4%1, ~ Multifl!\llOth Gr..x R~aining Percentage(SS.6'Mo) By I Ith Gucle Retention Rfle (93,G"MoI. I=f: Mutt~ 11th G.-.de R ~.ining Percentlge(U&'Mo) By CombOned 12th Grade Drcpout/Continuing Studenl Retention Ri te (U6'Mo). ~ OMde On-- Time G .."""u (57.9:/6) by On-- Time G""""liOll Rale (7Q.l%) to Estimlte the Number d 9th Graderslhlt Stlned Hi!#> School in Ih e FI~ d :/QOO (el.SS7). ffi C~e the Total Gradullu by AMng the On· TKne GrIOJate. 157,926) to Ihe Lale GraOJlle. (3.458). 10 DMde tho Total Grl"" "IO$(61 .394) bylhe E~imaled Fall20009th Gra~ (82,657)locC1'f'lP,>le the ExtendedGrI",,"tion Rfle (74.3%), SCiiOOL ~ON7fiiIiEGi, IIIUN RATE I'fiOii Retention Rare Grade-Lwei (100%- D:: -R';;;'- Remaining ~ ;;: 'N 0"tm_ , •• G... , 11th Grnde (2002-"2003) 12th Grnde (2002-"20031 Dropouts :::::.; ~OJ - ::z::::: 12\t1 Grade DropouI$ (lI.:;:tu a~ 12th G,aders Serv~_ _ _7S, 7S3 6.105 CD - " ,." 6.'" "~ 2 20.2'% rn rn B 7$.~ l", .-A "" -"" --to- 6'.7% On-Time G,ad Rate 12th Grnde (2002-"20031 12\t1 Grade Re peaters (Actual) 9.176 QJ 12.1% Continuing Students 12\t1 Grade<s Served (Actual) 7~7S3 , ~ 12tt> G r_ ~~_r:~~ 12tt> Grade Continuing Stude nt Grgyp (Students Staying in ScIIooI Mo.-eThan !tie Standa rd Number 01 Year.) . Oropoyt Rate (8,1'l1li) Mt112th GradeContinuing Student Rate (121'l1li) , , It::tO'1111 (Retemion Rate), , OropoutIContinuing Stude<1t Rates (16.4'l1li) !rom ll)O'!1. , , Retention Rate (938'l1li) ev 10th G,ade Retention Rate (94.1'l1li). Grade l1e<1'laining Peroentage (86.3%) ev 11111 Grade Retention Rate (93,3'l1li) , Grade l1e<1'laining Peroentage(62,4'1111) 9yC<lmbined 12th Grade OropOUtlContinuing $t:ude<lll1etention Rate (79,6'l1li). ED-OIG/A09G0009 Page 18 of 21 Attachment 2: WOSPI Comments to Draft Report SUPERINTENDENT OF PUBLIC INSTRUCTION DR. TERRY BERGESON OLD CAPITOL BUILDING. PO BOX 47200. OLYMPIA WA 98504-7200 . http://www.k12.wa.us October 23, 2006 Ms. Gloria Pilotti Regional Inspector General for Audit United States Department of Education Office of Inspector General 501 I Street, Suite 9-200 Sacramento, CA 95814 RE: Audit Control Number ED-OIG/A09G0009 Dear Ms. Pilotti, I am writing in response to the Department's August 30, 2006 Draft Audit Report, entitled Data Quality of Washington Consolidated State Perfonnance Reports. The following is Washington Office of Superintendent of Public Instruction's (WQSPI) response, as well as our corrective actions planned/taken and/or additional information in response to the findings and recommendations included in your draft report. (Our responses are provided in italics.) Finding No.1 WOSPI Used Unreliable Data to Calculate Graduation and Dropout Rates WOSPI concurs with the finding that errors were found in the three sampled districts and recognizes that additional controls over the quality oj data collected are necessary. However, wefeel the generalizationJrom these districls, which were selected because they were the most at risk of having errors, to the entire state may over estimate the extent oj data errors statewide. The years reviewed during the audit were just after the passage oj NeLB, which greatly magnified the impact this data collection has regarding AYP detenninations, thereby, increasing the need for ensuring the most accurate data possible. We agree that some data reported may be inaccurate, but WaSPI has instituted many quality control measures in order to ensure reliable data. Nevertheless, we agree that the steps recommended by the OIG wouldfurther enhance the accuracy of the data WaSPI uses to calculate the rates, and we plan to take those steps. ED-OIG/A09G0009 Page 19 of 21 Ms. Gloria Pilotti Page 2 October 23, 2006 Other comments: Regarding the statement on page 5 "Students on the Spokane Public Schools' P2IO report were not included in the statewide student-level data file." Based on one of our data edit checks, our files indicate that these students were not eligible under the current definition to be reported as dropouts. These students were included in the "extended grad rate" category based on their expected graduation date. Recommendation 1.1 - Identify the specific difficulties that district information technology staff encountered when extracting student data from district student information systems for transmittal to WOSPI and take action to mitigate the difficulties. We are in the process of moving to a model that will completely eliminate districts having to manually upload data to wasp/. We will extract the districts' data by pulling it directly from their systems. We are currently piloting this model with three districts. Targeted Completion Date: 2007-08 school year Recommendation 1.2 - Advise the WSIPC of the deficiency noted in the WESPaC reporting module and take the necessary actions to ensure that districts using the WESPaC system properly report the graduation status of students who took classes at more than one school. We will contact WS1PC and relay this information. Targeted Completion Date: October 31,2006 Recommendation 1.3 - Continue to provide annual training to districts and take steps to target the training to district and school personnel who enter enrollment codes to ensure consistent understanding of enrollment status definitions and required supporting documentation. If feasible, the training should be provided online to enable new hires and others to easily access the training. We plan on continuing our annual training and will work on ensuring we target those school personnel who work with the data. We have started offering K-20 video conference data meetings this year on a monthly basis to address new and emerging issues surrounding data reporting, which includes enrollment codes. These are offered at all nine Educational Service District (ESD) locations, video capable districts, and audio capable districts and are well attended, which shows the districts are striving to ensure they provide the most accurate data possible. The video conferences are also available for download through our ESDs. We have discussed the possibility of having some of our education related associations who provide district related training to add this as a topic to their planned training schedule and will look into this option further. In addition, we have already had discussions regarding providing training on-line and will continue to further review this option. Targeted Completion Date: Ongoing ED-OIG/A09G0009 Page 20 of 21 Ms. Gloria Pilotti Page 3 October 23, 2006 Recommendation 1.4 - Assess whether the standard data checks perfonned by WOSPI staff to identify incomplete or unusual student numbers in district data should be expanded to include additional data analyses, such as confirming that the P210 reports and the statewide student-level data file include students in each enrollment category for each district and school. We currently provide a significant number of data analyses for reasonableness. We will review our data checks that are currently performed to determine whether we can expand those to ensure the most reliable data possible. Targeted Completion Date: April 2007 Recommendation 1.5 - Issue a notice advising districts of the importance of adequate internal controls for ensuring that enrollment statuses are accurately recorded and that students meet the minimum credits for graduation. We will emphasize the importance of adequate internal controls in our annual training sessions as well as our guidance that outlines graduation reporting requirements and instructions. The importance of accurate data is also reinforced in our monthly data meetings mentioned above, and will be conveyed to district superintendents through the appropriate communications. Targeted Completion Date: Ongoing Recommendation 1.6 - Develop and implement a review process for ensuring that school staff accurately records student enrollment statuses and retain required documentation. WaSPI could consider reviewing enrollment statuses of selected students during state on-site monitoring reviews and having the WSAO expand its review of graduate status during single audits to include review of support for other enrollment statuses. We are in the process of identifying ways to increase monitoring oj graduation rate data at the district level, which could include one or a combination of activities, such as: specialized staff to perform on-site reviews, adding afunction to the Consolidated Program Review process, performing risk assessments based on data review, etc. In addition, we have had recent meetings with our State Auditor's Office regarding their expanding work in the area of student level data reporting during their single audits of school districts. Targeted Completion Date: We anticipate having a pian in place by February 2007. The implementation of that plan will be an ongoing function. ED-OIG/A09G0009 Page 21 of 21 Ms. Gloria Pilotti Page 4 October 23. 2006 Finding No.2 WOSPI Reported Dropout Rates That Were Not Computed Using the Required Definition of a Dropout We concur with this finding. We used the same definition as found in NCLB for calculating Graduation Rates, which includes GED completers in the definition of dropouts. We believe lhal having two different calculations creates some confusion among the users of that in/onnation, but we understand the reasons surrounding the decision to exclude GED completersJrom the dropout rate. Recommendation 2.1- We recommend that the Assistant Secretary for Elementary and Secondary Education require WaSPI to adhere to the CSPR instructions for reporting dropout rates. WOSP! will report dropout informatioll ill its CSPR in accordance with the NCES reporting requirements. Targeted Completion Date: At the time the NCES report is due. "Other Matter" - WOSPl's graduation rate formulas were not accurately reflected in the State's Consolidated State Application Accountability Workbook 771is was corrected in an amendment submitted on August 6, 2006 which can be found at http://www.kJ2.wa.us/ESENpubdocs/A YPworkbookrevisionsAu 1:06. doc. I appreciate the opportunity to respond to your office's recommendations to assist us in ensuring the most accurate and reliable data is used in reporting graduation and dropout rates to the Department. Si!lcerely, /s/ Cathy ~vidson. Ed.D: Deputy Superintendent Learning and Teaching
Data Quality Review of Washington Consolidated State Performance Reports
Published by the Department of Education, Office of Inspector General on 2006-11-14.
Below is a raw (and likely hideous) rendition of the original report. (PDF)