UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL 1999 BRYAN STREET, HARWOOD CENTER, SUITE 1440 DALLAS, TEXAS 75201-6817 PHONE: (214) 661-9530 AUDIT FAX: (214) 661-9531 INVESTIGATION FAX: (214) 661-9589 June 7, 2006 Control Number ED-OIG/A06F0021 Dr. Rick Melmer - Secretary South Dakota Department of Education 700 Governors Drive Pierre, SD 57501 Dear Dr. Melmer: This Final Audit Report, entitled Data Quality Review of the South Dakota Consolidated State Performance Report presents the results of our audit. The purpose of the audit was to determine whether the South Dakota Department of Education’s required reporting of dropout and graduation rates in the 2003-2004 Consolidated State Performance Report were supported by reliable data and met the requirements of the Elementary and Secondary Education Act (ESEA). Our review covered the reporting period of July 1, 2003 – June 30, 2004. BACKGROUND Sections 9302 and 9303 of the ESEA, as amended by the No Child Left Behind Act of 2001 (NCLB), provides to States the option of applying for and reporting on multiple ESEA programs through a single consolidated application and report. The Consolidated State Performance Report (CSPR) includes the following ESEA programs: • Title I, Part A, Part B, Subpart 3, Part C, Part D, and Part F • Title II, Part A and Part D • Title III, Part A • Title IV, Part A, Subparts 1 & 2 and Part B • Title V, Part A • Title VI, Section 6111 and Part B Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations J ED-OIG/A06F0021 Page 2 of 12 The NCLB CSPR consists of two information collections. Part I of the 2003-2004 CSPR must be submitted in January of 2005 and requests information related to the five ESEA Goals. Part II of the 2003-2004 CSPR, due to the U.S. Department of Education (the Department) by April 15, 2005, consists of information related to State activities and the outcomes of specific ESEA programs. The five ESEA Goals established in the June 2002 Consolidated State Application are: • Performance Goal 1: By 2013-2014, all students will reach high standards, at a minimum, attaining proficiency or better in reading/language arts and mathematics. • Performance Goal 2: All limited English proficient students will become proficient in English and reach high academic standards, at a minimum, attaining proficiency or better in reading/language arts and mathematics. • Performance Goal 3: By 2005-2006, all students will be taught by highly qualified teachers. • Performance Goal 4: All students will be educated in learning environments that are safe, drug free, and conducive to learning. • Performance Goal 5: All students will graduate from high school. South Dakota adopted a comprehensive data collection system, the Student Information Management System (SIMS) Net, in 1998 to electronically gather student data such as test scores, attendance, and student status. Each student is assigned a unique identifier in SIMS Net that matches student demographics and has the capacity to track the status and location of each student. SIMS Net is a web-based student-level data collection system where student data records are stored in a centralized data warehouse. South Dakota officials believe that SIMS Net allows the state to collect and analyze more accurate and comprehensive student data, to meet Federal and State reporting requirements. Prior to the adoption of NCLB in 2001, South Dakota had not reported graduation or dropout rates. South Dakota officials started collecting graduation and drop data through SIMS Net from its districts to comply with NCLB. Prior to NCLB, South Dakota officials requested district officials to provide an electronic file of data at their convenience. South Dakota utilized the graduation and drop data provided by its districts for statistical purposes. For the school years 1999-2000 through 2001-2002, South Dakota officials stated that they have a database of the graduation and drop data reported by its districts. All school districts in South Dakota converted to the SIMS Net system except for the two largest districts, Sioux Falls School District (SFSD) and Rapid City School District (RCSD). These districts had already developed comprehensive data reporting systems and chose not to convert to the SIMS Net. Except for SFSD and RCSD, school and district officials input the student data directly into SIMS Net using a web-based format. Student data in SFSD and RCSD are maintained on each district software vendor’s server until uploaded into SIMS Net. These uploads occur on a routine basis. Below are South Dakota’s definitions of terms used when discussing graduation and dropout rates. ED-OIG/A06F0021 Page 3 of 12 • High School Completer - individual awarded a high school diploma. This would not include a student who receives a non-standard diploma (e.g. a General Education Development (GED) or certificate of completion). • Dropout - individual who a. Was enrolled in school at some time during the previous school year; and b. Was not enrolled at the beginning of the current school year; and c. Has not graduated from high school or completed a state- or district-approved educational program; and d. Does not meet any of the following exclusionary conditions: 1) Transferred to another public school district, private school, or state- or district-approved educational program (including correctional or health facility programs); 2) Temporary absence due to suspension or school-excused illness; or 3) Death. • Cohort - individuals who were high school completers in 2002-2003. For clarity in our discussion, we defined a student that left school in 2002-2003 and did not return in school year 2003-2004 as a “Leaver.” • Leaver - The status of a student who was enrolled or in attendance during a school year but stopped attending or did not return the next school year are reported using codes that describe the circumstances of the student’s departure. A school leaver is categorized as a dropout, or student who withdrew to: (a) enroll in another public or private school in the state; (b) enroll in a school outside the state; (c) enroll in a college; or (d) enter home schooling. For our review, we selected the three largest school districts in South Dakota—RCSD, SFSD, and Watertown School District (WSD) and visited the three largest high schools within RCSD and SFSD districts. We also visited the only high school in WSD, for a total of seven high schools. AUDIT RESULTS South Dakota met the requirements of ESEA by reporting dropout and graduation rates. However, the State reported its 2003-2004 graduation rate by using a one-year cohort definition in its calculation instead of the four-year cohort suggested by the Department. In addition, we determined that for the seven schools reviewed, South Dakota collected unreliable data to support the graduation and dropout rates reported in the 2003-2004 CSPR. In its comments to the draft report, South Dakota did not concur with Finding 1 and its recommendation, but did concur with Finding 2 and its recommendation. The comments are summarized at the end of each finding. The full text of South Dakota’s comments on the draft report is included as an Attachment A to the report. ED-OIG/A06F0021 Page 4 of 12 FINDING 1 - South Dakota’s 2002-2003 Approved Graduation Rate Does Not Meet NCLB Requirements South Dakota reported a 2002-2003 graduation rate in its 2003-2004 CSPR using a one-year cohort alternative graduation rate, which only captured dropouts in grade 12 and excluded dropouts in grades 9-11. South Dakota’s 2002-2003 graduation rate was calculated using a formula of a one-year cohort of high school completers in 2002-2003 divided by the sum of high school completers plus dropouts for that one year. South Dakota officials stated that the one-year cohort definition was used because the State had not been required to collect graduation and dropout data prior to NCLB. South Dakota officials stated graduation and dropout data had been collected prior to NCLB for statistical purposes only. We requested access to the graduation and dropout database for 1999-2000 through 2001-2002 that was collected so we could recalculate the graduation rate using a four-year cohort. South Dakota officials did not provide us access stating, “we are not convinced that this data will yield any useful, valid and accurate graduation calculations.” Approved Definition Does Not Meet NCLB As explained in the South Dakota Accountability Workbook submitted to and approved by the Department, the one-year cohort graduation rate definition would only be used for the 2002-2003 school year. Since South Dakota had not reported a graduation rate prior to NCLB, State officials are building a graduation rate database over a four-year period based on the following schedule. In school year 2002-2003 include 12 grade data only; in school year 2003-2004 include 11-12 grade data; in school year 2004-2005 include 10-12 grade, and in school year 2005-2006 full implementation with the inclusion of data for grades 9-12 grades. By the end of school year 2005-2006, the graduation rate definition will be a follows: High School Completers in Year 4 Dropouts (Grade 9, year 1 + Grade 10, year 2 + Grade 11, year 3 + Grade 12, year 4) + HS Completers, Year 4 Although the Department reviewed and approved South Dakota’s alternative graduation rate definition, the definition does not meet the NCLB required definition of a graduation rate. A significant difference is that the South Dakota method does not track students using a cohort over time (from entry to graduation). Instead, the formula used simply calculates an annual rate then adds four years of annual rates together. The denominator includes only graduates and dropouts for each year, leaving out a potentially large number of students in other categories such as those not graduating on time but continuing their education and students pursuing a GED certificate. When the denominator excludes students of the cohort, the resulting graduation rate is inflated or higher than actual. This calculation produces a graduation rate that is unlikely to be truly representative and would, therefore, also be inconsistent with the NCLB requirement to be valid and reliable. NCLB defines graduation rates as the percentage of students who graduate from high school with a regular diploma in the standard number of years. The standard number of years is determined by a state and is generally based on the structure of the school. For example, a high school with grades 9 through 12 would have 4 as its standard number of years while a school with grades 10 ED-OIG/A06F0021 Page 5 of 12 through 12 would have 3 as its standard number of years. This method of measure is called a longitudinal graduation rate. It is a cumulative indicator derived from student-level information and reported as a function of entry year and graduation year. This method uses a cohort definition to track students through the years spent in high school. The cohort definition compares the number of 12th grade graduates with a standard diploma, with the number of students enrolled as 9th graders 4 years earlier, while also taking into account those who left the cohort, such as those who transferred in and out. A formula statement of a cohort is: Cohort - Students who started high school (i.e., ninth grade) plus student transfers in, less student transfers out in year Y; plus student transfers in, less student transfers out in year Y+1; plus student transfers in, less student transfers out in year Y+2; plus student transfers in, less student transfers out in year Y+3. One-Year Cohort Inflates Reported Graduation Rate Section 200.19 of the Title I regulations issued under the NCLB Act on December 2, 2002, defines graduation rate to mean: • The percentage of students, measured from the beginning of high school, who graduate from public high school with a regular diploma (not including a GED or any other diploma not fully aligned with the State’s academic standards) in the standard number of years; or, • Another more accurate definition developed by the State and approved by the Secretary in the State plan that more accurately measures the rate of students who graduate from high school with a regular diploma; and • Avoids counting a dropout as a transfer. The regulations define a four-year cohort that starts tracking students when they begin high school as a freshman and continuing throughout the four years of high school, each year adding students that transfer into South Dakota and subtracting students that transfer or drop out during the four years. The regulations also indicate that States can use a more accurate definition than the four-year cohort, if approved by the Department. Each State has flexibility, however, in determining how its graduation rate will be specifically calculated as long as the rate is, as the law requires, “valid and reliable.” South Dakota developed and received Departmental approval for an alternative definition using a one-year cohort for 2002-2003. However, the alternative definition did not provide a more accurate graduation rate; instead, the definition inflated the reported graduation rate. This inflation occurred because South Dakota focused on only one year of data, grade 12. By only including data from grade 12 in the calculation, officials excluded all students that dropped out in grades 9-11 from the calculation. The seven schools that we reviewed reported 1,680 students in the 9-12 grades that were reported by school officials as leaving school and not returning to school in 2002-2003. These leavers included transfers between South Dakota districts, transfers out of South Dakota, and dropouts, among others. We found that 1,372 of the 1,680 (82 percent) students reported as leavers were students in the 9–11 grades. Further, 362 of the 1,680 reported ED-OIG/A06F0021 Page 6 of 12 leavers were reported as dropouts. Of the 362 reported dropouts, 291 (80 percent) were in the 9- 11 grades. Using a one-year cohort for the 2002-2003 graduation rate, South Dakota officials incorrectly reported the 96 percent graduation rate in 2002-2003 by excluding all reported drops in grades 9-11, which constituted 80 percent of the reported drops at the seven schools we visited. Recommendation We recommend that the Assistant Secretary for Elementary and Secondary Education require the South Dakota Department of Education to develop and implement a graduation definition that meets NCLB. South Dakota’s Comments South Dakota did not concur with the finding or recommendation. South Dakota stated that the method used for calculating its graduation rate was clearly described in the Consolidated State Application Accountability Workbook and approved by the United States Department of Education on June 3, 2003. They further stated the approved methodology was “consistent with section 200.19 of the Title I regulations” and was “prescribed by the National Center for Educational Statistics”(NCES). OIG’s Response Nothing in South Dakota’s comments changed our finding or recommendation. We are aware that the Department approved South Dakota’s graduation methodology reported in its Consolidated State Application Accountability Workbook; however, the methodology does not meet NCLB, specifically, Section 200.19 of the Title I regulations. Section 200.19 allows for states to either report a longitudinal graduation rate (percentage of students who graduate from high school with a regular diploma in the standard number of years) or an approved graduation rate that more accurately measures the rate of students who graduate from high school with a regular diploma. South Dakota chose the latter, an alternative approach that was supposed to be more accurate. However, South Dakota’s alternative method was not more accurate. First, the alternative graduation definition does not track students using a cohort over time (from entry to graduation). Instead, the formula used simply calculates an annual rate then adds four years of annual rates together. Second, the denominator includes only graduates and dropouts for each year, leaving out a potentially large number of students in other categories such as those not graduating on time but continuing their education, and students pursuing a GED certificate. When the denominator excludes students of the cohort, the resulting graduation rate is inflated or higher than actual. We believe that South Dakota’s method, although approved by the Department, produces an unreliable and invalid graduation rate that is inconsistent with the NCLB requirement to be valid and reliable. Also, we are aware that NCES prescribed a graduation rate methodology in August 2002. However, the NCES methodology does not meet NLCB standards because the methodology did not require tracking students from freshman entry through high school graduation. ED-OIG/A06F0021 Page 7 of 12 FINDING 2 - Data Used for Graduation and Dropout Rates Were Inaccurate South Dakota’s data for dropout and graduation rates reported in the 2003-2004 CSPR were not sufficiently accurate to produce reliable rates. Specifically, South Dakota incorrectly classified or had insufficient supporting documentation to validate the reported dropout rate data. Graduation Rate Data Were Not Always Reliable To review data used in the graduation rate calculation, we selected two samples. The first sample of 237 was of the cohort high school completers in school year 2002-2003. The graduate sample data populate the numerator of the cohort graduation rate formula. Three (1.2 percent) of the sampled 237 high school completers were incorrectly classified as graduates. Specifically, of the three exceptions: • Two were full-time private school students that enrolled for one class at the public high schools; and • One completer graduated in 2003-2004, not 2002-2003. The remaining 234 graduates were fully supported by high school transcripts. However, we discovered in our reviews of cohort leavers and campus-wide leavers, discussed in detail below, students reported as transfers by school officials, based on documentation obtained at the schools, were graduates of the 2002-2003 reporting year. Graduate students incorrectly reported as transfers constituted 11 percent (7 of the 64 leavers) of the cohort leavers and 4 percent (6 of the 171 leavers) of the campus-wide leavers sampled. As a result of these exception rates, we determined the data in the cohort were not sufficiently accurate to produce a reliable graduation rate. The second sample was of cohort leavers in school year 2002-2003. Leavers include dropouts, as well as transfers between South Dakota districts, transfers out of South Dakota, and home schoolers, among others. South Dakota officials only included dropouts in its denominator of the graduation rate formula. However, to determine if school officials correctly classified and reported dropouts, our random sample included all students reported by school officials as not returning (leavers) to school in grade 12 in school year 2002-2003. We reviewed a sample of 64 cohort leaver records and determined 43 (67 percent) were incorrectly classified or had inadequate or no supporting documentation for the drop status. Specifically, the student records we reviewed for 20 exceptions contradicted the reported classification and were adequate to conclude that: • 17 (26 percent) students were incorrectly classified as transfers. Nine students dropped out of school, seven students graduated from the reporting school in 2002-2003 and one student was a continuing student at the reporting school the next year; and • three (five percent) students were incorrectly classified as dropouts but should have been classified as transfers. ED-OIG/A06F0021 Page 8 of 12 Further, we determined that 23 (36 percent) of the 64 leavers had inadequate or no supporting documentation. Without sufficient supporting documentation we could not determine whether the leaver codes were accurate or there was inadequate or no supporting documentation for the leaver classification. As a result of this exception rate, we determined data in the cohort were not sufficiently accurate to produce reliable graduation rates. Dropout Rate Data Were Not Always Reliable To review data used in the dropout rate calculation, we selected a third sample. The annual dropout rate is calculated by dividing the number of grade 9-12 dropouts during the school year by the accumulative enrollment in those grades during the school year. The third sample was of campus-wide leavers in school year 2002-2003. To review data factored into the annual dropout rate calculation, we selected a sample of 171 students from the 9-12 grades that were reported by school officials as leavers for the reporting year. These leavers included transfers between South Dakota districts, transfers out of South Dakota, and dropouts, among others. We reviewed the sample of 171 campus-wide leaver records and determined 97 (57 percent) were incorrectly classified or had inadequate or no supporting documentation for the drop status. Of the 97 exceptions, 54 (32 percent) were incorrectly classified. The student records we reviewed at the schools contradicted the reported classification and were adequate to conclude that two students, incorrectly classified as dropouts, should have been a transfer and a continuing student. The remaining 52 students were incorrectly classified as a type of transfer of which: • 30 should have been classified as dropouts; • 6 should have been classified as graduates; • 1 student should have classified as a continuing student; and • 15 students should have classified as another type of transfer. Further, of the 97 exceptions, 43 (25 percent) had inadequate or no supporting documentation. Without sufficient supporting documentation we could not determine whether the campus-wide dropout codes were accurate or there was inadequate or no supporting documentation for the leaver classification. As a result of this exception rate, we determined data in the cohort were not sufficiently accurate to produce reliable dropout rates. Guidance and Oversight by South Dakota Officials Is Needed These deficiencies occurred because South Dakota officials did not provide adequate guidance, training, oversight, and monitoring to calculate graduation and dropout rates. Since the adoption of the SIMS Net in 1998, the only guidance provided to schools regarding graduation and dropout data were SIMS Newsletters published by South Dakota. South Dakota officials have published 21 newsletters in seven school years. In relation to graduation and drop rates, the Newsletters communicate revisions in data reporting codes and deadlines. No guidance is provided to school officials on the correct classification and reporting of leaver data based on supporting documentation. South Dakota also did not provide guidance on training requirements for new or current employees responsible for data reporting. District officials at one of the three districts stated that, for data reporting staff, the district provides two to four hours of new employee training and one to two hours of update training each school year. On the other hand, ED-OIG/A06F0021 Page 9 of 12 school officials interviewed in the remaining two districts stated that training was not provided on data reporting. School officials stated that the State did provide a two to four hour training session on SIMS Net when it was adopted in 1998. South Dakota provides a contact person at the State for school officials if they had questions on data reporting. However, school officials we interviewed rely more on experienced data reporting staff at the schools or districts. South Dakota officials stated numerous data reliability queries are performed on data reported by the schools and when necessary, school officials are contacted to resolve discrepancies in the data. However, South Dakota officials provided no documentation of these queries. District officials we interviewed were only aware of the duplicate student count query performed by the State. These queries are the only monitoring South Dakota officials perform on the graduation and dropout data reported by school officials. The NCLB Act of 2001, Public Law 107-110, enacted January 8, 2002, places emphasis on and strengthens the accountability for results. It also increases the importance of data and the need for the Department to have data that are reliable and valid. Unreliable data cause graduation and dropout rates to be inaccurate, either overstated or understated. It is important that data are reliable because the graduation and dropout rates are performance measures considered by the Department, the State, and the public in comparison to other States' performance, and the information is also used to assess school, district, and State accountability. As recipients of Title I grants, State officials agree to establish and maintain internal controls designed to reasonably ensure compliance with Federal laws, regulations, and program compliance requirements. OMB Circular A-133 Compliance Supplement, Part 6, is intended to assist non-Federal entities in complying with these requirements by describing, for each type of compliance requirement, the objectives of internal control and certain characteristics of internal control that, when present and operating effectively, may ensure compliance with program requirements. Part 6 discusses five components of internal control that should reasonably assure compliance with the requirements of Federal laws, regulations, and program compliance requirements. Two of the five components are Control Activities and Monitoring. Control Activities are the policies and procedures that help ensure that management’s directives are carried out. Specifically, the supplement mentions: • Operating policies and procedures clearly written and communicated. • Computer and program controls should include: a. Data entry controls, e.g., edit checks. b. Exception reporting. c. Reviews of input and output data. Monitoring is a process that assesses the quality of internal control performance over time. Some of the monitoring steps outlined in the supplement are: • Periodic site visits performed and checks performed to determine whether procedures are being followed as intended. • Follow up on irregularities and deficiencies to determine the cause. • Internal quality control reviews performed. ED-OIG/A06F0021 Page 10 of 12 • Management meets with program monitors, auditors, and reviewers to evaluate the condition of the program and controls. • Internal audit routinely tests for compliance with Federal requirements. Recommendation We recommend that the Assistant Secretary for Elementary and Secondary Education require the South Dakota Department of Education to develop and implement improved procedures on classifying and documenting graduate and leaver data, training data-reporting staff, and data collection oversight and monitoring. South Dakota’s Comments South Dakota concurred with this finding and stated that the following steps will be implemented to improve graduation and dropout data reliability: • Policies and procedures relative to reporting graduation and dropout data will be published in upcoming issues of the SIMS Newsletter. • Office of Data Collection staff will analyze input and output data, and compare to fall enrollment data and prior year data to identify variances. • Office of Data Collection staff will monitor data for significant changes. When changes are identified; staff will follow up with districts to determine the cause, and remedy irregularities if warranted. OBJECTIVES, SCOPE, AND METHODOLOGY The overall objective was to determine whether the South Dakota Department of Education’s required reporting of dropout and graduation rates in the 2003-2004 CSPR were supported by reliable data and met the requirements of the ESEA. Specifically, we determined whether the— • data for high school completers were accurate and documented; • leaver data in the 2003-2004 academic reporting year were accurate and documented; and • leaver data in the reporting year for leavers were accurate and documented. To accomplish our objective, we— • reviewed written policies and procedures for monitoring school supplied data; • reviewed applicable laws, regulations, and other guidance; • interviewed South Dakota officials, as well as, selected school and district officials; and • reviewed student files at the selected high schools. ED-OIG/A06F0021 Page 11 of 12 We judgmentally selected Rapid City, Sioux Falls, and Watertown School Districts for review because they were the largest districts in the South Dakota. We then selected the three largest high schools in these districts for review. However, since Watertown district only had one high school, we were limited to that one high school for a total of seven high schools in three districts. For testing purposes, we had South Dakota extract from SIMS Net a database of students in the 2002-2003 academic year from each of the seven high schools including transfers in and out of schools. From this extract, we created a database of high school completers and another database of cohort leavers from the 2002-2003 school year data. • From the high school completers, we drew a random 10 percent sample with a maximum of 50 and a minimum of 10 students for review at each high school from a universe of 2,256 students. • From the cohort leaver data, we drew a 10 percent random sample of students (with the same minimum/maximum as above) that left during the 2003-2004 academic reporting year at each high school to ensure they were properly classified from a universe of 232 students. We also obtained another extract of leavers that left the high school campus (campus-wide leavers) during the 2003-2004 academic reporting year. • From this extract, we drew a 10 percent random sample with a maximum of 50 and a minimum of 10 students at each high school to ensure they were properly classified from a universe of 1,680 students. To achieve our audit objective, we relied, in part, on computer-processed data related to the student information contained in the South Dakota’s SIMS Net database. At Infinite Campus, the contractor for SIMS Net, comparability tests were conducted by our Computer Assisted Assessment Technologies (CAAT) staff on the one-year cohort extract used for our audit work to the contractor’s managed source database. The CAAT staff deemed the database to be sufficiently reliable for the purposes of our audit. At the seven schools, we also verified the completeness of data by comparing source records to computer-generated data, and verified the authenticity by comparing computer-generated data to source documents. Based on these tests, we concluded that data were sufficiently reliable to be used in meeting the audit’s objective. We conducted our fieldwork at the State, RCSD, SFSD, and WSD between August 15, 2005, and October 26, 2005. An exit conference was held with South Dakota officials on February 8, 2006. Our audit was performed in accordance with generally accepted government auditing standards appropriate to the scope of audit described above. ED-OIG/A06F0021 Page 12 of 12 ADMINISTRATIVE MATTERS Statements that managerial practices need improvements, as well as other conclusions and recommendations in this report, represent the opinions of the Office of Inspector General. Determinations of corrective action to be taken will be made by the appropriate Department of Education officials. If you have any additional comments or information that you believe may have a bearing on the resolution of this audit, you should send them directly to the following Education Department official, who will consider them before taking final Departmental action on this audit: Henry Johnson Assistant Secretary Office of Elementary and Secondary Education U.S. Department of Education 400 Maryland Ave., SW Washington, DC 20202 It is the policy of the U. S. Department of Education to expedite the resolution of audits by initiating timely action on the findings and recommendations contained therein. Therefore, receipt of your comments within 30 days would be appreciated. In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. Sincerely, /s/ Sherri L. Demmel Regional Inspector General for Audit Attachment Attachment April 20, 2006 Sherri L. Demmel, Regional Inspector General for Audit United States Department of Education Office of Inspector General 1999 Bryan Street, Harwood Center, Suite 1440 Dallas, TX 75201-6817 RE: Control Number ED-OIG/A06F0021 Dear Ms. Demmel: This correspondence serves as the South Dakota Department of Education’s written comments in response to the findings and recommendations contained in the draft audit report entitled Data Quality Review of the South Dakota Consolidated State Performance Report dated March 17, 2006. Our comments relative to the findings and recommendations are as follows: Finding 1- South Dakota’s 2002-2003 Approved Graduation Rate Does Not Meet NCLB Requirements. Recommendation We recommend that the Assistant Secretary for Elementary and Secondary Education require the South Dakota Department of Education to develop and implement a graduation definition that meets NCLB. We do not concur with the finding or recommendation. South Dakota’s method for calculating our graduation rate is described in our Consolidated State Application Accountability Workbook and was approved by the United States Department of Education on June 3, 2003. The pertinent workbook element is as follows: Critical Element 7.1 What is the State definition for public high school graduation rate? Methodology for Calculation Graduation Rate: The below formula will be fully implemented in four years. It is South Dakota’s intention to build the database needed to calculate this rate for all subgroups over a four year period based on the following schedule. In school year 2003 include 12th grade data only; in school year 2004 include 11th and 12th grade data; in school year 2005 include 10th through 12th grade and in school year 2006 full implementation with the inclusion of data for grades 9th through 12th grades. The formula to be utilized is as follows: High School Completers in Year 4 Dropout (Gr. 9, year 1 + Gr. 10, year 2 +Gr. 11, year 3 +Gr. 12, year 4) + HS Completers, Year 4 This proposed calculation is based on the recommendation of NCES in a publication “Public High School Dropouts and Completers from Common Core of Data: School Year 1998-99 through 1999-2000”. This rate will be reported and utilized for purposes of determining AYP for all students (in the aggregate) and reported for disaggregated subgroups. Although you state that the above described methodology does not meet the NCLB required definition of a graduation rate, it was clearly described in the accountability workbook and approved by the agency (USDOE) which regulates NCLB. Further, the Consolidated State Performance Report, Parts I and II, for reporting on school year 2003-2004 states the following directive: “The Secretary approved each State’s definition of the graduation rate, consistent with section 200.19 of the Title I regulations, as part of each State’s accountability plan. Using the definition of the graduation rate that was approved as part of your State’s accountability plan, in the following chart please provide graduation rate data for the 2002-2003 school year”.(page 42) Quite simply, we did just that. The 2003-2004 Consolidated State Report contains the same language. As noted in your report, we now use four years of student data in calculating our graduation rate. The methodology we use is prescribed by the National Center for Educational Statistics (an organization supported by USDOE) and approved by the federal agency, as previously noted. We believe our approach does meet NCLB standards. The accountability workbook element and a copy of the USDOE approval letter are enclosed. Finding 2- Data Used for Graduation and Dropout Rates Were Inaccurate. Recommendation We recommend that the Assistant Secretary for Elementary and Secondary Education require the South Dakota Department of Education to develop and implement improved procedures on classifying and documenting graduate and leaver data, training data reporting staff, and data collection oversight and monitoring. Your report states that our data for dropout and graduation rates reported in the 2003-2004 Consolidated State Performance Report were not sufficiently accurate to produce reliable rates, specifically dropout rate data. Based on the results of the file reviews completed by your staff, we concur with the finding. Staff sampled cohort high school completers and cohort leavers in school year 2002-2003. Prior to NCLB, school districts were not required to report graduation and dropout data, and thus were not held accountable for the data. It is not a surprise that the data was not wholly accurate given the transition to reporting data for accountability purposes. Our Office of Data Collection staff provides technical assistance to school staff regarding data reporting and contacts school staff to resolve discrepancies primarily through telephone and e-mail contacts. We recognize that our oversight can be improved. To that end, we will implement the following steps: • Policies and procedures relative to reporting graduation and dropout data will be published in upcoming issues of the SIMS Newsletter. • Office of Data Collection staff will analyze input and output data, and compare to fall enrollment data and prior year data to identify variances. • Office of Data Collection staffwill monitor data for significant changes. When changes are identified, staffwill follow up with districts to determine the cause, and remedy irregularities if warranted. Polices and procedures specified in the SIMS Newsletter will be implemented immediately. We will compare fall 2005 12th graders to 2006 graduates to check for variances. Because this is the first year we will have a four year database, data monitoring will begin with FY07 data. All actions will be ongoing. Thank you for the opportunity to comment on the report. If you need further information, please contact Deborah Barnett at telephone 605-773-4708 or at email@example.com. Sincerely, Rick Melmer Secretary
Data Quality Review of the South Dakota Consolidated State Performance Report
Published by the Department of Education, Office of Inspector General on 2006-06-07.
Below is a raw (and likely hideous) rendition of the original report. (PDF)