oversight

Data Quality Review of the South Dakota Consolidated State Performance Report

Published by the Department of Education, Office of Inspector General on 2006-06-07.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                          UNITED STATES DEPARTMENT OF EDUCATION

                                OFFICE OF INSPECTOR GENERAL
                                   1999 BRYAN STREET, HARWOOD CENTER, SUITE 1440 

                                                 DALLAS, TEXAS 75201-6817 

                                                    PHONE: (214) 661-9530 

                                AUDIT FAX: (214) 661-9531 INVESTIGATION FAX: (214) 661-9589



                                                        June 7, 2006


                                                                                            Control Number
                                                                                            ED-OIG/A06F0021

Dr. Rick Melmer - Secretary
South Dakota Department of Education
700 Governors Drive
Pierre, SD 57501

Dear Dr. Melmer:

This Final Audit Report, entitled Data Quality Review of the South Dakota Consolidated State
Performance Report presents the results of our audit. The purpose of the audit was to determine
whether the South Dakota Department of Education’s required reporting of dropout and
graduation rates in the 2003-2004 Consolidated State Performance Report were supported by
reliable data and met the requirements of the Elementary and Secondary Education Act (ESEA).
Our review covered the reporting period of July 1, 2003 – June 30, 2004.




                                                  BACKGROUND 



Sections 9302 and 9303 of the ESEA, as amended by the No Child Left Behind Act of 2001
(NCLB), provides to States the option of applying for and reporting on multiple ESEA programs
through a single consolidated application and report.

The Consolidated State Performance Report (CSPR) includes the following ESEA programs:

    •     Title I, Part A, Part B, Subpart 3, Part C, Part D, and Part F
    •     Title II, Part A and Part D
    •     Title III, Part A
    •     Title IV, Part A, Subparts 1 & 2 and Part B
    •     Title V, Part A
    •     Title VI, Section 6111 and Part B




        Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations
J
ED-OIG/A06F0021 	                                                                   Page 2 of 12


The NCLB CSPR consists of two information collections. Part I of the 2003-2004 CSPR must
be submitted in January of 2005 and requests information related to the five ESEA Goals. Part II
of the 2003-2004 CSPR, due to the U.S. Department of Education (the Department) by April 15,
2005, consists of information related to State activities and the outcomes of specific ESEA
programs. The five ESEA Goals established in the June 2002 Consolidated State Application
are:

   • 	 Performance Goal 1: By 2013-2014, all students will reach high standards, at a
       minimum, attaining proficiency or better in reading/language arts and mathematics.
   • 	 Performance Goal 2: All limited English proficient students will become proficient in
       English and reach high academic standards, at a minimum, attaining proficiency or better
       in reading/language arts and mathematics.
   • 	 Performance Goal 3: By 2005-2006, all students will be taught by highly qualified
       teachers.
   • 	 Performance Goal 4: All students will be educated in learning environments that are
       safe, drug free, and conducive to learning.
   • 	 Performance Goal 5: All students will graduate from high school.

South Dakota adopted a comprehensive data collection system, the Student Information
Management System (SIMS) Net, in 1998 to electronically gather student data such as test
scores, attendance, and student status. Each student is assigned a unique identifier in SIMS Net
that matches student demographics and has the capacity to track the status and location of each
student. SIMS Net is a web-based student-level data collection system where student data
records are stored in a centralized data warehouse. South Dakota officials believe that SIMS Net
allows the state to collect and analyze more accurate and comprehensive student data, to meet
Federal and State reporting requirements.

Prior to the adoption of NCLB in 2001, South Dakota had not reported graduation or dropout
rates. South Dakota officials started collecting graduation and drop data through SIMS Net from
its districts to comply with NCLB. Prior to NCLB, South Dakota officials requested district
officials to provide an electronic file of data at their convenience. South Dakota utilized the
graduation and drop data provided by its districts for statistical purposes. For the school years
1999-2000 through 2001-2002, South Dakota officials stated that they have a database of the
graduation and drop data reported by its districts.

All school districts in South Dakota converted to the SIMS Net system except for the two largest
districts, Sioux Falls School District (SFSD) and Rapid City School District (RCSD). These
districts had already developed comprehensive data reporting systems and chose not to convert
to the SIMS Net. Except for SFSD and RCSD, school and district officials input the student data
directly into SIMS Net using a web-based format. Student data in SFSD and RCSD are
maintained on each district software vendor’s server until uploaded into SIMS Net. These
uploads occur on a routine basis.

Below are South Dakota’s definitions of terms used when discussing graduation and dropout
rates.
ED-OIG/A06F0021 	                                                                      Page 3 of 12


   • 	 High School Completer - individual awarded a high school diploma. This would not
       include a student who receives a non-standard diploma (e.g. a General Education
       Development (GED) or certificate of completion).
   • 	 Dropout - individual who
           a. 	 Was enrolled in school at some time during the previous school year; and
           b. 	 Was not enrolled at the beginning of the current school year; and
           c. 	 Has not graduated from high school or completed a state- or district-approved
                educational program; and
           d. 	 Does not meet any of the following exclusionary conditions:
                   1) Transferred to another public school district, private school, or state- or
                       district-approved educational program (including correctional or health
                       facility programs);
                   2) Temporary absence due to suspension or school-excused illness; or
                   3) Death.
   • 	 Cohort - individuals who were high school completers in 2002-2003.

For clarity in our discussion, we defined a student that left school in 2002-2003 and did not
return in school year 2003-2004 as a “Leaver.”

   • 	 Leaver - The status of a student who was enrolled or in attendance during a school year
       but stopped attending or did not return the next school year are reported using codes that
       describe the circumstances of the student’s departure. A school leaver is categorized as a
       dropout, or student who withdrew to: (a) enroll in another public or private school in the
       state; (b) enroll in a school outside the state; (c) enroll in a college; or (d) enter home
       schooling.

For our review, we selected the three largest school districts in South Dakota—RCSD, SFSD,
and Watertown School District (WSD) and visited the three largest high schools within RCSD
and SFSD districts. We also visited the only high school in WSD, for a total of seven high
schools.




                                     AUDIT RESULTS 



South Dakota met the requirements of ESEA by reporting dropout and graduation rates.
However, the State reported its 2003-2004 graduation rate by using a one-year cohort definition
in its calculation instead of the four-year cohort suggested by the Department. In addition, we
determined that for the seven schools reviewed, South Dakota collected unreliable data to
support the graduation and dropout rates reported in the 2003-2004 CSPR.

In its comments to the draft report, South Dakota did not concur with Finding 1 and its
recommendation, but did concur with Finding 2 and its recommendation. The comments are
summarized at the end of each finding. The full text of South Dakota’s comments on the draft
report is included as an Attachment A to the report.
ED-OIG/A06F0021                                                                       Page 4 of 12



FINDING 1 - South Dakota’s 2002-2003 Approved Graduation Rate Does Not Meet
NCLB Requirements

South Dakota reported a 2002-2003 graduation rate in its 2003-2004 CSPR using a one-year
cohort alternative graduation rate, which only captured dropouts in grade 12 and excluded
dropouts in grades 9-11. South Dakota’s 2002-2003 graduation rate was calculated using a
formula of a one-year cohort of high school completers in 2002-2003 divided by the sum of high
school completers plus dropouts for that one year. South Dakota officials stated that the
one-year cohort definition was used because the State had not been required to collect graduation
and dropout data prior to NCLB. South Dakota officials stated graduation and dropout data had
been collected prior to NCLB for statistical purposes only. We requested access to the
graduation and dropout database for 1999-2000 through 2001-2002 that was collected so we
could recalculate the graduation rate using a four-year cohort. South Dakota officials did not
provide us access stating, “we are not convinced that this data will yield any useful, valid and
accurate graduation calculations.”

Approved Definition Does Not Meet NCLB

As explained in the South Dakota Accountability Workbook submitted to and approved by the
Department, the one-year cohort graduation rate definition would only be used for the 2002-2003
school year. Since South Dakota had not reported a graduation rate prior to NCLB, State
officials are building a graduation rate database over a four-year period based on the following
schedule. In school year 2002-2003 include 12 grade data only; in school year 2003-2004
include 11-12 grade data; in school year 2004-2005 include 10-12 grade, and in school year
2005-2006 full implementation with the inclusion of data for grades 9-12 grades. By the end of
school year 2005-2006, the graduation rate definition will be a follows:

                                 High School Completers in Year 4
                        Dropouts (Grade 9, year 1 + Grade 10, year 2 + Grade 11,
                         year 3 + Grade 12, year 4) + HS Completers, Year 4

Although the Department reviewed and approved South Dakota’s alternative graduation rate
definition, the definition does not meet the NCLB required definition of a graduation rate. A
significant difference is that the South Dakota method does not track students using a cohort over
time (from entry to graduation). Instead, the formula used simply calculates an annual rate then
adds four years of annual rates together. The denominator includes only graduates and dropouts
for each year, leaving out a potentially large number of students in other categories such as those
not graduating on time but continuing their education and students pursuing a GED certificate.
When the denominator excludes students of the cohort, the resulting graduation rate is inflated or
higher than actual. This calculation produces a graduation rate that is unlikely to be truly
representative and would, therefore, also be inconsistent with the NCLB requirement to be valid
and reliable.

NCLB defines graduation rates as the percentage of students who graduate from high school with
a regular diploma in the standard number of years. The standard number of years is determined
by a state and is generally based on the structure of the school. For example, a high school with
grades 9 through 12 would have 4 as its standard number of years while a school with grades 10
ED-OIG/A06F0021 	                                                                       Page 5 of 12


through 12 would have 3 as its standard number of years. This method of measure is called a
longitudinal graduation rate. It is a cumulative indicator derived from student-level information
and reported as a function of entry year and graduation year. This method uses a cohort
definition to track students through the years spent in high school.

The cohort definition compares the number of 12th grade graduates with a standard diploma,
with the number of students enrolled as 9th graders 4 years earlier, while also taking into account
those who left the cohort, such as those who transferred in and out. A formula statement of a
cohort is:

       Cohort - Students who started high school (i.e., ninth grade) plus student transfers in, less
       student transfers out in year Y; plus student transfers in, less student transfers out in year
       Y+1; plus student transfers in, less student transfers out in year Y+2; plus student
       transfers in, less student transfers out in year Y+3.


One-Year Cohort Inflates Reported Graduation Rate

Section 200.19 of the Title I regulations issued under the NCLB Act on December 2, 2002,
defines graduation rate to mean:

   • 	 The percentage of students, measured from the beginning of high school, who graduate
       from public high school with a regular diploma (not including a GED or any other
       diploma not fully aligned with the State’s academic standards) in the standard number of
       years; or,
   • 	 Another more accurate definition developed by the State and approved by the Secretary
       in the State plan that more accurately measures the rate of students who graduate from
       high school with a regular diploma; and
   • 	 Avoids counting a dropout as a transfer.

The regulations define a four-year cohort that starts tracking students when they begin high
school as a freshman and continuing throughout the four years of high school, each year adding
students that transfer into South Dakota and subtracting students that transfer or drop out during
the four years. The regulations also indicate that States can use a more accurate definition than
the four-year cohort, if approved by the Department. Each State has flexibility, however, in
determining how its graduation rate will be specifically calculated as long as the rate is, as the
law requires, “valid and reliable.”

South Dakota developed and received Departmental approval for an alternative definition using a
one-year cohort for 2002-2003. However, the alternative definition did not provide a more
accurate graduation rate; instead, the definition inflated the reported graduation rate. This
inflation occurred because South Dakota focused on only one year of data, grade 12. By only
including data from grade 12 in the calculation, officials excluded all students that dropped out in
grades 9-11 from the calculation. The seven schools that we reviewed reported 1,680 students in
the 9-12 grades that were reported by school officials as leaving school and not returning to
school in 2002-2003. These leavers included transfers between South Dakota districts, transfers
out of South Dakota, and dropouts, among others. We found that 1,372 of the 1,680 (82 percent)
students reported as leavers were students in the 9–11 grades. Further, 362 of the 1,680 reported
ED-OIG/A06F0021                                                                      Page 6 of 12


leavers were reported as dropouts. Of the 362 reported dropouts, 291 (80 percent) were in the 9-
11 grades. Using a one-year cohort for the 2002-2003 graduation rate, South Dakota officials
incorrectly reported the 96 percent graduation rate in 2002-2003 by excluding all reported drops
in grades 9-11, which constituted 80 percent of the reported drops at the seven schools we
visited.

Recommendation

We recommend that the Assistant Secretary for Elementary and Secondary Education require the
South Dakota Department of Education to develop and implement a graduation definition that
meets NCLB.


South Dakota’s Comments

South Dakota did not concur with the finding or recommendation. South Dakota stated that the
method used for calculating its graduation rate was clearly described in the Consolidated State
Application Accountability Workbook and approved by the United States Department of
Education on June 3, 2003. They further stated the approved methodology was “consistent with
section 200.19 of the Title I regulations” and was “prescribed by the National Center for
Educational Statistics”(NCES).

OIG’s Response

Nothing in South Dakota’s comments changed our finding or recommendation. We are aware
that the Department approved South Dakota’s graduation methodology reported in its
Consolidated State Application Accountability Workbook; however, the methodology does not
meet NCLB, specifically, Section 200.19 of the Title I regulations. Section 200.19 allows for
states to either report a longitudinal graduation rate (percentage of students who graduate from
high school with a regular diploma in the standard number of years) or an approved graduation
rate that more accurately measures the rate of students who graduate from high school with a
regular diploma. South Dakota chose the latter, an alternative approach that was supposed to be
more accurate.

However, South Dakota’s alternative method was not more accurate. First, the alternative
graduation definition does not track students using a cohort over time (from entry to graduation).
Instead, the formula used simply calculates an annual rate then adds four years of annual rates
together. Second, the denominator includes only graduates and dropouts for each year, leaving
out a potentially large number of students in other categories such as those not graduating on
time but continuing their education, and students pursuing a GED certificate. When the
denominator excludes students of the cohort, the resulting graduation rate is inflated or higher
than actual. We believe that South Dakota’s method, although approved by the Department,
produces an unreliable and invalid graduation rate that is inconsistent with the NCLB
requirement to be valid and reliable.

Also, we are aware that NCES prescribed a graduation rate methodology in August 2002.
However, the NCES methodology does not meet NLCB standards because the methodology did
not require tracking students from freshman entry through high school graduation.
ED-OIG/A06F0021 	                                                                     Page 7 of 12




FINDING 2 - Data Used for Graduation and Dropout Rates Were Inaccurate

South Dakota’s data for dropout and graduation rates reported in the 2003-2004 CSPR were not
sufficiently accurate to produce reliable rates. Specifically, South Dakota incorrectly classified
or had insufficient supporting documentation to validate the reported dropout rate data.

Graduation Rate Data Were Not Always Reliable

To review data used in the graduation rate calculation, we selected two samples. The first
sample of 237 was of the cohort high school completers in school year 2002-2003. The graduate
sample data populate the numerator of the cohort graduation rate formula. Three (1.2 percent) of
the sampled 237 high school completers were incorrectly classified as graduates. Specifically, of
the three exceptions:

   • 	 Two were full-time private school students that enrolled for one class at the public high
       schools; and
   • 	 One completer graduated in 2003-2004, not 2002-2003.

The remaining 234 graduates were fully supported by high school transcripts. However, we
discovered in our reviews of cohort leavers and campus-wide leavers, discussed in detail below,
students reported as transfers by school officials, based on documentation obtained at the
schools, were graduates of the 2002-2003 reporting year. Graduate students incorrectly reported
as transfers constituted 11 percent (7 of the 64 leavers) of the cohort leavers and 4 percent (6 of
the 171 leavers) of the campus-wide leavers sampled.

As a result of these exception rates, we determined the data in the cohort were not sufficiently
accurate to produce a reliable graduation rate.

The second sample was of cohort leavers in school year 2002-2003. Leavers include dropouts,
as well as transfers between South Dakota districts, transfers out of South Dakota, and home
schoolers, among others. South Dakota officials only included dropouts in its denominator of the
graduation rate formula. However, to determine if school officials correctly classified and
reported dropouts, our random sample included all students reported by school officials as not
returning (leavers) to school in grade 12 in school year 2002-2003. We reviewed a sample of 64
cohort leaver records and determined 43 (67 percent) were incorrectly classified or had
inadequate or no supporting documentation for the drop status. Specifically, the student records
we reviewed for 20 exceptions contradicted the reported classification and were adequate to
conclude that:

   • 	 17 (26 percent) students were incorrectly classified as transfers. Nine students dropped
       out of school, seven students graduated from the reporting school in 2002-2003 and one
       student was a continuing student at the reporting school the next year; and
   • 	 three (five percent) students were incorrectly classified as dropouts but should have been
       classified as transfers.
ED-OIG/A06F0021                                                                       Page 8 of 12


Further, we determined that 23 (36 percent) of the 64 leavers had inadequate or no supporting
documentation. Without sufficient supporting documentation we could not determine whether
the leaver codes were accurate or there was inadequate or no supporting documentation for the
leaver classification. As a result of this exception rate, we determined data in the cohort were
not sufficiently accurate to produce reliable graduation rates.

Dropout Rate Data Were Not Always Reliable

To review data used in the dropout rate calculation, we selected a third sample. The annual
dropout rate is calculated by dividing the number of grade 9-12 dropouts during the school year
by the accumulative enrollment in those grades during the school year. The third sample was of
campus-wide leavers in school year 2002-2003. To review data factored into the annual dropout
rate calculation, we selected a sample of 171 students from the 9-12 grades that were reported by
school officials as leavers for the reporting year. These leavers included transfers between South
Dakota districts, transfers out of South Dakota, and dropouts, among others. We reviewed the
sample of 171 campus-wide leaver records and determined 97 (57 percent) were incorrectly
classified or had inadequate or no supporting documentation for the drop status. Of the 97
exceptions, 54 (32 percent) were incorrectly classified. The student records we reviewed at the
schools contradicted the reported classification and were adequate to conclude that two students,
incorrectly classified as dropouts, should have been a transfer and a continuing student. The
remaining 52 students were incorrectly classified as a type of transfer of which:

   •   30 should have been classified as dropouts;
   •    6 should have been classified as graduates;
   •    1 student should have classified as a continuing student; and
   •   15 students should have classified as another type of transfer.

Further, of the 97 exceptions, 43 (25 percent) had inadequate or no supporting documentation.
Without sufficient supporting documentation we could not determine whether the campus-wide
dropout codes were accurate or there was inadequate or no supporting documentation for the
leaver classification.

As a result of this exception rate, we determined data in the cohort were not sufficiently accurate
to produce reliable dropout rates.

Guidance and Oversight by South Dakota Officials Is Needed

These deficiencies occurred because South Dakota officials did not provide adequate guidance,
training, oversight, and monitoring to calculate graduation and dropout rates. Since the adoption
of the SIMS Net in 1998, the only guidance provided to schools regarding graduation and
dropout data were SIMS Newsletters published by South Dakota. South Dakota officials have
published 21 newsletters in seven school years. In relation to graduation and drop rates, the
Newsletters communicate revisions in data reporting codes and deadlines. No guidance is
provided to school officials on the correct classification and reporting of leaver data based on
supporting documentation. South Dakota also did not provide guidance on training requirements
for new or current employees responsible for data reporting. District officials at one of the three
districts stated that, for data reporting staff, the district provides two to four hours of new
employee training and one to two hours of update training each school year. On the other hand,
ED-OIG/A06F0021 	                                                                     Page 9 of 12


school officials interviewed in the remaining two districts stated that training was not provided
on data reporting. School officials stated that the State did provide a two to four hour training
session on SIMS Net when it was adopted in 1998. South Dakota provides a contact person at
the State for school officials if they had questions on data reporting. However, school officials
we interviewed rely more on experienced data reporting staff at the schools or districts.

South Dakota officials stated numerous data reliability queries are performed on data reported by
the schools and when necessary, school officials are contacted to resolve discrepancies in the
data. However, South Dakota officials provided no documentation of these queries. District
officials we interviewed were only aware of the duplicate student count query performed by the
State. These queries are the only monitoring South Dakota officials perform on the graduation
and dropout data reported by school officials.

The NCLB Act of 2001, Public Law 107-110, enacted January 8, 2002, places emphasis on and
strengthens the accountability for results. It also increases the importance of data and the need
for the Department to have data that are reliable and valid. Unreliable data cause graduation and
dropout rates to be inaccurate, either overstated or understated. It is important that data are
reliable because the graduation and dropout rates are performance measures considered by the
Department, the State, and the public in comparison to other States' performance, and the
information is also used to assess school, district, and State accountability.

As recipients of Title I grants, State officials agree to establish and maintain internal controls
designed to reasonably ensure compliance with Federal laws, regulations, and program
compliance requirements. OMB Circular A-133 Compliance Supplement, Part 6, is intended to
assist non-Federal entities in complying with these requirements by describing, for each type of
compliance requirement, the objectives of internal control and certain characteristics of internal
control that, when present and operating effectively, may ensure compliance with program
requirements. Part 6 discusses five components of internal control that should reasonably assure
compliance with the requirements of Federal laws, regulations, and program compliance
requirements. Two of the five components are Control Activities and Monitoring.

Control Activities are the policies and procedures that help ensure that management’s directives
are carried out. Specifically, the supplement mentions:

   • 	 Operating policies and procedures clearly written and communicated.
   • 	 Computer and program controls should include:
          a. 	 Data entry controls, e.g., edit checks.
          b. Exception reporting.
          c. 	 Reviews of input and output data.

Monitoring is a process that assesses the quality of internal control performance over time.
Some of the monitoring steps outlined in the supplement are:

   • 	 Periodic site visits performed and checks performed to determine whether procedures are
       being followed as intended.
   • 	 Follow up on irregularities and deficiencies to determine the cause.
   • 	 Internal quality control reviews performed.
ED-OIG/A06F0021 	                                                                     Page 10 of 12


   • 	 Management meets with program monitors, auditors, and reviewers to evaluate the
       condition of the program and controls.
   • 	 Internal audit routinely tests for compliance with Federal requirements.

Recommendation

We recommend that the Assistant Secretary for Elementary and Secondary Education require the
South Dakota Department of Education to develop and implement improved procedures on
classifying and documenting graduate and leaver data, training data-reporting staff, and data
collection oversight and monitoring.


South Dakota’s Comments

South Dakota concurred with this finding and stated that the following steps will be implemented
to improve graduation and dropout data reliability:

   • 	 Policies and procedures relative to reporting graduation and dropout data will 

       be published in upcoming issues of the SIMS Newsletter. 

   • 	 Office of Data Collection staff will analyze input and output data, and compare 

       to fall enrollment data and prior year data to identify variances. 

   • 	 Office of Data Collection staff will monitor data for significant changes. When 

       changes are identified; staff will follow up with districts to determine the cause, 

       and remedy irregularities if warranted. 





                  OBJECTIVES, SCOPE, AND METHODOLOGY 



The overall objective was to determine whether the South Dakota Department of Education’s
required reporting of dropout and graduation rates in the 2003-2004 CSPR were supported by
reliable data and met the requirements of the ESEA. Specifically, we determined whether the—

   • 	 data for high school completers were accurate and documented;
   • 	 leaver data in the 2003-2004 academic reporting year were accurate and documented; and
   • 	 leaver data in the reporting year for leavers were accurate and documented.

To accomplish our objective, we—

   •	   reviewed written policies and procedures for monitoring school supplied data;
   •	   reviewed applicable laws, regulations, and other guidance;
   •	   interviewed South Dakota officials, as well as, selected school and district officials; and
   •	   reviewed student files at the selected high schools.
ED-OIG/A06F0021 	                                                                     Page 11 of 12


We judgmentally selected Rapid City, Sioux Falls, and Watertown School Districts for review
because they were the largest districts in the South Dakota. We then selected the three largest
high schools in these districts for review. However, since Watertown district only had one high
school, we were limited to that one high school for a total of seven high schools in three districts.

For testing purposes, we had South Dakota extract from SIMS Net a database of students in the
2002-2003 academic year from each of the seven high schools including transfers in and out of
schools. From this extract, we created a database of high school completers and another
database of cohort leavers from the 2002-2003 school year data.

   • 	 From the high school completers, we drew a random 10 percent sample with a maximum
       of 50 and a minimum of 10 students for review at each high school from a universe of
       2,256 students.
   • 	 From the cohort leaver data, we drew a 10 percent random sample of students (with the
       same minimum/maximum as above) that left during the 2003-2004 academic reporting
       year at each high school to ensure they were properly classified from a universe of 232
       students.

We also obtained another extract of leavers that left the high school campus (campus-wide
leavers) during the 2003-2004 academic reporting year.

   • 	 From this extract, we drew a 10 percent random sample with a maximum of 50 and a
       minimum of 10 students at each high school to ensure they were properly classified from
       a universe of 1,680 students.

To achieve our audit objective, we relied, in part, on computer-processed data related to the
student information contained in the South Dakota’s SIMS Net database. At Infinite Campus,
the contractor for SIMS Net, comparability tests were conducted by our Computer Assisted
Assessment Technologies (CAAT) staff on the one-year cohort extract used for our audit work to
the contractor’s managed source database. The CAAT staff deemed the database to be
sufficiently reliable for the purposes of our audit. At the seven schools, we also verified the
completeness of data by comparing source records to computer-generated data, and verified the
authenticity by comparing computer-generated data to source documents. Based on these tests,
we concluded that data were sufficiently reliable to be used in meeting the audit’s objective.

We conducted our fieldwork at the State, RCSD, SFSD, and WSD between August 15, 2005, and
October 26, 2005. An exit conference was held with South Dakota officials on February 8, 2006.

Our audit was performed in accordance with generally accepted government auditing standards
appropriate to the scope of audit described above.
ED-OIG/A06F0021                                                                      Page 12 of 12




                            ADMINISTRATIVE MATTERS



Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

If you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Education Department
official, who will consider them before taking final Departmental action on this audit:

                              Henry Johnson
                              Assistant Secretary
                              Office of Elementary and Secondary Education
                              U.S. Department of Education
                              400 Maryland Ave., SW
                              Washington, DC 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.


                                              Sincerely,


                                              /s/
                                              Sherri L. Demmel
                                              Regional Inspector General
                                                for Audit

Attachment
                                                                      Attachment




April 20, 2006




Sherri L. Demmel, Regional Inspector General for Audit
United States Department of Education
Office of Inspector General
1999 Bryan Street, Harwood Center, Suite 1440
Dallas, TX 75201-6817

RE: Control Number ED-OIG/A06F0021

Dear Ms. Demmel:

This correspondence serves as the South Dakota Department of Education’s written
comments in response to the findings and recommendations contained in the draft audit
report entitled Data Quality Review of the South Dakota Consolidated State Performance
Report dated March 17, 2006. Our comments relative to the findings and recommendations
are as follows:

Finding 1- South Dakota’s 2002-2003 Approved Graduation Rate Does Not Meet
NCLB Requirements.

Recommendation

We recommend that the Assistant Secretary for Elementary and Secondary Education
require the South Dakota Department of Education to develop and implement a
graduation definition that meets NCLB.

We do not concur with the finding or recommendation. South Dakota’s method for
calculating our graduation rate is described in our Consolidated State Application
Accountability Workbook and was approved by the United States Department of Education
on June 3, 2003. The pertinent workbook element is as follows:

Critical Element 7.1 What is the State definition for public high school graduation rate?
Methodology for Calculation Graduation Rate:

The below formula will be fully implemented in four years. It is South Dakota’s intention to
build the database needed to calculate this rate for all subgroups over a four year period
based on the following schedule. In school year 2003 include 12th grade data only; in
school year 2004 include 11th and 12th grade data; in school year 2005 include 10th through
12th grade and in school year 2006 full implementation with the inclusion of data for grades
9th through 12th grades.

The formula to be utilized is as follows:

High School Completers in Year 4

Dropout (Gr. 9, year 1 + Gr. 10, year 2 +Gr. 11, year 3
+Gr. 12, year 4) + HS Completers, Year 4

This proposed calculation is based on the recommendation of NCES in a publication
“Public High School Dropouts and Completers from Common Core of Data: School Year
1998-99 through 1999-2000”.

This rate will be reported and utilized for purposes of determining AYP for all students (in
the aggregate) and reported for disaggregated subgroups.

Although you state that the above described methodology does not meet the NCLB required
definition of a graduation rate, it was clearly described in the accountability workbook and
approved by the agency (USDOE) which regulates NCLB. Further, the Consolidated State
Performance Report, Parts I and II, for reporting on school year 2003-2004 states the
following directive:

“The Secretary approved each State’s definition of the graduation rate, consistent with
section 200.19 of the Title I regulations, as part of each State’s accountability plan. Using
the definition of the graduation rate that was approved as part of your State’s accountability
plan, in the following chart please provide graduation rate data for the 2002-2003 school
year”.(page 42)

Quite simply, we did just that. The 2003-2004 Consolidated State Report contains the same
language. As noted in your report, we now use four years of student data in calculating our
graduation rate. The methodology we use is prescribed by the National Center for
Educational Statistics (an organization supported by USDOE) and approved by the federal
agency, as previously noted. We believe our approach does meet NCLB standards. The
accountability workbook element and a copy of the USDOE approval letter are enclosed.

Finding 2- Data Used for Graduation and Dropout Rates Were Inaccurate.

Recommendation
We recommend that the Assistant Secretary for Elementary and Secondary Education
require the South Dakota Department of Education to develop and implement
improved procedures on classifying and documenting graduate and leaver data,
training data reporting staff, and data collection oversight and monitoring.

Your report states that our data for dropout and graduation rates reported in the 2003-2004
Consolidated State Performance Report were not sufficiently accurate to produce reliable
rates, specifically dropout rate data. Based on the results of the file reviews completed by
your staff, we concur with the finding.

Staff sampled cohort high school completers and cohort leavers in school year 2002-2003.
Prior to NCLB, school districts were not required to report graduation and dropout data, and
thus were not held accountable for the data. It is not a surprise that the data was not wholly
accurate given the transition to reporting data for accountability purposes. Our Office of
Data Collection staff provides technical assistance to school staff regarding data reporting
and contacts school staff to resolve discrepancies primarily through telephone and e-mail
contacts.

We recognize that our oversight can be improved. To that end, we will implement the
following steps:

        • 	 Policies and procedures relative to reporting graduation and dropout data will
            be published in upcoming issues of the SIMS Newsletter.
        • 	 Office of Data Collection staff will analyze input and output data, and compare
            to fall enrollment data and prior year data to identify variances.
        • 	 Office of Data Collection staffwill monitor data for significant changes. When
            changes are identified, staffwill follow up with districts to determine the cause,
            and remedy irregularities if warranted.

Polices and procedures specified in the SIMS Newsletter will be implemented immediately.
We will compare fall 2005 12th graders to 2006 graduates to check for variances. Because
this is the first year we will have a four year database, data monitoring will begin with
FY07 data. All actions will be ongoing.


Thank you for the opportunity to comment on the report. If you need further information,
please contact Deborah Barnett at telephone 605-773-4708 or at deb.barnett@state.sd.us.

Sincerely,




Rick Melmer
Secretary