oversight

Audit of Selected Portions of the U.S. Department of Education's Oversight of the Consolidated State Performance Reports

Published by the Department of Education, Office of Inspector General on 2008-04-04.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                   UNITED STATES DEPARTMENT OF EDUCATION
                                          OFFICE OF INSPECTOR GENERAL

                                                                                                                        Audit Services

                                                            April 4, 2008
                                                                                                                Control Number
                                                                                                              ED-OIG/A06H0001


Kerri L. Briggs
Assistant Secretary
Office of Elementary and Secondary Education
U.S. Department of Education
400 Maryland Avenue, SW
Washington, DC 20202


Dear Dr. Briggs:

This Final Audit Report, entitled Audit of Selected Portions of the U.S. Department of
Education’s Oversight of the Consolidated State Performance Reports, presents the results of our
audit. The purpose of the audit was to determine whether the Department of Education’s Office
of Elementary and Secondary Education (OESE) provided sufficient oversight of graduation and
dropout rates submitted by states in their Consolidated State Performance Reports to ensure the
rates were supported by reliable data. Our review covered the reporting period of July 1, 2003,
through June 30, 2004.




                                                      BACKGROUND 



Sections 9302 and 9303 of the Elementary and Secondary Education Act of 1965 (ESEA), as
amended by the No Child Left Behind Act of 2001 (NCLB), Public Law 107-110, enacted
January 8, 2002, provide to states the option of applying for and reporting on multiple ESEA
programs through a single consolidated application and report.




 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
Final Report
ED-OIG/A06H0001                                                                                    Page 2 of 13


The NCLB Consolidated State Performance Reports (CSPRs) are required to be submitted
annually to the Department and consist of two information collections. Part I of the CSPR
provides information from the prior school year related to the five ESEA Goals.1 Part II of the
CSPR consists of information related to state activities and the outcomes of specific ESEA
programs.

States report graduation rates for all students and by student subgroups in Part I of the CSPR.
The CSPR instructions require the state to report graduation rates computed in accordance with
the definition approved as part of the state’s accountability plan (i.e., approved Consolidated
State Application Accountability Workbook).

Graduation Rate is identified as a specific accountability indicator for secondary education and is
defined in Section 1111 (b)(2)(C)(vi) of NCLB “. . . as the percentage of students who graduate
from secondary school with a regular diploma in the standard number of years . . . .” (emphasis
added)

The National Center for Education Statistics (NCES)2 preferred indicator calculates graduation
rates by dividing the number of cohort graduates in the reporting year by the cohort for that year.
A cohort is defined as “Students who started high school (i.e., ninth grade) plus student transfers
in, less student transfers out in year Y; plus student transfers in, less student transfers out in year
Y+1; plus student transfers in, less student transfers out in year Y+2; plus student transfers in,
less student transfers out in year Y+3.”

The CSPR instructions also require states to report the annual dropout rates for high school
students using the definition of a high school dropout from the NCES. The annual dropout rate
is calculated by dividing the number of grade 9-12 dropouts during the school year by the total
number of students served in those grades during the school year. The classification (leaver
code) of a student’s status (e.g., graduate, dropout, transfer, or continuing) determines their usage
in the graduation and dropout rate calculations.




1
  Performance Goal 1: By 2013-2014, all students will reach high standards, at a minimum attaining proficiency or
better in reading/language arts and mathematics. Performance Goal 2: All limited English proficient students will
become proficient in English and reach high academic standards, at a minimum attaining proficiency or better in
reading/language arts and mathematics. Performance Goal 3: By 2005-2006, all students will be taught by highly
qualified teachers. Performance Goal 4: All students will be educated in learning environments that are safe, drug
free, and conducive to learning. Performance Goal 5: All students will graduate from high school.
2
  NCES is a part of the Institute of Education Sciences within the U.S. Department of Education. NCES is the
primary federal entity responsible for collecting and analyzing data related to education.
Final Report
ED-OIG/A06H0001                                                                      Page 3 of 13




                                     AUDIT RESULTS 



OESE could have provided better oversight to ensure that graduation and dropout rates submitted
by states in their Consolidated State Performance Reports were supported by reliable data.
Specifically, we determined that the Department did not put enough emphasis on data reliability
and comparability across states.

In its comments to the draft report, OESE generally agreed with the finding and most of the
recommendations. The comments are summarized at the end of the finding. The full text of
OESE’s comments on the draft report is included as an Attachment to the report. Based on
OESE’s comments, we modified four of our recommendations.


FINDING – More Emphasis Needed on Data Reliability and Comparability Across
          States
We conducted four audits (Oklahoma, South Dakota, Texas, and Washington) of the reliability
of the graduation and dropout rates submitted in the states’ 2003-2004 CSPRs. The data used by
the states to compute graduation and dropout rates were not always sufficiently reliable.

Graduation Rates Not Supported by Reliable Data

To determine if states were reporting graduation rates that were supported by reliable data, we
selected two samples of students at each state audited—one of graduates and one of cohort or
graduating year leavers. We compared our samples of graduates reported by each state to the
supporting documentation and found that the reporting of the number of graduates for all four
states was accurate and supportable. However, in comparing our second sample of dropouts and
transfers during the cohort period to the supporting documentation, we found significant errors in
the numbers of dropouts and transfers reported by all four states. Additionally, only one state
used a true cohort to compute the graduation rate. The other three states used data from only one
year instead of four years of data tracked longitudinally that would be necessary to calculate a
cohort.

Although approved by the Department, the different graduation rate formulas used by the three
states did not provide a graduation rate that was consistent with NCLB. The formulas used by
the three states were—

   •	 A one year cohort alternative graduation rate, which only captured dropouts in grade 12
      and excluded dropouts in grades 9-11;
   •	 A synthetic graduation rate, which used dropout data for grades 9-12 for one school year;
      and
   •	 A four-year cohort reported in the state’s Accountability Workbook that was in fact
      computed with data from a single year.
Final Report
ED-OIG/A06H0001                                                                                      Page 4 of 13

These formulas were given approval with no qualifications by the Department and thus would
require no further review and approval unless changed by a state in its submission of a new
Consolidated State Application Accountability Workbook. Therefore the three states could
continue to use graduation rate formulas that were not consistent with NCLB. We acknowledge
that at the time NCLB was enacted only a few states had the capacity to calculate true cohort
graduation rates because most states’ data systems were not capable of tracking individual students
across years. We believe provisional approval of graduation rate formulas for these states would
provide a means to track states while they are developing longitudinal tracking systems.3

Additionally, three of the four states erroneously included students who took longer than the
standard number of years to graduate in the numerator of the graduation rate calculations.
Students who take longer than the standard number of years to graduate do not meet the
definition of an “on-time graduate” and should not be included in the calculation. Further, the
states did not include students who did not graduate but continued their high school education
into the following year in their graduation rate calculations. Continuing students should be
included in the denominator of the graduation rate calculation because they are part of the total
population served. Including students in the numerator when they should not be included and
excluding students from the denominator when they should be included result in inflated
graduation rates. Lastly, three of the four states did not have a data collection system in place to
compute graduation rates in accordance with NCLB.

As a result of these problems, we determined that the graduation rates for all four states were not
supported by reliable data.

Dropout Rates Not Supported by Reliable Data

To determine if the four states’ dropout rates were supported by reliable data, we selected
samples of students in grades 9-12 who were reported as dropouts for the reporting year,
transfers between districts, and transfers out of state. We compared our samples to supporting
documentation. The dropout rate for only one state was supported by sufficiently reliable data.
The three remaining states’ dropout rates were not supported by reliable data because the states
did not provide adequate guidance, did not adequately monitor the districts, and did not
adequately train personnel.

Department Oversight and Guidance Needs To Be Strengthened

We reviewed the applicable portions of the Accountability Workbooks for the 50 states, the
District of Columbia, and Puerto Rico. Only 11 of 52 (21 percent) Accountability Workbooks
(1) used an actual cohort where the students were tracked during their progression through high
school and (2) were in compliance with the law. The majority of the Accountability Workbooks
(32) used an estimated/synthetic cohort formula. The remaining 9 Accountability Workbooks
used 7 different formulas which did not track a cohort and, among other things, did not account
for continuing students, excluded General Education Development (GED) certificate recipients


3
 A data system which tracks a cohort of students over five to seven years, from the time they enter Grade 9 or
Grade 7 until the fall following their anticipated graduation date. (Texas Education Agency, Secondary School
Completion and Dropouts in Texas Public Schools 2002-03)
Final Report
ED-OIG/A06H0001                                                                        Page 5 of 13

from the denominator, used only one year of data, or did not use the NCES definition of a
dropout.

OESE reviewed graduation rates as part of its Title I monitoring. We reviewed the 52 reports
covering school years 2003-2004 through 2005-2006. Only one report identified an error in the
calculation of the state’s graduation rate, GED students were erroneously included as graduates.
This is in sharp contrast to the results of our audits, where none of the four states we reviewed
had reliable data to support their graduation rates, and one state was not using the graduation rate
formula approved by the Department.

We also reviewed the current Student Achievement and School Accountability Programs
Monitoring Plan for Formula Grant Programs and noted steps to check for the presence of
procedures addressing data quality of elements within Adequate Yearly Progress (AYP) and state
report cards. However, we did not find any steps designed to check the accuracy of the
graduation rate formulas or the accuracy of the underlying supporting data.

Department officials were very careful not to require individual student tracking because they do
not believe NCLB requires it. Although NCLB does not specifically require individual student
tracking, the only way to know if a student has graduated in the standard number of years is to
track an individual student from entry into high school through graduation. Since the
implementation of NCLB in 2002, the Secretary and Deputy Secretary issued 32 Policy Letters
that deal with some aspect of NCLB. None of the policy letters provide any specific guidance to
the states on graduation or dropout rates or the importance of a longitudinal tracking system. If
the Department had been more assertive in requiring states to implement a longitudinal student
tracking system shortly after the enactment of NCLB, all states now could have four years of
student data. Instead, less than a quarter of the states are using a system that complies with the
requirements of the law.

The Department issued Non-Regulatory Guidance, titled Improving Data Quality for Title I
Standards, Assessments, and Accountability Reporting, in April 2006. The Department states in
this document—

       A fundamental piece of any data quality infrastructure is a standardized set of
       precise data definitions that all providers use. A “data dictionary,” which
       identifies all data elements and describes their content, coding options, and
       format, is essential to establishing consistent collection and reporting. Adhering
       to a standard data dictionary improves data quality by fostering interoperability of
       different reporting systems and promoting the use of comparable data across the
       entire State.

The Department has not developed a standard set of data definitions that all states would be
required to follow. Because there is no standard set of data definitions for the states to follow,
states are handling some identical student statuses in differing manners. An example observed
during our audits related to students returning to their home country. One state requires only a
withdrawal form signed by a parent/guardian and a school official and then that student is
counted as a transfer. Another state requires a transcript request from the student’s new school
Final Report
ED-OIG/A06H0001                                                                         Page 6 of 13

before that student can be counted as a transfer; otherwise, the student must be counted as a
dropout.

NCES published a technical report in August 2006, titled “Users Guide to Computing High
School Graduation Rates” (NCES 2006-604). It states that NCLB

        . . . included an on-time graduation rate as an accountability reporting
        requirement. The accurate reporting of such a rate requires student record data on
        student progression from grade to grade, data on graduation status, and data on
        students who transfer in and out of a school, district, or state during the high
        school years, or in other words cohort data (National Institute of Statistical
        Sciences (NISS) 2004 Task Force, NCES 2005-105).

The NCLB places emphasis on and strengthens accountability for results. It also increases the
importance of the Department having reliable and valid data. The data reliability problems we
identified occurred because the Department did not provide adequate guidance on such things as
tracking students from entry into high school through graduation and the standardization of the
data elements used in the graduation rate formulas. Additionally, the Department did not assess
the reliability of the graduation rate data submitted by the states. Lastly, the Department
approved graduation rate formulas that did not track true cohorts.

Allowing states to use varying formulas not only results in graduation rates that are inaccurate,
but it also results in graduation rates that are not comparable between states. It is important that
the data be reliable because states, and the public, use the graduation and dropout rates in
evaluating schools’ performances. The information can also be used to assess school, district,
and state accountability. Finally, as one of the indicators in AYP, inaccurate graduation rates
could result in schools being identified as not making AYP when they are or vice versa.

Recommendations

We recommend that the Assistant Secretary for Elementary and Secondary Education—

1.1 	   Stress to states the importance of using a longitudinal student tracking system.

1.2 	   Disapprove any future graduation rate formulas that do not use a true cohort except for
        states that are in the process of developing data systems that would enable them to
        calculate a cohort graduation rate. For those states, we recommend granting provisional
        approval pending full implementation of the necessary data systems.

1.3 	   Continue to impress upon states the importance of data quality and the need to provide
        regular guidance, training, and monitoring of their local educational agencies.

1.4 	   Publish a list of data definitions that states will be required to use in NCLB reports. In
        addition, the published list should include procedures on how school districts should
        handle different classifications of students, (e.g., transfers and returning to home country)
        and the documentation requirements to support the classification.
Final Report
ED-OIG/A06H0001                                                                        Page 7 of 13

1.5 	   Continue to develop and implement written policies and procedures to monitor states that
        are in the process of developing data collection systems to ensure the systems will collect
        student data in accordance with the requirements in NCLB.

1.6	    Periodically review its procedures to monitor the accuracy of the data used by the states
        and districts to compute graduation rates and make any necessary adjustments.


OESE’s General Comments

In its response, OESE generally agreed with our finding, acknowledging that the subject of the
IG Report, graduation rates, is an important NCLB accountability measure. However, OESE
stated that the report, which looks at data from the 2003-04 school year, does not accurately
reflect the data capacity of states at the time NCLB was signed into law in January 2002 and
does not mention the progress that has been made since.

OESE agreed that reliable graduation rates are a key factor in holding high schools accountable for
graduating their students within four years. It further believes that a high-quality, uniform
graduation rate would improve the reliability of graduation rate data among the states.
However, OESE stated it was concerned that the draft report does not provide sufficient
perspective on the matter with regard to the early years of implementing NCLB. The OIG audit
focuses on state graduation data from the 2003-04 school year, the first year that the new
accountability measures under NCLB, including measuring graduation rates, were in effect.

OESE further stated that it is important to acknowledge that, at the time NCLB was enacted, only a
few states had the capacity to calculate true cohort graduation rates because most states’ data
systems were not capable of tracking individual students across years. OESE also pointed out that
states have made substantial progress in developing their data systems for collecting and reporting
information on graduation rates.

OIG’s Response to General Comments

We are aware that most states did not have the systems in place to capture longitudinal data and
that progress is being made in this area. However, our focus was the reliability of the data. Our
report addresses the lack of supporting documentation for and the non-comparability of the data
being reported. This is why our recommendations address the development of data collection
systems which will provide reliable and comparable information.

OESE’s Comments on the Finding

OESE agreed that states need to continue their efforts to improve the quality of data used to
compute graduation and dropout rates. However, OESE stated the report should make clear that
NCLB did not establish a single method of computing graduation rates. There is no one
definition of graduation rate that complies with NCLB, and NCLB does not mandate a definition
that is comparable across states.
Final Report
ED-OIG/A06H0001                                                                         Page 8 of 13

OESE further stated that with the states’ progress in data systems and the recent coalescence
around the National Governors Association’s (NGA's) suggested graduation rate definition, it
acknowledged the benefits of establishing a more uniform and accurate definition of graduation
rate. Also, the Department's reauthorization proposal would require use of the NGA definition.
The NCES has calculated and reported the "averaged freshman graduation rate" (AFGR) for each
state to provide more accurate, comparable graduation rate information across all states. In
addition, OESE takes seriously issues that compromise data quality and has augmented its
review of data quality.

OIG’s Response

It is apparent that the Department desires comparability when it developed the AFGR. The
Deputy Secretary stated in a press release in November 2005, when the AFGR was posted, that it
was an “important first step towards a national perspective on the success of our high schools . . .
and is comparable across states.” While NCLB does not specifically require states to use
definitions of graduation rates that can be compared across states, the intent of Congress was for
the public to use the information to compare one school to others, even across state lines. The
House Committee stated in House Report 107-404 that:

            the Committee intends for the Director to consider what would comprise
            an appropriate uniform standard, building upon prior efforts by the
            Department and the National Research Council, by which States could
            report dropout and graduation data, as well as examine longitudinal
            measurements that follow students from 7th grade through graduation
            from secondary school, and support more accurate and consistent
            measures that avoid problems such as distinguishing between transfers
            and dropouts that will allow for meaningful comparisons across schools,
            districts, and States.

OESE’s Comments on Recommendations and OIG’s Response

Recommendation 1.1	 Stress to states the importance of using a longitudinal student tracking
                    system.

 OESE
 Comments          OESE did not comment on Recommendation 1.1.

 OIG
 Response          No change to the recommendation.

Recommendation 1.2	 Disapprove any future graduation rate formulas that do not use a true
                    cohort.

 OESE              OESE stated it recognizes the value and preference for a cohort but
 Comments          reasonable flexibility is still necessary for states who are unable to calculate a
                   cohort graduation rate until necessary data systems are in place.
Final Report
ED-OIG/A06H0001                                                                     Page 9 of 13

 OIG 	            We are aware that the Department has the flexibility to waive requirements
 Response 	       under the law. However, in our opinion granting a provisional approval
                  would be more appropriate than a full approval. Therefore, we modified the
                  recommendation to “Disapprove any future graduation rate formulas that do
                  not use a true cohort except for states that are in the process of developing
                  data systems that would enable them to calculate a cohort graduation rate.
                  For those states, we recommend granting provisional approval pending full
                  implementation of the necessary data systems.”

Recommendation 1.3	 Impress upon states the importance of data quality and the need to
                    provide regular guidance, training, and monitoring of their local
                    educational agencies.

 OESE             OESE agreed with Recommendation 1.3, noting that OESE already has 

 Comments         augmented efforts to further impress upon states the importance of data 

                  quality. 


 OIG 	            Based on OESE’s comments, we have modified Recommendation 1.3 for
 Response 	       OESE to “Continue to impress upon states the importance of data quality and
                  the need to provide regular guidance, training, and monitoring of their local
                  educational agencies.”

Recommendation 1.4	 Publish a list of data definitions that states will be required to use in
                    NCLB reports. In addition, the published list should include procedures
                    on how school districts should handle different classifications of
                    students, (e.g., transfers and returning to home country) and the
                    documentation requirements to support the classification.

 OESE 	           OESE agreed having states use uniform data definitions for NCLB reporting
 Comments 	       is a worthwhile goal but needs to be balanced with the flexibility the statute
                  grants to states. However, it stated the non-regulatory guidance, noted in the
                  body of this report, more appropriately addressed these concerns.

 OIG 	            Flexibility for states to define data elements negates the likelihood of having
 Response 	       data that is comparable. Further, the Non-Regulatory Guidance indicates that
                  a standardized set of data definitions is a fundamental piece of any data
                  quality infrastructure but does not identify or define the data elements. In
                  addition to noting the importance of standard definitions, OESE should define
                  those data elements that all states are required to report to the Department.
                  We did not change the recommendation.
Final Report
ED-OIG/A06H0001                                                                         Page 10 of 13

Recommendation 1.5	 Develop and implement written policies and procedures to monitor
                    states that are in the process of developing data collection systems to
                    ensure the systems will collect student data in accordance with the
                    requirements in NCLB.

 OESE 	            OESE stated it is not clear that the Department has the authority to monitor
 Comments 	        the development of state data systems. It further stated that during its 

                   monitoring of states it examines whether states have the data necessary to 

                   meet NCLB requirements and require corrective actions when they do not. 


 OIG 	             While OESE is not clear it has the authority, its response indicates that it
 Response 	        requires states to take corrective action if the states do not have data
                   necessary to meet NCLB requirements. It appears from its response that it is
                   requiring states to have the data necessary to meet NCLB requirements. We
                   have therefore modified the recommendation for OESE to “Continue to
                   develop and implement written policies and procedures to monitor states that
                   are in the process of developing data collection systems to ensure the systems
                   will collect student data in accordance with the requirements in NCLB.”

Recommendation 1.6	 Modify the monitoring procedures to check the accuracy of the data used
                    by the states and districts to compute graduation rates.

 OESE              OESE agreed with Recommendation 1.6, stating it has already revised its
 Comments          monitoring indicators for reviewing the procedures states and school districts
                   have in place to ensure data quality.

 OIG 	             Based on OESE’s comments, we have modified Recommendation 1.6 to
 Response 	        “Periodically review its procedures to monitor the accuracy of the data used 

                   by the states and districts to compute graduation rates and make any 

                   necessary adjustments.” 


OESE Summary of Issues Identified in Individual State Audit Reports

As part of its response, OESE reiterated several positions taken in the state audits.

OIG Response to Summary of Issues

Comments regarding prior audits were addressed during the process of issuing the respective
reports. Those reports can be obtained from the OIG Audit Service Reading Room on the
internet at http://www.ed.gov/about/offices/list/oig/areports.html or will be provided upon
request. The states reported on, their audit control numbers, and issue dates are: Texas (Audit
Report A06F0020, dated March 21, 2006); South Dakota (Audit Report A06F0021, dated May 7,
2006); Oklahoma (Audit Report A06G0008, dated October 23, 2006); and Washington (Audit
Report A09G0009, dated November 14, 2006.)
Final Report
ED-OIG/A06H0001                                                                     Page 11 of 13

OESE Miscellaneous Comments

OESE reiterated its desire to emphasize that no one specific definition of graduation rate is
required by the current statute or regulations and that the report also appears to imply that it
should require states to implement longitudinal student tracking systems though NCLB does
not require such systems. OESE stated that it was not clear whether the problem the report is
attempting to present is what was approved in states' Accountability Workbooks or their
implementation of what was approved.

OIG Response to Miscellaneous Comments

While NCLB does not specifically require states to use definitions of graduation rates that can be
compared across states, the intent of Congress was for the public to use the information to
compare one school to others, even across state lines. While OESE emphasizes that no one
definition is required, in its General Comments at the beginning of its response, it agreed with
the OIG position stating “. . . that a high-quality, uniform graduation rate would improve the
reliability of graduation rate data among States.” Also, it has proposed requiring every state to
use the cohort graduation rate endorsed by the NGA during reauthorization of NCLB. In
addition, in our Perspective Paper, An OIG Perspective on Improving Accountability and
Integrity in ESEA Programs, issued in October, 2007, we address amending the General
Education Provisions Act to establish standard definitions for data quality terms in ESEA and
other laws authorizing Federal education programs.

One of our concerns dealt with the approval of Accountability Workbooks with graduation rate
formulas that did not track a true cohort. We are aware of the waiver authority. However,
granting a provisional approval would be more appropriate than a full approval.




                  OBJECTIVE, SCOPE, AND METHODOLOGY



The objective of our audit was to determine whether OESE provided sufficient oversight of
graduation and dropout rates submitted by states in their CSPRs to ensure the rates were
supported by reliable data.

To accomplish our objective, we—

   •	 Reviewed applicable laws, regulations, and other guidance;
   •	 Reviewed pertinent pages from the Accountability Workbooks for all 50 states, the
      District of Columbia, and Puerto Rico;
   •	 Reviewed Policy Letters issued by the Secretary and Deputy Secretary;
   •	 Reviewed the Department’s monitoring procedures;
   •	 Analyzed the results of our audits of four states;
   •	 Reviewed 52 Title I monitoring reports prepared by OESE; and
   •	 Interviewed OESE officials.
Final Report
ED-OIG/A06H0001                                                                     Page 12 of 13


We conducted our fieldwork at the Department of Education in Washington, D.C., from
November 6, 2006, through December 6, 2006. An exit conference was held with OESE
officials on April 16, 2007. We provided the draft audit report to OESE on June 18, 2007.
OESE provided a response to the draft audit report on August 7, 2007. OESE subsequently
requested to revise its response and provided its revised response on February 13, 2008.

Our audit was performed in accordance with generally accepted government auditing standards
appropriate to the scope of audit described above.



                            ADMINISTRATIVE MATTERS



Corrective actions proposed (resolution phase) and implemented (closure phase) by your office
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System (AARTS). ED policy requires that you develop a final corrective action plan
(CAP) for our review in the automated system within 30 days of the issuance of this report. The
CAP should set forth the specific action items, and targeted completion dates, necessary to
implement final corrective actions on the findings and recommendations contained in this final
audit report. An electronic copy of this report has been provided to your Audit Liaison Officer.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
six months from the date of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

If you have any questions, please call Gary D. Whitman, Regional Inspector General for Audit,
at 312-730-1620.


                                             Sincerely,



                                             /s/
                                             Keith West
                                             Assistant Inspector General
                                             for Audit


Attachment
Final Report 

ED-OIG/A06H0001                                  Page 13 of 13 





                  Attachment – OESE Response 

                                                                                                            Attachment



                          UNITED STATES DEPARTMENT OF EDUCATION
                               OFFICE OF ELEMENTARY AND SECONDARY EDUCATION




                                                                                 FEB 13 2008



TO:               George A. Rippey
                  Acting Assistant Inspector General for Audit

FROM:              Kerri L. Briggs, Ph.D /s/

SUBJECT:          Draft Audit Report, entitled Audit of Selected Portions of the U.S. Department of
                 Education's Oversight of the Consolidated State Performance Reports, Control
                 Number ED-OIG/A0 6H0001

We appreciate the opportunity to review and provide comments on the draft audit report entitled
Audit of Selected Portions of the U.S. Department of Education's Oversight of the
Consolidated State Performance Reports. Attached are the Office of Elementary and
Secondary Education's updated comments on the draft report, which are intended to replace
earlier comments that we provided. Please let me know if you have questions.




Attachment




                             400 MARYLAND AVE., S.W. WASHINGTON, D.C. 20202
                                              www.ed.gov
        Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation.
                                                                                               Attachment
Draft Audit Report, Audit of Selected Portions of the U.S. Department of Education's Oversight of the
Consolidated State Performance Reports (Control Number ED-OIG/A06H0001)

General Comments

In this draft audit report, the Office of Inspector General (OIG) examines a very important topic:
graduation rates, which are a critical accountability measure under the No Child Left Behind Act
(NCLB). Through its audit work, OIG raises a number of issues for consideration by policymakers, both
within the Department and in Congress during the upcoming reauthorization of NCLB. Specifically, OIG
recommends the use of a uniform definition of graduation rate in order to compare graduation rates
among States. OIG also recommends that such uniform definition follow a cohort of students from entry
in ninth grade through twelfth grade who graduate with a regular diploma in the standard number of
years, and track students who transfer into the cohort and those who leave the cohort to ensure that they
enroll in another school or educational program that confers a regular diploma; otherwise, they must be
counted as dropouts. OIG also highlights the need for common definitions of the components used to
calculate graduation rates to facilitate accurate computation and emphasizes the need for high-quality
reliable data in reporting graduation rates, particularly regarding dropouts and transfers.

We agree with OIG that reliable graduation rates are a key factor in holding high schools accountable for
graduating their students within four years. Moreover, we believe that a high-quality, uniform graduation
rate would improve the reliability of graduation rate data among the States. That is why the
Administration proposed in its Blueprint for reauthorizing NCLB that the reauthorization include
requiring every State to use the cohort graduation rate endorsed by the National Governors Association
(NGA). As the Secretary recently noted in her speech at the National Press Club on the sixth anniversary
of NCLB, if Congress does not act quickly to reauthorize the law, she intends to take available
administrative actions to address areas of concern to her, including a uniform graduation rate. Although
we are hopeful that reauthorization will occur this year, we are also exploring what existing authority we
may have with regard to securing more uniform and reliable graduation rate data. We appreciate that the
OIG helped to focus attention on this important issue.

We are concerned, however, that the draft report does not provide sufficient perspective on the matter
with regard to the early years of implementing NCLB. The OIG audit focused on State graduation data
from the 2003-04 school year, the first year that the new accountability measures under NCLB, including
measuring graduation rates, were in effect. The Department's regulations published in December 2002
made clear that a State must measure graduation rate from the beginning of high school. At the same
time, the regulations did not require a uniform definition in order to afford flexibility and because there
was not clear consensus at that time as to what that definition should be. Moreover, the Department did
not mandate that States adopt a definition that would require individual tracking of students across grades,
taking into account the data systems that States had at the time.

It is important to acknowledge that, at the time NCLB was enacted, only a few States had the capacity to
calculate a true cohort graduation rate because their data systems were not capable of tracking individual
students across years. Even under the Department's more modest regulations, in 2003-04, few States had
the data collection and reporting capability to calculate a graduation rate that measured students from
entry in ninth grade through high school. Moreover, they did not have data for the 12th grade class in
2003-04 on dropouts, and transfers, etc. from the prior three years. As a result, the Department used its
"transition authority" to grant most States permission under section 4(c) of NCLB to use "proxy" or
"transitional" definitions until States could accumulate four years of data on a given class. The audit
report would be improved significantly if it acknowledged this perspective more fully, so that readers
fully understand the context in which the OIG's findings emerged.

                                                     1
                                                                                               Attachment
In addition, the report should note that substantial progress has been made by States in developing their
data systems for collecting and reporting information on graduation rates since 2003-04. We believe it is
important for readers to know that improvements have been made to mitigate the level of concern from
the earliest years of the implementation of NCLB that is cited in the draft report.

Specific Comments on Findings and Recommendations are as follows:

1.   Finding (page 3): More Emphasis Needed on Data Reliability, Comparability. and Oversight

The Department agrees that States need to continue their efforts to improve the quality of data used to
compute graduation and dropout rates. As with any new endeavor, it takes time and iterative efforts to
effectively implement a new system or process.

The report should make clear, however, that NCLB did not establish a single method of computing
graduation rates. There is no one definition of graduation rate that complies with NCLB and NCLI3 does
not mandate a definition that is comparable across states, a point made in OESE's determinations
regarding the audit report on South Dakota (see additional comments below). It would be very helpful if
the report were clearer on this important fact; otherwise, it may be misleading.

Taking into account the progress States have made in developing more sophisticated data systems and the
recent coalescence around the NGA's suggested definition, we acknowledge the benefits of establishing a
more uniform and accurate definition of graduation rate. As noted above, the Department's
reauthorization proposal would require use of the NGA definition. Moreover, the Department' s National
Center for Educational Statistics (NCES) has calculated and reported the "averaged freshman graduation
rate" for each State to provide more accurate, comparable graduation rate information across all States. In
addition, we note that issues that compromise data quality are also concerns OESE takes seriously and
that OESE has augmented its review of data quality, both through monitoring efforts and other efforts
including the provision of guidance on data quality.

2.   Recommendations (page 6).

Recommendation 1.2: Disapprove any future graduation rate formulas that do not use a true cohort. The
process OESE has established for reviewing and approving State graduation rate formulas centers on
State Accountability Workbooks. All States have approved graduation rate definitions in their current
Accountability Workbooks. Any State wishing to change its definition of graduation rate must request an
amendment to its Accountability Workbook, and we review the amendment requests against the current
statutory and regulatory requirements. We also recognize the value and preference for a cohort
graduation rate. However, although every State has made progress in enhancing its data capacity
(including support and funding from the Department through the Institute of Educational Sciences (IES)
Statewide Longitudinal Data Systems Grants), reasonable flexibility is still necessary for States who are
unable to calculate a cohort graduation rate until necessary data systems are in place. For example, even
if we were to require that States adopt the NGA definition of graduation rate today, we believe States
would need until the 2012-13 school year to fully implement it, given the need to document students who
transfer to another educational program that culminates in the award of a regular high school diploma in
order to distinguish them from dropouts.

Recommendation 1.3: Impress upon States the importance of data quality and the need to provide regular
guidance, training, and monitoring of their local educational agencies. We agree with the importance of
data quality and note that OESE already has augmented efforts to further impress upon States the
importance of data quality. These augmented efforts include revisions to the monitoring indicators (i.e.,
that focus on data quality and holding States accountable for monitoring activities in their LEAs), the
release of Improving Data Quality for Title I Standards, Assessments, and Accountability Reporting:
Guidelines for States, LEAs, and Schools, Non-Regulatory Guidance, April 2006, and the use of more
systematic procedures to check data submitted through the Consolidated State Performance Reports.

                                                      2

                                                                                                 Attachment
Additionally, we have supported the development and enhancement of State data systems through the
Statewide Longitudinal Data System (SLDS) Grant Program, administered by the Department's IES. As
of June 2007, 27 States have received grains totaling nearly S115 million to increase their ability to
efficiently and accurately manage, analyze, and use education data, including individual student records.
The Department has also worked extensively with States to streamline and improve submission of State
and district data, including achievement data through the EDFacts project. One of the specific purposes
of this initiative is to "improve State data capabilities by providing resources and technical assistance."

Recommendation 1.4: Publish a list of data definitions that States will be required to use in NCLB
reports. In addition, the published list should include procedures on how school districts should handle
different classifications of students. (e.g. transfers and returning to home country) and the documentation
requirements to support the classification. We agree that having States use uniform data definitions for
NCLB reporting is a worthwhile goal. However, on this issue, we need to balance the authority the
statute grants to the Department and the flexibility it provides to States. We believe the guidance OESE
issued last year in Improving Data Quality for Title I Standards, Assessments, and Accountability
Reporting: Guidelines for States, LEAs, and Schools, Non-Regulatory Guidance, April 2006 strikes the
right balance of providing direction to States yet permitting them to work through the very complicated
issues of defining and collecting data that yield reliable and valid measures of student and school
performance.

Recommendation 1.5:Develop and implement written policies and procedures to monitor States that are in
the process of developing data collection systems to ensure the systems will collect student data in
accordance with the requirements in NCLB. It is not clear that the Department has the authority to
monitor the "development" of State data systems. OESE does examine whether States have the data
necessary to meet NCLB requirements, and OESE requires corrective actions from States when this is not
the case. The Department also has provided funding through the Statewide Longitudinal Data System
Grants to States who wish to enhance their data systems, which includes the implementation of individual
student records.

Recommendation 1.6: Modify the monitoring procedures to check the accuracy of the data used by the
States and districts to compute graduation rates. OESE has revised its monitoring indicators for reviewing
the procedures States and districts use to ensure data quality. These revisions provide for a more in-depth
examination of the procedures for ensuring data quality in the areas of assessments (e.g., administration
and scoring), data disseminated, and AYP data.


Summary of Issues Identified in Individual State Audit Reports

The report would be strengthened by providing context for State reporting of graduation rate data in 2003-
04. For example, South Dakota, one of the four case study states for this audit, is one such State that is
now able to use an improved data system and has amended the graduation rate section of its
Accountability Workbook since data were collected for this report (see page 28 of the current South
Dakota Accountability Workbook at www.ecl.gov/admins/leal/account/stateplans03/sdcsa.doc).

Finding 1 Approved Graduation Rate Does Not Meet NCLB Requirements

South Dakota, like most States, did not have four years of data available in 2003-04, the initial year of
NCLB implementation and the year audited, to calculate a graduation rate in accordance with the
regulatory definition in §200.19(a)(1)(i)(A). (Although SDDE had collected graduation data in prior
years, it did so for reporting purposes only. It did not use these data for NCLB accountability purposes
because it believed the data were not sufficiently reliable.) As a result, the Secretary used his transition
authority in section 4(c) of NCLB to permit the orderly transition from requirements under the Improving
America's Schools Act to NCLB.


                                                       3
                                                                                                Attachment
In a letter to SDDE dated July 1, 2003, former Under Secretary Eugene Hickok authorized SDDE to
calculate graduation rate on the basis of only 12th grade graduates and dropouts and to continue to do so,
adding a grade each year, until it had four years of data to calculate a cohort graduation rate consistent
with the regulatory definition. The letter also provided approval of South Dakota's plan, consistent with
§200.19 of the Title I regulations, to use a definition of graduation rate that would follow a cohort of
students from entry in ninth grade through graduation in four years. It emphasized that South Dakota's
graduation rate had to include all recipients of any type of certificate or diploma (as well as students who
had dropped out of or transferred into a high school) in the denominator and could include only those
students receiving a regular diploma in the standard number of years in the numerator.

Given that SDDE, like most other States, received permission from the Secretary to deviate from the
statutory and regulatory definition because it did not have sufficient data in the early years of NCLB to
calculate a proper rate, we disagree with the auditors' finding that South Dakota's 2003-04 graduation
rate was non-compliant and that South Dakota's approved graduation rate definition violates NCLB's
requirements.

As noted above, there is no one definition of graduation rate that complies with NCLB. The auditors
favor a definition like the one developed by the NGA, as do we now. We have encouraged States to
consider the use of the NGA definition, and have proposed it as the definition for NCLB reauthorization.
The NGA definition is an acceptable way to compute graduation rate and may, in fact, have some
advantages over the definition used by SDDE. By actually requiring schools to focus specifically on
"transfers in" and "transfers out," it would likely lead to better data quality as students who dropout of
school are sometimes mistakenly coded as transfers. However, South Dakota's definition is consistent
with the current statute and regulations. South Dakota's definition:

        ƒ   Tracks a cohort of students from entry in 9th grade through graduation with a regular
            diploma after 12th grade.
        ƒ   Excludes dropouts as high school completers, as required by §200.19(a)(1)(ii). The dropouts who
            are excluded each year are from the class that began in a given school in 9th grade. That is
            why SDDE needed to phase in its definition over four years; in 2002-2003, SDDE had only
            data for 12th grade completers plus dropouts in that cohort of students.
        ƒ   Takes into consideration the students that the auditors contend are not included.
            -	 South Dakota's definition does not count students who graduate with a GED as
                 graduates; rather, they are included in the definition of "dropout" because a student cannot
                 qualify to take the GED unless the student has already dropped out.
            -	 South Dakota's definition of "dropout" does not include "transfers to another public
                 school district, private school, or state- or district-approved educational program
                 (including correctional or health facility programs); temporary absence due to suspension
                 or school-excused illness; or death."
            -	 South Dakota's definition of "dropout" includes students who have not "completed a
                 state approved educational program." This language includes students who exit high
                 school with only a certificate, rather than a regular diploma, and those who do not
                 graduate in the standard number of years.


Miscellaneous Comments

We wish to emphasize that no one specific definition of graduation rate is required by the current statute
or regulations; rather the regulations allow States some flexibility in defining graduation rate. Therefore,
we suggest that the report acknowledge the flexibilities in the law related to the findings and
recommendations. The report also appears to imply that we should require States to implement
longitudinal student tracking systems (see page 5), though NCLB does not require such systems.
Findings and recommendations that do not take into account the proper legal context, including the
flexibilities allowed by NCLB, are not very helpful.

                                                       4
                                                                                               Attachment
It would be helpful if the methodology section indicated how the four case-study States were selected.

Paragraph spanning pages 3 to 4: It is not clear whether the problem the report is attempting to present is
what was approved in States' accountability workbooks or their implementation of what was approved.

Page 4, Reference to the lack of State data system capacity: This needs to be put in context of the 2003-04
school year. Additionally, the report should acknowledge the improvements that have been put in place
by numerous States to enhance their data systems since the passage of NCLB.

Page 4, second and third full paragraphs, references to data being unreliable: It appears that the concern
is that there was not enough information to document reliability; this does not confirm unreliability,
however.

Page 10, Washington, graduation rates, and last sentence: It's not clear what "these deficiencies" are.

We wish to incorporate by reference all of the comments we have made on the draft audits that are part of
this management audit (i.e., South Dakota, Texas, Oklahoma and Washington) as well as comments made
by Department staff during interviews and conference calls with OIG staff. These additional comments,
particularly those on the audit reports for the four states reviewed, should be applied to this report.

We appreciate your efforts in this important area.




                                                      5