oversight

Status of State Student Assessment Systems and the Quality of Title I School Accountability Data.

Published by the Department of Education, Office of Inspector General on 2002-08-30.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

A Joint Audit Report on

The Status of State Student
Assessment Systems and the Quality
of Title I School Accountability Data
A product of the Texas State Auditor’s Office




                                                      U.S. General Accounting Office




                                                United States Department of Education
                                                           Office of Inspector General




                                                                State Auditor’s Office
                                                                       State of Texas




                                                   Department of the Auditor General
                                                      Commonwealth of Pennsylvania




August 2002                                                       City of Philadelphia
SAO Report No. 02-064                                         Office of the Controller
  Key Points of Report

                             A Joint Audit Report on the Status of State Student
                            Assessment Systems and the Quality of Title I School
                                            Accountability Data
                                                             August 2002

                         Overall Conclusion
                         The majority of states have not complied with the 1994 Title I student assessment
                         requirements of the Elementary and Secondary Education Act (ESEA). As a result,
                         many states may not be positioned well for timely implementation of the new
                         requirements of the 2002 No Child Left Behind Act's ESEA reauthorization. ESEA, which
                         will provide $10.3 billion in fiscal year 2002 to improve the educational achievement of
                         children at risk, allows the U.S. Department of Education (Education) to withhold up to
                         25 percent of administrative funds from states that are not in compliance. At the local
                         level, schools may face significant consequences, such as restructuring, should their
                         accountability data fail to demonstrate adequate yearly progress over certain time
                         frames.

                         Accountability data, which measures compliance with Title I requirements, can be
                         improved at the federal, state, and local levels. Accurate, complete, valid, and timely
                         information is critical to ensure that funding and decisions regarding adequate
                         progress are based on reliable data.

                         State Accountability Systems: Accountability
                         systems are used for developing information to           We compiled state level information
                                                                                  for our joint audit from the following:
                         identify Title I campuses that are not making
                         adequate yearly educational progress. As of              •    A survey of the 50 states, District of
                                                                                       Columbia, and Puerto Rico
                         March 2002, 17 states had complied with the 1994
                         ESEA Title I student assessment requirements;            •    Interviews with Education’s data
                         however, 35 states (including the District of                 processing contractor
                         Columbia and Puerto Rico) had not. Congress              •    Detailed audits of the school
                         gave states until the beginning of the 2000–01                improvement data in three of the
                                                                                       largest states—California,
                         school year to meet those requirements.
                                                                                       Pennsylvania, and Texas
                         Many states are not adequately monitoring critical       References in this report to “many
                         components of their accountability systems,              states” are taken either from the survey
                                                                                  of all states or from interviews with
                         including local education agencies’ (LEA) data
                                                                                  Education’s data processing
                         reporting procedures, to ensure that accountability      contractor. References to only one,
                         data are reliable. LEAs and campuses also need to        two, or three states apply to the
                         make improvements to ensure the quality of the           individual audits we conducted in
                         accountability data they report to their state           California, Texas, and Pennsylvania.
                         education agencies (SEAs).

                         State Title I Reporting Systems: Two of the three large states we audited were reporting
                         to Education inaccurate or incomplete information on Title I schools in need of
                         improvement. These two states lacked adequate procedures and controls to ensure
                         the reporting of reliable information. All three states lacked adequate documentation
                         of procedures and controls for reporting on Title I schools in need of improvement.

                         The Department of Education’s Methods for Ensuring Accountability Data Quality:
                         Education’s accountability data controls can be improved. The deadline for state Title
                         I reporting may be unrealistic given the complexity of gathering and reporting school
                         accountability data. Education needs a systematic process for enforcing the deadline
                         that it sets. Education can also make enhancements to its state monitoring visits and to
                         the single audit requirements to help ensure the quality of state Title I data.


Lawrence F. Alwin, CPA
Key Points of Report

            Key Facts and Findings
            •   State implementation of accountability systems varies widely:
                –   As of March 2002, 35 states had not complied with 1994 Title I assessment and
                    accountability requirements.
                –   Assessment requirements most frequently unmet include requirements to (1)
                    assess all students and (2) break out assessments by subcategories such as
                    gender, race, and disability status.
                –   Because the accountability system of one large state we audited did not yet
                    include all Title I schools in need of improvement, the SEA in that state may
                    have underreported the number of schools in need of improvement.
            •   State officials identified certain factors that affect the implementation of
                assessment and accountability systems:
                –   Factors that further implementation are active participation and support from
                    governors, legislators, and business leaders; development of high-level
                    committees, focus groups, and new state legislation; collaboration and
                    coordination at all levels of public education administration; and state level
                    expertise.
                –   Factors that hinder implementation are inadequate funds and prior
                    investments in systems that predated and conflicted with the 1994
                    requirements.
            •   Many states are not adequately monitoring critical components of their
                accountability systems:
                –   States are not ensuring they receive accurate, complete assessment results
                    from test contractors.
                –   One state was not adequately monitoring student assessment test
                    participation.
                –   When administering assessment tests, many states are not ensuring LEA
                    accommodation of students with limited English proficiency and students with
                    disabilities.
                –   States are not monitoring LEA accountability data quality.
            •   Campuses and LEAs have significant weaknesses in data-reporting procedures and
                quality controls:
                –   Error rates in campus reporting prevent full assurance regarding accountability
                    data.
                –   Errors in manual processes for data from 259 campuses in one LEA were not
                    identified or corrected before data were used to report schools in need of
                    improvement. In addition, this LEA had not maintained supporting
                    documentation for school improvement data it reported to Education, which is
                    a violation of record retention requirements. The LEA also used inappropriate
                    sampling methods to report student assessment results.
            •   Some states are not reporting accurate, complete, or timely information to
                Education:
                –   SEA noncompliance with Title I program and reporting requirements results in
                    misreported or underreported school improvement data:
                    ▪   Two states misinterpreted Title I reporting requirements.
                    ▪   One state’s accountability system did not apply to all Title I campuses.
Key Points of Report
                    ▪   Two states failed to develop and maintain supporting documentation as
                        required by federal regulation.
                    ▪   Two states failed to report significant data quality issues as required by
                        Education.
                    ▪   Many states failed to meet Education’s December 1 reporting deadline.
                –   States lack systematic procedures and controls for developing and reporting
                    reliable accountability information, including school improvement data:
                    ▪   Two states lacked systematic procedures and controls to ensure data
                        reliability.
                    ▪   All three states we audited lacked adequate documentation for their data
                        procedures and controls.
            •   Improvements are needed in data quality controls at Education:
                –   Education needs to monitor and address the problem of late state Title I data
                    submissions.
                –   Education returned 40 state submissions for 1998–99 because of anomalies in
                    the data.
                –   Education published the 2000 and 2001 Title I reports 22 months after the state
                    data submission deadline.
                –   Education was not using its state monitoring visits to assess data quality.
                –   Education has not distributed its current data quality standards to states.
                –   The OMB Compliance Supplement does not require reviews of states’
                    accountability data.
Table of Contents

            Executive Summary ..................................................................................................... 1


            Introduction: The Joint Audit of Title I Accountability Data
            Identified Improvements to Be Made at Federal, State,
            and Local Levels ......................................................................................................... 15

            Section 1:
            State Implementation of Accountability Systems Varies
            Widely ............................................................................................................................... 15

            Section 2:
            State Actions to Ensure Accurate Assessment Test
            Scoring and Student Accommodations (Where
            Applicable) Are Limited .......................................................................................... 19

            Section 3:
            Campuses and LEAs Have Significant Weaknesses in
            Data Reporting and Quality Control Procedures ........................................ 22

            Section 4:
            Some States Are Not Reporting Accurate, Complete, or
            Timely Information to Education .......................................................................... 25

            Section 5:
            Improvements Can Be Made in Data Quality Controls at
            Education........................................................................................................................ 33

            Section 6:
            There Are Additional Data Collection Options for
            Education to Consider .............................................................................................. 36


            Appendices
                       1 - Objective, Scope, and Methodology.......................................................... 38

                       2 - Joint Audit Reports and Contact Information ............................................ 40

                       3 - Accountability and Assessment Requirements Under the
                          1994 and 2001 Reauthorizations of Title I ...................................................... 42
This page intentionally left blank.
Executive Summary
                        O    ur joint audits found variation in
                             states’ implementation of student
                        assessment and school accountability
                                                                          requirements. The 2001 No Child Left
                                                                          Behind Act’s reauthorization of ESEA has
                                                                          deadlines that vary according to the specific
                        systems. We also found variation in               requirements in the reauthorization (see
                        states’ methods for ensuring the quality of       Appendix 3).
                        the accountability data they use to report
                        on students’ and schools’ performance             Because the majority of states have not met
                        and on schools in need of improvement.            the requirements of the 1994 law, it appears
                        (See Appendix 2 for a list of the six audit       that many states may not be positioned well
                        reports and contact information.)                 to meet the schedule for implementing the
                                                                          new requirements of the 2001
                        States report accountability information to       reauthorization. Education may withhold 25
                        students, parents, school districts, state        percent of administrative funds if a state does
                        officials, and the U.S. Department of             not comply with these requirements.
                        Education (Education).
                                                                          The two assessment requirements that states
                        We compiled state level information for           most frequently have not met are (1) the
                        our joint audit from interviews with state        requirement to assess all schools and students
                        Title I officials, Education’s data               and (2) the requirement to break out
                        processing contractor, and 50 responses to        assessment results by subcategories such as
                        a detailed survey conducted across all            gender, race, and disability status. One of the
                        states, the District of Columbia, and             three states we audited in detail did not yet
                        Puerto Rico. We also audited school               have a school accountability system in place
                        improvement data in three large states—           for all schools. Because of this, the state
                        California, Pennsylvania, and Texas—in            education agency (SEA) in this state may
                        greater detail. In addition, we reviewed          have underreported the number of Title I
                        Education’s controls over data quality.           schools in need of improvement.

                        State Implementation of                           Certain Factors Affect the
                        Accountability Systems Varies                     Implementation of Accountability
                        Widely                                            Systems

                            As of March 2002, 17 states were in           States we surveyed and state officials we
                            compliance with the 1994 Title I              interviewed indicate that the following
                            assessment requirements of the                factors are key to the successful
                            Elementary and Secondary Education Act        implementation of compliant assessment and
                            (ESEA). These requirements were               accountability systems:
                                           intended to provide the
                                           basis for the development      •    Active participation and support from
                 What Is Title I?
                                           of accountability systems.          governors, legislators, and business
     Title I of the ESEA (reauthorized
     by the No Child Left Behind Act)
                                           Thirty-five states                  leaders to make compliance with
     is the largest source of federal      (including the District of          assessment and accountability system
     funding for public education          Columbia and Puerto                 requirements a high priority
     and will provide states with $10.3    Rico) were not in
     billion in fiscal year 2002 to
                                           compliance.                    •    Development of high-level committees;
     improve the educational
     achievement of children at risk.
                                                                               focus groups; and new, enabling state
     Title I serves about 12.5 million   Congress gave states until            legislation
     children in all 50 states, the      the beginning of the
     District of Columbia, and Puerto
     Rico.
                                         2000–01 school year to
                                         fully implement those



                            A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002              SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 1
Executive Summary
            •   The ability of agencies and different         inaccurate identification of schools in need of
                levels of public education                    improvement.
                administration to collaborate and
                coordinate their efforts to meet              One of the three states we audited in detail
                requirements                                  reported to Education inaccurate information
                                                              on adequate yearly progress and Title I
            •   The availability of state level
                                                              schools in need of improvement. This
                expertise that can facilitate the
                                                              happened because the state did not detect
                implementation of an accountability
                                                              errors in its accountability database and in
                system
                                                              the test contractor’s compilation of student
                                                              assessment results before it submitted its
            States cite inadequate funding as the
                                                              report.
            primary barrier hindering the
            implementation of compliant assessment
                                                              Another of the three states we audited in
            and accountability systems. Some states
                                                              detail was not adequately monitoring test
            reported investments in assessment
                                                              participation by all eligible and qualified
            systems that predated and conflicted with
                                                              students. This left unaddressed the risk that
            the 1994 Title I reauthorization. Building
                                                              not all students were receiving the tests they
            support and obtaining funding for a new
                                                              were eligible and qualified to take. The
            system can take several years. In
                                                              inadequate monitoring also created the
            addition, the SEA in one small state
                                                              possibility that school accountability ratings
            reported that its small staff lacked the
                                                              were not being accurately calculated and that
            technical expertise to develop a new
                                                              schools in need of improvement were not
            system.
                                                              being accurately reported to Education.
            Many States Are Not                               Test Exemptions and Accommodations.
            Adequately Monitoring Critical                    Most states report that they have established
            Components of Their                               standards for ensuring that (1) any
            Accountability Systems to                         exemptions for students with limited English
            Ensure Reliable Data                              proficiency are justified and (2) students with
                                                              disabilities are accommodated when these
            Test Scoring. Most states are taking              students take assessment tests. However,
            some action to ensure that Title I student        states’ efforts to ensure adherence to these
            assessment tests are scored accurately.           standards are limited.
            However, these measures do not always
            provide adequate assurance of accurate            Some states reported that their primary
            scoring.                                          means of verifying the existence of
                                                              appropriate accommodations was to compare
            Most states (44) hire contractors to score        the number of students who receive
            Title I assessment tests, and about one-          accommodation in the current year with the
            third of these states (16) report that they       number from the previous year. Because of
            monitor the scoring the contractors               high student mobility and students’ changing
            perform. Some states have found scoring           status with regard to English proficiency, this
            errors and, in some cases, these errors           comparison has limitations. It addresses only
            have had serious negative consequences            the presence of accommodations rather than
            for schools and students. In one state,           their appropriateness.
            contractor-scoring errors resulted in
            children’s unnecessarily being required to        State Monitoring of LEA Procedures to
            attend summer school. In another state,           Ensure Data Quality. Many states were not
            contractor-scoring errors resulted in the         monitoring or otherwise supervising their
                                                              local education agencies’ (LEAs) procedures


                A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 2       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA           AUGUST 2002
Executive Summary
                 and controls to ensure the quality of the        Some States Are Not Reporting to
                 accountability data they report. States use      Education Accurate, Complete, or
                 LEA data to identify and report schools in
                                                                  Timely Information on Schools in
                 need of improvement to Education.
                                                                  Need of Improvement
                 Campuses and LEAs Have
                                                                  Noncompliance and Weaknesses in
                 Significant Weaknesses in Data                   Procedures and Controls for Ensuring
                 Reporting Procedures and                         Reliability of Title I Data. Regardless of an
                 Quality Controls                                 SEA’s procedures and controls to ensure the
                                                                  quality of the accountability data it collects,
                 Individual campuses and LEAs vary                processes, and reports, its state accountability
                 significantly with regard to the procedures      information is unreliable at least to the extent
                 and controls they use to ensure the quality      that the LEAs’ data are unreliable. One of
                 of the accountability data they report to        the three states we audited in detail has had a
                 their SEAs. In one state, the campuses           strong, fully automated accountability system
                 and LEAs had numerous controls to help           in place since 1991. This state has an
                 ensure the quality of accountability data.       automated process for reporting reliable
                 However, statistical error rates in their        information on schools in need of
                 documentation and reporting of                   improvement to Education. However, error
                 accountability data were sufficiently high       rates in reporting Title I data at campuses and
                 to prevent full assurance regarding the          LEAs prevented us from providing full
                 quality of that state’s accountability           assurance regarding that state’s
                 information.                                     accountability information.

                 In another state, the largest urban LEA          The other two states we audited in detail
                 relied on manual processes to gather and         underreported or misreported information on
                 report accountability data. This resulted        schools in need of improvement, or the
                 in errors that were not identified or            accuracy of their reported data could not be
                 corrected before the data were used to           determined. This occurred because of the
                 determine school accountability ratings          following conditions:
                 and to identify schools in need of
                 improvement.                                     •    Accountability system that did not
                                                                       include all Title I campuses
                 Additionally, this LEA and its campuses
                 were not complying with the record               •    Misinterpretation of the requirement to
                 retention requirements of the Title I                 report all schools in improvement status,
                 program. Therefore, data the LEA                      regardless of when they are identified for
                 reported to the SEA and to Education for              improvement
                 the 1999–2000 school year were not               •    Failure to develop and retain supporting
                 supported by source documentation and                 documentation as required by federal
                 could not be verified.                                regulations
                                                                  •    Failure to report data quality issues (such
                 This LEA also allowed some of its                     as LEAs’ use of sampling to report
                 campuses to determine student assessment              accountability data) in the designated
                 results by inappropriate and inconsistent             section in the report to Education
                 sampling methods, resulting in unreliable
                 accountability data for those campuses.          •    Accountability ratings based on
                                                                       undetected errors in the accountability
                                                                       database and the compilation of




                    A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 3
Executive Summary
                assessment results from the test              Education’s data processing contractor for
                contractor                                    state Title I data reported that it had to return
            •   Inadequacies in or the absence of             the data submissions of 40 states for the
                systematic procedures and controls            1998–99 school year for correction and
                for developing and reporting reliable         revision. This further hindered Education’s
                data to Education                             ability to submit timely reports to Congress
                                                              and states on Title I, Part A program
            All three states we audited in detail lacked      performance.
            adequate documentation of their
            procedures and quality controls for               Improvements Are Needed in
            developing and reporting to Education             Data Quality Controls at Education
            their Title I performance data and school
            improvement information. The absence              Education did not publish reports for the
            of such documentation creates a risk of           1997–98 and 1998–99 school years until 22
            inconsistency and interruption in                 months after accountability data was due
            reporting in the event of an unexpected           from the states. This delay was caused by
            interruption in business processes or staff       states’ late data submissions and poor data
            changes, absences, or departures.                 quality, as well as by the absence of adequate
                                                              procedures at Education to ensure timely
            Delayed Submission of State Title I               receipt, review, and publication of
            Data. Although one state had a strong,            accountability data.
            fully automated accountability system,
            this state’s requirements and time lines for      Examples of steps Education can take to
            processing LEA data did not match well            improve the quality of accountability data
            with Education’s reporting time lines.            include the following:
            Because of this, the state reported to
            Education student assessment data from
            the report year but dropout and attendance        •    Education should establish a systematic
            data from the previous year. Even using                process to enforce the deadline for states
            that data, the state could not meet                    to submit accountability data. For the
            Education’s December 1 deadline to                     most recent reporting year, Education did
            submit data because of the time needed to              not initiate actions on late submissions of
            properly aggregate, disaggregate, and                  state data until three months after the
            verify final accountability data.                      December 1 due date. Education took no
                                                                   action against states that repeatedly
            Each of the other two states we audited in             submitted data late or states that were
            detail submitted its Consolidated State                non-responsive.
            Performance Report to Education after             •    Education should assess the quality of
            the deadline. Our audit determined that                accountability data during its state
            only four states met the deadline for                  monitoring visits. Education relies on a
            submission to Education for the 1998–99                contractor to perform edit checks of
            school year. Education’s State ESEA                    accountability data, but this control alone
            Title I Participation Information for                  is not enough to ensure the quality of
            1998–99: Final Summary Report (2001)                   accountability data.
            reports that one-fifth of the states              •    Education should expand the
            submitted reports as late as 18 months                 requirements in the OMB Compliance
            after the deadline.                                    Supplement to include a review of
                                                                   accountability data controls during single
                                                                   audits. Currently, Education does not
                                                                   require states to formally attest to the



                A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 4       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
Executive Summary
                     validity and reliability of their                  currently the method used for assessing
                     accountability data.                               student performance through the
                                                                        National Assessment of Educational
                 There Are Additional Data                              Progress (NAEP), a 30-year longitudinal
                 Collection Options for                                 study that uses a small statistical sample
                                                                        from each state. The new law requires
                 Education to Consider                                  that students in grades 4 and 8 take the
                                                                        NAEP exams in reading and math every
                 This audit determined that most states
                                                                        other year beginning in 2002–03 (subject
                 submit Title I data late, and these
                                                                        to federal financing).
                 submissions require revisions because of
                 errors in the data. Additionally, because
                 states follow their own criteria for              Summary of Objective, Scope,
                 evaluating school accountability and              and Methodology
                 identifying schools not making adequate
                 yearly progress, aggregation of these data        The overall objective of this audit was to
                 at the national level is not meaningful.          determine whether the accountability data
                                                                   states use to report on Title I, Part A program
                 Some of the state Title I officials we            performance are accurate, complete, valid,
                 interviewed suggested alternatives to the         and timely. In particular, we examined the
                 present reporting requirements of                 data used to identify and report on schools in
                 Education’s Consolidated State                    need of improvement.
                 Performance Report. These alternatives
                 would allow the collection of reliable            We conducted the audits resulting in this
                 information for evaluating Title I in a           joint report as part of a project of the U.S.
                 timely manner:                                    Comptroller General’s Domestic Working
                                                                   Group. Our goal was to provide
                 •   Education could consider developing           recommendations that Education, SEAs, and
                     a more realistic deadline for the             LEAs could use to improve the quality of
                     Consolidated State Performance                accountability data used to evaluate the
                     Report on Title I programs. The               effectiveness of Title I, Part A funds in
                     deadline could take into account the          improving the performance of at-risk
                     time states need to process and verify        students.
                     their Title I data. The deadline should
                     be one that Education can monitor             The following were participants in this joint
                     and enforce.                                  effort:

                 •   Education could evaluate how
                                                                   •    The U.S. General Accounting Office
                     individual states are and are not
                     meeting the objectives of the federal         •    The U.S. Department of Education’s
                     law, instead of attempting to                      Office of Inspector General
                     aggregate state results to a national         •    The Texas State Auditor’s Office
                     summary level. This would allow
                     Congress to identify successes and            •    The Pennsylvania Department of the
                     weaknesses in the program as enabled               Auditor General
                     by statute and implemented by states.         •    The Office of the City Controller,
                 •   To evaluate program performance                    Philadelphia
                     nationwide, Education could use
                     sampling (stratified if necessary) as
                     an alternative to gathering all program
                     information from all states. This is



                     A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 5
Executive Summary
                        Summary of Findings, Recommendations, and Management Responses

                   Finding                              Recommendation                                Response

                                   Implementation of State School Accountability Systems

    Based on the status of states’            No recommendation made.
    compliance with the requirements
    of the 1994 ESEA reauthorization,
    many states may not be well
    positioned to meet the additional
    requirements in the 2001 No Child
    Left Behind legislation. Thirty-five
    states (including the District of
    Columbia and Puerto Rico) have
    not met the 1994 assessment and
    accountability requirements.

    One state’s accountability system         Education’s assistant secretary for         The SEA stated that its
    did not include 11 percent of the         Elementary and Secondary                    alternative accountability
    state’s Title I campuses in the 1999–     Education should:                           system for alternative and very
    2000 school year. (The campuses                                                       small schools will be fully
                                              •   Ensure that the SEA fully
    excluded were alternative                                                             implemented by fall 2003.
                                                  implements its alternative
    campuses and very small                                                               Management controls will be
                                                  accountability system and has in
    campuses.) As a result, the state’s                                                   comprehensive and the data
                                                  place appropriate
    SEA may have underreported to                                                         from the system will be reliable
                                                  management controls over the
    Education the number of Title I                                                       and valid. The SEA intends to
                                                  reliability, validity, and timeliness
    schools in need of improvement.                                                       review all schools receiving Title I
                                                  of performance data from that
                                                  system.                                 funds in fall 2002 and to perform
                                                                                          subsequent annual reviews to
                                              •   Ensure that the SEA includes all        identify schools in need of
                                                  Title I schools in its review to        program improvement.
                                                  identify schools in need of
                                                  improvement.


                                  State Oversight and Monitoring of Accountability Systems

    Despite the enhanced emphasis             To enhance confidence in state              Management generally
    on assessment results, states still       assessment results, Education should        concurs.
    appear to be struggling with              include in its compliance reviews a
                                                                                          Education will include in its
    ensuring that assessment data are         check on the controls states have in
                                                                                          monitoring activities a review of
    complete and correct.                     place to ensure proper test scoring
                                                                                          a state’s compliance in
                                              and the effective implementation of
                                                                                          monitoring the technical quality
                                              these controls by states.
                                                                                          of products delivered by its
                                                                                          contractor and its procedures to
                                                                                          ensure the accuracy of
                                                                                          assessment data prior to
                                                                                          dissemination.

    One state was not adequately              The SEA should implement needed             Management agrees and will
    monitoring student assessment             controls to evaluate and help ensure        develop methods to help ensure
    participation to ensure full              full participation of eligible, qualified   full participation in student
    participation by all eligible,            students in required tests.                 assessment testing as required
    qualified students as required by                                                     by law.
    state law.




                             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 6                    SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 AUGUST 2002
Executive Summary
                         Summary of Findings, Recommendations, and Management Responses

                   Finding                           Recommendation                             Response

    There
    SEA
    reviewing,
    accuracy
    assessment
    scoring
         was
          was
            results.
             properly
               of
               no
               ortest
                  the
                   verifying
                   evidence
                      contractor’s
                       state’s
                        supervising,
                             the
                               student
                               that one     The SEA should:                          Management generally
                                                                                     concurs.
                                            •   Enhance procedures for
                                                reviewing reported test scores to    The SEA agrees that the data
                                                detect errors in the test            should be reviewed more
                                                contractor’s data.                   accurately; however, a detailed
                                            •   Document the procedures and          review was not possible due to
                                                maintain the documentation.          the time constraints of reporting
                                                                                     deadlines. Documented
    Reviews of SEAs’ monitoring of LEAs                                              procedures for these processes
    found that some SEAs were not                                                    have not been established but
    sufficiently monitoring LEAs’                                                    will be with the implementation
    procedures and controls or                                                       of the No Child Left Behind Act.
    otherwise exercising adequate
    oversight to ensure the quality of      Two audits included                      The responses below correspond
    the accountability data the LEAs        recommendations to SEAs.                 to each of the two audits:
    report to the states.
                                            In one audit, the recommendations
                                            included the following:
                                            •   Implement monitoring or              Management disagrees.
                                                auditing procedures at LEAs to       Education’s data processing
                                                review the LEAs’ controls over       contractor reviews and
                                                and the accuracy and                 approves the data submitted to
                                                completeness of Title I data they    Education. The data submitted
                                                submit to the SEA in order to        to Education is final. Auditor’s
                                                ensure that reliance on that         Comment: The SEA should
                                                data is justified for reporting      discuss with Education staff
                                                schools in need of improvement       methods for correcting
                                                to Education.                        information submitted that is
                                                                                     later found to be incorrect.
                                            •   Identify LEA data quality issues     Management does not
                                                and report them in the               specifically address this
                                                designated section of the            recommendation in its response.
                                                Consolidated State Performance
                                                Report.


                                            In the second audit, the                 Management concurs with both
                                            recommendations included the             recommendations.
                                            following:                               Management describes plans,
                                            •   Maximize the value of audits of      provides time lines, and names
                                                LEAs and campuses for data           parties responsible for ensuring
                                                quality problems.                    the implementation of needed
                                                                                     improvements.
                                            •   The SEA and LEAs should
                                                communicate and collaborate
                                                to focus training on specific data
                                                quality problems identified by
                                                this audit, by monitoring, and by
                                                subsequent audits.




                           A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002             SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 7
Executive Summary
                          Summary of Findings, Recommendations, and Management Responses

                   Finding                              Recommendation                             Response

                                        LEA and Campus Accountability Data Quality
    Statistical error rates in campus and      SEA and LEAs should collaborate in       Management concurs.
    LEA documentation
    reporting
    were   sufficiently
                of accountability
                        highand
                             to prevent
                                 in data
                                    the full   developing measures to ensure the
                                                                                        Management describes plans,
    assurance regarding the quality of         continuing improvement of
                                                                                        provides time lines, and names
    one large state’s accountability           accountability data quality.
                                                                                        parties responsible for ensuring
    information.
                                               LEAs should increase supervision over    the implementation of needed
                                               data-gathering and reporting             improvements.
    In one of the states we audited in         processes to improve data reliability.
    detail, most LEAs submitted
    accountability data to the SEA             No recommendation made.
    electronically, using a Web-based
    application that identified all fatal
    errors for correction before final
    submission.



    One large urban LEA had                    LEAs should:                             The SEA responded as follows:
    numerous and significant errors in
                                               •   Automate the data-gathering          •   For 2002, all data will be
    its school accountability
                                                   and reporting process to the             processed electronically.
    information. This LEA and its 259
                                                   extent possible.                     •   Documentation was in
    campuses relied on manual
    procedures for gathering and               •   Develop and implement written            place in 2001 and 2002.
    recording accountability data. The             procedures for developing and        •   For 2002, all data will be
    absence of written procedures and              reporting school accountability          electronically collected and
    quality controls for developing the            data.                                    shared. There will be no
    data, and the absence of                   •   Implement data quality                   need for individual teachers
    procedures such as reviews to                  assurance standards that                 or schools to report data.
    ensure data quality, allowed errors            include a systematic review by a
    in the identification of schools in            second party, a review of data
    need of improvement to remain                  for reasonableness, and the
    undetected.                                    identification and correction of
                                                   data anomalies due to errors.

    The same LEA discussed above               LEAs should seek approval for any        The LEA will seek approval from
    allowed some of its campuses to            methods used for reporting               the SEA if necessary. The LEA
    report student assessment results          assessment results based on less than    does not intentionally send less
    using inappropriate and                    the entire data population.              than complete population data.
    inconsistent sampling methods,
                                               The SEA and LEAs should identify and
    resulting in unreliable performance
                                               report data quality issues in the
    data for those campuses.
                                               designated section of the
                                               Consolidated State Performance
                                               Report.




                             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 8                    SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA              AUGUST 2002
Executive Summary
                         Summary of Findings, Recommendations, and Management Responses

                   Finding                             Recommendation                             Response

    The same LEA discussed above             The LEA should better safeguard and     The LEA has distributed to all
    was not in compliance with the           retain financial and programmatic       schools and area academic
    federal reporting requirements for       information by:                         offices a copy of the LEA’s
    record retention. It had not                                                     records retention policy.
                                             •   Implementing strategies to
    maintained source documentation
                                                 prevent, mitigate, or recover
    for student performance or for
                                                 from identified physical risks
    schools identified as needing
    improvement. Additionally,               •   Monitoring and enforcing
    summary-level records to calculate           compliance with the record-
    annual school performance had                retention requirements of the
    been destroyed in storage.                   Title I program
    Therefore, validating the school
    improvement data the LEA
    reported to the SEA was not
    possible.


                             State Title I Reporting Systems for Schools In Need of Improvement

    One state has a fully automated          The SEA is commended for its school
    school accountability system and         accountability system.
    an automated process for
                                             The SEA and LEAs should work
    reporting schools in need of
                                             together to identify and address
    improvement to Education. It has
                                             data quality issues at LEAs.
    numerous and comprehensive
    procedures and controls to help
    ensure the reliability of school
    accountability data. However,
    campus and LEA errors continue to
    compromise the quality of the
    state’s accountability data.

    Because of a misinterpretation of        The SEAs should report all Title I      In one state, management
    the requirement, two states              schools identified as needing           concurs. The SEA will report all
    audited in detail did not report the     improvement, including both newly       Title I schools identified for
    total of all schools currently in need   and previously identified schools, in   improvement and will maintain
    of improvement. Instead they             their Consolidated State                all supporting documentation
    reported only those schools              Performance Report.                     for three years.
    identified as in need of
                                                                                     Another state did not concur
    improvement during the report
                                                                                     and stated that no one with
    year. One state underreported this
                                                                                     Education or Education’s data
    number by 40 percent in the
                                                                                     processing contractor had
    1999—2000 school year.
                                                                                     pointed this out in the past. This
                                                                                     state asserted that it relied on
                                                                                     this review process. Auditors
                                                                                     recommended that this state
                                                                                     consult with Education on this
                                                                                     matter.




                           A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002             SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                   PAGE 9
Executive Summary
                         Summary of Findings, Recommendations, and Management Responses

                   Finding                             Recommendation                             Response

    One state may have                        See above recommendation and
    underreported to Education the            the response under Implementation
    Title I schools in need of                of State School Accountability
    improvement because it had not            Systems.
    yet included alternative and very
    small campuses. These campuses
    comprised 11 percent of the
    state’s Title I campuses in the
    1999–2000 school year.

    Two states had not complied with          The SEAs should:                        One SEA responded that it will
    federal record retention                                                          maintain all supporting
                                              •   Retain documentation to
    requirements that they retain                                                     documentation for three years.
                                                  support development of the
    source documentation to support
                                                  reported data.                      The other SEA stated that a
    the data they reported to
                                              •   Ensure that LEAs comply with the    copy of the records retention
    Education on schools in need of
                                                  record retention requirements of    policy has been distributed to all
    improvement.
                                                  the Title I program.                schools and to area academic
                                                                                      offices.

    All three states had accountability       SEAs should strengthen data             SEAs did not specifically respond
    data quality issues, but the SEAs in      verification procedures and             to this summary
    these states did not identify or          communications with LEAs and            recommendation.
    report these issues in the                should detect and evaluate data
    designated section of the                 quality issues at both the state and
    Consolidated State Performance            local levels. SEAs should disclose
    Report. This violates a data quality      data quality issues in the annual
    requirement for full disclosure of all    report to Education.
    issues that could affect the quality
    of the Title I data being reported.

    Many states are not able to finalize      See the first recommendation under      See the first response under U.S.
    school accountability data in time        U.S. Department of Education            Department of Education
    to meet the December 1 deadline           Quality Controls for State              Quality Controls for State
    for submission of the Consolidated        Accountability Data below.              Accountability Data below.
    State Performance Report to
    Education.

    One state reported to Education           The SEA should ensure that test data    Management generally concurs
    incorrect data on adequate yearly         from the test contractor is properly    and is revising its written
    progress and Title I schools in need      and timely reviewed by both the SEA     procedures regarding data
    of improvement because of                 and LEAs to ensure completeness         verification. Time constraints
    undetected errors in its                  and accuracy before it is compiled      were a major factor in the data
    accountability database and in            for Table C-1 of the Consolidated       submission during the year
    the test contractor’s compilation of      State Performance Report.               covered by this audit.
    student assessment results. When
    the errors were corrected, the state
    did not correct its Title I report to
    Education.




                             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 10                   SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA            AUGUST 2002
Executive Summary
                        Summary of Findings, Recommendations, and Management Responses

                  Finding                            Recommendation                             Response

    SEAs lack systematic procedures        SEAs should develop systematic           One state reported that, since
    for identifying schools in need of     procedures for developing school         the audit, it has completed and
    improvement or controls, including     improvement information and              implemented written
    analytical and supervisory review,     adequate controls to ensure the          procedures for developing and
    to ensure reliable school              reliability of the data. Controls        reviewing the data on Title I
    improvement data. As a result,         should include a supervisory, an         schools in need of improvement.
    some states are reporting              analytical, and a reasonableness
    inaccurate or incomplete school        review of the data against sources
    improvement information to             and across years to identify errors
    Education.                             and anomalies.
    One state underreported the total
    number of schools in need of
    improvement from the 1996–97
    school year through the 1999–2000
    school year by a percentage
    ranging from 40 percent to 86
    percent. This occurred because
    the SEA did not have sufficient
    management controls to ensure
    that the reported data were
    reliable and valid.

    All three states audited in detail     The SEAs should develop and              Management concurs.
    lacked adequate documentation          maintain current documentation of
                                                                                    SEAs plan to develop and
    of procedures and controls for (1)     all manual and automated
                                                                                    maintain documentation of
    developing the data on schools in      procedures and controls for
                                                                                    procedures and controls for
    need of improvement, (2) reporting     preparing and ensuring the quality of
                                                                                    preparing the Consolidated
    data to Education in the               the data submitted in the
                                                                                    State Performance Report. One
    Consolidated State Performance         Consolidated State Performance
                                                                                    SEA reported that it completed
    Report, and (3) ensuring that they     Report on schools in need of
                                                                                    and implemented written
    were reporting reliable information.   improvement.
                                                                                    procedures and controls in
                                                                                    February 2002.

                      U.S. Department of Education Quality Controls for State Accountability Data

    Management controls over the           Education’s assistant secretary for      Management generally
    timely publication of school           Elementary and Secondary                 concurs.
    improvement data need to be            Education and the undersecretary
                                                                                    The offices of the
    strengthened. Approximately 40         should strengthen management
                                                                                    Undersecretary and of the
    states were asked to revise their      controls to ensure the timely receipt,
                                                                                    Elementary and Secondary
    initial 1998–99 data submission        review, and publication of
                                                                                    Education are working on
    because of conditions identified       performance data concerning
                                                                                    developing more efficient
    during an edit. Education              schools identified for improvement.
                                                                                    follow-up procedures.
    published the 2000 (1997–98 data)
                                                                                    Education plans to provide
    and 2001 (1998–99 data) reports
                                                                                    technical assistance to states in
    approximately 22 months after the
                                                                                    implementing new reporting
    data were due from the states.
                                                                                    requirements of the No Child
                                                                                    Left Behind Act.




                         A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002           SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                   PAGE 11
Executive Summary
                       Summary of Findings, Recommendations, and Management Responses

                  Finding                             Recommendation                             Response

    Education needs to strengthen its        Education’s assistant secretary for     Management generally
    process for ensuring that Title I        elementary and secondary                concurs.
    school improvement data are              education and the undersecretary
                                                                                     The Office of Elementary and
    reliable and valid.                      should:
                                                                                     Secondary Education (OESE)
                                             •   Develop and implement written       plans to develop written
                                                 procedures to assess, during        procedures to assess the
                                                 monitoring visits to SEAs and       reliability and validity of school
                                                 LEAs, whether school                improvement data during
                                                 improvement data are reliable       monitoring visits to states.
                                                 and valid.                          Furthermore, OESE hopes to
                                             •   Distribute Education’s Data         improve the quality of federal
                                                 Quality Standards to SEAs and       data, as well as minimize the
                                                 encourage them to provide the       burden on states, by
                                                 standards to LEAs for their use.    implementing a performance-
                                                                                     based data management
                                             •   Include audit procedures in the
                                                                                     initiative.
                                                 OMB Compliance Supplement
                                                 to review controls over Title I,
                                                 Part A school improvement data
                                                 at LEAs and SEAs during annual
                                                 single audits.




                            A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   PAGE 12                  SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
Executive Summary

                                          Summary of Additional Observations

   Factors that further success in implementing state assessment and accountability systems:
   •   Active involvement and support by state leaders
   •   Coordination among the levels of public education administration
   •   Depth of assessment expertise at the state level

   Factors that hinder success in implementing state assessment and accountability systems:
   •   Inadequate funding
   •   Prior investments in systems that predated and conflicted with the 1994 requirements

   Areas of greatest noncompliance with assessment requirements:
   •   The requirement to assess all students and schools
   •   The requirement to break out assessment results by subcategories such as gender, race, and disability status

   Limited state measures to ensure appropriate testing exemptions and accommodations:
   •   Most states report that they have developed standards for LEAs to follow to justify exemptions of students
       with limited English proficiency and to ensure appropriate accommodations for students with disabilities.
       However, states reported few actions that would ensure that their guidelines were followed. SEA on-site
       monitoring visits could be used to assess the appropriateness of LEA policy and practice with regard to
       testing accommodations. However, in a recent review, the U.S. General Accounting Office found that
       states varied dramatically on how often they conducted on-site visits. The average time between visits
       ranged from 2 years or fewer (in 6 states) to more than 7 years (in 17 states).

   Failure to describe data quality issues in designated section of Consolidated State Performance Report to
   Education:
   •   Two states did not report significant data quality issues regarding schools needing improvement in the
       designated section of the Consolidated State Performance Report.

   Failure to report states that have changed assessment systems from one year to the next:
   •   In Education’s State ESEA Title I Participation Information: Final Summary Report for each school year,
       Education’s Performance and Evaluation Services Division has not identified states that have changed their
       assessment systems from year to year. Without such disclosure, year-to-year comparisons can mislead the
       reader or cause the reader to question the reliability of the data.

   Failure to report or the misreporting of Title I LEAs in need of improvement:
   •   One large state did not report the number of Title I LEAs identified for improvement in its performance report
       for the 1996–97 and 1999–2000 Consolidated State Performance Reports. For both the 1997–98 and 1998–99
       school years, this state reported the same numbers for the total number of Title I LEAs and the number of Title
       I LEAs in need of improvement.




                          A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
   AUGUST 2002            SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 13
                         This page intentionally left blank.




          A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 14   SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA    AUGUST 2002
Introduction: The Joint Audit of Title I Accountability Data Identified
Improvements to Be Made at Federal, State, and Local Levels

              This report summarizes the work of five public auditing offices that collaborated to
              audit the quality of Title I accountability data gathered and reported at three
              governmental levels: local education agencies (LEAs), state education agencies
              (SEAs), and the U.S. Department of Education (Education). The following were the
              participants in the audit:

              •        U.S. General Accounting Office
              •        U.S. Department of Education’s Office of Inspector General
              •        Texas State Auditor’s Office
              •        Pennsylvania Department of the Auditor General
              •        Office of the City Controller, Philadelphia

              We conducted audits of Title I accountability data quality, specifically focusing on the
              reliability, validity, and timeliness of information on schools in need of improvement
              reported to and used by Education in reports to Congress. We surveyed all states;
              interviewed state Title I officials; and conducted detailed audits in California,
              Pennsylvania, Texas, and the Philadelphia School District. We also reviewed
              Education’s controls over data quality. Our audit results are intended to provide
              useful information to all LEAs, SEAs, and the federal government about improving
              the quality of Title I accountability data nationwide. (See Appendix 2 for a list of the
              six audit reports and contact information.)


Section 1:
State Implementation of Accountability Systems Varies Widely

              Our joint audits found variation among states’:
                                                                                      Title I, Part A
              •        Implementation of school accountability         The Title I, Part A program was enacted
                                                                       under the Elementary and Secondary
                       systems.
                                                                       Education Act (ESEA) and was amended
                                                                       by the Improving America’s Schools Act of
              •        Methods for ensuring the quality of the
                                                                       1994.
                       accountability data they use to report on
                                                                       The 1994 reauthorization of ESEA required
                       students’ and schools’ performance and on       each state to develop an assessment
                       schools in need of improvement.                 system for all students and all schools to
                                                                       identify schools that were not meeting
                                                                       state standards for adequate yearly
              Accountability information measures school               progress in student performance.
              performance based on state-established criteria,      The 2001 reauthorization of ESEA
              which include standards for student performance on    augments the assessment and
              statewide student assessment tests. It may also       accountability requirements that states
              include additional indicators, such as school         must implement and increases the stakes
                                                                    for schools that fail to make adequate
              attendance and dropout rates. States use              yearly progress.
              accountability information to determine schools
              making adequate yearly progress and to identify
              schools in need of improvement. States report accountability information to students,


                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                     PAGE 15
          parents, school districts, state officials, and the general public. They also report
          accountability information on Title I schools in their annual Consolidated State
          Performance Report to Education.

          Title I of the Elementary and Secondary Education Act (ESEA), which was
          reauthorized by the 2001 No Child Left Behind Act, is the largest source of federal
          funding for public education. Title I will provide states with $10.3 billion in fiscal
          year 2002 to improve the educational achievement of children at risk. Title I serves
          about 12.5 million children in all 50 states, the District of Columbia, and Puerto Rico.


          Section 1-A:
          States Lacking Accountability Systems Risk Losing Federal Title I
          Funds

          As of March 2002, 17 states were in compliance with the 1994 Title I student
          assessment requirements of ESEA. Thirty-five states (including the District of
          Columbia and Puerto Rico) were not in compliance with those requirements (see
          Table 1 below).

          Table 1

                             Status of States’ Compliance With 1994 Title I
                             Assessment Requirements as of March 2002

                  Compliant (17)                           Noncompliant (35)
          Colorado                        Alabama                        Nebraska
          Delaware                        Alaska                         Nevada
          Indiana                         Arizona                        New Hampshire
          Kansas                          Arkansas                       New Jersey
          Louisiana                       California                     New Mexico
          Maine                           Connecticut                    New York
          Maryland                        District of Columbia           North Dakota
          Massachusetts                   Florida                        Ohio
          Missouri                        Georgia                        Oklahoma
          North Carolina                  Hawaii                         Puerto Rico
          Oregon                          Idaho                          South Carolina
          Pennsylvania                    Illinois                       South Dakota
          Rhode Island                    Iowa                           Tennessee
          Texas                           Kentucky                       Utah
          Vermont                         Michigan                       Washington
          Virginia                        Minnesota                      West Virginia
          Wyoming                         Mississippi                    Wisconsin
                                          Montana
          Source: Title I: Education Needs to Monitor States’ Scoring, U.S. General Accounting
                  Office, April 2002




             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 16      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                       AUGUST 2002
                     Congress gave states until the beginning of the 2000–01 school year to fully
                     implement those requirements. The status of the 35 states that were not in compliance
                     as of March 2002 was as follows:

                     •       Thirty states held a waiver extending the 2000–01 deadline.

                     •       Five states had been asked by Education to enter into compliance agreements that
                             establish the date by which they must be in compliance before being subject to
                             losing some Title I administrative funds.
                      Documents from Education show that data for the disabled, migrant, and
                      economically disadvantaged subcategories are the most common subgroups excluded
                                  from campus, LEA, and state reports. In addition, many states are
 The assessment requirements      behind in other areas such as aligning assessments to state content
 that states most frequently
                                  standards and including all schools and students in the accountability
 have not met are the
 following:                       system. For example, one of the three large states we audited in detail
 •    Assess all schools and      had not yet included all Title I schools in its accountability system as
      students                    required by law.
 •   Break out assessment
     results by the following         The 2001 No Child Left Behind Act’s reauthorization of ESEA includes
     subcategories of students:
                                      deadlines that vary according to the specific requirements in the
     -   Gender
                                      reauthorization. States can receive a one-year extension beyond those
     -   Race
                                      deadlines only in the case of a “natural disaster or precipitous and
     -   Ethnicity
                                      unforeseen decline in the financial resources of the state.” Because the
     -   English proficiency
                                      majority of states have not met the requirements of the 1994 law, it
         status
                                      appears that many states may not be positioned well to meet the schedule
     -   Migrant status
                                      for implementing the new requirements.
     -   Disability status
     -   Economic
         disadvantage
                                    Education may withhold funds if a state does not meet the terms of its
                                    compliance agreement. The 1994 legislation was not specific in the
                                    amount of administrative funds that could be withheld, but the 2001
                     legislation states that Education must withhold 25 percent of state administrative
                     funds until a state meets the 1994 requirements (including the terms of any time line
                     waivers or compliance agreements).


                     Section 1-B:
                     Several Factors Affect the Implementation of Compliant
                     Accountability Systems

                     Responses from 50 state Title I directors to a survey, as well as information gained
                     from interviews with state officials, revealed four primary factors that further or
                     hinder compliance with 1994 Title I requirements.

                     Factors Furthering Implementation:

                     Efforts of State Leaders

                     Officials we interviewed in states that had attained compliance with 1994 Title I
                     requirements said that their respective governors, legislators, and business leaders
                     made compliance a high priority. They described the development of high-level
                     committees, new state legislation, and other measures to raise the visibility and


                          A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002               SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA               PAGE 17
          priority of school accountability. One governor spearheaded a plan that used
          commissions to develop content standards and assessments aligned with those
          standards.

          Some state officials reported that SEA leaders had initiated organizational changes to
          facilitate implementation of accountability systems. One SEA reorganized according
          to function (rather than funding stream) to enhance coordination. In one case, SEA
          managers who did not support changes necessary to achieve compliance with Title I
          were replaced with more supportive staff.

          One large state also noted the importance of state leadership and collaboration among
          levels of government and education in developing and implementing the automated
          information systems necessary to operate the state’s assessment and accountability
          systems.

          Coordination Among Staff and Levels of Administration

          More than 80 percent of the state Title I officials identified the ability of agencies and
          different levels of public education administration to coordinate their efforts as a
          factor that helped them meet requirements. Two states reported that having
          assessment and Title I staff share working space enhanced their ability to achieve Title
          I compliance. One state emphasized the importance of coordination and collaboration
          among state and local information managers, administrators, and teachers in the
          development and continued improvement of the data quality within accountability
          systems.

          Title I and other state officials in states that had met the 1994 Title I requirements
          noted that they had made great efforts to secure the support and participation of other
          state officials, local administrators, educators, and the public. Several officials
          reported holding public meetings and focus groups to obtain input from parents,
          teachers, and local administrators. They also reported conducting public relations
          campaigns to educate the public about the importance of complying with Title I
          requirements. One state conducted focus groups and hearings over a period of six
          years, and it conducts annual conferences to allow local education officials to address
          issues and gain advice from experts.

          Availability of Expertise at the State Level

          More than 80 percent of the state Title I directors identified the availability of state
          level expertise as a factor that facilitated their efforts to meet Title I requirements.
          Training for teachers and district personnel was a key factor in achieving success in
          the administration of assessment and accountability systems. Two states use regional
          education centers to train and provide technical assistance on assessments and
          standards to local staff.

          Factors Hindering Implementation:

          Half of the survey respondents identified inadequate funding as an obstacle in moving
          toward compliance. Noncompliant states cited this problem more often than
          compliant ones. Officials from noncompliant states also reported that progress toward
          compliance with Title I requirements was stalled because of previous investments they



             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 18      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA              AUGUST 2002
              had made in assessment systems that predated and conflicted with the 1994 Title I
              reauthorization. One example of this conflict was the shift from norm-referenced
              assessments to criterion-referenced assessments. Respondents specified that building
              support and obtaining funding for a new system was time-consuming, and it took their
              respective states several years to change from old systems to new, compliant ones.
              One survey respondent from a very small state noted that, due to the state’s size, the
              SEA has a small staff that lacks the technical expertise to develop a new system.


Section 2:
State Actions to Ensure Accurate Assessment Test Scoring and
Student Accommodations (Where Applicable) Are Limited

              Most states are taking some action to ensure that Title I student assessment tests are
              scored accurately, that any exemptions for students with limited English proficiency
              are justified, and that students with disabilities are receiving appropriate testing
              accommodations to gather an accurate assessment of their abilities. However,
              although they reported establishing standards and guidelines for LEAs to follow, they
              reported few activities to ensure that these guidelines were followed.


              Section 2-A:
              State Monitoring of Test Contractor Scoring Is Inadequate to
              Ensure Accuracy

              Monitoring methods states use to ensure the accuracy of test contractors’ scoring and
              reporting do not always provide adequate assurance of complete and accurate scoring
              results. Several states reported problems with scoring and calculation errors,
              regardless of whether they had monitoring procedures in place. These states reported
              that local district officials, parents, and state agency staff discovered the errors. The
              errors affected students, families, and school and district resources—in some cases
              significantly. One state sent thousands of children to summer school based on
              erroneous scores the test contractor calculated. In another case, based on a test
              contractor’s errors, a state incorrectly identified several schools as needing
              improvement, a designation that carried with it both bad publicity and the extra
              expense of providing technical assistance to schools that did not need it.

              Of the 44 states that hire contractors for test scoring, 16 (one-third) have no
              monitoring mechanism to ensure the accuracy of the contractor’s test scoring and
              reporting. Of the 28 states that reported they do use one or more monitoring
              mechanisms, 15 reported that they monitor the contractor’s scoring by comparing a
              sample of original student test results to the contractors’ results. Some also reported
              comparing current test scores with those from previous years to identify significant
              variations, but this comparison may, in some cases, be informal. Several states
              indicated they are still relying on contractor self-monitoring to ensure accurate
              scoring. For example, one of the three large states we audited in detail had submitted
              to Education incorrect information on adequate yearly progress and Title I schools in
              need of improvement because of errors in its accountability database and in the test
              contractor’s compilation of student assessment results. When the state discovered the




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 19
          errors and corrected its accountability ratings, it did not submit corrected data to
          Education.

          States that were in compliance with Title I requirements generally had more-complete
          monitoring systems that included measures such as using technical advisory
          committees to review results, conducting site visits, and following a sample of tests
          through the scoring and reporting process. One large state simulates six complete
          district student populations and test answer documents to check the contractor’s
          scoring and reporting for all possible types of errors. Three states direct staff or hire
          third parties to conduct independent audits of test scoring.

          Education is obligated under the Federal Managers’ Financial Integrity Act of 1982
          and the Single Audit Act to ensure that states that receive federal funds comply with
          statutory and regulatory requirements for monitoring contractors. However,
          Education currently takes limited action regarding states’ monitoring of assessment
          test contractors. Education has an important vehicle for such oversight: compliance
          reviews of state programs that it conducts on a four-year cycle, which includes a
          weeklong, on-site visit. However, Education’s Office of Inspector General has
          reported the following deficiencies in this review:

          •        Insufficient time to conduct the reviews
          •        Lack of knowledge among Education staff about areas they review
          •        Lack of consistency in how Education staff conduct the reviews

          Senior Education officials report that Education is developing a new achievement-
          focused monitoring process that it will pilot during fiscal year 2002. However, a
          senior Education official who is working on the redesign of the compliance reviews
          reported that the current draft plans do not include specific checks on state monitoring
          of assessment scoring. Confidence in the accuracy of test scoring is critical to the
          acceptance of the test results used in assessing school performance.

          Education should include in its new compliance reviews a check of (1) the controls
          states have in place to ensure proper test scoring and reporting of results and (2)
          effective implementation of these controls by states.


          Section 2-B:
          States’ Efforts to Ensure Compliance With Guidelines for
          Accommodating All Students Are Limited

          According to our surveys and interviews, 33 states have taken at least minimal actions
          to ensure that assessment test exemptions for students with limited English
          proficiency are justified; 41 states have taken actions to ensure that assessment test
          accommodations for students with disabilities are appropriate. Most states have
          developed standards for LEAs to follow in accommodating these students so that
          assessments can yield accurate measures of their performance. However, states report
          they have made limited efforts to ensure that LEAs follow these guidelines. For
          example, 17 states reported that they compare the number of students with limited
          English proficiency tested within a given year against the number for the previous
          year. Because of high student mobility and students’ changing status with regard to



              A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 20       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
              English proficiency, this comparison has obvious limitations. It is also not evident
              how such comparisons would allow states to ascertain the appropriateness of the
              accommodations.

              Survey results and interviews indicated that relatively more states are taking actions to
              monitor accommodations for students with disabilities, but relatively fewer states are
              taking actions to monitor accommodations for students with limited English
              proficiency. While 25 states reported that they have standards for accommodating
              students with limited English proficiency, 36 have standards for accommodating
              students with disabilities. State officials report that this is the case because LEAs
              have been able to build on steps they have taken under the Individuals with
              Disabilities in Education Act (IDEA) to document accommodations needed by
              students with disabilities. More states are still in the process of developing alternative
              assessments or standards to accommodate students with limited English proficiency.

              SEAs conduct cyclical or risk-based monitoring of the implementation of all of their
              programs. This same monitoring process could be used to assess the appropriateness
              of LEA policy and practice with regard to testing accommodations. However, in a
              recent review, the U.S. General Accounting Office found that states varied
              dramatically on how often they conducted on-site visits. The average time between
              visits ranged from 2 years or fewer (in 6 states) to more than 7 years (in 17 states).


              Section 2-C:
              Some States Are Not Adequately Monitoring LEAs or Otherwise
              Exercising Oversight of LEA Procedures and Controls to Ensure
              Accountability Data Quality

              We identified a number of practices and procedures that helped ensure data quality
              when we reviewed SEA and LEA procedures for (1) gathering, processing, and
              reporting accountability data and (2) ensuring data quality. Best practices included
              the following:

              •        Automated attendance-recording systems for teachers
              •        Required reconciliations between campus and LEA reports and source
                       documentation
              •        Comprehensive manuals providing data definitions, procedures, and
                       requirements
              •        Ongoing training provided by the SEA, regional education service centers,
                       and LEAs
              •        LEA and campus officials designated as coordinators for the data quality of
                       accountability indicators
              •        LEA superintendent validation of LEA data submitted to the SEA
              •        Electronic editing systems for LEAs to use in submitting data to the SEA
              •        Independent or SEA audits of LEA data quality




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 21
             •        Sanctions (built in to the accountability system) for LEAs and campuses with
                      significant data quality problems

             However, the states we audited were not adequately monitoring the procedures and
             controls their LEAs have in place to ensure the quality of the accountability data they
             reported to the SEAs. One of the states we audited in detail provided comprehensive
             guidance and training to LEA personnel, but it did not monitor or otherwise verify
             LEA data submitted to the SEA. Our audit revealed significant error rates in Title I
             campus and LEA accountability data submissions in that state, which may have
             affected the quality of the school improvement data this SEA reported to Education.
             Instead of monitoring or auditing LEA data at the time of submission, this state’s SEA
             relied on a special data inquiry unit to identify and audit LEAs and campuses with
             potential data quality problems after accountability ratings had been determined. The
             SEA then lowered accountability ratings for LEAs that had significant data quality
             problems. However, the data inquiry unit was not reporting its results in a way that
             allowed the SEA to quantify error rates and types, determine the significance of LEA
             data quality problems, or address these problems.

             In another state we audited in detail, we found that the SEA was not monitoring its
             LEAs for data quality or otherwise becoming aware of problems with LEA data
             quality. The audit we conducted at this state’s largest urban LEA revealed serious
             data quality problems that the SEA had not acknowledged in its Consolidated State
             Performance Report to Education for the 1999–2000 school year.

             As mentioned previously, states conduct cyclical monitoring of the implementation of
             their programs at LEAs. These monitoring visits could include reviews of
             accountability data procedures and controls. However, some states may not be
             visiting LEAs frequently enough to provide ongoing assurance about LEA data
             quality. States should at least become aware of the potential problems. Then they
             would be able to develop some method for evaluating and disclosing the extent to
             which poor LEA data quality could be affecting the information on schools in need of
             improvement reported within the state and to Education.


Section 3:
Campuses and LEAs Have Significant Weaknesses in Data Reporting
and Quality Control Procedures

             LEAs from state to state differ in terms of the degree of automation and quality
             controls they use in gathering, processing, and reporting basic Title I accountability
             data. Their systems range from highly automated processes to mostly manual
             processes, and their systematic procedures and controls range from established and
             well documented to almost nonexistent.




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 22          SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA            AUGUST 2002
              Section 3-A:
              Data Quality Problems Persist at LEAs and Campuses in One State
              With Automated Reporting and Editing Systems

              As mentioned in Section 2-C, we found significant statistical error rates in the
              gathering and reporting of basic accountability data at Title I campuses and LEAs in
              one of the states we audited in detail—a state that is recognized as having one of the
              best accountability information systems in the country. Extended to all Title I
              campuses statewide, these error rates prevented us from providing full assurance
              regarding the quality of the state’s accountability information used within the state
              and reported to Education.

              The SEA in this state does not monitor or otherwise verify the accountability data the
              LEAs submit to it. Instead, it provides thorough and annually updated data
              definitions, standards, and guidelines to the LEAs, as well as ongoing training. It also
              has implemented a secure automated data transfer process. The data transfer process
              requires the LEAs to use a Web-based edit application for each data submission to
              identify and correct all fatal errors and two other types of errors. This procedure
              ensures that data is logically correct, but it cannot address the weaknesses in data
              quality controls that exist during the data gathering and reporting process.

              In the 1999–2000 school year, this state used three base accountability indicators:
              student assessment results, attendance rates, and dropout rates. We found an error rate
              of 7 percent in the coding of student assessment test answer documents for students’
              “tested” or “not tested” status. We also found certain unmonitored areas with the
              potential for careless or purposeful miscoding that would result in eligible students’
              being unaccounted for with regard to required testing. Additionally, we found a 10
              percent error rate in the documentation and coding of school leaver reasons (including
              dropouts) and a 0.55 percent error rate in attendance reporting. The error rate in
              reporting student attendance and absence was relatively low, but extended statewide,
              it meant that as much as $27 million in state public education funds was allocated on
              the basis of reported but undocumented student attendance.

              These data quality problems suggest the need for enhanced training and procedures at
              campuses and LEA central offices, including improved supervisory review, to help
              ensure the reliability of the data LEAs report to their SEAs. Campuses can implement
              routine reconciliations between source documentation and campus reports to the LEA
              and can perform additional review of the coding of students’ answer documents for
              test participation status. There are additional steps the SEA can take to ensure campus
              and LEA data quality, such as data quality audits followed by sanctions, if necessary.
              However, as mentioned previously, the states we reviewed were not monitoring LEAs
              and campuses for Title I data quality.


              Section 3-B:
              Manual Reporting Procedures for Which There Is No Oversight or
              Review Produce Poor Data Quality

              In the large LEA we audited, 259 campuses and the LEA were gathering, recording,
              and reporting accountability data manually. This resulted in numerous errors of




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                PAGE 23
          multiple types that remained undetected as the data were processed and reported to the
          SEA.

          Moreover, it was not possible to test or verify the accuracy, completeness, validity, or
          timeliness of the data. The LEA and campuses had not complied with federal record
          retention requirements and had maintained almost no supporting documentation for
          the data that they submitted to the SEA and that the SEA would use to determine
          schools in need of improvement. Both the original source documents and the campus
          summary sheets for student assessment results were missing. Some of this
          documentation had been lost as a result of damage, and some of it had not been
          maintained in the first place. The limited testing we were able to conduct using the
          estimated 4 percent of available records indicated that there were problems with the
          validity and reliability of the information reported to the LEA for the 1999–2000
          school year.

          Additionally, this LEA had allowed its campuses to use inconsistent and perhaps
          inappropriate sampling methods for reporting student assessment results for middle
          and high schools. The SEA had not sanctioned the use of sampling, and we could not
          verify the manner in which or the extent to which the sampling had occurred.

          There was no evidence that supervisors or other staff reviewed data before submitting
          it to the next level in the process. No one at campuses or at the LEA compared
          classroom summary sheets to source documents, checked on the completeness of the
          summaries, or verified that all classrooms were included in the campus summaries
          sent to the LEA.

          Moreover, at this LEA, one person was responsible for manually entering all the
          campus data—more than 8,000 entries—into an Excel spreadsheet. This person also
          manually keyed in data to the formatted Title I report. However, no one reviewed the
          accuracy of the spreadsheet formulas or totals, nor did anyone review the accuracy
          and completeness of the data reported. No one reviewed the spreadsheet totals or the
          final report for anomalies such as unusual year-to year differences or omissions.

          Examples of the types of errors we discovered during our audit included the
          following:

          •        Schools failed to report student assessment results for significant numbers of
                   students, and the LEA reported incomplete data to the SEA.
          •        On LEA summary spreadsheets, there was a significant discrepancy between
                   the number of grades reported for math assessment and the number of grades
                   reported for reading assessment.
          •        Nine schools had no data on the number of students tested. The SEA reported
                   that it prepared the report from orally conveyed information without
                   completing the spreadsheet.
          •        Serious data entry errors for one high school occurred when data from campus
                   summary sheets were keyed into the LEA spreadsheet.
          •        Information for one entire classroom of assessment results was missing from
                   the LEA Title I spreadsheet.




              A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 24       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA            AUGUST 2002
              In another state we audited in detail, LEAs were not required to submit data for the
              SEA to use in preparing its Consolidated State Performance Report until shortly
              before the report was due to Education. Additionally, the LEAs manually submitted
              their data, which were often inconsistent and took time to reconcile. As a result, the
              SEA could not submit its report by the December 1 deadline. Beginning in the
              2001–02 school year, however, the SEA reports that LEAs started submitting data to
              the SEA electronically and were required to resolve edit discrepancies prior to
              submission.

              To improve data quality, LEAs should:

              •        Identify and address risks to the security of records required for compliance
                       with Title I law.
              •        Implement strategies to prevent, mitigate, or recover from loss of required
                       documentation.
              •        Monitor and enforce campus and LEA compliance with federal record
                       retention requirements.
              •        Develop and implement written procedures for campuses and the LEA to
                       follow in developing and reporting Title I performance data.
              •        Develop and implement data quality controls that include reconciliations with
                       source documentation; systematic review of data for reasonableness,
                       accuracy, and completeness; and identification of anomalies that could be
                       errors.
              •        Automate, to the extent possible, Title I accountability data gathering,
                       processing, and reporting procedures.
              •        Use monitoring and audit results to focus resources, training, and oversight on
                       identified data quality problems.

              In addition, SEAs should monitor LEAs to ensure they are using appropriate methods
              for collecting and reporting accountability data.


Section 4:
Some States Are Not Reporting Accurate, Complete, or Timely
Information to Education

              The SEAs in two of the three large states we audited were not reporting to Education
              accurate, complete, valid, or timely Title I accountability data on schools in need of
              improvement. In the third state we audited, the state’s school improvement
              information may have been compromised by campus and LEA data quality problems.

              States’ submissions of incorrect Title I data to Education result in the use of unreliable
              data to prepare Education’s Title I performance reports to the U.S. Congress. Because
              of the lateness and poor quality of many states’ submissions, Education’s reports to
              Congress for the 1997–98 and 1998–99 school years were not published until 22
              months after the state data submission deadline. Consequently, Congress may have




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                   PAGE 25
          been making Title I policy and funding decisions on the basis of inconsistent,
          unreliable, and out-of-date data.

          Our audits allowed us to identify three primary causes of poor state Title I
          accountability data quality:

          •        Significant errors in basic accountability data reported to SEAs by campuses
                   and LEAs
          •        SEA noncompliance with the Title I law and reporting requirements
          •        Absence of systematic procedures and basic controls at SEAs to help ensure
                   the quality of state accountability data developed and reported within the state
                   and to Education

          In Sections 4-A through 4-C, we provide additional details on the causes of and the
          recommendations for eliminating problems in the accountability information
          development and reporting process.


          Section 4-A:
          Significant Errors in LEA Accountability Data Reported to the SEAs
          Continue to Affect the Quality of Data That SEAs Report to
          Education

          As mentioned in Section 2-C, our audits of three states determined that SEAs may not
          be adequately monitoring or otherwise exercising adequate oversight over LEA
          procedures and controls to ensure the quality of the accountability data they report to
          the state. Regardless of an SEA’s procedures and controls to ensure the quality of the
          accountability data it collects, processes, and reports, its state accountability data are
          incorrect at least to the extent that the LEAs’ data are incorrect.

          One of the three states we audited has a strong, fully automated accountability system
          in place. This state also has an automated process for calculating and reporting
          information on schools in need of improvement to Education. The state’s assessment
          and accountability systems received approval by Education in March 2002. The SEA
          reliably processes data on schools in need of improvement and reports this
          information on the basis of the data it receives from the LEAs. As noted previously,
          however, error rates in data gathering and reporting at campuses and LEAs prevented
          us from providing full assurance regarding the quality of this state’s accountability
          data.

          SEAs need to develop some method for identifying data quality problems (and the
          extent of those problems) at LEAs and need to report that information in the section of
          the Consolidated State Performance Report designated for data quality issues. To
          continually improve accountability data quality, SEAs and LEAs should collaborate
          on addressing identified problems.




              A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 26       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA              AUGUST 2002
              Section 4-B:
              Noncompliance Within Accountability Systems and Reporting
              Procedures Results in Errors in Title I Data Reported to Education

              In two states we audited in detail, we determined that some of the data quality
              problems we identified in the school improvement data reported to Education resulted
              from the SEAs’ noncompliance with the Title I law and federal reporting
              requirements.

              Incomplete School Accountability System. One SEA had an accountability
              system in place for determining school performance and identifying schools in need of
              improvement. The system was not yet in compliance with the Title I statute, however,
              because its accountability system did not include all Title I schools. The system had
              not yet been extended to include alternative campuses or very small campuses in the
              state. As a result, this state failed to consider the performance status of 543 (11
              percent) of the 4,868 Title I schools and may have underreported to Education the
              number of Title I schools in need of improvement.

              Misinterpretation of Title I Reporting Requirements. Two SEAs were
              misinterpreting the federal definition of schools in need of improvement. This
              definition includes schools identified as needing improvement in the current year and
              schools remaining in that category from prior years. These two states were
              underreporting the number of Title I schools in need of improvement because they
              were reporting only schools identified as needing improvement in the current year.

              Failure to Develop and Retain Supporting Documentation. As described
              previously, two SEAs were not in compliance with federal record retention
              requirements. This meant that data could not be verified and corrected before
              submission to Education or after submission, when adjustments were necessary or
              when data quality audits were being conducted. In the case of one state, it was not
              possible to determine whether the school improvement information it submitted to
              Education was accurate.

              Failure to Report Data Quality Issues. Two SEAs should have been but were not
              sufficiently aware of significant issues regarding the quality of their Title I
              accountability data. As a result, neither state reported these issues in the designated
              section of the Consolidated State Performance Report. The third SEA we audited
              knew that there were problems with the quality of LEA data, but it did not know the
              extent of the problem or report it in the Title I report to Education. SEAs that fail to
              report circumstances that affect the quality of data are not in compliance with Title I
              reporting requirements. This also violates a data quality requirement for full
              disclosure of all information that would affect the interpretation and use of the data.

              Inability to Meet Title I Reporting Timelines With Reliable, Timely Data. SEAs in
              the three states we audited were not able to submit their Consolidated State
              Performance Reports to Education by the December 1 deadline. The state with the
              most established student assessment and accountability systems and the most fully
              automated accountability information system must use data from two different years
              to determine accountability ratings each year. This state must use student assessment
              data from the current year, but it must use dropout and attendance data (which were
              also base accountability indicators) from the previous year.



                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 27
          In this state, student assessment data are not final until the end of November,
          following the administration of the spring and summer assessment test series.
          However, these data are not useful because the Title I law and reporting requirements
          call for final accountability calculations to identify schools not making adequate
          yearly progress. Given the additional time required to aggregate, disaggregate, and
          verify final accountability data and to review the Consolidated State Performance
          Report for all entitlement programs, the final report is not ready by December 1.
          Because this is the case in a state that Education officials and contractors describe as
          having one of the best accountability information systems in the country, it is likely
          that timely submission of the Consolidated State Performance Report is a problem in
          other states as well. According to the SEA in another state we audited, LEAs were
          not required to submit data that the SEA used to prepare its Consolidated State
          Performance Report until shortly before the report was due to Education. In our audit
          of Title I data quality at Education, we found that another state was not able to finalize
          its data submission until two weeks before Education published the 1998–99 school
          year report in October 2001 because the largest LEA in that state was late in
          submitting its data to the SEA.

          Education’s State ESEA Title I Participation Information for 1998–99: Final
          Summary Report (2001) indicates that many states do not meet the December 1
          deadline. For example, only four states met that deadline for the 1998–99 school year.
          One-fifth of all states submitted their Consolidated State Performance Reports for that
          school year more than 18 months late.

          Additionally, the data processing contractor Education retains to receive and edit state
          Title I data informed us that, for that same school year, approximately 40 states were
          asked to revise their initial data submissions because of conditions identified during
          the contractor’s editing process. Moreover, one of the three states we audited in detail
          was submitting preliminary data based on unreviewed assessment results. It was not
          sending final, corrected data to Education.

          Education originally set the December 1 deadline to meet a legislative requirement for
          receiving annual performance reports on federal education programs no later than
          March 31 of each year. This date coincides with Education’s submission of its annual
          financial report to the Office of Management and Budget (OMB). However, because
          of states’ late submissions and the absence of adequate procedures by Education to
          ensure timely receipt, review, and publication of school improvement data, Education
          did not publish the reports for the 1997–98 and 1998–99 school years until 22 months
          after the data were due from the states. (See Section 5 for a discussion of
          improvements Education could make in its procedures for obtaining timely and
          reliable state Title I data.)

          Each SEA should include all Title I schools in the state’s review to identify and report
          schools in need of improvement. Specifically, SEAs should:

          •        Report all Title I schools with the status of needing improvement, not just
                   those identified during the current year.
          •        Ensure that they and LEAs maintain source documentation for and prepare
                   adequate documentation of each year’s development of school improvement




              A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 28       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
                       information. SEAs and LEAs need to retain documentation for three years, as
                       required by federal regulations.
              •        Communicate with their LEAs and identify data quality issues at local and
                       state levels. They should disclose these issues in the designated section of the
                       Consolidated State Performance Report as necessary.

              For a recommendation regarding late state submissions, see Section 5-A.

              Section 4-C:
              States Need to Develop and Implement Systematic Procedures
              and Controls for Gathering, Processing, and Reporting Reliable
              Title I Accountability Data

              In one of the states we audited in detail, the SEA reported having numerous and
              comprehensive procedures and controls in place to protect the quality of the
              accountability data it received from the state’s campuses and LEAs as it processed and
              reported that data to Education. Our tests of campus and LEA data as it progressed
              through that SEA’s automated processes and on to the Consolidated State
              Performance Report indicated that the SEA was protecting the integrity of the LEA
              data it received and used to determine schools in need of improvement. This state’s
              data quality procedures and controls included the following:

              •        Strategic planning and resource allocation for mission-critical, enterprise
                       information assets
              •        Current, published data definitions, standards, and procedures
              •        Published and updated agency policies on security, confidentiality, and
                       information resources
              •        Automated tools for reporting secure data and for identifying and correcting
                       errors
              •        Ongoing training and technical assistance for LEAs
              •        Progressive oversight, review, and data correction
              •        Data quality audits followed by sanctions when necessary

              However, as reported previously, the errors we found in campus and LEA
              accountability data reported to that SEA prevented us from providing full assurance
              regarding the quality of the state’s accountability data. Like other SEAs we reviewed,
              the SEA did not monitor its LEAs for accountability data quality. Instead, it relied on
              audits of LEAs and campuses for data quality after their accountability ratings had
              been established, followed by sanctions and lowered accountability ratings when
              necessary.

              We recommended that the SEA and LEAs cooperate to identify and address persistent
              LEA data quality problems. We also identified four ways this SEA could monitor
              student assessment participation to ensure full participation of all eligible, qualified
              students. Additionally, we identified ways the SEA could enhance its accountability
              reporting to make it more accessible to users and to inspire greater public confidence
              in accountability information for policy- and decision-making.


                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 29
          Inadequate Procedures and Controls for Developing Accountability
          Information. In the other two states we audited in detail, we did not find evidence of
          systematic procedures or basic controls for gathering, calculating, and reporting
          accountability and school improvement information. Problems with the basic data
          procedures included weaknesses in or the absence of:

          •        Oversight and monitoring of LEA procedures for gathering and reporting
                   accountability data to the SEA or lack of SEA awareness of data quality
                   problems at LEAs
          •        Documentation of the collection and development of Title I school
                   improvement information and retention of such documentation for the
                   required time
          •        Evidence and documentation of analytical, reasonableness, and independent
                   supervisory reviews of the data to ensure accuracy, completeness, and validity
                   prior to use within the state and prior to submission to Education

          In one state, these weaknesses in controls resulted in significant underreporting of the
          number of Title I schools in need of improvement. Over four years, this state reported
          to Education only from 14 to 60 percent of the schools identified as needing
          improvement in its database. Some schools were not reported because of the
          misinterpretation of the reporting requirement, but this does not account for all of the
          incomplete reporting. A proportion of the incomplete reporting also occurred because
          there was an absence of sufficient procedures and data quality controls for developing
          and reporting information to ensure its accuracy, completeness, and validity.

          As mentioned previously, another of these two states submitted its Consolidated State
          Performance Report to Education without detecting errors in its accountability
          database or in the test contractor’s compilation of student assessment results on which
          school performance had been based. After the SEA corrected its accountability
          database, our audit testing identified 24 campuses that were incorrectly reported in the
          database and 10 Title I LEAs that were not recorded in the database at all. Therefore,
          this state had not correctly reported to Education adequate yearly progress and Title I
          schools in need of improvement. The SEA lacked basic data quality controls,
          including established procedures, documentation of calculations, maintenance of
          records, and analytic, reasonableness, and supervisory reviews.

          Inadequate or Nonexistent Documentation of Procedures and Controls for
          Developing and Reporting Title I Accountability Data. All three states we audited
          in detail lacked adequate documentation of their procedures and data quality controls
          for gathering, processing, and reporting information on schools in need of
          improvement to Education. This weakness ranged from an occasional gap in
          documentation to a complete absence of such documentation.

          Without written documentation for data sources, definitions, business rules, manual
          and automated procedures, change and test procedures, or security and quality
          controls, there is an absence of history and accountability for reported data. This
          increases the risk of reporting inconsistent, unreliable data and of interruption in the
          case of disaster or absence or loss of staff.




              A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 30       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
                    See Table 2 for an overview of the strengths and weaknesses in data processing and
                    quality controls at three states and at Education for state Title I data. We prepared this
                    report from information in the individual reports listed in Appendix 2.
   Table 2
   Compliance and Data Quality Controls Vary

   Comparison of Compliance and Data Quality Controls for Reporting Title I Schools in Need of Improvement in
                   Consolidated State Performance Reports for the 1999–2000 School Year

                   Criterion                          California   Pennsylvania           Texas        Education
   Compliance with Title I Requirements

   Compliance with Title I reporting            No                 No               Waiver            Not
   requirement to report all schools with                                           because of        audited
   status of needing improvement, not just                                          state’s annual
   schools identified with this status in the                                       school
   report year                                                                      improvement
                                                                                    requirements
   Compliance with Title I requirement to       No                 No               Yes               Not
   include all schools in evaluation of                                                               audited
   schools in need of improvement,
   including alternative and very small
   campuses
   Compliance with Title I reporting            No                 No               No*               Not
   requirement to describe data quality                                                               audited
   issues in designated section of
   Consolidated State Performance
   Report
   Compliance with record retention             No                 No               Yes               Not
   requirements of Title I program                                                                    audited
   Timely submission of Consolidated State      No                 No               No                No
   Performance Report and controls to
   help ensure timely submission

   Data Quality Controls
   Automated state school accountability        Yes                No               Yes               Yes
   system for determining schools making
   and not making adequate yearly
   progress

   Systematic procedures and controls           No                 No               No                No
   developed and implemented for
   identifying schools in need of
   improvement and for ensuring the
   quality of the data
   Documentation of manual and                  No                 No               No                Not
   automated procedures and data                                                                      audited
   quality controls for identifying and
   reporting schools in need of
   improvement internally and to
   Education




                       A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002            SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 31
                    Summary of Overall Data Quality Controls and State Oversight and Monitoring
                    Criterion                          California       Pennsylvania           Texas         Education
   Overall Data Quality Controls

   Sufficient overall controls in place to        No                   No                No                 No
   ensure reliable data on schools in need
   of improvement

   State Oversight and Monitoring of Accountability System
   Monitoring of full participation in            Not audited          Not audited       No                 Not
   student assessment testing by all                                                                        audited
   eligible, qualified students
   Monitoring and verification of student         Not audited          No                Yes                Not
   assessment test scores determined and                                                                    audited
   reported by test contractor
   Enforcement of timely submission of            Not audited          No                Yes                Not
   student assessment data from test                                                                        audited
   contractor
   Oversight and monitoring of LEAs’ and          Not audited          No                No                 No
   SEA’s accountability data processing
   procedures and controls to ensure Title I
   data submitted to SEA and Education
   are reliable, valid, and timely

   * The SEA did not mention data quality weaknesses at campuses and LEAs in the designated section of the Consolidated
    State Performance Report.



                    To strengthen systematic procedures and controls for gathering, processing, and
                    reporting reliable Title I accountability data, SEAs should:

                    •           Ensure that test data from test contractors are adequately reviewed, and
                                document the review process before the data are used to determine and report
                                schools in need of improvement to Education.
                    •           Improve their information systems, procedures, and controls to ensure the
                                accuracy, completeness, validity, and timeliness of their school accountability
                                information reported within the state and to Education. Controls should
                                include adequate supervisory, analytical, and reasonableness review of the
                                data prior to publication or submission to Education.
                    •           Correct data as soon as possible if they discover errors in their Title I report to
                                Education.
                    •           Maintain thorough documentation of all their manual and automated systems
                                and data quality controls for developing and reporting accountability
                                information within the state and to Education.

                    In addition, SEAs and LEAs should collaborate in continually improving state
                    accountability data. SEAs should strengthen data verification procedures and
                    communications with LEAs. They should adequately monitor and audit LEA data
                    procedures and quality controls to ensure the completeness and accuracy of the Title I
                    data they submit to the SEAs.



                        A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 32                 SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  AUGUST 2002
Section 5:
Improvements Can Be Made in Data Quality Controls at Education

              We assessed whether Education has controls to ensure that state Title I, Part A school
              improvement data are accurate, complete, valid, and timely. We determined that:

              •        Management controls over timely publication of school improvement data
                       need to be strengthened.
              •        Management controls to ensure that Title I school improvement data are
                       reliable and valid need to be strengthened.

              To meet its administrative responsibilities and to report performance information to
              Congress, Education requires states to submit annual performance data. The reporting
              instrument, the Consolidated State Performance Report, requires states to provide
              data for eight formula grant programs, including Title I, Part A. Education requires
              states to submit their reports by December 1 with Title I information based on
              accountability and school improvement data from the previous school year.

              Within Education, the Office of Elementary and Secondary Education (OESE)
              administers the Title I program of the ESEA. This includes the collection, review, and
              monitoring of the performance information the states provide. The Performance
              Evaluation Services division (PES) helps guide the Title I program priorities by
              evaluating the effectiveness, efficiency, and equity of the programs. OESE and PES
              have shared responsibility for school improvement data that SEAs report to
              Education.

              Education hired a contractor to analyze and edit the participation, services, and
              achievement data from the Title I portion of the Consolidated State Performance
              Report for the 1997–98 and 1998–99 school years. The contractor consolidated the
              analysis of the states’ data into the State ESEA Title I Participation Information
              Summary Report. This report presented the scope of the Title I program services for
              two successive school years.


              Section 5-A:
              Management Controls Over the Timely Publication of School
              Improvement Data Need to Be Strengthened

              As mentioned in Section 4, Education did not publish the State ESEA Title I
              Participation Information Summary Report for the 1997–98 and 1998–99 school years
              until approximately 22 months after the deadline for state data submissions.

              OESE does not have a systematic process in place to enforce the December 1 deadline
              with clear and frequent reminders to states. There are no established time frames or
              regular follow-up procedures for obtaining late data submissions. OESE staff
              reported that they generally contact state officials by telephone and do not keep
              consistent records of these contacts. For 1999–2000, the most recent school year for
              required reporting, OESE program staff specified that they did not initiate actions on
              late submissions of state data until three months after the December 1 due date.




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA              PAGE 33
                         Additionally, they took no action against states that repeatedly submitted data late or
                         against states that were nonresponsive.

                         Table 3 details data submission statistics from the log that OESE staff maintained to
                         record the receipt and status of state consolidated reports for the 1998–99 school year.
                         This table indicates that 48 states submitted their Consolidated State Performance
                         Reports after the December 1 deadline.

    Table 3

                   Timeliness of States’ Data Submission to Education for the 1998–99 School Year

                                                             Submission Category
                                                                                                     Total States
                                             On Time           5–30        31–120     121–270         Reporting
                                                             days late    days late   days late

    Number of States Reporting in
                                                 4              17             16        15               52a
    Each Category
    a   Alaska was omitted from the log, thus reducing the number of reporting
        entities (including the District of Columbia and Puerto Rico) to 52.

    Source: Improving Title I Data Integrity for Schools Identified for Improvement, U.S. Department of Education’s Office
            of Inspector General Philadelphia, Pennsylvania, March 2002


                         Because the subsequent year’s log was incomplete, it was not possible to determine
                         the extent of late accountability data submissions by states for the 1999–2000 school
                         year.

                         Education’s Assistant Secretary for Elementary and Secondary Education and the
                         Undersecretary should strengthen management controls to ensure the timely receipt,
                         review, and publication of performance data concerning schools identified for
                         improvement.


                                                                Section 5-B:
                 Data Quality Standards                         Education Needs Stronger Management
                      March 2000                                Controls to Ensure that Title I School
Validity:         Data adequately represent                     Improvement Data Are Reliable and
                  performance.
Accurate          Data definitions and counts are
                                                                Valid
Description:      correct.
Editing:          Data are correct, internally consistent,      The general instructions in Education’s Consolidated
                  and without mistakes.                         State Performance Report require that all states must
Calculation:      Measured amounts are accurately               submit this report. Each state report is due December
                  computed using the right numbers
                                                                1 of each year and should reflect data for the previous
                  and formulas.
Timeliness:       Data are recent and reported in time
                                                                school year. To help ensure that performance data are
                  to inform policy action.                      of the highest quality, Education adopted data quality
Reporting:        Full disclosure is made. Full disclosure      standards for its program managers (see text box). (In
                  can be met in part by documenting             March 2001, Education published an expanded and
                  data collection processes.
                                                                revised list of eight data quality standards in its 2000
Source: U.S. Department of Education                            Performance Report and 2002 Annual Plans. For this
        (http://www.ed.gov/pubx/AnnualPlan2001/                 joint audit, however, the audit teams used the six
        Append1.doc)
                                                                standards published in March 2000.)




                             A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 34                      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                   AUGUST 2002
              Education’s processing contractor for state Title I data reported that, for the 1998–99
              school year, it asked approximately 40 states to revise their initial data submissions
              because of anomalies detected through the electronic edit process. Our audit work
              concerning school improvement in one large state determined that this editing process
              alone was not adequate to ensure that the published school improvement data the state
              submitted were reliable and valid.

              There are specific controls Education could implement to help ensure greater accuracy
              of state accountability data. For example, the data processing contractor and OESE
              staff advised us that they were unable to check the accuracy of the Title I performance
              data without requesting supporting documentation from states. However, collecting
              such documentation from the states would require approval from the OMB.

              During state monitoring visits, OESE has not requested or reviewed school
              improvement data to ensure that the data were reliable and valid. The OESE reports
              that it will be piloting an achievement-focused monitoring process during fiscal year
              2002. The monitoring visits will be based on a performance matrix, and school
              improvement data will be reviewed during those visits. Such monitoring will allow
              the OESE to identify weaknesses and call for improvements in a state’s data quality
              controls, as the audits summarized in this report have done.

              The OESE does not require states to validate the performance data they submit by
              requiring the appropriate state official to attest that the data reported on the
              Consolidated State Performance Report are reliable and valid. This attestation could
              provide a significant additional check on the quality of state data.

              The Single Audit Act of 1984 established requirements for audits of states, local
              governments, and Indian tribal governments that administer federal financial
              assistance programs. The OMB provides a Compliance Supplement with guidance to
              assist auditors in determining compliance requirements relevant to the single audit.
              However, Education has not included steps to review school improvement data quality
              controls in this supplement.

              In addition, at the time of this audit, Education had not distributed to SEAs its draft of
              data quality standards so that SEAs and LEAs could use these standards as a guide.

              An additional audit observation is that the State ESEA Title I Participation
              Information for 1997–98 and 1998–99: Final Summary Reports do not identify the
              states that changed their assessment systems. A change in a state’s assessment system
              used to measure schools’ performance may affect the number of schools identified as
              needing improvement. As a result, year-to-year comparisons of the data, without
              adequate disclosure of a change in the state’s assessment system, could lead a reader
              or decision-maker to draw incorrect conclusions or to question the validity and
              reliability of the data. We suggest that PES identify in the report the states that
              change assessment systems from one period to the next.

              The recently passed No Child Left Behind Act of 2001 strengthens school
              accountability for student performance and defines consequences for schools that do
              not make adequate yearly progress. This increases the importance of reliable, valid
              accountability data. Improving management controls over the quality of state
              accountability data will help to ensure greater data quality.



                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 35
             To ensure that Title I School improvement data are reliable and valid, Education’s
             Assistant Secretary for Elementary and Secondary Education and the Undersecretary
             should:

             •        Distribute Education’s data quality standards to SEAs and encourage them to
                      provide the standards to LEAs.
             •        Develop and implement written procedures to assess during monitoring visits
                      to SEAs and LEAs whether school improvement data are reliable and valid.
             •        Include audit procedures in the OMB Compliance Supplement to review
                      controls over Title I, Part A school improvement data at LEAs and SEAs
                      during annual single audits.


Section 6:
There Are Additional Data Collection Options for Education to
Consider

             This audit determined that the lateness, incompleteness, and inaccuracy of states’ Title
             I data in their Consolidated State Performance Reports prevent Education from
             providing timely evaluations of the Title I, Part A programs for use by states and the
             U.S. Congress.

             Some state Title I officials we interviewed noted that a major intention and effect of
             the enabling legislation for title programs is the integration of funding and service
             delivery across many instructional programs for many different types of at-risk
             students. According to these state officials, the Title I, Part A tables in the
             Consolidated State Performance Report require an arbitrary disaggregation of all
             information that has been integrated. The report requires input and output indicators
             and measures for each title program that may not accurately capture the variables or
             allow evaluation of overall performance and desirable outcomes for all programs
             combined.

             In addition, reporting for schoolwide programs covers all students, including gifted
             and talented students and other categories of students who are not at risk. Title I, Part
             A funds are awarded to schools with high percentages of economically disadvantaged
             students and, although this category overlaps with at-risk students, it is not necessarily
             the same. According to some Title I officials, this blurring of inputs results in a
             blurring of the effects of Title I, Part A funding and programs, which makes it equally
             difficult to attribute outcomes to specific programs. Therefore, the present reporting
             system limits the validity of conclusions about performance of both the overall Title I,
             Part A program and its individual components.

             Some state Title I officials we interviewed also noted that it would be helpful to have
             defined overall educational outcomes and then measure each program against those
             outcomes. This would allow a more meaningful evaluation of the effects of federal
             funding on student learning. According to these state officials, the previous ESEA
             and the No Child Left Behind Act, signed into law in January 2002, do not clearly
             define or quantify expected outcomes and their measures. The officials believe the




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 36          SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             AUGUST 2002
              No Child Left Behind Act leaves program outcomes, measure definitions, and
              reporting requirements to Education to define by rule.

              Education is currently developing rules for the consolidated state applications for title
              funding that give the states the flexibility to develop their own performance targets for
              measuring progress. However, these targets and reporting requirements would
              supplement Education’s requirements for all 50 states to report on numerous variables
              and indicators for all the complex federal programs for public education. States are
              not measuring indicators consistently, which makes it impossible to draw conclusions
              about performance nationwide. Under the new law, limited capacity to draw
              nationwide conclusions will be offered by a new requirement that states participate in
              the National Assessment of Educational Progress (NAEP) for fourth- and eighth-grade
              reading and math.

              State Title I officials in one large state we audited in detail compare Education’s
              reporting requirements with those of another report on state compensatory education
              (which complements Title I, Part A programs). The state legislature defined in law
              the expected outcomes and the measurements required to report on those outcomes.
              Reporting is simple and straightforward and is not unduly burdensome on campus,
              LEA, or SEA personnel. It serves the state legislature’s purposes in evaluating the
              effect of state funding on improving the academic performance of at-risk students.

              State Title I officials suggest two alternatives to the burdensome reporting
              requirements of the Consolidated State Performance Report:

              •        Education could evaluate how individual states are and are not meeting the
                       objectives of the federal law, instead of attempting to aggregate state results to
                       a national summary level. This would allow Congress to identify successes
                       and weaknesses in the program as enabled by statute and implemented by
                       states.
              •        To evaluate program performance nationwide, Education could use sampling
                       (stratified if necessary) to evaluate program performance nationwide. This
                       would be an alternative to gathering all program information from all states,
                       which is currently the method for assessing the national effect of student
                       performance through the NAEP, a 30-year longitudinal study using a small
                       statistical sample from each state.




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                  PAGE 37
Appendix 1:
Objective, Scope, and Methodology

              The overall objective of this audit was to determine whether data that states reported
              on Title I, Part A program performance were accurate, complete, valid, and timely.
              We conducted the audits resulting in this joint report as part of a project of the U.S.
              Comptroller General’s Domestic Working Group. Our goal was to examine the
              quality of the data used for identifying Title I schools in need of improvement. The
              participants in this joint effort were the U.S. General Accounting Office (GAO), the
              U.S. Department of Education’s Office of Inspector General (ED-OIG), the Texas
              State Auditor’s Office (SAO), the Pennsylvania Department of the Auditor General,
              and the Office of the City Controller, Philadelphia.

              Each participant defined a specific role in the audit:

              •        The GAO surveyed all states and conducted detailed interviews with several
                       regarding their experiences in implementing major provisions of Title I.
              •        The ED-OIG conducted an assessment of data quality at the state and local
                       levels in California and conducted additional work on control processes at the
                       Department of Education.
              •        The SAO conducted a detailed assessment of data quality at the state and local
                       levels in Texas.
              •        The Pennsylvania Department of the Auditor General assessed data at the
                       state level, and the Philadelphia City Controller’s Office pursued the same
                       goal within the city of Philadelphia.

              Our methodology included the following procedures:

              •        Interviews with Education and state and local public education officials
              •        Interviews with officials at Education’s regional assistance centers
              •        Interviews with officials at the Council of Chief State School Officers
              •        Survey of state Title I directors
              •        Survey of a state’s LEA public education information managers
              •        Reviews of legislation, rules, and regulations
              •        Review of peer reviews and reports completed or commissioned by
                       Education’s Planning and Evaluation Service
              •        Review of state consolidated plans and performance reports
              •        Review of relevant previous audit reports
              •        Review of major manual and automated information systems
              •        Review of state and local policies and procedures governing data collection,
                       processing, and reporting
              •        Testing of data quality at campuses, LEAs, and SEAs




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 38           SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA               AUGUST 2002
              •        Analysis of Title I data reported by SEAs and LEAs
              •        Analysis of data quality and security controls
              •        Coordination among our audit partners who provided information relative to
                       their states’ activities

              Our criteria included the following:

              •        State and federal law and regulations
              •        Education and SEA data quality definitions
              •        SEA rules and regulations
              •        SEA standards, requirements, policies, and procedures




                  A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002       SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA            PAGE 39
Appendix 2:
Joint Audit Reports and Contact Information

              U.S. General Accounting Office

              Title I: Education Needs to Monitor States’ Scoring (General Accounting Office,
              GAO-02-393, April 2002)

              Contact: Marnie Shaul, Director, Education, Workforce, and Income Security

              Telephone: (202) 512-7215         E-mail: shaulm@gao.gov

              Web site: http://www.gao.gov

              U.S. Department of Education

              Improving Title I Data Integrity for Schools Identified for Improvement (U.S.
              Department of Education Office of Inspector General, ED-OIG/A03 – B0025, March
              2002)

              Contact: Bernard Tadley, Regional Inspector General for Audit - Philadelphia

              Telephone: (215) 656-6279         E-mail: mshaul@gao.gov

              Web site: http://www.ed.gov/offices/OIG/Areports.htm

              Pennsylvania Department of Education

              An Audit of the Pennsylvania Department of Education (PDE) Title I Performance
              Data (Pennsylvania Department of the Auditor General, June 2002)

              Contact: Richard Jordan, Director, Bureau of Federal Audits

              Telephone: (717) 783-2858         E-mail: Richard_Jordan@auditorgen.state.pa.us

              Web site: http://www.auditorgen.state.pa.us

              School District of Philadelphia

              An Audit of the School District of Philadelphia’s Management Controls for Compiling
              and Reporting Performance Data that Measured the Effectiveness of the Title I, Part
              A Program in Philadelphia for the 1999–2000 School Year (Office of the City
              Controller, Philadelphia, April 2002)

              Contact: Albert Scaperotto, Deputy Controller

              Telephone: (215) 686-6684         E-mail: Albert.Scaperotto@phila.gov

              Web site: http://www.philadelphiacontroller.org




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 40          SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA         AUGUST 2002
              Texas Accountability Information System

              An Audit Report on the Quality of the State’s Public Education Accountability
              Information (Texas State Auditor’s Office, SAO Report No. 02-044, May 2002)

              Contact: Carol Smith, Audit Manager

              Telephone: (512) 936-9500                E-mail: csmith@sao.state.tx.us

              Web site: http://www.sao.state.tx.us/Reports

              California Department of Education

              California Department of Education Needs to Report Reliable and Valid Data on Title
              I Schools Identified for Improvement (U.S. Department of Education Office of
              Inspector General, ED-OIG/A09-B0019, February 2002)

              Contact: Gloria Pilotti, Regional Inspector General for Audit - Sacramento

              Telephone: (916) 930-2399                E-mail: gloria.pilotti@ed.gov

              Web site: http://www.ed.gov/offices/OIG/Areports.htm

              California Department of Education

              California Department of Education’s Management Controls Over Performance Data
              for Identifying Title I Schools for Improvement [(U.S. Department of Education Office
              of Inspector General, ED-OIG/A09-C0002, March 2002) and (Corrected Response
              Letter April 2002)]

              Contact: Gloria Pilotti, Regional Inspector General for Audit - Sacramento

              Telephone: (916) 930-2399                E-mail: gloria.pilotti@ed.gov

              Web site: http://www.ed.gov/offices/OIG/Areports.htm




                 A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002      SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA             PAGE 41
Appendix 3:
Accountability and Assessment Requirements Under the 1994 and
2001 Reauthorizations of Title I


                     1994 Requirements                                     2001 Requirements

    Developing standards for content and performance

    Develop challenging standards for what students         In addition, develop standards for science
    should know in math and reading or language             content by 2005. The same standards must be
    arts. In addition, for each of these standards,         used for all children.
    states should develop performance standards
    representing three levels: partially proficient,
    proficient, and advanced. The standards must be
    the same for all children. If the state does not
    have standards for all children, it must develop
    standards for Title I children that incorporate the
    same skills, knowledge, and performance
    expected of other children.

    Implementing and administering assessments
    Develop and implement assessments aligned with          Add assessments aligned with the content and
    the content and performance standards in at             performance standards in science by the 2007–
    least math and reading or language arts.                08 school year. These science assessments must
                                                            be administered at some time in each of the
                                                            following grade ranges: from grades 3 through
                                                            5, 6 through 9, and 10 through 12.
    Use the same assessment system to measure Title I       Use the same assessment system to measure
    students as the state uses to measure the               Title I students as the state uses to measure the
    performance of all other students. In the absence       performance of all other students. If the state
    of a state system, a system that meets Title I          provides evidence to the secretary that it lacks
    requirements must be developed for use in all Title     authority to adopt a statewide system, it may
    I schools.                                              meet the Title I requirement by adopting an
                                                            assessment system on a statewide basis and
                                                            limiting its applicability to Title I students or by
                                                            ensuring that the Title I local educational
                                                            agency (LEA) adopts standards and aligned
                                                            assessments.
    Include in the assessment system multiple               Unchanged
    measures of student performance, including
    measures that assess higher-order thinking skills
    and understanding.




                       A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 42                SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA               AUGUST 2002
                     1994 Requirements                                      2001 Requirements

    Administer assessments for math and reading in           Administer reading and math tests annually in
    each of the following grade spans: from grades 3         grades 3 through 8, starting in the 2005–06
    through 5, 6 through 9, and 10 through 12.               school year (in addition to the assessments
                                                             previously required sometime within grades 10
                                                             through 12).
                                                             States do not have to administer math and
                                                             reading or language arts tests annually in
                                                             grades 3 through 8 if Congress does not provide
                                                             specified amounts of funds to do so, but states
                                                             have to continue to work on the development
                                                             of the standards and assessments for those
                                                             grades.
                                                             Have students in grades 4 and 8 take the
                                                             National Assessment for Educational
                                                             Performance (NAEP) exams in reading and
                                                             math every other year beginning in 2002–03, as
                                                             long as the federal government pays for it.
    Implement controls to ensure the quality of the          Unchanged
    data collected from the assessments.

    Including students with limited English proficiency and with disabilities in assessments
    Assess students with disabilities and limited English    By 2002–03, annually assess the language
    proficiency according to standards for all other         proficiency of students with limited English
    students.                                                proficiency. Students who have attended a U.S.
                                                             school for 3 consecutive years must be tested in
    Provide reasonable adaptations and
                                                             English unless an individual assessment by the
    accommodations for students with disabilities or
                                                             district shows testing in a native language will be
    limited English proficiency, to include testing in the
                                                             more reliable.
    language and form most likely to yield accurate
    and reliable information on what they know and
    can do.

    Reporting data
    Report assessment results according to the               Unchanged
    following: by state, LEA, school, gender, major
    racial and ethnic groups, English proficiency,
    migrant status, disability, and economic
    disadvantage.

    LEAs must produce for each Title I school a              Provide annual information on the test
    performance profile with disaggregated results           performance of individual students and other
    and must publicize and disseminate these to              indicators included in the state accountability
    teachers, parents, students, and the community.          system by 2002–03. Make this annual information
    LEAs must also provide individual student reports,       available to parents and the public and include
    including test scores and other information on the       data on teacher qualifications. Compare high-
    attainment of student performance standards.             and low-poverty schools with respect to the
                                                             percentage of classes taught by teachers who
                                                             are "highly qualified," as defined in the law, and
                                                             conduct similar analyses for subgroups listed in
                                                             previous law.




                        A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
AUGUST 2002             SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 PAGE 43
                      1994 Requirements                                         2001 Requirements

    Measuring improvement
    Use performance standards to establish a                      In addition to showing gains in the academic
    benchmark for improvement referred to as                      achievement of the overall school population,
    "adequate yearly progress." All LEAs and schools              schools and districts must show that the
    must meet the state's adequate yearly progress                following subcategories of students have made
    standard, for example, having 90 percent of their             gains in their academic achievement: pupils
    students performing at the proficient level in                who are economically disadvantaged, have
    math. LEAs and schools must show continuous                   limited English proficiency, are disabled, or
    progress toward meeting the adequate yearly                   belong to a major racial and ethnic group. To
    progress standard. The state defines the level of             demonstrate gains among these subcategories
    progress a school or LEA must show. Schools that              of students, school districts measure their
    do not make the required advancement toward                   progress against the state's definition of
    the adequate yearly progress standard can face                adequate yearly progress.
    consequences, such as the replacement of the
                                                                  States have 12 years for all students to perform
    existing staff.
                                                                  at the proficient level.

    Consequences for not meeting the adequate yearly progress standard
    LEAs are required to identify for improvement any             New requirements are more specific as to what
    schools that fail to make adequate yearly                     actions an LEA must take to improve failing
    progress for 2 consecutive years and to provide               schools. Actions are defined for each year the
    technical assistance to help failing schools                  school continues to fail leading up to the 5th
    develop and implement required improvement                    year of failure when a school must be
    plans. After a school has failed to meet the                  restructured by changing to a charter school,
    adequate yearly progress standard for 3                       replacing school staff, or state takeover of the
    consecutive years, LEAs may take corrective                   school administration. The new law also provides
    action to improve the school.                                 that LEAs offer options to children in failing
                                                                  schools. Depending on the number of years a
                                                                  school has been designated for improvement,
                                                                  these options may include going to another
                                                                  public school with transportation paid by the
                                                                  LEA or using Title I funds to pay for supplemental
                                                                  tutoring services.
    Source: U.S. General Accounting Office, P.L. 103–382 and P.L. 107–110.




                         A JOINT AUDIT REPORT ON THE STATUS OF STATE STUDENT ASSESSMENT
PAGE 44                  SYSTEMS AND THE QUALITY OF TITLE 1 SCHOOL ACCOUNTABILITY DATA                 AUGUST 2002
Copies of this report have been distributed to the following:


Legislative Audit Committee
The Honorable James E. “Pete” Laney, Speaker of the House, Chair
The Honorable Bill Ratliff, Lieutenant Governor, Vice Chair
The Honorable Rodney Ellis, Senate Finance Committee
The Honorable Florence Shapiro, Senate State Affairs Committee
The Honorable Robert Junell, House Appropriations Committee
The Honorable Rene O. Oliveira, House Ways and Means Committee

Office of the Governor
The Honorable Rick Perry, Governor

Members of the Audit Collaboration*
U.S. General Accounting Office
U.S. Department of Education, Office of the Inspector General
Pennsylvania Department of the Auditor General
Office of the City Controller, Philadelphia

State Board of Education
Ms. Grace Shore, Chair
Mrs. Geraldine Miller, Vice Chair
Ms. Alma A. Allen, Ed.D.
Mrs. Mary Helen Berlanga
Mr. Joe J. Bernal, Ph.D.
Mr. David Bradley
Dr. Don McLeroy
Mr. Dan Montgomery
Dr. Richard B. Neill
Mr. Rene Nunez
Ms. Rosie Collins Sorrells, Ed.D.
Mrs. Judy Strickland
Ms. Cynthia A. Thornton, Secretary
Mr. Chase Untermeyer
Mr. Richard Watson

Texas Education Agency
Dr. Felipe Alanis, Commissioner of Education



* Collaborative members will distribute according to their respective procedures.
This document is not copyrighted. Readers may make additional copies of this report as
needed. In addition, most State Auditor’s Office reports may be downloaded from our Web
site: www.sao.state.tx.us.

In compliance with the Americans with Disabilities Act, this document may also be requested
in alternative formats. To do so, contact Production Services at (512) 936-9880 (Voice), (512)
936-9400 (FAX), 1-800-RELAY-TX (TDD), or visit the Robert E. Johnson Building, 1501 North
Congress Avenue, Suite 4.224, Austin, Texas 78701.

The State Auditor’s Office is an equal opportunity employer and does not discriminate on the
basis of race, color, religion, sex, national origin, age, or disability in employment or in the
provision of services, programs, or activities.

To report waste, fraud, or abuse in state government call the SAO Hotline: 1-800-TX-AUDIT.