oversight

Best Practices for Management Controls Over Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001

Published by the Department of Education, Office of Inspector General on 2004-02-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                           UNITED STATES DEPARTMENT OF EDUCATION 

                                                OFFICE OF INSPECTOR GENERAL 

                                                                     REGION V 

                                                             III NORTH CANAL, SUITE 940 

                                                               CHICAGO, ILLINOIS 60606 


                                                                    FAX: (312) 353-0244
    Audit                                                                                                                                     Investigation
(312) 886-6503                                                                                                                               (312) 353-7891




             Management Information Report
             STATE AND LOCAL NO. 04-01

             DATE:                 February 03, 2004

             TO:            Raymond J. Simon
                           ri?~ Elementary and Secondary Education


                 FROM:             Richar.·
                                   Regional Inspector General for Audit

                 SUBJECT:          Best Practices for Management Controls Over Scoring of the State
                                   Assessments Required Under the No Child Left Behind Act of2001


                 This Management Information Report presents information that the U.S. Department
                 of Education (Department) may find beneficial in helping states to ensure data quality
                 and reliability. Our review objectives were to (1) identify any current management
                 controls required by the Department over scoring of the state assessments under the No
                 Child Left Behind Act of2001, and (2) provide the Department information on the types
                 of management controls over scoring used for the National Assessment of Educational
                 Progress (NAEP) and the benefits of having standards for a minimum level of controls.

                                                            REVIEW RESULTS

                 The Department should consider developing and issuing best practices for management
                 controls over scoring of state assessments required under the No Child Left Behind Act of
                 2001, such as those used for NAEP, by states that it has identified as having cost effective
                 controls, or the Model Contractor Standards and State Responsibilities for State Testing
                 Programs. While the Department has issued Information Quality Guidelines, it has not
                 issued specific guidance regarding the need for management controls over scoring of
                 state assessments. The No Child Left Behind Act of2001, Public Law 107-110, enacted
                 January 8, 2002, places more emphasis on the accountability of results and defines
                 consequences for schools that do not make adequate yearly progress. Interviews with
                 officials from the Office of Elementary and Secondary Education (OESE), NCS Pearson
                 (NCS), Educational Testing Service (ETS), our NAEP audit work, and review of prior
                 audit reports indicate a need for the Department to consider developing and issuing best
                 practices for management controls over scoring of state assessments.



                       Our mission Is to promote the efficiency, effectiveness, and integrity of the Department's programs and operations.
MANAGEMENT INFORMATION REPORT                                         STATE AND LOCAL NO. 04-01




The No Child Left Behind Act of 2001, Public Law 107-110, Section 1111(b)(3)(C)(iii),
indicates state assessments shall be used for purposes for which such assessments are
valid and reliable, and be consistent with relevant, nationally recognized professional and
technical standards. In our opinion, for the assessment to be reliable, the data should be
accurate.

Sound business practices require a minimum level of management controls over scoring
the assessments conducted in response to the No Child Left Behind Act of 2001.
Management controls over scoring should increase data quality and data reliability, which
are important aspects of the Government Performance and Results Act. The design of
each state’s assessment should include some minimum level of management controls
over scoring to ensure data quality and reliability. The types of controls would vary by
the type of assessment and scoring process (electronic or manual) and type of question
(multiple-choice and/or constructed response). These controls should cover the receipt
and control, data quality, scoring, and analysis processes. The receipt and control process
would cover the front end of the scoring process. For example, were all the assessments
administered accounted for and available for scoring? The data quality process would
cover (1) quality assurance steps taken to ensure all assessment booklets were properly
entered into the scoring process and the questions were tied to the correct answer key, (2)
monitoring during the scoring process, and (3) data quality assurance steps during and
after scoring. The scoring process for constructed response type questions would
consider controls such as scorer qualifications or interrater reliability (a measure of scorer
consistency). The analysis process would cover quality assurance steps designed to
ensure that the data analysis was performed properly and that data abnormalities were
identified and resolved. For example, plausibility checks could be performed to compare
data to expectations, historical precedent, and data obtained through other analysis
methods to make sure the results make sense.

The Department and States Could Benefit From Developing and Issuing Best
Practices for Management Controls Over Scoring State Assessments

Officials from OESE informed us that the Department does not require any management
control standards over scoring of state assessments because it lacks the statutory and
regulatory authority to require standards. All states have gone through on-site
compliance reviews at least once based on the 1994 reauthorization standards. The
Department is currently updating its Peer Reviewer Guidance for Evaluating Evidence of
Final Assessments Under Title I of the Elementary and Secondary Education Act to
incorporate aspects for the No Child Left Behind Act of 2001. The purpose of this
guidance is (1) to inform states of useful evidence that complies with Title I of the
Elementary and Secondary Education Act of 1965, as amended (Title I), final assessment
requirements; and (2) to guide teams of peer reviewers who will examine evidence
submitted by the states and advise the Department on whether a state has met Title I
requirements. The intent of these requirements is to help states develop comprehensive
assessment systems that provide accurate and valid information for holding districts and
schools accountable for student performance against state standards. While the guidance


February 2004                 Best Practices for Management Controls Over                   Page 2 of 8
         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                                         STATE AND LOCAL NO. 04-01


requires peer reviewers to examine the evidence that states’ administration, scoring,
analysis, and reporting procedures meet high standards, it leaves this qualitative judgment
to the reviewer. Developing and issuing best practices for management controls over
scoring would assist the reviewer and help the states develop assessment systems that
provide accurate and valid information. The Department issued the Information Quality
Guidelines to the states in February 2003 to provide policy and procedural guidance for
maximizing the quality, objectivity, utility, and integrity of the information that it
disseminates. Our review of the guidelines disclosed that they addressed high-level
standards. The guidelines did not require management controls over scoring of state
assessments. In general, the guidelines sought to have the states take steps to ensure that
the data submitted to the Department was accurate and reliable. The method used to
ensure data accuracy and reliability was left to the states to determine. Similar to the
Information Quality Guidelines, the Department has the authority to issue guidelines for
management controls over scoring.

Management Controls Over Scoring State Assessments Would Address Known and
Potential Problems

Our NAEP audit work and prior audit work performed by the Office of Inspector General
(OIG), General Accounting Office (GAO), and Texas State Auditor’s Office have
identified the need for states to improve their data quality and reliability controls. We
performed audit work to determine whether management controls over scoring of the
NAEP 2000 assessment were in place and adequate to provide reasonable assurance that
the assessment results could be relied on during the period October 1, 1999, through
September 30, 2000. We determined that the management controls over scoring of the
NAEP 2000 assessment were adequate and generally working as intended.

However, discussions with ETS and NCS officials, as part of that audit, indicated
management controls over scoring state assessments varied from state to state and there
were no uniform minimum levels of management controls. States apply fewer and less
complex management controls over scoring compared to NAEP. State funding,
timelines, and the complexity of the assessments affected each state’s use of management
controls over scoring. State assessments generally required fewer management controls
over scoring than NAEP because they tended to be less complicated and easier to score.
States used more multiple-choice questions to lower the cost, the time required, and the
risk of error. While the scope of the NAEP assessment is different than the scope of state
assessments, many of the NAEP management controls could be used as a model of best
practices for controls over scoring of the state assessments. (See the Appendix to this
report for additional information on NAEP 2000 audit work.)

Prior audits by OIG and GAO identified the need for the Department to (1) improve its
management controls over school improvement data to ensure the data are reliable, valid,
and timely; (2) monitor state scoring of assessments; and (3) facilitate the sharing of
information on experiences to reduce state expenses. Prior audit work performed by the
OIG and GAO disclosed that the monitoring methods states use to ensure the accuracy of
test contractors’ scoring and reporting do not always provide adequate assurance of


February 2004                 Best Practices for Management Controls Over                   Page 3 of 8
         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                                         STATE AND LOCAL NO. 04-01


complete and accurate scoring results. Most states hire a contractor to score Title I
assessments. About one-third of these states did not monitor the scoring performed by
the contractor. Some states that hire contractors have found errors in the contractors’
scoring, and in some cases, these errors have had serious negative consequences for
schools and students. Several states reported problems with scoring and calculation
errors, regardless of whether they had monitoring procedures in place. These states
reported that local district officials, parents, and state agency staff discovered the errors.
(See the Appendix to this report for additional information on review of prior audit
reports.)

Scoring contractors’ errors can result in serious negative consequences for schools and
students and lead to costly litigation and negative public opinion. Scoring errors could
potentially jeopardize the successful implementation of the No Child Left Behind Act of
2001. Reports that include scoring errors would erroneously tell parents, communities,
educators, and boards of education that schools are or are not doing well. Scoring errors
may also have an effect on adequate yearly progress reported to the Department by the
states. The Department uses adequate yearly progress in its decision making process,
which could affect funding decisions made by states related to needed improvement,
corrective action, or restructuring measures.

School districts and schools that fail to make adequate yearly progress toward statewide
proficiency goals will, over time, be subject to improvement, corrective action, and
restructuring measures aimed at getting them back on course to meet state standards.
Schools that meet or exceed adequate yearly progress objectives or close achievement
gaps will be eligible for State Academic Achievement Awards. (See Background for
additional information on school improvement, corrective action, and restructuring
measures.)

The Education Leaders Council and Accountability Works have assisted participating
states in developing Model Contractor Standards and State Responsibilities for State
Testing Programs (Model) to communicate expectations for the development and
administration of state testing programs. The Model indicates that with high school
diplomas, monetary awards for schools and school systems dependent on test results, it is
imperative that state assessments be of high quality, meet professional standards for best
practice, be delivered in a timely manner, and be scored accurately. With increasingly
tight budgets, it is similarly imperative that assessment programs be developed and
implemented in an efficient and cost effective manner without sacrificing quality.

Suggestion for Enhancing Management Controls over Scoring of State Assessments

We suggest that OESE consider developing and issuing best practices for management
controls over scoring of state assessments required under the No Child Left Behind Act of
2001, such as those used for NAEP, states that OESE has identified as having cost
effective controls, or the Model Contractor Standards and State Responsibilities for State
Testing Programs.



February 2004                 Best Practices for Management Controls Over                   Page 4 of 8
         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                                         STATE AND LOCAL NO. 04-01





                                       BACKGROUND

No Child Left Behind Act of 2001

On January 8, 2002, President Bush signed into law the No Child Left Behind Act of
2001. The Act contains the President’s four basic education reform principles: stronger
accountability for results, increased flexibility and local control, expanded options for
parents, and an emphasis on teaching methods that have been proven to work. An
“accountable” education system involves several critical steps. These steps generally
require states to (1) create standards, (2) test students’ progress toward the standards, and
(3) hold schools, districts, and themselves accountable for making adequate yearly
progress toward meeting the state’s standards. They also provide that performance will
be publicly reported in district and state report cards, and that there will be consequences
for districts and schools that continually fail to make adequate yearly progress toward the
standards.

School districts and schools that fail to make adequate yearly progress toward statewide
proficiency goals will, over time, be subject to improvement, corrective action, and
restructuring measures aimed at getting them back on course to meet State standards.
Schools that meet or exceed adequate yearly progress objectives or close achievement
gaps will be eligible for State Academic Achievement Awards. The No Child Left
Behind Act of 2001 significantly strengthens the school improvement provisions under
section 1116 of Title I.

School Improvement (schools that fail to make adequate yearly progress for two
consecutive years are subject to improvement measures)

The No Child Left Behind Act of 2001, section 1116(b) establishes the School
Improvement requirements.

ƒ   Requires schools identified for improvement to develop two-year improvement plans
    incorporating strategies from scientifically-based research on how to strengthen the
    core academic subjects and address the specific issues that caused the school to be
    identified for improvement.

ƒ   Requires schools identified for improvement to reserve annually at least 10 percent of
    their Title I, Part A, funds for professional development that directly addresses the
    problems that led to identification for improvement.

ƒ   Requires local education agencies (LEA) to immediately provide students attending
    schools identified for improvement the option of attending another public school,
    including a public charter school, that is not identified for improvement. LEAs must
    provide or pay for transportation to the new school, within certain limits (see 20
    percent requirement below).



February 2004                 Best Practices for Management Controls Over                   Page 5 of 8
         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                                          STATE AND LOCAL NO. 04-01


ƒ   Permits students attending schools in the second year of school improvement (failure
    to make adequate yearly progress for three consecutive years) to use Title I funds to
    obtain supplemental educational services from a State approved public- or private-
    sector provider of their choice. The school improvement provisions cap the per-child
    cost of such services at the lesser of the LEA per-child Title I, Part A, allocation or
    the cost of services.

ƒ   Requires LEAs to “promptly” notify parents of eligible students attending schools
    identified for improvement, corrective action, or restructuring of their option to
    transfer their child to a better public school or to obtain supplemental educational
    services.

ƒ   Requires LEAs to give priority to low-achieving students from low-income families
    in making available choice and supplemental educational services. Only low-income
    children are eligible for supplemental services.

ƒ   Requires LEAs to use an amount equal to 20 percent of their Title I, Part A,
    allocations to pay for transportation of students exercising a choice option or
    obtaining supplemental educational services for eligible students. If an LEA reserves
    such funds from its Title I, Part A, allocation, the LEA may not reduce allocations to
    schools identified for corrective action or restructuring by more than 15 percent.

ƒ   Permits a student who transferred to another school under these provisions to remain
    in that school through its highest grade, but the LEA is required to provide
    transportation to the new school only as long as the student’s original school is
    subject to school improvement, corrective action, or restructuring.

Corrective Action

The No Child Left Behind Act of 2001, section 1116(b)(7) establishes the Corrective
Action requirements.

ƒ   Strengthens corrective action (required after four years of failing adequate yearly
    progress) to include actions more likely to bring about meaningful change at the
    school, such as replacing school staff responsible for the continued failure to make
    adequate yearly progress, comprehensive implementation of a new curriculum
    (including professional development), and reorganizing the school internally.
    Corrective action schools also must continue to provide choice and supplemental
    services options to their students.




February 2004                 Best Practices for Management Controls Over                   Page 6 of 8 

         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001

MANAGEMENT INFORMATION REPORT                                          STATE AND LOCAL NO. 04-01



Restructuring

The No Child Left Behind Act of 2001, section 1116(b)(8) establishes the Restructuring
requirements.

ƒ   Adds a new restructuring requirement for schools that fail to respond to corrective
    actions. If a school fails to make adequate yearly progress after one year of corrective
    action, it must begin planning for restructuring, which involves fundamental changes
    such as reopening the school as a public charter school, replacing all or most of the
    school’s staff, or turning operation of the school over to a private management
    company with a demonstrated record of effectiveness, and implement its restructuring
    plan the following year. Schools identified for restructuring also must continue to
    provide choice and supplemental services options to their students.

Government Performance and Results Act

This review falls under the context of the Government Performance and Results Act
(GPRA), specifically data quality and reliability. The Department’s Strategic Plan for
2002 – 2007, goals one and six, are related to our review objectives and GPRA
requirements.

Goal One, Create a Culture of Achievement, is related as it pertains to linking federal
education funding to accountability for results. The No Child Left Behind Act of 2001
will strengthen Title I accountability by requiring states to implement statewide
accountability systems covering all public schools and students. These systems must be
based on challenging state standards in reading and mathematics, annual testing for all
students in grades 3 through 8, and once in grades 10 through 12, and annual statewide
progress objectives ensuring that all students reach proficiency within 12 years.
Assessment results and state progress objectives must be broken out by poverty, race,
ethnicity, disability, and limited English proficiency to ensure that no group is left
behind. School districts and schools that fail to make adequate yearly progress toward
statewide proficiency goals will, over time, be subject to improvement, corrective action,
and restructuring measures aimed at getting them back on course to meet state standards.
Schools that meet or exceed adequate yearly progress objectives or close achievement
gaps will be eligible for State Academic Achievement Awards.

Goal Six, Establish Management Excellence, is related in two ways. First, developing
and maintaining management controls, in this case, controls over scoring state
assessments, establish excellence. Second, achieving budget and performance integration
by linking funding decisions to results, establishes excellence. As indicated previously,
schools that fail to make adequate yearly progress will be subject to improvement,
corrective action, and restructuring measures, which could affect the state’s funding
decisions.




February 2004                 Best Practices for Management Controls Over                   Page 7 of 8 

         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001

MANAGEMENT INFORMATION REPORT 	                                       STATE AND LOCAL NO. 04-01



                OBJECTIVE, SCOPE, AND METHODOLOGY
The objectives of our review were to (1) identify any current management controls
required by the Department over scoring of the state assessments under the No Child Left
Behind Act of 2001 and (2) provide the Department information on the types of
management controls over scoring used for NAEP and the benefits of having standards
for a minimum level of controls. To accomplish our review objectives, we

1. 	    interviewed officials from OESE, NCS, and ETS to gain an understanding of
        current management control standards over scoring of the state assessments
        required under the No Child Left Behind Act of 2001,

2. 	    reviewed background materials, such as:
               a. 	  Prior OIG, GAO, and Texas State Auditor’s Office audit reports;
               b. 	  No Child Left Behind Act of 2001;
               c. 	  U.S. Department of Education Strategic Plan for 2002 – 2007;
               d. 	  U.S. Department of Education, Information Quality Guidelines;
               e. 	  Peer Reviewer Guidance For Evaluating Evidence of Final
                     Assessments Under Title I of the Elementary and Secondary Act,
                     dated November 1999;
               f. 	  Minnesota State Performance Report for the 2000-2001 school
                     year;
               g. 	  State of West Virginia Consolidated State Application
                     Accountability Workbook, approved April 11, 2003;
               h. 	  Consolidated State Performance Report for State Formula Grant
                     Programs under the Elementary and Secondary Education Act and
                     Goals 2000: Educate America Act, For reporting on School Years
                     1999-2000 and 2000-2001, dated October 26, 1999;
               i. 	  Planning and Evaluation Service, State ESEA Title I Participation
                     Information for 1999-2000, Final Summary Report, dated 2002;
               j. 	  Planning and Evaluation Service, State Education Indicators with a
                     Focus on Title I 1999-2000; and

3. 	    relied on prior OIG audit work, Review of Management Controls Over Scoring of
        the National Assessment of Educational Progress 2000, Audit Control Number:
        ED-OIG/A05-C0010.

We performed our fieldwork at OESE’s offices in Washington, DC, from April 29, 2003,
through May 1, 2003. We held an exit conference with OESE officials on June 16, 2003.
Our review was performed in accordance with established OIG standards found in
Inspector General Bulletin No. 91-4 (standards for performing services other than audits)
that are appropriate to the scope of the review described above.

No response from your office is necessary regarding the information contained herein. If
you would like to discuss the information presented in this memorandum or obtain
additional information, please contact me at (312) 886-6503.

February 2004                 Best Practices for Management Controls Over                   Page 8 of 8
         Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                  STATE AND LOCAL NO. 04-01                  APPENDIX


     ADDITIONAL MANAGEMENT CONTROL INFORMATION

NAEP 2000 Audit Work

We performed extensive audit work to determine whether management controls over
scoring of the NAEP 2000 assessment were in place and adequate to provide reasonable
assurance that the assessment results could be relied on during the period October 1,
1999, through September 30, 2000. Based on the work performed, we determined that
the management controls over scoring of the NAEP 2000 assessment were adequate and
generally working as intended. For additional details, see the OIG final audit report
entitled Review of Management Controls Over Scoring of the National Assessment of
Educational Progress 2000, issued June 2003, under Audit Control Number: ED-
OIG/A05-C0010.

Review of Prior Audit Reports

The prior audit reports that we reviewed included (1) OIG report, Improving Title I Data
Integrity for Schools Identified for Improvement, dated March 2002; (2) GAO report,
Education Needs to Monitor States' Scoring of Assessments, dated April 2002; (3) Texas
State Auditor’s Office report, The Quality of the State's Public Education Accountability
Information, dated May 2002; (4) Joint Audit Report on The Status of State Student
Assessment Systems and the Quality of Title I School Accountability Data, dated August
2002; and (5) GAO report, Title I - Characteristics of Tests Will Influence Expenses;
Information Sharing May Help States Realize Efficiencies, dated May 2003. The entire
reports are available on the OIG, GAO, and Texas State Auditor’s Office web sites
respectively, www.ed.gov/offices/oig/Areports.htm, www.gao.gov, and
www.sao.state.tx.us.

OIG Final Audit report, Improving Title I Data Integrity for Schools Identified for
Improvement, Audit Control Number ED-OIG/A03-B0025, dated March 2002,
recommended that the Department take steps to improve its management controls over
school improvement data to ensure the data are reliable, valid, and timely. The
Department generally concurred with the findings and recommendations.

GAO report, Education Needs to Monitor States' Scoring of Assessments, GAO-02-393,
dated April 2002, recommended that when the Department monitors state compliance
with federal programs, it include checks for contractor monitoring related to Title I, Part
A. Specifically, the Department should include in its new compliance reviews a check on
the controls states have in place to ensure proper test scoring and the effective
implementation of these controls by states. The Department agreed with GAO’s
recommendation.

Texas State Auditor’s Office audit report, The Quality of the State's Public Education
Accountability Information, Report No. 02-044, dated May 2002, identified specific
improvements that can be made to further enhance data collection, processing, and

February 2004             Best Practices for Management Controls Over                      Page 1 of 2
        Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001
MANAGEMENT INFORMATION REPORT                  STATE AND LOCAL NO. 04-01                  APPENDIX


reporting for student assessment records and results and for the calculation and reporting
of passing rates for accountability purposes. The audit report identified the need for
Texas to monitor the assessment test vendor’s data security and quality controls.
Management agreed with the recommendations.

The Joint Audit Report, on The Status of State Student Assessment Systems and the
Quality of Title I School Accountability Data, SAO Report No. 02-064, dated August
2002, indicated that monitoring methods states use to ensure the accuracy of test
contractors’ scoring and reporting do not always provide adequate assurance of complete
and accurate scoring results. Of the 44 states that hire contractors for test scoring, 16
(about one-third) have no monitoring mechanism to ensure the accuracy of the
contractor’s test scoring and reporting. Management concurred with most of the findings.

GAO report, Title I - Characteristics of Tests Will Influence Expenses; Information
Sharing May Help States Realize Efficiencies, GAO-03-389, dated May 2003, is a GAO
study of the costs associated with implementing the test required under the No Child Left
Behind Act of 2001. GAO recommended that the Department facilitate the sharing of
information on states' experiences in attempting to reduce expenses. The Department
agreed with GAO’s recommendation.




February 2004             Best Practices for Management Controls Over                      Page 2 of 2 

        Scoring of the State Assessments Required Under the No Child Left Behind Act of 2001