oversight

Data Quality Review of Washington Consolidated State Performance Reports

Published by the Department of Education, Office of Inspector General on 2006-11-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                       UNITED STATES DEPARTMENT OF EDUCATION 

                            OFFICE OF INSPECTOR GENERAL

                                             501 I STREET, SUITE 9-200
                                          SACRAMENTO, CALIFORNIA 95814

                                     PHONE (916) 930-2388 • FAX (916) 930-2390


                                                    November 14, 2006
                                                                                               Control Number
                                                                                               ED-OIG/A09G0009

Dr. Terry Bergeson
State Superintendent of Public Instruction
State of Washington
Office of the Superintendent of Public Instruction
Old Capitol Building, P.O. Box 47200
Olympia, WA 98504-7200

Dear Dr. Bergeson:

This Final Audit Report, entitled Data Quality Review of Washington Consolidated State
Performance Reports, presents the results of our audit. The purpose of the audit was to
determine whether graduation and dropout rates reported to the U.S. Department of Education
(Department) by the State of Washington’s Office of the Superintendent of Public Instruction
(WOSPI) in the Consolidated State Performance Reports for 2003-2004 and 2004-2005 are
supported by reliable data and meet the requirements of Section 1111(b)(2) of the Elementary
and Secondary Education Act of 1965, as amended, and 34 Code of Federal Regulations
§ 200.19 (a) and (c).




                                                   BACKGROUND 



The Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind
Act of 2001 (NCLB), requires states receiving ESEA, Title I, Part A funds to implement a
statewide accountability system. The accountability system must include a high school
graduation rate indicator – a required factor in determining whether local educational agencies
and high schools made adequate yearly progress in students’ achievement of state academic
standards.

States report graduation rates for all students and by student subgroups in the Consolidated State
Performance Report (CSPR) – the required annual state report to the Secretary. The CSPR
instructions require the state to report graduation rates computed in accordance with the
definition approved as part of the State’s accountability plan (i.e., approved Consolidated State
Application Accountability Workbook). The CSPR instructions also require states to report the
annual dropout rates for high school students using the National Center for Education Statistics’
definition of a high school dropout. States reported graduation and dropout rates for school year
2002-2003 in the 2003-2004 CSPR. Rates for school year 2003-2004 were reported in the
2004-2005 CSPR. The Department uses information in the CSPR to prepare reports on progress


          Our mission is to promote the efficiency, effectiveness, and integrity of the Department’s programs and operations.
ED-OIG/A09G0009                                                                        Page 2 of 21

in meeting strategic plan goals, individual program performance, and States’ progress in
implementing NCLB.

In the definition of “adequate yearly progress” at ESEA § 1111(b)(2)(C)(vi), Congress defined
graduation rate as “the percentage of students who graduate from secondary school with a
regular diploma in the standard number of years.” The implementing Federal regulation at
34 Code of Federal Regulations (C.F.R.) § 200.19(a)(1)(i) further defines the graduation rate to
mean—
       (A) The percentage of students, measured from the beginning of high school, who
           graduate from high school with a regular diploma (not including an alternative
           degree that is not fully aligned with the State’s academic standards, such as a
           certificate or a GED) in the standard number of years; or
       (B) Another definition, developed by the State and approved by the Secretary in
           the State plan, that more accurately measures the rate of students who
           graduate from high school with a regular diploma as defined in paragraph
           (a)(1)(i)(A) of this section.

The cited regulation also states that “[i]n defining [the] graduation rate, the State must avoid
counting a dropout as a transfer.” The regulation at 34 C.F.R. § 200.19(c) requires states to
ensure that academic indicators, such as the graduation rate, are valid and reliable; consistent
with relevant, nationally recognized professional and technical standards, if any; and consistent
throughout the State within each grade span.

In September 2005, WOSPI received approval from the Department to use an extended
graduation rate for adequate yearly progress (AYP) purposes rather than a rate based on students
who graduated within a standard number of years (i.e., on-time graduation rate). The extended
graduation rate includes students who graduated in more than the standard four years as
graduates in the calculation of the rate.

WOSPI received about $157 million and $171 million in Title I, Part A funds in fiscal years
2003 and 2004, respectively. In its 2003-2004 CSPR, WOSPI reported an on-time graduation
rate for all students of 65.7 percent and an annual dropout rate of 6.7 percent based on student
data for school year 2002-2003. In its 2004-2005 CSPR, WOSPI reported an on-time graduation
rate for all students of 70.1 percent, an extended graduation rate for all students of 74.0 percent,
and an annual dropout rate of 5.8 percent based on student data for school year 2003-2004.
Attachment 1 provides the formulas and calculations WOSPI used to compute the on-time and
extended graduation rates reported in the CSPRs.
ED-OIG/A09G0009 	                                                                     Page 3 of 21




                                     AUDIT RESULTS 



WOSPI complied with the ESEA requirement to provide an annual report to the Secretary by
submitting the 2003-2004 and 2004-2005 CSPRs, which included the State’s graduation and
dropout rates. However, student data used by WOSPI to calculate the reported rates was not
reliable and the required definition of a “high school dropout” was not used to calculate the
reported dropout rates. The OTHER MATTER section of this report presents our concerns with
the presentation in the State of Washington’s Consolidated State Application Accountability
Workbook of the State’s formulas for computing graduation rates for accountability purposes.

In its comments to the draft report, WOSPI concurred with our findings and agreed to implement
our recommendations. In response to the other matter, WOSPI submitted an amendment to the
State’s Accountability Workbook, but the amendment did not fully address our concerns about
the clarity of WOSPI’s description of the graduation rate used for AYP purposes. WOSPI’s
comments are summarized at the end of each finding. The full text of WOSPI’s comments on
the draft report are included as Attachment 2 to the report.


FINDING NO. 1 – W
                	 OSPI Used Unreliable Data to Calculate Graduation and
                Dropout Rates

WOSPI did not use reliable data to calculate graduation and dropout rates reported in the
2003-2004 and 2004-2005 CSPRs. WOSPI acknowledged that the data used to calculate rates
reported in the 2003-2004 CSPR was of questionable accuracy and subsequently took steps to
improve the data. Our review of the student records at three school districts found that data used
in calculating the rates reported in the 2004-2005 CSPR was also unreliable. WOSPI needs to
take additional actions to improve the accuracy of the data used in calculating the graduation and
dropout rates.

The ESEA § 1111(b)(2)(D) and the implementing regulations at 34 C.F.R. § 200.19(c) require
states to ensure high school graduation rates are reliable and consistent throughout the state
within each grade span. The Government Accountability Office’s Assessing the Reliability of
Computer Processed Data states data are reliable when they are (1) complete (they contain all of
the data elements and records needed for the engagement) and (2) accurate (they reflect the data
entered at the source or, if available, in the source documents). A subcategory of accuracy is
consistency. Consistency refers to the need to obtain and use data that are clear and well-defined
enough to yield similar results in similar analyses. For example, if data are entered at multiple
sites, inconsistent interpretation of data rules can lead to data that, taken as a whole, are
unreliable.

WOSPI used information from a statewide student-level data file to compute graduation and
dropout rates reported in the CSPR. The statewide data file for each school year was created
from data files submitted in the fall of each year by the State’s 249 school districts with high
schools. The individual district data files (referred to as P210 Reports) contained the enrollment
ED-OIG/A09G0009                                                                                   Page 4 of 21

status for students enrolled in the previous school year. To create the P210 report, the districts
extracted student data from the district’s student information system and converted the district’s
data codes to the State’s data codes. WOSPI staff performed various reasonableness,
completeness, and accuracy checks of data in the P210 reports.

WOSPI categorizes students in the 9th through 12th grade into the following enrollment
categories based on the student’s enrollment status code in the statewide data file:

           On-Time Graduate: Students who receive a diploma in the expected year of graduation.1

           Late Graduate: Students who receive a diploma after their expected year of graduation.

           Dropout: Students who intentionally leave school for any reason, except death, before
           receiving a diploma and who do not transfer to another school or educational program.
           Students whose status is unknown because they are no longer enrolled, but who are not a
           confirmed transfer or dropout. For purposes of graduation and dropout rate calculations,
           WOSPI includes students who receive General Education Development (GED)
           certificates in the dropout category.

           Transfer (confirmed): Students who depart school and whose transcripts/student records
           have been requested by a gaining school, or students who file an “Intent to Receive
           Home-Based Instruction.” Students who move outside the United States before
           completing high school graduation requirements are also considered confirmed transfers.
           For purposes of graduation and dropout rate calculations, WOSPI includes deceased
           students in the transfer category.2

           Continuing Student: Students who do not have an enrollment status of graduate or other
           type of completer (e.g., GED certificate), transfer, dropout, unknown, or deceased.
           A subgroup of this category is the “extended students,” who are 12th grade students who
           are enrolled beyond their expected graduation year.

The number of students enrolled in each grade (less transfers), the number of students in each
grade in the dropout category, the number of students in the extended student subgroup, and the
number of students in the on-time graduate and late graduate categories are used to calculate the
State’s graduation and dropout rates and, for accountability purposes, the graduation rates for
each school and district in the State of Washington. WOSPI issues an annual report on the
State’s graduation and dropout statistics that provides schools and districts with the actual
student data used to compute individual school and district graduation and dropout rates. The
report for school year 2003-2004 was issued in September 2005.




1
  “Graduate” includes students who receive a regular high school diploma, students who receive an adult diploma
from a community college program, and students with disabilities who complete their Individualized Education Plan
(IEP). For school year 2003-2004, the statewide data file included 244 adult diploma recipients and 120 IEP
completers.
2
    For school year 2003-2004, the statewide data file included 101deceased students.
ED-OIG/A09G0009 	                                                                      Page 5 of 21


WOSPI Had Concerns Regarding the
Reliability of Data Used to Calculate Rates
Reported in the 2003-2004 CSPR

WOSPI officials advised us that the data used to calculate the rates for school year 2002-2003
were not accurate, or consistent throughout the state, as required by the ESEA. In its report titled
Graduation and Dropout Statistics for Washington’s Counties, Districts, and Schools: School
Year 2002-03, dated September 2004, WOSPI stated that “some districts have found errors in the
information provided for some of their students” and that district and school “[r]ates that are
extremely high or low may also reflect inaccurate reporting.” Due to concerns about data
reliability, WOSPI took the following actions to improve the quality of data for school year
2003-2004.

   • 	 WOSPI published the P210 Collection Data Manual for the 2003-2004 School Year in
       August 2004. The manual used a new format to present data element definitions that
       identified the various parameters (e.g., data type, valid codes, code descriptions, business
       rules, and input examples) for each data field.

   • 	 WOSPI required districts to report an enrollment status code for every student. If the
       enrollment status code field was blank for any student, the district’s P210 file was
       returned for correction. Previously, the enrollment status code field defaulted to a
       continuing student status when the field was left blank.

   • 	 WOSPI provided additional training via video teleconference to district staff on the P210
       reports. Documentation showed WOSPI held four P210 training sessions (September and
       October 2004) for the school year 2003-2004 reporting period.

Since WOSPI acknowledged that the data for school year 2002-2003 were not accurate and had
subsequently taken steps to improve data, we focused our review on the reliability of the data
used to calculate the graduation and dropout rates for school year 2003-2004, which were
reported in the 2004-2005 CSPR.

WOSPI Also Used Unreliable Data to Calculate
Rates Reported in the 2004-2005 CSPR

Our review of student enrollment data in the statewide student-level data file for three school
districts (Seattle Public Schools, Spokane Public Schools, and Edmonds School District) found
that a group of students at Spokane Public Schools was not included in the statewide file and
student enrollment statuses in the statewide file were not accurate for the three districts.

Students on the Spokane Public Schools’ P210 Report Were Not Included in the Statewide
Student-level Data File. Spokane Public Schools’ staff provided us with its final P210 report for
school year 2003-2004. The P210 report included 94 students in the 12th grade who were in the
dropout category (i.e., enrollment statuses of dropouts, unknown, or GED completers). The
statewide file contained no 12th grade students in the dropout category for Spokane Public
Schools for school year 2003-2004. With the inclusion of the 12th grade dropouts, Spokane
  ED-OIG/A09G0009                                                                                        Page 6 of 21

  Public Schools’ on-time graduation rate for school year 2003-2004 would decrease from
  85.1 percent to 81.0 percent and its extended graduation rate would decrease from 86.4 percent
  to 82.3 percent.

  Student Enrollment Statuses in the Statewide Student-level Data File Were Not Accurate. Our
  review of school year 2003-2004 student data extracted from the statewide file for selected high
  schools in three school districts identified inaccurate or unsupported enrollment codes for
  141 of the 760 students in our sample (18.6 percent).

        Table 1: Results of Review of Student Enrollment Status for the School Year 2003-2004
                              for Selected High Schools in Three Districts
Enrollment Category Shown in    On-time    Late                                               Extended       Total
         Statewide File        Graduates Graduates Transfers Dropouts                         Students (a) Students
Number of Students in Category   2,979      60       1,462     430                               225       5,156 (b)
Number of Students Reviewed       303       60        208 (c)   90                                99          760
School Documentation Confirmed
                                  289       50        126       80                                  74           619
Enrollment Codes
Inaccurate Enrollment Codes       14        10         82       10                                  25           141
Error Rate                                4.6%          16.7%        39.4%         11.1%          25.3%         18.6%

                                            Description of Error Type
Enrollment Should Have Been
Coded as:
       Transfer                             ---           ---          ---           10           ---           10
       Graduate                             ---           ---          ---           ---          19            19
       Dropout                              ---           ---          82            ---           6 (d)        88
       Total                                ---           ---          82            10           25            117
Student Had Less than Minimum
                                            14            10           ---           ---          ---           24
Credit Hours to Graduate (e)
Total Errors                                14            10           82            10           25            141

(a) Continuing students that were enrolled beyond their expected graduation year.
(b) Total does not include continuing students who were enrolled in 9th, 10th, and 11th grade. This group was not
    included in our sampling universe. We considered this group to be at lowest risk for inaccurate enrollment statuses.
(c) Our computerized analysis of student data for the 1,462 transferred students confirmed that 840 of the students had
    been reported as enrolled by another school. We selected the sampled students from the remaining 622 students.
(d) The six errors did not impact graduation rates since extended students are included with dropouts in the calculation
    of the grade-level dropout rates used to derive the on-time and extended graduation rates.
(e) Each district established the minimum number of credits that a student needed for graduation. For students in the
    on-time and late graduate groups, we reviewed student records to confirm that students met the district’s minimum
    credits for graduation. We were unable to determine from the records whether the 24 students cited as errors
    received graduation diplomas.
ED-OIG/A09G0009 	                                                                      Page 7 of 21

While we were not able to identify the specific cause of each enrollment status error, the
following conditions and weaknesses in District and WOSPI processes may have caused or
contributed to the occurrence of the errors:

   • 	 The P210 Collection Data Manual for the 2003-2004 School Year (P210 Manual) was
       issued in August 2004 and included new enrollment status coding requirements
       applicable to the school year that ended in August 2004.

   • 	 Information technology and administrative staff at the three districts described the
       extraction and conversion process for creating the P210 Report as difficult.

   • 	 A reporting module in the Edmonds School District’s student information system, which
       was used to create the District’s P210 Report, did not show the proper enrollment status
       for students who took classes at more than one school in the district. Specifically, we
       noted that Edmonds’ student information system (WESPaC) identified 12th grade
       students enrolled full-time at one high school (home high school), who took at least one
       class at another high school, as continuing students at the end of 12th grade even if the
       student graduated from the home high school. In the State of Washington, 214 school
       districts use the WESPaC student information system, which was developed by the
       Washington School Information Processing Cooperative (WSIPC).

   • 	 Based on our interviews with school staff, we concluded that staff at seven of the nine
       high schools did not fully understand enrollment status definitions and documentation
       requirements. For example, staff entered a transfer status for students pursuing a GED
       and relied on statements from a responsible adult as evidence for a transfer. The P210
       Manual states that transfer codes are only used for confirmed transfers to another school
       (i.e., transcripts/records have been requested by another school or student filed an Intent
       to Receive Home-Based Instruction). The P210 Manual instructs districts to enter an
       unknown enrollment status code when the sole evidence is hearsay reports and
       unsubstantiated statements of intent to transfer. We also found that, for students with an
       unknown status in the district’s student information system, school staff did not change
       the enrollment status to transfer after the district received a records request from another
       school (i.e., confirmed transfer).

   • 	 School staff in Spokane Public Schools altered the graduation year of students that were
       not promoted to the next grade. WOPSI uses the graduation year, which is entered at the
       time the student enters high school, to identify students who graduated in the standard
       number of years. As a result of altering the graduation year, Spokane did not report any
       extended students in school year 2003-2004. On March 30, 2006, Spokane sent an email
       instructing school staff to not change the graduation year in the student record and to
       check student records and make changes, as necessary, to reflect the student’s original
       graduation year.

   • 	 Spokane Public Schools information technology staff inadvertently changed student
       records in the district’s student information system to show an enrollment status of
       graduate for students with less than 21 credits and a graduation year of 2004.
ED-OIG/A09G0009 	                                                                                       Page 8 of 21

    • 	 In the districts we reviewed, a school counselor determines whether a student has met the
        minimum credits for graduation based on information in the student’s school transcript.
        There was no independent review by a second person or edit check in the district’s
        student information system to validate the counselor’s determination.

    • 	 Reviews of student enrollment statuses recorded in districts’ student information systems
        were limited. WOSPI performed various reasonableness, completeness, and accuracy
        checks of data submitted in the P210 reports. The Washington State Auditor’s Office
        (WSAO) analyzed trends in graduation rates for districts subject to the audit requirements
        of the Single Audit Act. When conducting school district audits, WSAO confirmed that
        the district submitted the required P210 reports and reviewed school records for selected
        students with an enrollment status of graduate in the district’s student information system.
        However, WSAO did not review support for other enrollment statuses and conducted no
        reviews of school records supporting the student’s enrollment status for districts that were
        not subject to the audit requirements of the Single Audit Act. WOSPI did not review
        student enrollment statuses as part of its on-site monitoring reviews nor did the three
        districts perform such reviews.

Our tests of the reliability of student-level data was limited to the 760 sampled students who
attended selected high schools in three districts. Thus, the results of our review of the sampled
students may not necessarily be representative of data errors statewide. However, if the results
of the 760 sampled students are representative of the reliability of data in other districts in the
statewide file, the State’s on-time graduation rate for school year 2003-2004 would decline from
70.1 percent to 64.7 percent and its extended graduation rate would decline from to 74.0 percent
to 68.0 percent.3 (The impact of enrollment status errors on the reported dropout rate is
presented in Finding No.2.) Thus, the Department may have relied on inaccurate rates to assess
the progress of the State of Washington’s Title I, Part A program. Additionally, WOSPI used
unreliable data to make AYP determinations for districts and high schools.

For school year 2004-2005, WOSPI collected data from districts using its new Core Student
Record System (CSRS). WOSPI issued the Core Student Record System Monthly Collection
Data Manual for the 2004-2005 School Year in May 2004. Districts continue to maintain their
own student information systems, but provide updated student data to CSRS on a monthly basis.
WOSPI is using student data in CSRS to calculate the graduation and dropout rates that will be
reported in the 2005-2006 CSPR. While the conversion to CSRS may address some of the
conditions found in our review, others will still need to be addressed since CSRS continues to
rely on data extracted from the districts’ student information systems. During our on-site visits
at school districts, district staff indicated that the procedures for preparing the monthly CSRS
data submissions were difficult and time consuming to perform.




3
 The revised rates are a non-statistical estimate of the impact of our audit results on the State’s graduation rates.
The revised rates were derived by adjusting the statewide total of students in each enrollment category using the
error rates from our sample and recalculating the rates using the State’s formulas and the adjusted statewide totals.
ED-OIG/A09G0009 	                                                                      Page 9 of 21

Recommendations

We recommend that the Assistant Secretary for Elementary and Secondary Education require
WOSPI to—
1.1 	   Identify the specific difficulties that district information technology staff encountered
        when extracting student data from district student information systems for transmittal to
        WOSPI and take action to mitigate the difficulties.
1.2 	   Advise the WSIPC of the deficiency noted in the WESPaC reporting module and take the
        necessary actions to ensure that districts using the WESPaC system properly report the
        graduation status of students who took classes at more than one school.
1.3 	   Continue to provide annual training to districts and take steps to target the training to
        district and school personnel who enter enrollment codes to ensure consistent
        understanding of enrollment status definitions and required supporting documentation.
        If feasible, the training should be provided online to enable new hires and others to easily
        access the training.
1.4 	   Assess whether the standard data checks performed by WOSPI staff to identify
        incomplete or unusual student numbers in district data should be expanded to include
        additional data analyses, such as confirming that the P210 reports and the statewide
        student-level data file include students in each enrollment category for each district and
        school.

1.5 	   Issue a notice advising districts of the importance of adequate internal controls for
        ensuring that enrollment statuses are accurately recorded and that students meet the
        minimum credits for graduation.

1.6 	   Develop and implement a review process for ensuring that school staff accurately record
        student enrollment statuses and retain required documentation. WOSPI could consider
        reviewing enrollment statuses of selected students during state on-site monitoring reviews
        and having the WSAO expand its review of graduate status during single audits to
        include review of support for other enrollment statuses.

WOSPI Comments

WOSPI concurred with the finding and described the action taken or planned for the
recommendations. WOSPI agreed that errors were present in the student data for the three
districts, but expressed concern that generalization from these districts to the entire state may
overestimate the extent of data errors statewide. WOSPI also noted that the review focused on
student data for a time period (school year 2003-2004) that occurred shortly after passage of
NCLB, which increased the need for ensuring that accurate data was available for AYP
determinations. WOSPI disagreed with information presented in the report in the paragraph
titled “Students on the Spokane Public Schools’ P210 Report Were Not Included in the
Statewide Student-level Data File.” Specifically, WOSPI asserted that dropouts from Spokane
Public Schools were included in the statewide student-level data file as extended students.
ED-OIG/A09G0009 	                                                                     Page 10 of 21

OIG Response

To address WOSPI’s concern regarding the generalization of the audit results, we added
language in the final report to emphasize the limited nature of our review and that the results
may not necessarily represent the extent of errors statewide. We made no other changes to the
information in the report. The WOSPI report supporting the student data used to calculate the
graduation rate for Spokane Public Schools for school year 2003-2004 showed no students in
either the dropout or extended student categories.


FINDING NO. 2 – W
                	 OSPI Reported Dropout Rates That Were Not Computed
                Using the Required Definition of a Dropout

In the 2003-2004 and 2004-2005 CSPRs, WOSPI reported dropout rates that were not computed
in adherence with the National Center for Education Statistics (NCES) definition of a dropout.
The CSPR instructions state—

       . . . States should use the annual event school dropout rate for students leaving a
       school in a single year determined in accordance with the NCES’ Common Core
       of Data.
       Consistent with this requirement, States must use NCES’ definition of “high school
       dropout,” An individual who: 1) was enrolled in school at some time during the
       previous school year; and 2) was not enrolled at the beginning of the current school
       year; and 3) has not graduated from high school or completed a state- or district-
       approved educational program; and 4) does not meet any of the following
       exclusionary conditions: a) transfer to another public school district, private school,
       or state- or district-approved educational program …, b) temporary absence due to
       suspension or school-excused illness; or c) death. [Emphasis added.]

The NCES definition of a dropout does not include students who completed a state- or district-
approved education program, such as a GED.

WOSPI used the following state definition of a dropout rather than the NCES definition when
calculating the dropout rate reported in the CSPRs.

       Dropouts are students who drop out of school for any reason, finish their schooling
       without a regular diploma, or whose status is “unknown” because they are no longer
       enrolled but are not confirmed transfers or dropouts. [Emphasis added.]

The state definition of a dropout includes students who received a GED (GED completers).
ED-OIG/A09G0009 	                                                                                       Page 11 of 21

By using the state definition, WOSPI overstated the state’s dropout rate in the CSPRs. WOSPI’s
calculation of the dropout rate for school year 2003-2004 included 18,365 dropouts of which
1,441 were GED completers. The exclusion of the GED completers in the calculation would
decrease WOSPI’s reported dropout rate from 5.8 percent to 5.4 percent. However, the dropout
rate increases when the rate is also adjusted for the enrollment status errors noted in Finding
No. 1. If the GED completers were excluded and the results of the sampled students reviewed in
our audit are representative of the reliability of data in other districts in the statewide file,
WOSPI’s reported dropout rate would increase from 5.8 percent to 7.7 percent.4

WOSPI staff considered the state definition of a dropout as the more appropriate definition for
reporting purposes to maintain consistency with the graduation rate defined at
ESEA § 1111(b)(2)(C), which does not include GED completers with graduates.

Recommendation

2.1 	    We recommend that the Assistant Secretary for Elementary and Secondary Education
         require WOSPI to adhere to the CSPR instructions for reporting dropout rates.

WOSPI Comments

WOSPI concurred with the finding and agreed to implement the recommendation.




4
  The revised rates are a non-statistical estimate of the impact of our audit results on the State’s reported dropout
rates.
ED-OIG/A09G0009                                                                                      Page 12 of 21




                                            OTHER MATTER 



WOSPI’s graduation rate formulas were not accurately reflected in the State’s Consolidated State
Application Accountability Workbook.5 Thus, the Department and other readers of the
Accountability Workbook were unaware of the actual formula used for AYP purposes. We
identified two reports that mistakenly concluded, based on the Accountability Workbook, that
the State of Washington used a graduation rate derived from data accumulated over a four-year
period.6 In fact, the State’s actual formula used student data exclusively from the most recently
completed school year to derive the rates for each of the four grade levels. The Accountability
Workbook needs to clearly disclose the formulas used to calculate high school graduation rates
used for AYP determinations and reported in the CSPRs.

In its comments on the draft report, WOSPI referred to an amendment to its Accountability
Workbook that was submitted in August 2006 for Departmental approval. The amendment
includes the actual formulas used by WOSPI to compute graduation rates. However, further
clarification is needed. The amendment does not state that WOSPI uses data solely from one
school year to compute the rate and use of the term “cohort group” could lead a reader to
incorrectly conclude that the State uses student data obtained over multiple school years.
In addition, the amendment incorrectly states that the extended graduation rate “may be used for
AYP purposes,” when, in fact, the extended graduation is used for such purposes. The
Department had not approved WOSPI’s amendment request as of the issuance of this report.




5
  The Washington Accountability Workbook was initially submitted to the Department in January 2003 and, after
revision, approved by the Department in June 2004. The Department approved subsequent amendments to the
Washington Accountability Workbook on September 1, 2005 and July 19, 2006.
6
 The National Institute of Statistical Sciences/Education Statistics Services Institute Task Force’s Final Report on
Graduation, Completion, and Dropout Indicators (NCES 2005-105) and the Urban Institute’s, Keeping Count and
Losing Count: Calculating Graduation Rates for All Students Under NCLB Accountability.
ED-OIG/A09G0009 	                                                                                     Page 13 of 21



                     OBJECTIVES, SCOPE, AND METHODOLOGY 



The objective of our audit was to determine whether graduation and dropout rates reported to the
Department by WOSPI in the 2003-2004 and 2004-2005 CSPRs are supported by reliable data
and meet the requirements of the applicable provisions of the ESEA and Federal regulations.
Specifically, we determined whether the—
    • 	 Graduation and dropout rates reported in the CSPRs for the “All Students” category
        were computed in accordance with the requirements of ESEA § 1111(b)(2) and
        34 C.F.R. §200.19 (a) and (c).
    • 	 Data supporting the graduation and dropout rates were sufficiently valid and reliable for
        the Department to use the rates to complete reports on its progress in meeting its strategic
        plan goals, individual program performance, and State’s progress in implementing
        NCLB.
To accomplish our objective, we—
    • 	 Reviewed applicable laws, regulations, and other guidance;
    • 	 Reviewed pertinent reports dealing with graduation and dropout rates published by
        Federal, state, and private organizations;
    • 	 Identified, assessed, and tested internal controls employed by WOSPI to improve the
        reliability of data supporting graduation and dropout rates;
    • 	 Interviewed officials and reviewed written policies and procedures related to graduation
        rates and dropout rates at WOSPI, three school districts, and nine high schools;
    • 	 Reviewed student files at nine high schools.
From the state’s eleven districts with more than 20,000 enrolled students in school year
2003-2004, we selected three districts for review using a risk-based approach. We selected
Edmonds School District and Seattle Public Schools because each met WOSPI’s high school
graduation rate goal of 66 percent using the extended graduation rate, but failed to meet the goal
using the on-time graduation rate employed in school year 2002-2003.7 We selected Spokane
Public Schools for review because of the district’s 19.4 percent increase in on-time graduation
rates from 2002-2003 to 2003-2004 (the largest increase for districts with more than
20,000 students). In each district, we selected for review the three high schools having at
least 1,300 students that met WOSPI’s extended graduation rate goal of 66 percent by the
smallest margin.

To select the student sample at each high school, we used the statewide student-level data file
that WOSPI utilized to compute the graduation and dropout rates in school year 2003-2004.
To assess the reliability of the data files, we compared the total records in the statewide file for
9th through 12th grades to enrollment counts reported by WOSPI. We confirmed that the totals
by category for each of nine high schools matched the totals used to compute graduation and


7
 For school year 2003-2004, high schools and districts in the State of Washington must have an extended
graduation rate of 66 percent or increase their rate by at least two percent from the prior school year to meet AYP.
    ED-OIG/A09G0009                                                                                                                       Page 14 of 21

    dropout rates for those schools. We also determined whether the school had records for sampled
    students that confirmed that the students were enrolled during the school year. Based on these
    tests, we concluded that the data were sufficiently reliable for use in selecting the sample.
    However, as noted in Finding No. 1 of the Audit Results section, we found that the student
    enrollment status information in the statewide file was not reliable.
    For the nine high schools selected for review, we extracted student-level data from the statewide
    student universe of 341,996 students in grades 9 through 12. Then, we segregated the students at
    each school into six groups: graduates, late graduates, transfers, dropouts, extended students, and
    continuing students (other than extended students). For the transfer group, we separated students
    into two subgroups: (1) students who had a gaining school enrollment code to support the losing
    school’s transfer code, and (2) students who did not have a gaining school enrollment code to
    support the losing school’s transfer code. We attributed the greatest risk of error to the second
    transfer subgroup and selected the sampled transfer students solely from that subgroup.
    We randomly selected students from the graduate, transfer subgroup, dropout, and extended
    student categories. For the selected students and all students in the late graduate category, we
    compared the student’s enrollment code to the school’s documentation supporting the code.
    We did not include students from the continuing student group in our sampling universe as we
    considered this group to be at lowest risk for inaccurate enrollment statuses.

      Table 2: School Year 2003-2004 Student Group and Sample Sizes by District and School
District      High School                On-time                  Late                          Transfers                   Dropouts                Extended
                                        Graduates               Graduates                                                                           Students
                                          Group Size

                                                       Tested

                                                                Group Size


                                                                             Tested


                                                                                       Group Size
                                                                                                    Group Size
                                                                                                    Unmatched


                                                                                                                   Tested

                                                                                                                             Group Size

                                                                                                                                           Tested

                                                                                                                                                    Group Size


                                                                                                                                                                   Tested
Edmonds       Edmonds-Woodway            309           31        8            8       114               54           12      98            10       27               10
              Lynnwood                   241           25       17           17       247              127           25      62            10       47               10
              Meadowdale                 304           31        7            7       134               71       (a) 71      45            10       49           (a) 49

Seattle       Ballard                    360           38       13           13       111                28         12       15            10       34              10
              Franklin                   328           33        0            0       162                29         17       63            10       25              10
              Roosevelt                  380           38       14           14        92                46         10       13            10       43              10
Spokane       Lewis & Clark              426           43             0          0     169              68          17       31            10             0            0
              North Central              292           30             0          0     194              73          20       28            10             0            0
              Rogers                     339           34             1          1     239             126          24       75            10             0            0
Totals                                  2979           303      60           60       1462             622        208       430            90       225             99
                                                                                                                            Total Students                       5,156
                                                                                                                            Total Tested                           760
(a) Due to the high error rates at Meadowdale High School, we reviewed 100 percent of students in the transfer and extended
groups.

    We performed our fieldwork at WOSPI offices in Olympia, Washington and administrative
    offices of the three districts and nine schools. We held an exit briefing with WOSPI officials on
    July 12, 2006. Our audit was performed in accordance with generally accepted government
    auditing standards appropriate to the scope of the review described above.
ED-OIG/A09G0009                                                                      Page 15 of 21




                            ADMINISTRATIVE MATTERS 




Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

If you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following Education Department
official, who will consider them before taking final Departmental action on this audit:

                              Henry Johnson
                              Assistant Secretary
                              Office of Elementary and Secondary Education
                              U.S. Department of Education
                              400 Maryland Ave., SW
                              Washington, DC 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued by the Office
of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.


                                             Sincerely,

                                             /s/

                                             Gloria Pilotti
                                             Regional Inspector General
                                              for Audit

Attachments
ED-OIG/A09G0009                                                                        Page 16 of 21

      Attachment 1: WOSPI Graduation Rate Formulas and Calculations
WOSPI uses the below formulas to calculate the on-time graduation rate and extended graduate
rate reported in the CSPR. The student numbers used in the formulas are based on the student’s
enrollment status at the end of the school year. The on-time graduation rate formula uses the
dropout rate in each grade level for that school year as a proxy for the rates in previous years.

                            WOSPI’s On-time Graduation Rate Formula
                                     (asterisk denotes multiplication)
On-time Graduation Rate:
                             100*(1 minus grade 9 dropout rate)*(1 minus grade 10 dropout rate)*
                             (1 minus grade 11 dropout rate)*
                             (1 minus grade 12 dropout rate minus grade 12 continuing rate)

Formula used to derive grade-level dropout rates for the above:
                  Number of students with a dropout, unknown, GED completer code
                       Total number of students served (minus transfers out)



                            WOSPI’s Extended Graduation Rate Formula
Extended Graduation Rate:                Number of on-time and late graduates
                                Number of on-time graduates divided by on-time graduation rate
ED-OIG/A09G0009                                                                                                                                                                            Page 17 of 21

The OIG prepared the following charts to assist the reader in understanding the computation of
graduation rates using the actual figures used by WOSPI.
                       SCHOOL YE AR 2003-2004 WOSPI ON- TIME AND EXTENDED GRA DUA TlON RATE CALCULA TlONS

                                                                                                                                  Relen/ioll
                                                                                                                                    Rate
                                                                                                 Grade-LeV91                       (100%-
                                                                                                     Dropout                       Dropout                       Remaining
       Grade,oroop                                                                                     Rate!>                       Rate)                        Pet'C(!ntaoe
       9th Grad. (2OO3_2QO.II           9th Grade Dropouts ("""'~)                      4.486
                                        9th Graders Served (}!dual!                    87.620    [II =       S.l%                [1J9H%

       1O!h Grad. (20037200()            10th Gr..x o..""",t, (.'.ctutl)         4.561           [D -        ,.'"               [!jS..4'1O
                                                                                                                                           t [!]               .!d9.G%
                                         10th Grlde<$ SeMd ("""'tI)              ~

       11th Gra" (2003-200()             11th Gr.cIe o..""",t. (Atluan
                                         11th ao-_S Serve<! ( _ )
                                                                                        U371I1
                                                                                       77. ~            •    6.4'10             mS3.5%~ ~ 8%
                                                                                                             "~
                                                                                                                                                                 L

       12th Grad. (2OO3·20N)
        D.-Op<),l1$


       12th Grad. (2OO3·ZOo()
                                         12th Gr.cIe o..""",t. (Adual)
                                         12th ao-lde<S SeM(! ("""'tI)

                                         12th Grlde Repeaters ("""'al)
                                                                                 ~[!J
                                                                                       59.252

                                                                                 6.995 [D
                                                                                                        "
                                                                                                        =
                                                                                                                 2
                                                                                                            10.1'10
                                                                                                                          lM%~                70
                                                                                                                                        U ,5'1O-->i                  10.''''        on:nm. GiOod ROI.        I
        Continuing Stud..-u              12th Gr_s Serwd ("""'tI)                ~

       Exten(l'" Grad Rot .             On· TIme Gn")Jl!U (ACtuet)
                                        On· TIme GraOJlti«l Rate                     -Tm[!] =8/F
                                                                                       7 ,1'Mo
                                                                                                                          E~;rnat..,   Nurntler (II 9th 0,..,.11' that $I   ~ed   Hiif' Schoo! in the Fan or 2000


                                             Uote ao-I ""ates (ACtuat)
                                             On· TIme ao-aOJato. (ACtual)
                                                                                        3.468
                                                                                       57.928       /
                                                                                                                                                             I
                                         Tol<IIGr_t<o.                   [!]           6t .m       10   =   U.l% E>rtended Go-;>duOlI«l ROI.
                                        E.M.,ledFIU2O(J() 9th Gr_.                     82.SS
i d Ulition Rlite Calculation Steps:
  I    C~e       Dropcut Rat. lor Each Grade Indthe 12th GradeC«l!inu;ng Student Go-oup (Student. Staying in School More Thin the Standa,d Nurntl..-orYeorsJ
 2 Con'Cline 12th Grade o.-q:,out Rate (6,3%) with 12th Gracie CC<>tinuing Stu"...,t Rate (10. 1'Mo),
~      Subtrael Gracle· Lewl Dropout Ratu           ~om   l00%(Retcntion R.te).
~      Sub/riel Cotri);ned 12th Grade         OropoutICon~nuing    Studenl Rat •• (16.4'Mo) from 10(I'Mo
I=f    M uIt~   9th Gracie   Relen~c<>   RIte (94.9'101 By 10th Gra cie    Relen~c<>   RIte (94.4%1,
~      Multifl!\llOth Gr..x      R~aining     Percentage(SS.6'Mo) By I Ith Gucle Retention Rfle (93,G"MoI.
I=f:   Mutt~    11th G.-.de      R ~.ining    Percentlge(U&'Mo) By CombOned 12th Grade Drcpout/Continuing Studenl Retention Ri te (U6'Mo).
~      OMde On-- Time G .."""u (57.9:/6) by On-- Time G""""liOll Rale (7Q.l%) to Estimlte the Number d 9th Graderslhlt Stlned Hi!#> School in Ih e                                 FI~   d :/QOO (el.SS7).
ffi    C~e       the Total Gradullu by AMng the On· TKne GrIOJate. 157,926) to Ihe Lale GraOJlle. (3.458).
 10 DMde tho Total Grl"" "IO$(61 .394) bylhe              E~imaled   Fall20009th     Gra~        (82,657)locC1'f'lP,>le the ExtendedGrI",,"tion Rfle (74.3%),


         SCiiOOL                                  ~ON7fiiIiEGi,                                                                             IIIUN RATE                                          I'fiOii
                                                                                                                                         Retention
                                                                                                                                           Rare
                                                                                                    Grade-Lwei                            (100%-

                                                                                                            D::                            -R';;;'-                    Remaining

~                                             ;;:                                                                'N
                                                                                                                                        0"tm_
       , •• G... ,



       11th Grnde (2002-"2003)


       12th Grnde (2002-"20031
         Dropouts
                                                                    :::::.; ~OJ -
                                                                    ::z:::::
                                              12\t1 Grade DropouI$ (lI.:;:tu a~
                                              12th G,aders Serv~_ _ _7S, 7S3
                                                                                6.105              CD
                                                                                                            -
                                                                                                            "
                                                                                                                 ,."
                                                                                                                 6.'"


                                                                                                                 "~   2      20.2'%
                                                                                                                                       rn
                                                                                                                                       rn
                                                                                                                                       B 7$.~
                                                                                                                                              l",
                                                                                                                                                  .-A
                                                                                                                                                   ""
                                                                                                                                                  -""
                                                                                                                                                          --to-             6'.7%          On-Time G,ad Rate
       12th Grnde (2002-"20031                12\t1 Grade Re peaters (Actual)              9.176   QJ           12.1%
        Continuing Students                   12\t1 Grade<s Served (Actual)                7~7S3

                ,
~                   12tt> G r_
                                                    ~~_r:~~ 12tt> Grade Continuing Stude nt Grgyp (Students Staying in ScIIooI Mo.-eThan !tie Standa rd Number 01 Year.) .
                                   Oropoyt Rate (8,1'l1li) Mt112th GradeContinuing Student Rate (121'l1li) ,
                                   ,                        It::tO'1111 (Retemion Rate),
                                         ,      OropoutIContinuing Stude<1t Rates (16.4'l1li) !rom ll)O'!1. ,
                        ,        Retention Rate (938'l1li)    ev 10th G,ade Retention Rate (94.1'l1li).
                       Grade l1e<1'laining Peroentage (86.3%)         ev 11111 Grade Retention Rate (93,3'l1li) ,
                       Grade l1e<1'laining Peroentage(62,4'1111) 9yC<lmbined 12th Grade OropOUtlContinuing $t:ude<lll1etention Rate (79,6'l1li).
ED-OIG/A09G0009                                                                                               Page 18 of 21

                Attachment 2: WOSPI Comments to Draft Report




                        SUPERINTENDENT OF PUBLIC INSTRUCTION
                   DR. TERRY BERGESON      OLD CAPITOL BUILDING. PO BOX 47200. OLYMPIA WA 98504-7200 . http://www.k12.wa.us




        October 23, 2006




        Ms. Gloria Pilotti
        Regional Inspector General for Audit
        United States Department of Education
        Office of Inspector General
        501 I Street, Suite 9-200
        Sacramento, CA 95814

        RE: Audit Control Number ED-OIG/A09G0009

        Dear Ms. Pilotti,

        I am writing in response to the Department's August 30, 2006 Draft Audit Report, entitled Data
        Quality of Washington Consolidated State Perfonnance Reports.

        The following is Washington Office of Superintendent of Public Instruction's (WQSPI)
        response, as well as our corrective actions planned/taken and/or additional information in
        response to the findings and recommendations included in your draft report. (Our responses are
        provided in italics.)

        Finding No.1        WOSPI Used Unreliable Data to Calculate Graduation and Dropout Rates

        WOSPI concurs with the finding that errors were found in the three sampled districts and
        recognizes that additional controls over the quality oj data collected are necessary. However,
        wefeel the generalizationJrom these districls, which were selected because they were the most at
        risk of having errors, to the entire state may over estimate the extent oj data errors statewide.

        The years reviewed during the audit were just after the passage oj NeLB, which greatly
        magnified the impact this data collection has regarding AYP detenninations, thereby, increasing
        the need for ensuring the most accurate data possible. We agree that some data reported may be
        inaccurate, but WaSPI has instituted many quality control measures in order to ensure reliable
        data. Nevertheless, we agree that the steps recommended by the OIG wouldfurther enhance the
        accuracy of the data WaSPI uses to calculate the rates, and we plan to take those steps.
ED-OIG/A09G0009                                                                                             Page 19 of 21




        Ms. Gloria Pilotti
        Page 2
        October 23, 2006


        Other comments: Regarding the statement on page 5 "Students on the Spokane Public Schools'
        P2IO report were not included in the statewide student-level data file." Based on one of our data
        edit checks, our files indicate that these students were not eligible under the current definition to
        be reported as dropouts. These students were included in the "extended grad rate" category
        based on their expected graduation date.

        Recommendation 1.1 - Identify the specific difficulties that district information technology staff
        encountered when extracting student data from district student information systems for
        transmittal to WOSPI and take action to mitigate the difficulties.

        We are in the process of moving to a model that will completely eliminate districts having to
        manually upload data to wasp/. We will extract the districts' data by pulling it directly from
        their systems. We are currently piloting this model with three districts.

        Targeted Completion Date: 2007-08 school year

        Recommendation 1.2 - Advise the WSIPC of the deficiency noted in the WESPaC reporting
        module and take the necessary actions to ensure that districts using the WESPaC system properly
        report the graduation status of students who took classes at more than one school.

        We will contact WS1PC and relay this information.

        Targeted Completion Date: October 31,2006

        Recommendation 1.3 - Continue to provide annual training to districts and take steps to target
        the training to district and school personnel who enter enrollment codes to ensure consistent
        understanding of enrollment status definitions and required supporting documentation. If
        feasible, the training should be provided online to enable new hires and others to easily access
        the training.

         We plan on continuing our annual training and will work on ensuring we target those school
        personnel who work with the data. We have started offering K-20 video conference data
        meetings this year on a monthly basis to address new and emerging issues surrounding data
        reporting, which includes enrollment codes. These are offered at all nine Educational Service
        District (ESD) locations, video capable districts, and audio capable districts and are well
        attended, which shows the districts are striving to ensure they provide the most accurate data
        possible. The video conferences are also available for download through our ESDs. We have
        discussed the possibility of having some of our education related associations who provide
        district related training to add this as a topic to their planned training schedule and will look
        into this option further. In addition, we have already had discussions regarding providing
        training on-line and will continue to further review this option.

        Targeted Completion Date: Ongoing
ED-OIG/A09G0009                                                                                           Page 20 of 21




        Ms. Gloria Pilotti
        Page 3
        October 23, 2006


        Recommendation 1.4 - Assess whether the standard data checks perfonned by WOSPI staff to
        identify incomplete or unusual student numbers in district data should be expanded to include
        additional data analyses, such as confirming that the P210 reports and the statewide student-level
        data file include students in each enrollment category for each district and school.

        We currently provide a significant number of data analyses for reasonableness. We will review
        our data checks that are currently performed to determine whether we can expand those to
        ensure the most reliable data possible.

        Targeted Completion Date: April 2007

        Recommendation 1.5 - Issue a notice advising districts of the importance of adequate internal
        controls for ensuring that enrollment statuses are accurately recorded and that students meet the
        minimum credits for graduation.

        We will emphasize the importance of adequate internal controls in our annual training sessions
        as well as our guidance that outlines graduation reporting requirements and instructions. The
        importance of accurate data is also reinforced in our monthly data meetings mentioned above,
        and will be conveyed to district superintendents through the appropriate communications.

        Targeted Completion Date: Ongoing

        Recommendation 1.6 - Develop and implement a review process for ensuring that school staff
        accurately records student enrollment statuses and retain required documentation. WaSPI could
        consider reviewing enrollment statuses of selected students during state on-site monitoring
        reviews and having the WSAO expand its review of graduate status during single audits to
        include review of support for other enrollment statuses.

        We are in the process of identifying ways to increase monitoring oj graduation rate data at the
        district level, which could include one or a combination of activities, such as: specialized staff to
        perform on-site reviews, adding afunction to the Consolidated Program Review process,
        performing risk assessments based on data review, etc. In addition, we have had recent meetings
        with our State Auditor's Office regarding their expanding work in the area of student level data
        reporting during their single audits of school districts.

        Targeted Completion Date: We anticipate having a pian in place by February 2007. The
        implementation of that plan will be an ongoing function.
ED-OIG/A09G0009                                                                                Page 21 of 21


     Ms. Gloria Pilotti
     Page 4
     October 23. 2006


     Finding No.2 WOSPI Reported Dropout Rates That Were Not Computed Using the
     Required Definition of a Dropout

     We concur with this finding. We used the same definition as found in NCLB for calculating
     Graduation Rates, which includes GED completers in the definition of dropouts. We believe lhal
     having two different calculations creates some confusion among the users of that in/onnation,
     but we understand the reasons surrounding the decision to exclude GED completersJrom the
     dropout rate.

     Recommendation 2.1- We recommend that the Assistant Secretary for Elementary and
     Secondary Education require WaSPI to adhere to the CSPR instructions for reporting dropout
     rates.

     WOSP! will report dropout informatioll ill its CSPR in accordance with the NCES reporting
     requirements.

     Targeted Completion Date: At the time the NCES report is due.

     "Other Matter" - WOSPl's graduation rate formulas were not accurately reflected in the
     State's Consolidated State Application Accountability Workbook

     771is was corrected in an amendment submitted on August 6, 2006 which can be found at
     http://www.kJ2.wa.us/ESENpubdocs/A YPworkbookrevisionsAu 1:06. doc.

     I appreciate the opportunity to respond to your office's recommendations to assist us in ensuring
     the most accurate and reliable data is used in reporting graduation and dropout rates to the
     Department.

     Si!lcerely,
     /s/
     Cathy ~vidson. Ed.D:
     Deputy Superintendent
     Learning and Teaching