oversight

Arizona Department of Education Management Controls Over IDEA, Part B-Special Education Performance Data.

Published by the Department of Education, Office of Inspector General on 2000-09-22.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

          Arizona Department of Education
              Management Controls Over
   IDEA, Part B- Special Education Performance Data



                                   FINAL AUDIT REPORT




                              Control Number ED-OIG/A09-A0001
                                        September 2000




Our mission is to promote the efficient                    U.S. Department of Education
and effective use of taxpayer dollars                      Office of Inspector General
in support of American education.                          Sacramento, California
                                   NOTICE
    Statements that management practices need improvement, as well as other
  conclusions and recommendations in this report, represent the opinions of the
Office of Inspector General. Determination of corrective action to be taken will be
            made by the appropriate Department of Education officials.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued
  by the Office of Inspector General are available, if requested, to members of the
 press and general public to the extent information contained therein is not subject
                              to exemptions in the Act.
                                        Table of Contents

Executive Summary ……………………………..….…...…....…...…. 1
Audit Results ……………………...………………………….….…… 3
Finding No. 1 – ADE and LEAs Did Not Have Management
                Controls in Place to Ensure the Reliability of
                Reported Performance Data for Exiting,
                Discipline and Personnel ....…………..……………….…....… 4
                      LEAs Did Not Report the Actual Number of Instances for
                         Performance Indicators....…………………………………….… 4
                      Reported Data Was Not Reviewed or the Reviews Were Inadequate....…. 6
                      Neither ADE nor the LEAs Had Documented Its
                        Data Collection Processes…………………………….....….……. 7
                      Recommendations ....……………………………….........……....... 7
                      ADE’s Comments ............................................................................. 7
                      OIG’s Response ............................................................................... 8

Finding No. 2 – ADE Did Not Ensure That Data Submitted by LEAs on
                Hardcopy Reports Were Accurately Reflected in
                Its Databases …………...………….…….….…....…..…. 8
                Recommendations …………...………….……...…....…..…. 9
                ADE’s Comments ............................................................................ 9

Finding No. 3 – ADE and LEAs Did Not Retain Supporting Documents for
                Reported Exiting, Discipline and Personnel Data …....….... 9
                Recommendation............………..…...…..……………..…….…... 10
                ADE’s Comments .......................................................................... 10

Finding No. 4 – LEAs’ Reliance on Manual Processes and
                Varying Data Collection Methods Could Impact
                ADE’s Ability to Provide Reliable Data …………...…….. 10
                Recommendation .…….……...………….…….....……..…. 12
                ADE’s Comments .......................................................................... 12

Background …………………………………...…….…..….….….… 13
Purpose, Scope and Methodology ..……………….…....…..…........ 14
Statement on Management Controls ……………...….…..….…..... 17
Attachment A – IDEA, Part B Program Objectives, Performance
            Indicators and Performance Data …….……......….…..... 18
Attachment B – ADE’s Comments to the Report .……..........….…....... 19
                                     Executive Summary

The Arizona Department of Education (ADE) needs to improve management controls over the
collection and reporting of performance data provided to the U.S. Department of Education
(Department). The Government Performance and Results Act of 1993 (GPRA) requires that
Federal agencies submit annual performance reports to Congress. The Office of Special
Education Programs (OSEP) within the Department’s Office of Special Education and
Rehabilitative Services (OSERS) administers programs funded under the Individuals with
Disabilities Education Act (IDEA), Part B. OSEP uses performance data reported by state
educational agencies in preparing the Department’s report to Congress on the outcomes of the
IDEA, Part B programs.

For reporting outcomes under the Department’s 2001 Annual Plan, OSEP uses state educational
agencies’ performance data for the following performance indicators:

         •   Graduation (exiting)
         •   Suspensions or expulsions (discipline)
         •   Qualified personnel (personnel)
         •   Earlier identification and intervention (intervention)
         •   Inclusive settings/Regular education settings (placement)

ADE is required by IDEA, Part B to submit this performance data to the Department.
Attachment A to this report shows the relationship between the IDEA, Part B program
objectives, performance indicators and performance data.

Our review of procedures and available documentation at ADE and three of its local educational
agencies (LEAs) identified weaknesses in the management controls covering performance data
for exiting, discipline and personnel. We found that ADE and/or the LEAs did not meet three of
the six data quality standards listed in the Department’s publication, Standards for Evaluating
the Quality of Program Performance Data (1999 Working Draft).1 ADE and/or the LEAs failed
to meet elements contained in the standards covering accurate descriptions, editing and reporting.
Specifically, we found that:

         •   The LEAs did not include all instances for the performance indicators,
         •   Neither ADE nor the three LEAs conducted reviews of the reported data or the
             reviews were inadequate and
         •   Neither ADE nor the three LEAs had documented its data collection processes.

We also found that ADE did not ensure that information submitted by LEAs on hardcopy reports
was accurately reflected in its databases. Our review found that ADE and two of the LEAs we
reviewed had not retained copies of its reports and related supporting documents for the three-
year period required by applicable Federal regulations. Reconstructed reports and supporting
documents, which were provided for our review, did not reconcile to the exiting, discipline and
personnel data previously reported to ADE. In addition, we concluded that LEAs’ reliance on

1
  In March 2000, the Department included the Data Quality Standards as an Appendix in Volume 1 of its publication
titled 1999 Performance Reports and 2001 Plans.


ED-OIG                              Control Number ED-OIG/A09-A0001                                      Page 1
manual processes and varying data collection methods could impact ADE’s ability to provide
reliable data. Due to these weaknesses in ADE’s and the LEAs’ management controls, we have
no assurance that ADE provided OSEP with reliable performance data for the 1998-1999 school
year.

We recommend that the Assistant Secretary for Special Education and Rehabilitative Services
require that ADE implement procedures to address the weaknesses in its management controls
over reported performance data. The Audit Results section of the report contains our specific
recommendations for each of the findings.

In its comments to the draft report, ADE generally concurred with the findings and described the
corrective action planned or taken. ADE’s comments are summarized in the report following
each finding. The full text of ADE’s comments is included as Attachment B.




ED-OIG                         Control Number ED-OIG/A09-A0001                           Page 2
                                      Audit Results

ADE uses two separate methods to accumulate the IDEA, Part B performance data that it reports
to OSEP. For exiting, discipline and personnel data, ADE compiles the data from reports
submitted by LEAs. For intervention and placement data, ADE extracts the data from a
statewide computerized database.

Exiting, discipline and personnel. ADE uses LEA-submitted information to compile the data
reported for the exiting, discipline and personnel performance indicators. The LEAs report the
information using the data collection format and instructions that OSEP provided to state
educational agencies. ADE enters information from the LEA reports into separate ACCESS files
for each performance indicator.

We concluded that ADE and LEAs did not have management controls in place that could be
relied upon to ensure that exiting, discipline and personnel data reported to OSEP for the
1998-1999 school year were reliable. Also, we found that ADE did not have procedures to
ensure that data from LEA reports were accurately entered in its ACCESS files. In addition,
LEAs did not retain copies of their reports and related supporting documents for the three-year
period required by applicable Federal regulations.

Intervention and placement. For these performance indicators, ADE uses its own computerized
informational database on children with disabilities in the State, which includes the age, primary
disability, ethnicity, gender and placement for each child. This data is referred to as the Special
Education Census. ADE collected the data from LEAs on December 1, 1998 and used it to
provide child counts for funding purposes. ADE extracts information from the census database
to provide OSEP with data for the intervention and placement performance indicators.

We found that ADE’s census database did not reflect all student withdrawal dates reported by an
LEA that submitted hardcopy reports to ADE. ADE informed us that the error was not systemic
but was caused by the LEA using the incorrect form. Except for this one instance, nothing came
to our attention during our limited assessment and testing of management controls that caused us
to doubt the acceptability of ADE’s reported performance data for intervention and placement
for the 1998-1999 school year.

In addition to the management control weaknesses, we noted that the LEAs’ reliance on manual
data collection processes and varying data collection methods could be barriers/obstacles that
may impact ADE’s ability to provide quality performance data.




ED-OIG                          Control Number ED-OIG/A09-A0001                             Page 3
Finding No. 1 – ADE and LEAs Did Not Have Management Controls in Place
                to Ensure the Reliability of Reported Performance Data for
                Exiting, Discipline and Personnel

ADE and LEAs did not have adequate management controls over the collection and reporting of
performance data for exiting, discipline and personnel for the 1998-1999 school year. The
Department’s publication, Standards for Evaluating the Quality of Program Performance Data
(1999 Working Draft), lists six data quality standards:

      •   Validity – Data adequately represent performance.
      •   Accurate Description – Definitions and counts are correct.
      •   Editing – Data are clean.
      •   Calculation – The math is right.
      •   Timeliness – Data are recent.
      •   Reporting – Full disclosure is made.

For each of the data quality standards, the publication provides examples of conditions that meet
or fail to meet the standard. The publication also provides a Data Quality Checklist for use by
primary data providers and secondary data managers.

We reviewed the collection and reporting procedures and available documentation at ADE and
three LEAs: Tucson Unified School District (Tucson), Deer Valley Unified School District (Deer
Valley) and Dysart Unified School District (Dysart). ADE and/or the LEAs failed to meet
elements contained in three of the standards: Accurate Description, Editing and Reporting. We
found that:

      •   Tucson and Deer Valley did not report the actual number of instances for
          performance indicators,
      •   Neither ADE nor the three LEAs conducted reviews of the reported data or
          the reviews were inadequate and
      •   Neither ADE nor the three LEAs had documented its data collection processes.

Due to the weaknesses in the management controls, we have no assurance that ADE provided
OSEP with reliable exiting, discipline and personnel data.

LEAs Did Not Report the Actual Number of
Instances for Performance Indicators

Tucson and Deer Valley did not report the actual number of instances for performance
indicators. Standard Two — Accurate Description of the Standards for Evaluating the Quality
of Program Performance Data ensures that definitions and counts are correct. One example of
failing this standard is when certain types of phenomena (e.g., private schools, children with
disabilities) are overlooked or not consistently included in counts. Another example of failing
this standard is the use of estimates instead of actual counts.




ED-OIG                         Control Number ED-OIG/A09-A0001                            Page 4
At Tucson, we found that a high school did not include all instances in the “dropped out”
category on its exiting data report and may not have used the correct age when identifying
students to include in the report. Also, a service center did not accurately report student
removals/suspensions/expulsions in its discipline data submitted to the district office.

         •   Exiting data. Tucson’s ten high schools each report exiting data to the district
             office. We reviewed the reports submitted by two of the high schools. The staff at
             one high school included six students who met the district’s definition of a drop out
             in the “dropped out” category. However, two other students who met the “dropped
             out” definition were categorized as “moved, not known to be continuing.” Also, the
             high school improperly used the student’s age at the end of the school year to
             determine whether the student should be included in the reported exiting data.
             OSEP instructed state educational agencies to use the OSEP IDEA, Part B 1998
             Data Dictionary when reporting performance data. The Data Dictionary states that
             “[a]ge is a child’s actual age in years on the date of the child count: December 1….”
             As a result, the high school may have improperly included some students who were
             13 years old and excluded some students who were 21 years old on December 1st.
             OSEP uses the data reported in the “dropped out” category for the following
             performance indicator: “…percentage [of children with disabilities] who drop out
             will decrease.” (The other high school that we reviewed had not retained its
             supporting documentation and the reconstructed records prepared for our review did
             not reconcile to its report. Therefore, we were unable to test the accuracy of the
             reported data.)

         •   Discipline data. Tucson has two service centers that report discipline data to the
             district office. Tucson procedures require schools to notify the service center when
             a child with disabilities is suspended or expelled. Each service center maintains a
             log of the reported disciplinary actions and used the logs to prepare the reported
             discipline data. Our review found that staff at one service center estimated the
             length of the suspension/expulsion actions for children with disabilities because
             disciplinary actions could be revised during manifestation hearings. OSEP uses the
             reported discipline data for the following performance indicator: “The percentage
             of children with disabilities who are subject to long-term suspension or expulsion,
             unilateral change in placement or change in placement if their current placement is
             likely to result in injury to someone, will decrease.”

Deer Valley did not include all instances in the exiting data submitted to ADE for the 1998-1999
school year.

         •   Exiting data. Deer Valley extracts the exiting data from a district-maintained
             student database. On its 1998-1999 school year report, Deer Valley reported
             40 graduates and 24 dropped students. Our tests found that the database contained
             information on 47 graduates and 29 dropped students for the school year. The
             district had used reports generated from the database on June 8, 1999 to prepare the
             report. According to staff in the district’s Information Services and Technology
             Department, the student information was incomplete on that date because of delays
             in entering data into the student database. We also found that Deer Valley
             improperly used the student’s age at the end of the school year to determine whether
             the student should be included in the reported exiting data.


ED-OIG                         Control Number ED-OIG/A09-A0001                             Page 5
At Dysart, we did not test the accuracy of data reported to ADE because the school district had
not retained supporting documentation for its reports and reconstructed documents provided for
our review did not reconcile to its reports.

Reported Data Was Not Reviewed
or the Reviews Were Inadequate

Neither ADE nor the three LEAs conducted reviews of the reported data or the reviews were
inadequate. Standard Three — Editing of the Standards for Evaluating the Quality of Program
Performance Data ensures that data are clean, that is, the data are correct, internally consistent
and without mistakes. The Data Quality Checklist for this standard lists reviews for primary data
providers and secondary data managers. The reviews include ensuring that:

   •     A person (who was familiar with the data, but was not the data preparer) had
         systematically reviewed the entered data.
   •     Managers have “eyeballed” the data to see if it is reasonable given what they know about
         earlier years and other respondents.
   •     Managers have discussed large changes or unusual findings with the primary data
         providers to see if the changes/findings might be due to editing errors.

LEAs Did Not Conduct Adequate Reviews of Reports Submitted to ADE. Staff at the school,
service center or district level summarized the performance data from original data sources. In
general, the reports prepared by district level staff were subjected to an independent review.
However, the LEAs did not have staff, who were independent of the preparer, review the
procedures and supporting data used to compile the data for the exiting, discipline and personnel
reports. We also noted that, in general, reports prepared at the school or service center level
were not subjected to any review prior to submission to district offices. The absence of adequate
independent reviews at the LEAs increased the risk that ADE collected and reported inaccurate
performance data.

ADE Did Not Review LEA Procedures and Supporting Documentation. ADE conducts
monitoring reviews of LEAs on a six-year cycle. ADE reviews the accuracy of the census
database information as part of its monitoring reviews. Also, independent auditors review the
census database information during the annual single audits of ADE and LEAs. However, ADE
does not review LEA procedures or supporting documentation for reported exiting, discipline
and personnel data. Also, the procedures and reported data are not reviewed as part of annual
single audits.

Title 34 Code of Federal Regulations §80.40 (a) states “...Grantees must monitor grant and
subgrant activities to assure compliance with applicable Federal requirements and that
performance goals are being achieved.” Without reviews of LEA procedures and tests of
supporting documentation, ADE has no assurance that exiting, discipline and personnel data
reported to OSEP and used for its monitoring of LEAs’ achievement of performance goals is
reliable.




ED-OIG                          Control Number ED-OIG/A09-A0001                            Page 6
Neither ADE nor the LEAs Had Documented
Its Data Collection Processes

ADE and the LEAs did not have written procedures for collecting, reviewing and reporting
performance data for the exiting, discipline and personnel indicators. Standard Six — Reporting
of the Standards for Evaluating the Quality of Program Performance Data ensures that full
disclosure is made. One example of meeting the standard is that data collection processes are
documented. Without written procedures, there is little assurance that the LEAs are providing
ADE with comparable and reliable data from year to year.

Recommendations

The Assistant Secretary for Special Education and Rehabilitative Services should:

1.1      Require ADE to develop written policies and procedures for collecting, reviewing and
         reporting performance data for exiting, discipline and personnel. The procedures should
         include reviews of LEA reports and periodic reviews of LEA procedures and supporting
         documentation.

1.2      Recommend to ADE that LEAs be required to maintain written documentation of their
         processes for collecting, reviewing and reporting data submitted to ADE for the exiting,
         discipline and personnel indicators. ADE should consider requiring that the LEA
         processes include that:

            •   A person (who was familiar with the data, but was not the data preparer)
                systematically reviews the entered data, including the supporting documentation.
            •   Managers with knowledge of the procedures used to collect the data “eyeball” the
                data to see if it is reasonable and that the managers evaluated significant changes
                or unusual findings.

1.3      Recommend to ADE that LEA officials certify the accuracy of the data submitted on their
         reports to ADE.

ADE’s Comments

In its comments to the draft report, ADE described the corrective action planned or taken. ADE
stated that it has hired new data management staff and is refining its procedure manual. ADE
also mentioned that it has issued a letter to all LEAs requiring written procedures and
certification of reported data. In addition, ADE plans to make training available for the LEAs.

However, ADE did not consider the weaknesses in its management controls to be significant.
ADE also stated that each LEA included in our review had verbally informed ADE that it
disagreed with various statements in the finding. The LEAs did not respond to the ADE’s
request for documentation to support their claims. In addition, ADE stated that it would have
been helpful to have received a copy of the Department’s Standards for Evaluating the Quality
of Program Performance Data prior to the audit so that the agency would have had an
opportunity to review or implement the standards.



ED-OIG                          Control Number ED-OIG/A09-A0001                             Page 7
OIG’s Response

Based on ADE’s comments on the draft report, we revised Recommendation 1.2 to recommend
that LEAs “maintain written documentation” of their processes rather than “certify” their
processes.


Finding No. 2 – ADE Did Not Ensure That Data Submitted by LEAs on
                Hardcopy Reports Were Accurately Reflected in Its
                Databases

ADE did not accurately enter in its databases the data that LEAs provided on hardcopy reports.
For intervention and placement, ADE extracts information from its census database to provide
OSEP with data. ADE collects data for the census database using electronic files or hardcopy
reports submitted by LEAs. Of the three school districts we reviewed, Tucson and Deer Valley
submitted electronic files to ADE, while Dysart submitted hardcopy reports. During our limited
assessment and testing of ADE’s processing of the electronic files, nothing came to our attention
that would cause us to doubt that the electronic data was properly recorded in the census
database. However, we found that ADE’s census database did not reflect all student withdrawal
dates reported by Dysart. Also, ADE did not have procedures to ensure that exiting, discipline
and personnel data provided by LEAs were accurately recorded in its ACCESS files.

ADE’s Census Database Did Not Reflect All Student Withdrawal Dates. Dysart provided us
with a copy of its report for December 1, 1998. The report contained the withdrawal dates for
students who had either left the school district or no longer required special education services.
Our comparison of Dysart’s report to information contained in ADE’s census database found that
student withdrawal dates were not reflected in the database for at least 27 students. As a result,
the 27 students were improperly included in ADE’s performance data for intervention and
placement. Also, the students were improperly included in ADE’s reported child count that was
used by OSEP to allocate IDEA funds to states. According to ADE’s School Finance
Department, the error was caused by Dysart not using the appropriate reporting form.

ADE Did Not Verify Its Data Entry Process for Exiting, Discipline and Personnel Data. ADE
staff entered data provided by LEAs on hardcopy reports into separate ACCESS files for each of
the three performance indicators. ADE used these databases to prepare its report to OSEP. ADE
did not have procedures to ensure that the data was accurately entered. Our tests found that data
had not been correctly entered in the database for one LEA. The personnel report submitted by
Dysart showed 31.5 full-time equivalents for special education teachers, while ADE’s database
listed only 20 for the school district. Without verification of the data entry process, there is no
assurance that ADE’s data for its over 350 LEAs were entered correctly in its databases.




ED-OIG                          Control Number ED-OIG/A09-A0001                            Page 8
Recommendations

The Assistant Secretary for Special Education and Rehabilitative Services should require ADE
to:

2.1      Implement procedures for verifying the accuracy of data entry for exiting, discipline and
         personnel data.

2.2      Return IDEA funds received based on improperly reported child counts.

ADE’s Comments

ADE concurred with our finding and recommendations. ADE stated that it is implementing
procedures to ensure greater accuracy and completeness of its data. Also, ADE stated that it is in
the process of re-verifying the child count for December 1, 1998.


Finding No. 3 – ADE and LEAs Did Not Retain Supporting Documents for
                Reported Exiting, Discipline and Personnel Data

ADE and two LEAs did not retain all supporting documentation for the reported exiting,
discipline and personnel data submitted to ADE for the 1998-1999 school year. Federal
regulations require that programmatic documentation be retained for three years.
34 CFR §80.42(b) states “... records must be retained for three years from the starting date
specified in paragraph (c) of this section.” Paragraph (c) states “the retention period for the
records of each funding period starts on the day the grantee or subgrantee submits to the
awarding agency its single or last expenditure report for that period.”

State-Level Supporting Documentation Not Retained. ADE could not provide us with the
spreadsheet used to track receipt of LEA reports on exiting, discipline and personnel data. ADE
used the spreadsheet to ensure that all LEAs provided reports for each of the three indicators.
Since ADE was unable to locate the spreadsheet, we could not confirm that ADE followed its
procedures and that all LEA-reported performance data were actually recorded in the databases.

LEA-Level Supporting Documentation Not Retained. Two of the three LEAs did not retain all
supporting documentation for performance data reported to ADE.

At Tucson, we found the school district had not retained all supporting documentation for the
reported exiting, discipline and personnel data.

         •    Exiting data. The school district had retained the reports received from each high
              school. However, one of the two high schools we tested had not retained the
              supporting documentation for its report.

         •    Discipline data. The two service centers that prepared the data had retained the logs
              of reported disciplinary actions. However, the service centers had not retained the
              documentation showing the names of the students selected from the logs who
              represented the counts reported to the district.
ED-OIG                          Control Number ED-OIG/A09-A0001                             Page 9
         •    Personnel data. The school district had not retained supporting documentation for
              the reported numbers.

For our review, Tucson staff provided reconstructed documents for the performance data listed
above. In all cases, the documents did not reconcile to the reported data.

At Dysart, we found that the school district had not retained any supporting documentation for
the exiting, discipline and personnel data reported to ADE. Dysart staff provided us with
reconstructed documents for the personnel data. However, the documents did not reconcile to
the reported data.

Without supporting documents showing student names, staff names or analyses of school
records, ADE staff, auditors or others conducting reviews do not have the necessary records to
evaluate the accuracy of LEA-reported performance data. Also, the LEAs would not have
information needed to evaluate the reported data if significant variances occurred in the
subsequent years’ data. As a result, the LEAs’ ability to identify reasons for the variances and
take action to improve or build upon its performance could be impacted.

Recommendation

The Assistant Secretary for Special Education and Rehabilitative Services should require ADE
to:

3.1      Retain all State-level supporting documentation and ensure that LEAs and their schools
         retain supporting documentation for reported exiting, discipline and personnel data for at
         least the retention period specified in 34 CFR §80.42 (b).

ADE’s Comments

ADE concurred with our finding and recommendation. ADE stated that state-level supporting
documentation is retained for the required amount of time, but the spreadsheet referred to in the
finding was apparently lost during an office move. ADE also stated that the LEAs have been
notified of the requirement to retain supporting documentation for the required period.


Finding No. 4 – LEAs’ Reliance on Manual Processes and Varying Data
                Collection Methods Could Impact ADE’s Ability to Provide
                Reliable Data

The three LEAs included in our review relied on manual processes to collect and report data for
one or more of the performance indicators. The following table shows those indicators where the
LEAs used manual processes:




ED-OIG                           Control Number ED-OIG/A09-A0001                            Page 10
             Performance Indicator          Tucson             Deer Valley           Dysart
                                            Online               Online             Electronic
         Intervention and Placement       Computerized         Computerized        Spreadsheet
                                            System               System                (b)
                                                                 Online
         Exiting                              Manual           Computerized          Manual
                                                                 System

         Discipline                           Manual              Manual             Manual


         Personnel                               (a)              Manual             Manual

      (a) Tucson did not maintain supporting records and current district staff could not provide an
          explanation of the process used to obtain reported data.
      (b) Dysart provided ADE with a hardcopy report that was generated from the electronic
          spreadsheet.

We also found that the manual processes used to collect and report exiting data varied among
schools within an LEA and among LEAs. For example, the processes used by Tucson varied
among its high schools and differed from the process used at Dysart.

         •     At Tucson, the ten high schools individually compiled the exiting data submitted to
               the district office. The two high schools that we reviewed used different methods
               for compiling data. Staff at one high school manually recorded exiting information
               taken from the district’s computerized database on an electronic spreadsheet. The
               other high school reconstructed the supporting data for its report by using a
               graduation list and “Addition and Deletion” forms contained in individual student
               files.

         •     At Dysart, a district clerk maintained an electronic spreadsheet containing
               information on the district’s population of children with disabilities. Staff
               determined the exit category based on the dates and student grade level on the
               spreadsheet and personal recollections of the students.

The use of manual processes and varying collection methods increases the risk that reported data
may not be accurate. ADE officials advised us of plans to implement an on-line statewide
system for maintaining information on children with disabilities who are receiving special
education services. The Arizona State legislature has not yet authorized implementation of the
system due to information privacy concerns. An on-line system could eliminate or reduce
reliance on manual processes and inconsistencies that may occur as a result of varying collection
methods.




ED-OIG                           Control Number ED-OIG/A09-A0001                                  Page 11
Recommendation

The Assistant Secretary for Special Education and Rehabilitative Services should:

4.1      Encourage ADE’s efforts to implement an on-line statewide system for maintaining
         information on children with disabilities and ensure that the system, when developed,
         provides data that can be readily summarized for use in reporting performance data.

ADE’s Comments

ADE concurred with our finding and recommendation. ADE stated that it is interested in
developing an on-line statewide system that collects, stores and summarizes student data, and is
also capable of tracking the movement of individual students between schools.




ED-OIG                         Control Number ED-OIG/A09-A0001                           Page 12
                                       Background

The GPRA, enacted on August 3, 1993, specifies the purposes of the Act which include:

         •   To help Federal managers improve service delivery, by requiring that they plan for
             meeting program objectives and by providing them with information about program
             results and service quality; and

         •   To improve congressional decision-making by providing more objective information
             on achieving statutory objectives and on the relative effectiveness and efficiency of
             Federal programs and spending.

GPRA requires that Federal agencies prepare a five-year strategic plan for their program
activities. Starting with fiscal year (FY) 1999, Federal agencies must prepare annual
performance plans and report on program performance.

The Department published its Strategic Plan 1998-2002 in September 1997. The Department’s
1999 Performance Report and 2001 Annual Plan was submitted to Congress in March 2000.
The 2001 Annual Plan contained nine performance indicators for the IDEA, Part B — Special
Education Program. The Department relies on state-reported data for measuring performance for
six of the nine listed indicators.

ADE is responsible for administering the IDEA, Part B — Special Education Program in the
State of Arizona. The State of Arizona has about 350 LEAs and charter schools and received
$66 million of IDEA, Part B funds for the 1999-2000 award year. ADE reported to OSEP that,
on December 1, 1998, the State had a total of 88,598 children who were receiving special
education services.




ED-OIG                          Control Number ED-OIG/A09-A0001                            Page 13
                      Purpose, Scope and Methodology


The purpose of the audit was to: (1) identify the process used by ADE to accumulate and report
IDEA, Part B performance data to OSEP, (2) determine whether ADE management controls
ensured that the performance data was reliable and (3) identify barriers or obstacles that may
impact ADE’s ability to provide quality performance data. Our review covered the State-
reported 1998-99 school year data for the performance indicators: inclusive settings and regular
education settings (placement), earlier identification and intervention (intervention), graduation
(exiting), suspensions and expulsions (discipline) and qualified personnel (personnel).

To accomplish our objectives, we interviewed State officials and staff responsible for collecting,
processing and reporting the performance data to OSEP. We evaluated ADE’s procedures to
ensure that LEA-reported data was accurately recorded in its electronic databases and that the
data reported to OSEP was supported by the information contained in ADE’s databases. As of
December 1, 1998, the State of Arizona had 353 LEAs, of which 34 LEAs were unified school
districts that reported 300 or more children for the IDEA, Part B program. We focused our
review on unified school districts since these LEAs included schools covering preschools and
grades K through 12. For three of the 34 LEAs, we reviewed the LEAs’ procedures and
interviewed LEA staff responsible for collecting and reporting performance data to ADE. To
ensure that we evaluated procedures at LEAs with varying populations of children with
disabilities, we grouped the 34 LEAs based on their reported child counts and randomly selected
an LEA from each group.

          Number
          of LEAS     Reported Child Count               Selected LEA
              2       Over 5,000                    Tucson Unified School District
             10       Between 1,000 and 5,000       Deer Valley Unified School District
             22       Between 300 and 999           Dysart Unified School District

In addition, we reviewed the audit working papers for ADE’s single audit for fiscal year ended
June 30, 1998 and the single audit reports for each selected LEA for the same period.

Intervention and placement. To achieve our audit objectives for these two performance
indicators, we extensively relied on computer-processed data extracted from ADE’s census
database for December 1, 1998. Our assessment of the reliability of the database was limited to
(1) gaining an understanding of the procedures used by ADE and the three LEAs to collect,
process and review the data, (2) testing ADE’s entry of data provided by the three LEAs and (3)
confirming that data provided by the three LEAs was supported by information contained in
school and student records.

To test ADE’s processing of reports submitted by Tucson and Deer Valley on electronic files, we
randomly selected students from the LEAs’ databases to confirm that student information was
accurately reflected on the December 1, 1998 extract from the census database. The following
table shows the total records for each LEA and the number of student records that were
reviewed.


ED-OIG                         Control Number ED-OIG/A09-A0001                             Page 14
                                                            Number of Students
                                    Records on LEA’s
                    LEA                                    For Whom Information
                                      Database (a)
                                                               Was Reviewed
           Tucson                         10,672                    50
           Deer Valley                    3,953                        26
          (a) Databases contained multiple records for individual students.

Dysart submitted the information to ADE on hardcopy reports. To test ADE’s processing of the
hardcopy reports, we confirmed that reported information for listed students was accurately
reflected on the December 1, 1998 extract.

To test the data provided by the three selected LEAs, we randomly selected students for each
LEA from ADE’s database. The following table shows the total students reported by each LEA
and the number of students reviewed:

                                                            Number of Students
                                      Total Students
                    LEA                                    For Whom Information
                                        Reported
                                                               Was Reviewed
           Tucson                          6,749                    50
           Deer Valley                     2,614                       20
           Dysart                            531                       20

Other than the error mentioned on page 8 regarding student withdrawal dates, nothing came to
our attention as a result of our limited assessment and tests that caused us to doubt the
acceptability of the data or the adequacy of ADE and LEA procedures for assuring the accuracy
of the data for intervention and placement information submitted to OSEP for the 1998-1999
school year.

Exiting, Discipline and Personnel. To achieve our audit objectives for these three performance
indicators, we (1) gained an understanding of the procedures used by ADE and the three LEAs to
collect, process and review the data, (2) confirmed that data reported to OSEP was supported by
information contained on ADE’s individual ACCESS files, (3) tested ADE’s entry of data
provided by the three LEAs into the ACCESS files and (4) confirmed that data provided by the
three LEAs was supported by information contained in school and student records. The
following table shows the total students/personnel reported by each LEA and the number of
student/personnel files reviewed:




ED-OIG                           Control Number ED-OIG/A09-A0001                        Page 15
                       Exiting                       Discipline                     Personnel
                 Total        Student           Total         Student          Total      Personnel
   LEA
                Students       Files           Students        Files         Personnel        Files
                Reported     Reviewed          Reported      Reviewed        Reported     Reviewed
Tucson            493            10              140             30             940            40
Deer Valley        147             10              10             13 (a)         313             10
Dysart               28             5               0             (b)             89             5
(a) Reviewed all students listed for the 1998-1999 school year on Deer Valley’s database for tracking
discipline hearings.
(b) Dysart did not provide the district’s discipline records for our review. Therefore, we were unable to
confirm the accuracy of the reported number.

From our assessment and tests, we concluded that we could not rely on the management controls
of ADE and LEAs to ensure the reliability of the performance data reported to OSEP for exiting,
discipline and personnel. The Audit Results section of the report provides details on our
findings.

We performed our fieldwork at ADE in Phoenix, Arizona and at Tucson, Deer Valley and Dysart
district and school offices. Fieldwork was conducted from October to November 1999 and in
January 2000. Our audit was performed in accordance with generally accepted government
auditing standards appropriate to the scope of the review described above.




ED-OIG                            Control Number ED-OIG/A09-A0001                                Page 16
                    Statement on Management Controls

As part of our review, we assessed the system of management controls, policies, procedures and
practices applicable to ADE’s process for collecting and reporting performance data for the
IDEA, Part B program as required by GPRA. Our assessment was performed to determine the
level of control risk for determining the nature, extent and timing of our substantive tests to
accomplish the audit objectives.

For the purpose of this report, we assessed and classified ADE’s significant controls related to
collection and reporting of performance data into the following categories:

   •     Guidance and technical assistance,
   •     Collection of data from LEAs,
   •     Data compilation and report preparation and
   •     Monitoring LEA data collection and reporting processes.

Because of inherent limitations, a study and evaluation made for the limited purpose described
above would not necessarily disclose all material weaknesses in the management controls.
However, our assessment disclosed management control weaknesses that adversely affected
ADE’s ability to report accurate performance data for GPRA. These weaknesses included failure
to report all instances for the performance indicators, lack of reviews or inadequate reviews of
reported data, lack of documentation on data collection processes, data submitted by LEAs on
hardcopy reports not accurately reflected in ADE databases and failure to retain supporting
documentation.




ED-OIG                          Control Number ED-OIG/A09-A0001                          Page 17
                                                                                                                                                           Attachment A
                        IDEA, Part B Program Objectives, Performance Indicators and Performance Data
                                                     2001 Annual Plan

                                                                                                                               PERFORMANCE DATA
    PROGRAM OBJECTIVE                                        PERFORMANCE INDICATOR                                         COLLECTED FROM OSEP FORMS
 All preschool children with disabilities     1.1 Inclusive settings. The percentage of preschool children with           SEAs report the number of students ages 3-5 by
 receive services that prepare them to            disabilities who are receiving special education and related services   age and educational placement
 enter school ready to learn.                     in inclusive settings will increase. (See below NOTE)

 All children who would typically be          2.1 Earlier identification and intervention. The percentage of              SEAs report number of children with disabilities
 identified as being eligible for special         children served under IDEA ages 6 or 7, compared to ages 6-21,          receiving special education by:
 education at age 8 or older and who are          will increase.                                                              § disability and age and
 experiencing early reading or behavioral                                                                                     § disability and ethnicity
 difficulties receive appropriate services
 earlier to avoid falling behind their
 peers.
 All children with disabilities have access   3.1 Regular education settings (school age). The percentage of              SEAs report the number of students ages 6-21, by
 to the general curriculum and                    children with disabilities ages 6-21 who are reported by states as      age category, disability and placement
 assessments, with appropriate                    being served in the regular education classroom at least 80 percent
 accommodations, supports and services,           of the day will increase. (See below NOTE)
 consistent with high standards.              3.3 Suspensions or expulsions. The percentage of children with              SEAs report the number of students suspended or
                                                  disabilities who are subject to long-term suspension or expulsion,      expelled, unilateral removed or removed based on
                                                  unilateral change in placement or change in placement if their          hearing by:
                                                  current placement is likely to result in injury to someone, will             § disability and basis of removal and
                                                  decrease.                                                                    § ethnicity and basis of removal
 Secondary school students with               4.1 Graduation. The percentage of children with disabilities exiting        SEAs report the number of students ages 14-21
 disabilities receive the support they need       school with a regular diploma will increase and the percentage who      that exited special education by:
 to complete high school prepared for             drop out will decrease.                                                      § age, disability and basis of exit,
 postsecondary education or employment.                                                                                        § age and basis of exit and
                                                                                                                               § ethnicity and basis of exit
 States are addressing their needs for        5.1 Qualified personnel. The number of states and outlying areas            SEAs report the number and type of teachers and
 professional development consistent              where at least 90 percent of special education teachers are fully       other personnel to provide special education and
 with their comprehensive system of               certified will increase.                                                related services for children ages 3-21. SEAs must
 personnel development.                                                                                                   report the number of staff:
                                                                                                                                § fully certified and
                                                                                                                                § not fully certified

NOTE: At the time we initiated the audit, the performance indicators 1.1 inclusive settings for preschool children and 3.1 regular education settings for school age children
were combined as one placement indicator.



ED-OIG                                                             Control Number ED-OIG/A09-A0001                                                                   Page 18
                                               Attachment B




         ADE’s Comments to the Report




ED-OIG       Control Number ED-OIG/A09-A0001         Page 19
                           REPORT DISTRIBUTION LIST


                                                           No. of
Auditee                                                    Copies
Ms. Lisa Graham Keegan                                       1
Superintendent of Public Instruction
Arizona Department of Education
1535 West Jefferson Street
Phoenix, Arizona 85007

Action Official
Ms. Judith Heumann                                           1
Assistant Secretary, Office of Special Education and
 Rehabilitative Services

Other ED Officials
Mr. Kenneth Warlick                                          1
Director, Office of Special Education Programs

Mr. Louis Danielson                                          1
Research to Practice Division
Office of Special Education Programs

Mr. Charles Miller                                           1
Supervisor, Post Audit Group, OCFO

Mr. Alex Wohl                                                1
Director, Public Affairs, OS

Office of Inspector General (electronically)
Inspector General                                             1
Assistant Inspector General for Audit                         1
Assistant Inspector General for Investigations                1
Assistant Inspector General for Analysis and Inspections      1
Regional Inspectors General for Audit                      1 each
Director, State and Local Advisory and Assistance Team        1
Director, Non-Federal Audit Team                              1