oversight

Information Report on Data Accumulated by SEAs and Reported to ED: ESEA/Title I and Perkins Vocational Education Programs.

Published by the Department of Education, Office of Inspector General on 2000-02-15.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

  Information Report on Data Accumulated by SEAs and Reported to ED:
        ESEA/Title I and Perkins Vocational Education Programs


                              INFORMATION REPORT




                             Control Number ED-OIG/S17-90009
                                        March 2000

Our mission is to promote the efficient                    U.S. Department of Education
and effective use of taxpayer dollars                        Office of Inspector General
in support of American education                                       Washington D.C.
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                                                                                         FINAL



                                                                    TABLE OF CONTENTS

EXECUTIVE SUMMARY...........................................................................................................................................2

INTRODUCTION..........................................................................................................................................................6

SEA-SUPPLIED DATA................................................................................................................................................9
  Processes for Collecting and Reporting Data.........................................................................................................9
  Controls to Ensure the Reliability of the Data .....................................................................................................11
  Limitations and Weaknesses of SEA-Supplied Data..........................................................................................13
    Timeliness of the Data.........................................................................................................................................13
    Consistency of the Data Over Time ...................................................................................................................14
    Comparability of the Data Across States ..........................................................................................................14
    ED Activities to Address Limitations and Weaknesses .................................................................................15
  Barriers and Obstacles to Improving the Quality of SEA-Supplied Data.......................................................16

TITLE I PERFORMANCE REPORTS....................................................................................................................17
  Participation Data......................................................................................................................................................17
  Status of Title I Schools ...........................................................................................................................................17
  Student Proficiency Levels ......................................................................................................................................18
  SEA Evaluation.........................................................................................................................................................19
  ED Activities to Improve the Data Collection Process for the Title I Program.............................................20

PERKINS PERFORMANCE REPORTS.................................................................................................................21
  Enrollment Data ........................................................................................................................................................22
  Performance Measurement Systems ......................................................................................................................24
  SEA Evaluation.........................................................................................................................................................25
  ED Activities to Improve the Data Collection Process for the Perkins program...........................................26

DATA ON ACADEMIC ACHIEVEMENT............................................................................................................27
 Administering the Assessment ...............................................................................................................................28
 Scoring of the Assessment ......................................................................................................................................29
 Reporting and Use of the Results...........................................................................................................................29

PLACEMENT DATA..................................................................................................................................................31
  Survey Method..........................................................................................................................................................31
  Records Search Method...........................................................................................................................................32

ED Comments on Report.............................................................................................................................................34

OBJECTIVES, SCOPE, AND METHODOLOGY.................................................................................................35

Appendix A – Background on Title I and Perkins Programs ................................................................................38
Appendix B – Statistical Profiles of the Five SEAs Visited..................................................................................42
Appendix C – Descriptions of the Systems for Assessment of Academic Achievement.................................43
Appendix D – Acronyms .............................................................................................................................................48
Appendix E – Terminology.........................................................................................................................................49
Appendix F – Publications Cited in this Report ......................................................................................................50
Appendix G – OESE’s Comments.............................................................................................................................51
Appendix H – OVAE’s Comments ...........................................................................................................................52




March 2000                                                                         -1 -                                                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                             FINAL



                                EXECUTIVE SUMMARY

We conducted a review to provide information to the U.S. Department of Education (ED)
on the processes used by state educational agencies (SEAs) to collect and report data to
ED. Our review focused on two of ED’s major state formula grant programs: Grants for
Schools Serving At-Risk Children (Title I program) and Vocational and Technical
Education Assistance to the States (Perkins program). To conduct our work, we
reviewed technical literature and visited five SEAs: California, Georgia, Massachusetts,
Minnesota, and New Jersey.          The five SEAs we selected are not statistically
representative and thus the results cannot be projected. However, the information we
obtained from these visits provides insight into the processes used by SEAs in providing
data to ED for the Title I and Perkins programs. Our review was not an audit of either
program or of any of the five states we visited.

Data obtained by ED from the SEAs for these two programs are used to monitor and
evaluate the programs. ED also plans to use some of the data in its annual performance
reports to Congress required by the Government Performance and Results Act of 1993
(GPRA or the Results Act). In addition, the authorizing legislation for the Perkins
program requires that ED report state-by-state comparisons of the Perkins performance
data. To be useful for these purposes, the data must be reliable, comparable across states,
consistent over time, and timely.

The results of our review are grouped into overall observations on SEA-supplied data and
more specific observations on Title I data, Perkins data, data on academic achievement,
and placement data.

Overall Observations on SEA-Supplied Data

Based on our review, we have the following observations about the data provided by
SEAs for the Title I and Perkins programs:

    •   The process of collecting data for both of these programs is complex. Thousands
        of entities are involved. Much of the data originates at the thousands of local
        education agencies (LEAs) that operate the programs. The LEAs then send
        reports to their SEAs, who in turn send reports to ED. The SEAs also collect data
        from other sources such as the testing contractor for the statewide academic
        assessment.

    •   Each SEA has its own unique processes for collecting data. At the SEAs visited,
        the method for collecting data from LEAs varied, from the submission of paper
        forms to the exchange of computer diskettes to transmission through the Internet.
        The amount of detail provided by the LEAs also varied from data on individual
        students to aggregate data for a district.




March 2000                                     -2 -                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                FINAL



    •   Each SEA has its own unique control structure. One SEA required a certification
        from the LEAs on the data submitted. Another conducted a quality control review
        of the testing contractor’s scoring of the statewide academic assessments.

    •   The data are not timely. The majority of states filed their Title I and Perkins
        performance reports for the program year 1996/1997 after the due date. Some
        states filed as late as four or five months after the due date, which is already set at
        least six months after the end of the program year. Thus, ED received some state
        Title I and Perkins program data almost a year after the program year-end.

    •   The data may not be consistent over time. At three of the states visited, spring
        1998 marked the initial use of the assessment instrument to measure academic
        achievement. These states will not have sufficient, consistent data on academic
        achievement to measure educational progress for several years.

    •   When used for national aggregation or comparison, such as GPRA reporting, the
        data are likely to not be comparable across states. In many cases, the states
        define how the data is collected and reported. For example, states select the
        assessment instruments for academic achievement and decide the number and
        meaning of the proficiency levels. States also decide whether placement data are
        obtained through searches of state unemployment compensation records or
        through surveys of former students.

The lack of comparability across states and the lack of consistency over time are to some
extent inherent in the process. Performance measurement is a dynamic process.
Congress has provided flexibility to states and local educational agencies that can affect
data collection. Improvements in data quality and timeliness may require new systems.
Designing, building, and maintaining systems requires significant human and financial
resources. In addition, some states are dealing with privacy concerns about what
information state databases can contain.

Because of the complexities of the processes, improvements will only come through the
joint efforts of states and ED. ED has begun working with states to improve data
collection through the Integrated Performance and Benchmarking System (IPBS). The
goal of the IPBS is to reduce paperwork and to streamline the federal education program
reporting system in such a way that it provides states, districts, school boards, and parents
with accurate, comparable information about federal program results.

Observations on Title I Data

For fiscal year 1999, ED plans to use information on the count of distinguished schools
and on student assessments to measure the performance of the Title I program for
reporting under the Results Act. However, that information may not be available from
Title I performance reports for fiscal year 1999. The authorizing legislation for the Title I



March 2000                                     -3 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                             FINAL



program does not require that a final assessment system be in place until program year
2000/2001. Some SEAs do not have a definition of adequate yearly progress, which is
needed to identify distinguished schools. In addition, some SEAs have not been
providing in their Title I performance reports information on student proficiency levels
because they have just started using their statewide assessment systems.

Observations on Perkins Data

For fiscal year 1999, ED is not planning to use state reports as a data source for GPRA
reporting on the Perkins program because the information needed would not be available
in those reports. For fiscal year 2000, ED is planning to use placement data from state
reports in its GPRA reporting on the Perkins program. Recent amendments to the
authorizing legislation of the Perkins program will require ED to report state-by-state
comparisons of performance information.

SEAs have had different definitions for vocational students for reporting on the Perkins
program. In addition, SEAs have not always had access to student level data on
vocational students and thus may have difficulty obtaining performance data on the
Perkins program. This situation poses a particular challenge to ED in meeting the
requirement to report state-by-state comparisons of information. To address this
challenge, ED has been working with states to develop an accountability framework.
This framework should assist states in moving toward comparable definitions.

Observations on Data on Academic Achievement

As part of our review, we obtained information on how the states we visited administered
their statewide academic assessments, scored the assessments, and reported and used the
results. Both the Title I and Perkins programs require data on academic achievement.
All five states we visited had a statewide assessment system that was required by state
statutes. In all five states, a testing contractor developed and scored the assessment, and
reported the results to the LEAs and the SEA. In one state, staff of the SEA conducted a
quality control check of the contractor’s work. All five states had test security measures
in place. However, the specific measures used varied in each state. All five states
indicated that the results are or will be used to identify schools and districts in need of
improvement.

Observations on Placement Data

We also obtained information on how the states collected data on the placement of
vocational education students. For the Perkins program, each state’s performance
measurement system must include a measure of placement in postsecondary education or
employment. Data on placement can be obtained from either a survey of former students
or from a search of state unemployment compensation and postsecondary records. In one
of the states we visited, the LEA conducted the survey using guidance provided by the



March 2000                                     -4 -                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



SEA. The SEA had set a minimum response rate. In another state, the search of records
was conducted by another entity and the results provided to the SEA.

ED Comments on Report

The Office of Elementary and Secondary Education (OESE) and the Office of Vocational
and Adult Education (OVAE) provided written comments on the draft of this report,
which are reprinted in appendices G and H, respectfully.

OESE noted that to improve data quality, in addition to the consolidated performance
report and the development of IPBS mentioned in the report, provisions related to
performance data have been included in the proposal for reauthorization of Elementary
and Secondary Education Act (ESEA).

OVAE stated that the results presented in this report are consistent with what OVAE has
learned during the course of working with states to implement the new Perkins
legislation. In their comments, OVAE provided more detailed information on the
requirements of the Perkins Act, efforts to improve data quality and comparability among
states, and plans for future GPRA reporting. OVAE noted that the new requirements
substantially increased the complexity of data collection. OVAE’s goal is to have a
vocational and technical education data system that is reliable, comparable among states,
consistent over time, and timely. To build that system, OVAE is working closing with
the states.

In addition, department officials noted that our work was limited to the SEA level and did
not include a review of the consistency of data collection within states. Department
officials also noted that by their nature state standards and assessment systems change
and, thus, consistency of data will always be an issue.

OESE and OVAE also provided technical comments that we incorporated where
appropriate throughout the text.




March 2000                                     -5 -                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                     FINAL



                                      INTRODUCTION

The U.S. Department of Education’s (ED) first performance report on fiscal year 1999
required by the Government Performance and Results Act (GPRA or Results Act) is due
in March 2000. The most common issue raised during our audit of ED’s implementation
of the Results Act was the availability of quality data for that first performance report and
for management decisions. 1 In June 1998, the General Accounting Office (GAO)
concluded that ED’s fiscal year 1999 Annual Plan did not provide sufficient confidence
that its elementary and secondary performance information will be credible because data
limitations were not included in the plan. 2 Much of that performance information will be
provided by sources external to ED, such as state educational agencies (SEAs).

In response to a request by ED and to follow-up on the earlier reports, we conducted a
review to identify:

    (1)   the processes used by SEAs to accumulate and report data to ED;
    (2)   the controls 3 used to ensure reliability4 of the data;
    (3)   limitations or weaknesses in that data; and
    (4)   barriers or obstacles to improving the quality of that data.

Our work focused on two of ED’s major state formula grant programs:

     • Grants for Schools Serving At-Risk Children (Title I program) authorized by Title
       I/Part A of the Elementary and Secondary Education Act (ESEA), and

     • Vocational and Technical Education Assistance to the States (Perkins program)
       authorized by Title I of the Carl D. Perkins Vocational and Technology Education
       Act of 1998 (Perkins III).

To obtain information on the data collected by SEAs and reported to ED, we reviewed
technical literature and visited five SEAs: California, Georgia, Massachusetts, Minnesota,
and New Jersey. Although the five SEAs selected are not statistically representative and
thus the results cannot be projected, they provide insight into the processes used by SEAs
in providing data to ED for the Title I and Perkins programs. A statistical profile of the
five SEAs we visited is included in Appendix B. Our work focused on the SEAs and not
on the local or federal level. Additional information on how we conducted this review is
included in the section “Objectives, Scope, and Methodology.”
1
  ED-OIG, 1998. Moving Towards A Results-Oriented Organization: A Report on the Status of
ED’s Implementation of the Results Act. (ED-OIG/A17-70007) Page 20.
2
  GAO, 1998. THE RESULTS ACT: Observations on the Department of Education’s Fiscal Year
1999 Annual Performance Plan. (GAO/HEHS-98-172R) Page 3.
3
  For purposes of this report, controls are what an entity does to provide reasonable assurance that
what should happen happens.
4
  For purposes of this report, reliability refers to the precision with which a phenomena is
measured. A measured value is considered reliable if it is accurate for its intended use.


March 2000                                      -6 -                             ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                       FINAL



The Title I program provides over $7 billion to schools, especially in low-income
communities, to improve education for children at risk of failing to achieve high
standards. In 1996, over 50,000 schools received Title I funds. ED distributes Title I
funds to the SEAs using a legislatively mandated formula. In turn, the SEAs distribute
the funds to local education agencies (LEAs) to support programs in schools.

The authorizing legislation for the Title I program requires that states have “high-quality,
yearly student assessments”5 for students served by the program. SEAs will report
disaggregated results6 from those assessments to ED in an annual performance report.
State and local assessments are a data source in the fiscal year 1999 Title I program
performance plan.

The Title I legislation also requires that                         GPRA Requirements
SEAs designate as “distinguished” any
Title I school that, for three consecutive             For fiscal years 1999 and 2000, ED submitted
                                                       an “Annual Plan” to Congress to meet the
years, has exceeded the state’s definition
                                                       requirements of the Results Act. Those
of adequate progress. The number of Title              Annual Plans contain “Program Performance
I schools designated as distinguished is an            Plans” for each of ED’s programs reported
indicator in the fiscal year 1999 Title I              individually or grouped by related program
program performance plan.                              purpose. ED’s Annual Plans include “Program
                                                       Performance Plans” for both the Title I and
The Perkins program provides over $1                   Perkins programs. Throughout this report, the
billion to develop more fully the academic,            phrase “program performance plans” refers to
vocational, and technical skills of                    those documents.
secondary and postsecondary students
enrolled in vocational and technical                   As allowed by OMB, ED decided to combine
                                                       its FY 1999 report with its FY 2001 plan. At
education programs. As with the Title I
                                                       the end of February 2000, ED issued a pre-
program, ED distributes Perkins funds to               publication copy of its combined report/plan.
the SEA or other designated state agency7              ED expects to issue a final version by the end
using a legislatively mandated formula.                of March 2000.
The designated state agency then
distributes the funds to LEAs and other
eligible recipients, such as community
colleges.



5
  An assessment is an exercise, such as a written test, portfolio, or experiment, that seeks to
measure a student’s skills or knowledge in a subject area.
6
  Disaggregated results are results broken down by subgroups, such as gender or student
economic status.
7
  A state agency other than the SEA can be the primary fiscal and reporting agency for the
Perkins program (e.g., the state agency responsible for colleges and universities). In some states,
more than one state agency administers the Perkins program. For example, the SEA may
administer the secondary programs and another state agency, the postsecondary programs.


March 2000                                      -7 -                               ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                              FINAL



Recent amendments, which first became effective for school year 1999-2000, to the
authorizing legislation of the Perkins program require states to identify core indicators in
four areas, establish levels of performance for those indicators, and report data to ED.
Those amendments also require states to describe in their state plan how the state “will
ensure” that the data they reported to ED is “complete, accurate, and reliable.” ED is “to
disseminate state-by-state comparisons of the information” to the public and Congress.
In return for this increased accountability, those amendments provide more flexibility to
the states; for example, fewer dollars are earmarked to specific programs.

For fiscal year 1999, the data sources in the Perkins program performance plan are
studies conducted by the Planning and Evaluation Service (PES) and the National Center
for Education Statistics (NCES). For fiscal year 2000, the data source for the indicator
related to student outcomes is state performance reports. The data sources for the other
indicators in the fiscal year 2000 program performance plan are studies conducted by
PES and NCES.

The Title I program is administered by ED’s Office of Elementary and Secondary
Education (OESE). The Perkins program is administered by the Office of Vocational and
Adult Education (OVAE). Additional information on the Title I and Perkins programs
and on the program performance plans for both programs is included in Appendix A.

Acronyms used in this report are listed in Appendix D. Appendix E contains definitions
of the technical terms used in this report.




March 2000                                     -8 -                       ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                FINAL



                                   SEA-SUPPLIED DATA

Below is a summary of our observations about the processes SEAs use to collect and
report data, the controls used to ensure the reliability of the data, the limitations and
weaknesses in the data, and the barriers and obstacles to improving the quality of the
data. In later sections of this report, we provide detailed information about program
reporting for the Title I and Perkins programs, data on academic achievement, and
placement data.

Processes for Collecting and Reporting Data

OIG Observations - The processes for gathering data on the Title I and Perkins programs
are complex because thousands of entities are involved. Each SEA has its own unique
processes for collecting the data.

Much of the data for the Title I program originates at the 50,000 participating schools.
Data is collected from the schools by the LEAs. The LEAs send reports to the SEA. In
turn, the SEAs send to ED annual performance reports. As with the Title I program,
much of the data for the Perkins program originates at the LEAs and the other eligible
recipients of Perkins funds. Reports from these local entities are also sent to the SEA or
other designated state agency. In turn, the state agencies send to ED annual performance
reports. Even when the SEA is responsible for both programs, separate divisions within
the SEA may be responsible for the two programs.

During our visits, we noted that:

    •   Data transfer from the LEAs to the SEAs varied, from the submission of paper
        forms to the exchange of diskettes to transmission through the Internet.

    •   The detail provided to the SEA by the LEAs varied from data on individual
        students to aggregate data for a district.

    •   A testing contractor developed and scored the statewide academic assessment and
        provided the results to both the LEAs and the SEA. Four of the five SEAs got the
        results at the student level in electronic form. The fifth SEA will get the results at
        the student level in electronic form in the future.

    •   Placement information was obtained through either a search of state
        unemployment compensation records and postsecondary records or a survey of
        former students.

The diagram below illustrates the basic processes for gathering performance data on the
Title I and Perkins programs. Since each state has its own governance and organizational




March 2000                                     -9 -                         ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                                                                                             FINAL



structure, the information trail represented below is generalized and does not illustrate the
processes used by any particular state.


                                                          LEA-SEA-ED-Congress Information Trail
         The processes for gathering and reporting data on the Title I and Perkins programs are complex because of the numerous state and local
         educational entities involved. The diagram below represents generalized processes for the gathering and reporting of data rather than
         the specific processes that are used by any of the five states we visited. The actual processes are unique to each state because each state
         has its own governance and organizational structure. Our work focused on the SEAs and not the local or federal level.

       LOCAL LEVEL:                                                      STATE LEVEL:                                                      FEDERAL LEVEL:
       One to 1,044 school districts                                     50 states plus territories        Annual Performance              ED Program offices
                                                                                                           Report for Title I (in
                                           Reports on enrollment,                                          future consolidated report
                                           participation in various                                        for ESEA programs)
                                           programs, use of                                                                                   OESE
                            LEA            funds,etc.                                SEA                                                                                     ED’s Annual
                                                                                                                                                                             Performance
                                                                                                                                                                             Report required
                                                                                                                                                                             by GPRA


                    LEA                                                                                                                                    OUS *                     Congress
                                                                                     SEA




           LEA
                                                                                              SEA                Annual Performance           OVAE
                                                                                                                  Report for Perkins


                                                                                                                                              * ED’s Office of the Under Secretary (OUS)
                                                                                                                                                will be responsible for preparing ED’s Annual
                                                                                                                                                Performance Report required by GPRA.
  Assessment Results
         (directly to                                                      Placement Information
     LEA and SEA)                                                          (either to LEA or SEA)
                                                                                                                                        Data/Reports

               Testing contractor for                                 State unemployment compensation                     This diagram only includes state reported data for the
                 assessment system                                        and postsecondary records                       Title I and Perkins programs. It does not include other
           (generally contracted by SEA)                                              or                                  sources of data on those programs, such as studies and
                                                                       Survey results of former students                  evaluations. It also does not include other federal grant
                                                                                                                          programs, such as those under IDEA, or other federal
                                                                                                                          data collections, such as those by NCES.




March 2000                                                                              - 10 -                                                   ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                  FINAL



Controls to Ensure the Reliability of the Data

OIG Observations - Because each state has its own processes, each state will have its
own system of controls to ensure the reliability of its data.

During our visits, we noted the following controls in the SEA processes.

    For data supplied by the LEAs:

        •    To detect errors in reports provided by LEAs, all five SEAs indicated that they
             ran edit checks. Such edit checks could include comparing the data to data
             from prior years.

        •    To prevent errors in reports provided by LEAs, one of the five SEAs required
             that the LEAs provide a certification that the data was “complete and
             accurate.”8

    For academic achievement data:

        •    To prevent errors and irregularies during the administration of the academic
             assessment, all five states had some security measures, such as designated
             coordinators or logs of test materials, over their statewide academic
             assessments.

        •    To detect errors in scoring, one of the five SEAs conducted a quality control
             check of the testing contractor’s scoring of the academic assessment.

        •    To encourage the inclusion of all students in the academic assessment, one of
             the five SEAs assigned the minimum score to any students who did not take
             the assessment.

During our work, we did not determine the effectiveness of these controls or of any other
controls used by the SEAs. However, we believe that, if properly implemented, the
above controls would help ensure the reliability of the data. The diagram below
illustrates the types of controls that could be used by a state in its processes for gathering
data for the Title I and Perkins programs. The diagram is generalized and does not
represent the system of controls used in any particular state.




8
  For the Title I report, SEA officials are to certify that the data is the “most accurate data
available.” For ED’s Annual Plan required by GPRA, ED program managers will be required to
either assert that the data used for their programs’s performance measurement are “reliable and
valid” or have plans for improvement.


March 2000                                    - 11 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                                                                                                 FINAL




                                                  Controls in the LEA-SEA-ED-Congress Information Trail

        Diagram depicts some of the controls that we found in the LEA-SEA-ED-Congress Information Trail for the Title I and Perkins
        programs. The diagram does not depict the system of controls used in any particular state since that system is unique to each state.
        During our work, we did not determine the effectiveness of these controls.
                             Control
                           Test security
                            measures

           Control
       Minimum score for             LEAs                Control
        absent students           Administers          Certification                                                                                       ED
                                   assessment          “complete &                                                           Annual
                                                         accurate”                                                         Performance   Control       Control
                      Control                                            Reports on enrollment,
                                                                                                                              Report       Edit   OESE Assertion
                                                                                                                                         checks        “valid &                Annual
                 Preliminary review                                      participation in various               Control                                reliable”
                                                                                                                                          of data                            Performance
                       by LEA.                                           programs, use of funds,etc.          Certification                                                     Report
    Testing                                                                                                  “most accurate
   Contractor                                                                Control                         data available”
                       Assessment results                                                    SEAs
   Develops,                                                               Edit checks
                                                                                        Prepares reports Control
                                                                                                                                                                       OUS          Congress
   scores, and                                                                of data
   reports on                                                                                  on         Review
                                                                            including Title I and Perkins    of
   assessment           Control                                            comparison programs for ED
                     Quality control                                                                       report                        Control     Control
                                                                             to prior                                                      Edit
     Control        check of scoring                                                                                      Annual                     Assertion
                                                                            year’s data                                 Performance      checks OVAE “valid &
  Manual check        by SEA staff                                                                                                        of data
                                                                                                                           Report                    reliable”
    of scoring
  when original
   answer sheet
  was scored by                                                                                                                                         Data/Reports
    scanning                                                               Placement data
  into computer                                                                                                                       Control: Instructions/Guidance

                                             State unemployment
                                                                       Survey results
                                              compensation and
                                                                     of former students
                                            postsecondary records                                                   This diagram only includes state reported data for the
                                                                                            Control
                            Control                                                         Minimum                 Title I and Perkins programs. It does not include other
                           Minimum              (The records search or                                              sources of data on those programs, such as studies and
                                                                                            response
                           match rate           survey could be done                                                evaluations. It also does not include other federal grant
                                                                                             rate for
                           for record           by either the LEAs or
                                                                                             surveys                programs, such as those under IDEA, or other federal
                            searches             the SEA.)
                                                                                                                    data collections, such as those by NCES.




March 2000                                                                                   - 12 -                                                 ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                       FINAL



Limitations and Weaknesses of SEA-Supplied Data

OIG Observations – The data are not timely. The data may not be consistent over time.
When used for national aggregation or comparison, such as GPRA reporting, the data
are likely to not be comparable across states.

Timeliness of the Data

We reviewed records maintained by ED on the receipt of performance reports for the
Title I and Perkins programs. For program year 1996/1997, those records indicate that all
but two of the Title I reports were received by ED after the due date of February 1998.
Program year 1996/1997 ended in June of 1997.

               Month Report Received          # of SEAs
                      February                     2
                      March                       28
                      April                        3
                      May                          7
                      June                        12 (one year after program year-end)
                      July                         1
                          Total9                  53

Only 17 of the 54 Perkins performance reports for program year 1996/1997 were
received by the due date of December 31, 1997. 10 Four were never received. As of
March 3, 1999, for program year 1997/1998, only 29 of the 54 Perkins performance
reports had been received by ED. These reports were due December 31, 1998. Part of
the lack of timeliness for the Perkins performance report could be that the paperwork
clearance for the form expired in January 1997. The clearance for the form was not
renewed because of pending reauthorization of the Perkins program’s authorizing
legislation. Instead, ED obtained a voluntary agreement with the states to use the form. 11

Historically, states have not been able to meet due dates for performance reports. Some
reports have arrived almost a year after the program year-end.

As part of our review, we did not conduct detailed work to determine the specific causes
for the lack of timeliness of the data. In the next section, we discuss some barriers and
obstacles to improving the overall quality of SEA-supplied data that we identified during
our review. ED has contracted for a study on the causes of the delay of the state Title I

9
  The 53 entities that submit Title I performance reports are the 50 states, the Bureau of Indian
Affairs, Puerto Rico, and the District of Columbia.
10
   The 54 entities that submit Perkins performance reports are the 50 states, 3 territories, and the
District of Columbia.
11
   Although the paperwork clearance had expired, under ED’s General Administrative
Regulations, SEAs were still required to supply the data.


March 2000                                      - 13 -                             ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                               FINAL



reports. Information from that study should assist ED in developing an appropriate
federal response.

Consistency of the Data Over Time

Data may not be consistent over time. States can change the methods they use to collect
data. For example, one of the states we visited changed from collecting placement data
using surveys of former students to searching state unemployment compensation and
other records. Any trend analysis of placement data for that state needs to consider the
effect the change in collection method may have on the trend.

As another example, the initial use of the assessment instrument to measure academic
achievement in three of the states we visited was spring 1998. Therefore, these states
may not have sufficient trend data to evaluate educational progress for a few years.

SEAs may change the method used to collect data. ED needs to know when SEAs make
these changes because it affects how the data is analyzed over time.

Comparability of the Data Across States

ED is planning to use some of the data reported by SEAs to ED in its annual performance
reports to Congress required by the Results Act. For the Title I program, ED is planning
on using data from state assessments and the status of Title I schools (meeting adequate
yearly progress, identified for improvement, or identified as distinguished). For the
Perkins program, ED is planning to use data from the state performance measurement
systems required by Perkins III.

A weakness in SEA data when it is used for national aggregation or comparison, such as
GPRA reporting, is the lack of comparability across states. For example, lack of
comparability in:

•    Who or what is counted or measured – For the Perkins program, SEAs determine
    how a vocational student is defined. Those definitions vary, both in the number of
    courses a student takes and in the grades covered.

•   How data is collected – For the Perkins program, the SEAs we visited varied in how
    they collect placement data. Some search state unemployment compensation records;
    while others survey former students.

•   How performance is measured. – For example, each of the five states we visited used
    a different assessment instrument to measure academic achievement.

The lack of comparability across states is to some extent inherent to the process.




March 2000                                    - 14 -                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                             FINAL



    •   Performance measurement is a dynamic process. Local, state, and federal
        governments make changes to the legislation that govern and organize the
        educational systems. These changes can affect what data are collected, when the
        data are collected, and how they are collected.

    •   Congress has provided flexibility to states and local educational agencies in
        operating federal programs in return for increased accountability. This flexibility
        can include allowing states to make decisions that affect what data is collected.

ED Activities to Address Limitations and Weaknesses

As part of an overall strategy for data quality when reporting under the Results Act, ED
has develop department-wide standards for performance indicator measurement. ED
began training department staff on those standards in Fall 1999. It is phasing in a
requirement that program managers examine the indicators and data for their programs to
determine their accuracy and validity and, as necessary, develop plans for improvement.




March 2000                                    - 15 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                              FINAL



Barriers and Obstacles to Improving the Quality of SEA-Supplied Data

Officials at one SEA mentioned that outdated hardware and software were obstacles to
improving data quality. Designing, building, and maintaining systems, especially those
capable of electronic transfer of data, requires significant resources, both financial and
human. One state indicated to us that it had several openings in information technology
positions. Those vacancies may slow the state’s development of electronic transfer of
data. Officials at another state indicated that addressing year 2000 concerns had delayed
improvements to its data systems.

Beyond the technical concerns of designing and building data systems, our review also
noted that some states are dealing with privacy concerns about the state databases
containing detailed student information, such as social security numbers. Generally,
social security numbers are needed to obtain placement data through searches of state
unemployment compensation records. Social security numbers can also be used as
unique student identifiers to prevent duplicate counts of students.

Officials in one state mentioned that for the Title I program ED does not provide enough
notice of changes to report format and content to allow the state to update its data
collection systems. In November 1998, the Council of Chief State School Officers Board
of Directors approved the following resolution:

        The implementation of any new or revised data collection instruments or
        categories, or the establishment/revision of any instructions associated with
        such instruments and categories, shall be optional for SEAs if final
        documents, with Office of Management and Budget (OMB) approval, are
        not issued by July 1 of the school year preceding the school year for which
        the data collection is requested. Further, this provision should be applicable
        to data collection requirements for all the various programs under the United
        States Department of Education, with the exception of surveys or other
        projects which do not impact state and/or local data systems. 12

Sometimes Congress mandates when a data collection will begin. For example, Perkins
III was enacted in October 1998 and effective for the program year that began July 1,
1999. Thus, ED and the states had only eight months to plan for the implementation of
the new legislative requirements.

The complexities of data collection require ED to work closely with the states in
developing data collection requirements. ED has begun to do so through activities such
as the Integrated Performance and Benchmarking System.



12
  Resolution of the Education Information Advisory Committee (EIAC) of the Council of Chief
State School Officers (CCSSO).


March 2000                                    - 16 -                       ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



                          TITLE I PERFORMANCE REPORTS

Summary - For fiscal year 1999, ED plans to use information on the count of
distinguished schools and on student assessments to measure the performance of the Title
I program for GPRA reporting. However, that information may not be available from
Title I performance reports for fiscal year 1999. The authorizing legislation for the Title
I program does not require that a final assessment system be in place until program year
2000/2001. Some SEAs do not have a definition of adequate yearly progress, which is
needed to identify distinguished schools. In addition, some SEAs have not been providing
in the Title I performance report any information on student proficiency levels because
they have just started using their statewide assessment systems.

SEAs have reported data annually to ED in a year-end Title I performance report.
Beginning with program year 1998/1999, SEAs must report data in a year-end
consolidated report for all ESEA13 formula grant programs. Both the single program and
the consolidated program reports request information on student participation, status of
Title I schools (meeting adequate yearly progress, identified for improvement, or
identified as distinguished), and student proficiency levels.

Participation Data

SEAs obtain participation data for the Title I program from LEAs. In the states we
visited, LEAs submitted the participation data to the SEA either in hardcopy, on SEA-
supplied diskettes, or through the Internet. The SEA-supplied diskettes contained
software that had built-in edit checks. The data from the diskettes was downloaded into
the SEA’s computer. When the data was supplied in hardcopy, the SEA keyed the data
into its computer. No matter which method is used to obtain the data, the SEAs ran edit
checks on the data entered into its computer. One SEA also required that the LEA include
a certification that the data provided was “complete and accurate.”

OIG Observations - The use of SEA-supplied software is likely to improve the quality of
the data because of the built-in edit checks and the reduced risk of error caused by re-
entering the data into the SEA’s computer from hardcopy. The use of a certification can
improve data quality because it establishes a means of accountability for the data.

Status of Title I Schools

We reviewed the year-end reports filed by the five SEAs we visited for program year
1996/1997. 14 In those reports:


13
   Elementary and Secondary Education Act. ESEA is the authorizing legislation for several
formula grant programs to the states including the Title I program.
14
   The program year 1996/1997 reports were the most recent reports available to us at the time of
our review.


March 2000                                    - 17 -                            ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                   FINAL



      •   All five SEAs provided a count of the schools identified for improvement.

      •   Two of the SEAs did not provide a count of the schools identified as
          distinguished.

One of the SEAs that did not have a count of distinguished schools indicated to us during
our visit that they did not currently have a definition of “adequate yearly progress.”15
Such a definition is needed to categorize Title I schools as “meeting adequate yearly
progress,” “identified for improvement,” or “identified as distinguished.” The other SEA
without a count indicated during our visit that so far only one school (not in program year
1996/1997) has been designated as distinguished because few schools had the required
three years of academic assessment data to even be eligible for consideration.

During our visits, we noted that two states obtained their count of schools identified as
distinguished as follows:

      •   In one state, elementary and secondary schools are only eligible to be recognized
          in the distinguished schools programs in alternate years. Once recognized, a
          school is not eligible to reapply for five years.

      •   In another state, the SEA provides a list of schools that are candidates for
          distinguished school status. The LEAs can nominate schools from that list but are
          not required to do so.

OIG Observations – For fiscal year 1999, ED plans to use the information on the count of
distinguished schools to measure the performance of the Title I program. However, that
information may not be available since not all states have a definition of adequate yearly
progress. Moreover, some school’s lack of three years of trend data and the
methodologies used by some states (such as the alternating between elementary and
secondary schools and the voluntary nominations) appears to render a trend in the
number of Title I schools “identified as distinguished” invalid for purposes of measuring
Title I program performance.

     Subsequent to our work, ED issued a pre-publication copy of its combine FY 1999 report and
     FY 2001 plan. In that document, ED dropped the indicator on the count of distinguished
     schools.

Student Proficiency Levels

We reviewed the year-end reports filed by the five SEAs we visited for program year
1996/1997. These reports did not include data on student proficiency levels. Although
the form requests data on student proficiency levels, the law does not require a final
15
  The SEA had submitted recommendations for defining “adequate yearly progress” to the state
legislature.


March 2000                                     - 18 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                               FINAL



assessment system until program year 2000/2001. The results of such assessments are
used to determine student proficiency levels.

In a separate section of this report, we discuss the processes and controls used to
administer and score academic assessments and report the results. During our visits, we
noted that:

    •   Spring 1998 was the first time three of the states administered their statewide
        assessment instruments.

    •   In all five states, both the LEAs and the SEA received the results of the statewide
        assessment from the contractor that scored the assessment.

    •   In four states, the SEA had the results from the statewide assessment at the
        student level in electronic form. In one of those states, the results are maintained
        for the SEA in a separate database at a contractor’s site. In the fifth state, the
        SEA will have the results at the student level in electronic form in the future.

In one state, the LEAs determined, within state guidelines, the levels of student
proficiency using their own methodology. Such methodologies can include additional
factors beyond the results of assessment scores on standardized tests, such as grades.
SEA officials noted that this system allows districts to use local standards appropriate for
their student population. The LEAs in this state submitted to the SEA, by school,
disaggregated numbers and percentages of students at each proficiency level. The SEA
manually entered the data from each LEA into its computer and ran edit checks.

OIG Observations – For fiscal year 1999, ED plans to use information on student
proficiency levels to measure performance of the Title I program. However, that
information may not be available since some states have just started using their statewide
assessment systems. Further, the authorizing legislation does not require that states have
a final assessment system in place until program year 2000/2001.

  Subsequent to our work, ED issued a pre-publication copy of its combined FY 1999 report
  and FY 2001 plan. In that document, ED noted that some states do not have the necessary
  data on student proficiency levels.


SEA Evaluation

During our visits to the SEAs, we asked state officials for their comments and opinions
on the Title I program and Title I reporting. Each of the following comments were made
by a SEA official. Since these comments were made by one individual, they may not be
representative:




March 2000                                    - 19 -                        ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



      • The Title I requirements were a driving force in the implementation of a statewide
        standards-based accountability system. This system will be used to determine
        student learning needs, improve school programs, and recognize outstanding
        academic achievements of students.

      • ED does not provide sufficient notice of changes in the format and content of the
        Title I report to allow the state to update its data collection system.

SEA officials in all five SEAs we visited mentioned using or planning to use assessment
data to identify schools in need of improvement.

ED Activities to Improve the Data Collection Process for the Title I Program

ED is working with several states on the Integrated Performance and Benchmarking
System (IPBS) for elementary and secondary program data collections. The goal of the
IPBS is to reduce paperwork and to streamline the federal education program reporting
system in such a way that it provides states, districts, school boards, and parents with
accurate, comparable information about federal program results.




March 2000                                    - 20 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                           FINAL



                         PERKINS PERFORMANCE REPORTS

Summary – For fiscal year 1999, ED is not planning to use state reports as a data source
for GPRA reporting on the Perkins program because the information needed would not
be available in those reports. For fiscal year 2000, ED is planning to use placement data
from state reports in its GPRA reporting on the Perkins program. Recent amendments to
the authorizing legislation of the Perkins program will require ED to report state-by-
state comparisons of performance information. SEAs have differed on how they define
vocational students for reporting on the Perkins program. In addition, SEAs have not
always had access to student level data on vocational students and thus may have
difficulty obtaining performance data on the Perkins program. This situation poses a
particular challenge to ED in meeting the requirement to report state-by-state
comparisons of information. To address this challenge, ED has been working with states
to develop an accountability framework. This framework should assist states in moving
toward comparable definitions.


Reporting for the Perkins program is more complicated than reporting for the Title I
program because it involves both secondary and postsecondary programs. Thus, local
entities, in addition to LEAs, receive Perkins funds and provide data on the program. In
addition, more than one state agency can be involved in the administration of the
program. For example, the SEA can administer the secondary programs, while the state
agency responsible for colleges and universities administers the postsecondary programs.

Appendix B contains the percentage of Perkins funds budgeted in fiscal year 1998 for the
secondary and postsecondary programs in each of the states we visited. In three of the
states, over 60 percent of the funds were used in secondary programs. In a fourth state,
the funds were divided equally between secondary and postsecondary programs. In the
fifth state, 35 percent of the funds were used in secondary programs. In that state, the
SEA was not the primary reporting agency for the Perkins program. Our work focused
on the SEAs and the secondary programs.

The authorizing legislation for the program was reauthorized in October 1998 only a few
months prior to the beginning of our review. The prior authorizing legislation for the
Perkins program (Perkins II) required a performance measurement system. Perkins III
expanded on those requirements.          Appendix A contains information about the
performance measurement requirements of the Perkins program. As we were conducting
our review, states were in the process of adjusting their systems to comply with Perkins
III.

The program year 1996/1997 Perkins performance report requested enrollment data and a
description of the state’s performance measurement system for its Perkins program. The
OMB clearance on the form expired on January 31, 1997. Due to the pending
reauthorization of the Perkins program, a new clearance was not obtained. Instead, ED



March 2000                                    - 21 -                    ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                      FINAL



obtained a voluntary agreement with the states to use the form. 16 A new form for the
performance report is being developed for program year 1999/2000, which will
incorporate the new Perkins requirements.

Enrollment Data

SEAs obtained enrollment data for the Perkins program from the LEAs using processes
similar to the Title I program: hardcopy or electronic, such as a SEA-supplied diskette.
The states varied in the detail that the LEAs provided the SEAs. For example, one SEA
received the social security numbers of the vocational students, while another SEA
received only district level counts of the vocational education students by gender,
special populations, and subject areas. The SEAs compared the current year data to data
from prior years. One SEA indicated that they also compared the data to other SEA
information it had on student enrollment.

We reviewed the Perkins performance                    Changing Nature of Vocational Education
reports for program year 1996/1997 for each
of the five states we visited. 17 The draft            In recent years, vocational education has
instructions for the performance report                increased the emphasis on academic skills.
                                                       The Perkins Act recognizes this change in
contained sample tables for the SEAs to use
                                                       vocational education by requiring academic
in reporting enrollment data. All five states          and technical achievement as an area of
submitted enrollment tables that were                  performance measurement.
modified from the sample enrollment tables
in the draft instructions for the report. The          In addition, recent education reforms have
draft instructions for the report also                 recognized the importance of providing all
requested explanatory information on the               students with training that prepares them for
data in the enrollment tables, for example,            employment.
how was enrollment defined. None of the
states provided all the requested explanatory          These changes in vocational education
information.      For example, two states              complicate the process of defining the
                                                       population of vocational students for data
provided no explanation of how enrollment
                                                       collection.
was defined.

SEAs define vocational students differently. For example, in one state we visited, a
vocational education student is defined as:

        “A student who is enrolled in a planned, sequenced, and organized system of
        coherent courses that leads to employment and/or advanced training.”

In another state we visited, a vocational education student is defined as:

16
   Although the paperwork clearance had expired, under ED’s General Administrative
Regulations, SEAs were still required to supply the data.
17
   The program year 1996/1997 reports were the most recent reports available to us at the time of
our review.


March 2000                                    - 22 -                             ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



        “A student who has declared that they are pursuing a program aimed at
        employment in a pre-baccalaureate setting, or enrolled in a one-course
        occupational program, or enrolled in a second course of a multi-course
        occupational program.”

Neither definition equates to the definition of a vocational education concentrator used
by ED in the fiscal year 2000 Perkins program performance plan: 18

        “A student who completes 3 or more Carnegie units19 in a single specific labor
        market preparation program area.”

OVAE conducted a study of the SEA definitions of vocational education students using
the 1994 enrollment charts. 20 This study grouped the definitions into two categories:

      • A student who took at least one vocational education course – 36 SEAs.

      • A student who was enrolled in a vocational education program – 15 SEAs. 21

The OVAE study also noted that states differed in the range of grades that SEAs included
when counting vocational students:

                  Range of Grades      # of SEAs Using that Range
                     Grades 6-12                    5
                             7-12                  19
                             9-12                  25
                            10-12                   1
                            11-12                   1
                      Total                        51

OIG Observations – Perkins III requires that ED disseminate state-by-state comparisons
of performance information. Currently, states do not have uniform definitions of
vocational education students. The legislation does not impose a definition for vocational
students. In addition, the legislation does not grant the Secretary the authority to require
states to adopt specific definitions. Instead, OVAE has been working with states to
develop a definition of a vocational education concentrator.



18
   This definition was used by ED in its fiscal year 2000 program performance plan for Perkins
prepared to meet the requirements of the Results Act. The definition has not been impose on the
states through regulation.
19
   A Carnegie unit is a standardized measure of class time equivalent to one fifty-minute course,
five times a week for an entire school year.
20
   Study of State Data on Vocational-Technical Education, 1993-1994. OVAE.
21
   The total is 51 for the 50 states and the District of Columbia.


March 2000                                    - 23 -                            ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



Performance Measurement Systems

The program year 1996/1997 reports of the states we visited included lists of the
performance areas to be measured, such as academic achievement or placement. In some
cases, the reports included the performance measure, and, in a few cases, the level of
outcome expected to be achieved. However, the reports generally did not include the
actual performance data for any of the measures. The prior authorizing legislation
(Perkins II) did not require states to report this information to ED.

During our visits, we noted the following
                                                       High Stakes are Associated with
about the development of performance
                                                       Perkins Reporting (which first became
measurement systems for vocational                     effective for school year 1999-2000)
education programs:
                                                       Sanctions – “If an eligible agency fails
    •   One SEA is in a consortium with                to meet the state adjusted levels of
        other states to develop core                   performance, has not implemented an
        competencies for technical skills.             improvement plan…, the Secretary may,
        The consortium was viewed as a cost-           after notice and an opportunity for a
        effective way to develop an                    hearing, withhold from the eligible
        examination to assess technical skills.        agency all, or a portion of, the eligible
        The target date for the first test is          agency’s allotment under this Title.”
                                                       (Perkins III, section 123(d)(2))
        2003.
                                                       Rewards – A state that exceeds its
    •   One SEA mentioned that they started            agreed upon performance levels for
        working in 1997 with the state agency          Perkins, the Adult Education and
        responsible      for    postsecondary          Family Literacy Act, and Workforce
        programs to develop a shared vision            Investment Act Title I is eligible to
        of kindergarten through 14-level               receive incentive grants. (Workforce
        education.     That effort involved            Investment Act, section 503)
        stakeholders throughout the state.

    •   In another state, the SEA is currently working with the state agency responsible
        for the postsecondary Perkins programs and the state’s Department of Labor to
        develop core indicators and levels of performance to satisfy the requirements of
        the Perkins Act and the Workforce Investment Act of 1998. The SEA is also
        involving local agencies in the development process.

Perkins III requires performance measurement of academic achievement and placement
among other core indicators. One source of data on academic achievement is statewide
assessments. In a separate section of this report, we discuss the processes and controls
used to administer and score assessments and report the results. In addition, in a separate
section of this report, we discuss the processes and controls for obtaining placement data
on vocational education students.




March 2000                                    - 24 -                            ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



During our visits, we noted the following about data collection for the vocational
education performance measurement systems:

    •   One SEA plans to use the academic assessment data for all tenth grade students
        who took the test instead of just vocational education students because SEAs will
        not be able to identify vocational education students out of the pool of students
        tested. The data for the measure of retention (students who do not drop out) will
        also be school-wide instead of solely for vocational education students.

    •   One SEA allows LEAs to choose from one of three options for how the LEA will
        measure academic achievement.

    •   One SEA receives the placement data on program completers from the LEAs in
        January of the following year, which is after the December due date of the
        Perkins report.

    •   One SEA recently switched from gathering placement data through surveys of
        former students to obtaining the data from searches of records.

    •   One SEA indicated that it compared information reported by LEAs to results of
        state board exams and other industry certifications as a way of gauging the school
        districts’ accuracy in reporting data to the SEA.

    •   One SEA mentioned joint planning sessions with other state agencies to
        determine the needs for collaborative data collections and data sharing.

OIG Observations – Building performance measurement systems is a difficult task
requiring the involvement of many stakeholders. Beyond developing the measures, states
face the challenge of obtaining credible data in a cost-effective manner. This is
especially challenging when the SEA does not have student level data.

SEA Evaluation

During our visits to the SEAs, we asked state officials for their comments and opinions
on vocational education, the Perkins program, and Perkins reporting. Each of the
following comments were made by a SEA official. Since these comments were made by
only one individual, they may not be representative.

         • Vocational technical education helps prepare all learners for continuing
           education.

         • The performance measurement system needs to satisfy not only Perkins
           requirements but should also be aligned with and viewed as a vital component
           of the overall accountability system for education.


March 2000                                    - 25 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                              FINAL



         • There is a need for additional guidance and standards on the use of
           employment service and unemployment data and student tracking.

         •   The information requested by ED is needed by the state also.

ED Activities to Improve the Data Collection Process for the Perkins program

ED has been working with the states to develop an accountability framework for
vocational education. This framework should assist states in moving toward comparable
definitions. This activity was, in part, in response to the requirement that ED
“disseminate state-by-state comparisons of the information” to the public and Congress.
OVAE provided additional information on its activities to improve data quality and
comparability among states in its response to this report, which is in Appendix H.




March 2000                                    - 26 -                        ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                  FINAL



                        DATA ON ACADEMIC ACHIEVEMENT

This section contains information on how the states we visited administered their
statewide academic assessments, scored the assessments, and reported and used the
results. As part of our work, we did not evaluate the validity of the academic assessments
used by the five states we visited nor did we evaluate the appropriateness of the use of
those assessments. Our work concentrated on the processes used to administer the
assessments, collect and report the results, and the controls used in those processes.

Summary - Both the Title I and Perkins programs require data on academic achievement.
All five states we visited had a statewide assessment system that was required by state
statutes. In all five states, a testing contractor developed and scored the assessment, and
reported the results to the LEAs and the SEA. In one state, staff of the SEA conducted a
quality control check of the contractor’s work. All five states had test security measures
in place. However, the specific measures used varied in each state. All five states
indicated that the results are or will be used to identify schools and districts in need of
improvement.


Both the Title I and Perkins programs require data on academic achievement. One source
of data on academic achievement is the results of tests (academic assessments).

All five of the states we visited had a statewide assessment system. All five systems
were required by state statutes. Those systems are described in Appendix C. Three of
these states used criterion-referenced tests. 22 The other two states used norm-referenced
tests. 23 One of these states plans to augment the norm-referenced test with questions
based on the state’s content standards. The other state, which is currently revising its
curriculum, plans to have a criterion-referenced test in use by spring 2000.

During our visits, we noted that:

     •   All five states involved a testing contractor in the system.

     •   In all five states, the testing contractor developed and scored the assessment, and
         reported the results to the LEAs and SEA. The assessments that were criterion-
         referenced to state standards were developed by the testing contractors with the
         oversight of the SEA.


22
   A criterion-referenced test is a test designed to determine whether each student has achieved
specific skills or concepts. Each individual is compared with a preset standard for acceptable
achievement. The performance of other examinees is irrelevant.
23
   A norm-referenced test is a test designed to rank each student with respect to the achievement
of others in broad areas of knowledge. Each individual is compared with other examinees and
assigned a score.


March 2000                                    - 27 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                  FINAL



    •   In four states, the SEA contracted with the testing contractor. In the fifth state,
        one that used a norm-referenced test, the LEAs contracted with the testing
        contractor.

    •   In all five states, the tests were administered by LEA staff.

Administering the Assessment

The first place that the validity and the reliability of the assessment data can be
compromised is during the testing process. All the SEAs that we visited had test security
measures in place. These measures included:

    ü   Designated assessment coordinators at districts and schools,
    ü   Training of coordinators and test proctors/administrators,
    ü   Policy and procedure manuals on test administration,
    ü   Logs maintained on the quantity and location of test materials,
    ü   Certifications of actions taken by responsible officials, and
    ü   Preprinted labels for shipment of test materials to the contractor.

Officials in one SEA mentioned that the state does not have a test manipulation penalty.
If irregularities are discovered in the administration of the tests, professional standards
boards handle the allegations administratively.

Testing in two states was simultaneous
throughout the state. In the other three states, a         IDEA requires inclusion
time period (two months, three weeks, and two
weeks) was given for when the districts were to            The Individuals with Disabilities
conduct the assessment. In one of those states,            Education Act Amendments of
                                                           1997 (IDEA) requires that students
testing was simultaneous within the school; that           with disabilities be included in
is, within a school all fourth graders would take          general state and district-wide
the reading test at the same time. To encourage            assessment programs. (section 612
districts to ensure that all students are tested,          (a) (17))
one SEA counted absent students in the
aggregate score as having the minimum score.

OIG Observations – While the test administration controls discussed above can reduce
the risk of errors and irregularies, ultimately, proper test administration depends on the
proficiency of those administering the assessment. The results of academic assessments
are important because, in some states, the results are used to rank schools. The results
can also inform decisions about funding and state accreditation. Therefore, it is
important that the tests are administered properly.




March 2000                                    - 28 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



Scoring of the Assessment

The testing contractors scored the assessment in all five states. Answers for multiple
choice questions were scanned into computers. Edit checks were done on the results. In
some cases, tests were checked manually and the results compared to the results
generated by the computer. The performance-based assessments (such as essays and
short-answer questions) were scored manually. In one state, staff of the SEA conducted a
quality control check by rescoring a sample of the assessments and comparing the results
to the contractor’s results. In two of the states we visited, school districts received
preliminary results to review for possible errors.

OIG Observations – While we did not verify its effectiveness, we believe that the quality
control check by SEA staff of contractor results would provide greater assurance
concerning the reliability of the data. We also believe that a preliminary review of the
results by school districts could help identify any errors in the scoring.

Reporting and Use of the Results

The testing contractor provided the results to the LEAs and SEAs in either hardcopy,
electronic format or both. SEAs publicly disseminated the disaggregated results by
LEAs and/or schools, often on the Internet. Individual results were provided to the
parents of the students. The student report provided to parents in one state contained:

     ü the results of the test (numerical scores and proficiency levels),
     ü specific comments about the student’s performance-based tasks (i.e. comments on
       the student’s essay),
     ü comparison of the student’s score to the average scores of the school, district and
       state, and
     ü explanatory information on the results (definitions of the proficiency levels and
       overview of the test content).

The student’s score was displayed as a range and a caution was included on the report
that the range represented the scores the student might receive if the test was taken more
than once.

Disaggregation of the data was based on the initial coding of the student demographics
(e.g., gender, ethnicity, low income, migrant, limited English proficiency (LEP), and
disability status) of the test document. In the states we visited, the test administrators
either coded the test documents by hand or affixed labels with pre-identified student
information to the test documents. In one of the states that used the coding by hand
method, state officials indicated that there was a minor risk that errors could be made in
coding the forms. One of the states that used the pre-identified student information has
instituted a no over-ride policy. Hand changes to the student information on the test
documents would not be used.



March 2000                                    - 29 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                 FINAL



All five states indicated that the results are or will be used to identify schools and districts
in need of improvement. In some cases, the use of the data is currently limited because
of the lack of trend data. States also plan to use the data to evaluate individual student
progress, generate information for accountability, and allow for comparison between
schools and districts. In one state, teachers are required to participate annually in a staff
development program on the use of tests within the instructional program designed to
improve the student’s academic achievement. In another state, in the future, a
satisfactory score on the statewide assessment will be necessary to receive a diploma.

OIG Observations – We believe that the usefulness of the data is enhanced when
contextual information is provided. In addition, we believe that cautions about data
limitations should be reported with the data. The results of assessments are often
analyzed by the disaggregation categories. Those analyses depend on the accuracy of the
initial coding of the test documents.




March 2000                                    - 30 -                         ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                              FINAL



                                        PLACEMENT DATA

This section contains information on how states we visited obtain placement data on
vocational education students.

Summary - For the Perkins program, each state’s performance measurement system must
include a measure of placement in postsecondary education or employment. Data on
placement can be obtained from either a survey of former students or from a search of
state unemployment compensation and postsecondary records. In one of the states we
visited, the LEA conducted the survey using guidance provided by the SEA. The SEA had
set a minimum response rate. In another state, the search of records was conducted by
another entity and the results provided to the SEA.


The authorizing legislation for the Perkins program requires that each state’s performance
measurement system include a measure of “placement in postsecondary education or
employment.” That legislation does not establish the method for collecting the data. 24

Placement data can be obtained from either a survey of former students or from a search
of state unemployment compensation and postsecondary records. Below is an example
of each method.

Survey Method

In one state we visited, the LEAs obtained placement data from former students. The
SEA provided guidance on how the survey was to be conducted including providing a
sample form to be sent to each student. Each year, the LEAs obtain the placement status
of current year graduates and prior year graduates. The status of current year graduates is
their placement status between June and September of that year. The status of the prior
year graduates is their placement status the following April or ten months after
graduation.

To obtain the data, the LEAs send surveys to each student. The survey asked the
student’s educational status (attending or not attending school) and employment status
(employed, full-time military, unemployed, or not in labor force). If employed, students
were also requested to provide information about whether their employment is related to
their vocational training and their salary level. The LEAs categorize the students
responses into one of the following six categories:

     (1) Military,
     (2) Employed in a related field (full or part-time),

24
  Unlike the Perkins Act, the Workforce Investment Act Title I requires the use of wage records to obtain
data: “In measuring the progress of the state on state and local performance measures, a state shall utilize
quarterly wage records, consistent with state law.” (section 136(f)(2))


March 2000                                          - 31 -                               ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                     FINAL



    (3)   Pursuing additional education,
    (4)   Employed in an unrelated field,
    (5)   Unemployed, and
    (6)   Not in the labor force.

The SEA’s instructions state that the students should be categorized as high as possible
on the above list.

The LEAs provide totals for each category by vocational program and demographics to
the SEA. The SEA set minimum response rates for the survey. The SEA indicated that it
was considering a review of LEAs’ records of the student surveys as a way of verifying
the placement data.

Records Search Method
                                                                 Privacy / Data Collection
Another SEA we visited obtained the
placement data from a search of state                  “To protect the privacy of families whose
unemployment           compensation       and          children are in school, states and the federal
postsecondary financial aid records. The               government has established strong legal
search was actually conducted by another               statutes to keep private the information in
entity in the state government and the results         education records that schools maintain on
provided to the SEA. Officials indicated               students.” (NCES/NFES. Protecting the
that they had difficulty tracking students             Privacy of Student Records: Guidelines for
who attend postsecondary institutions out of           Education Agencies. 1997)
state and do not receive financial aid.
                                                       These privacy statutes in some states
                                                       preclude the SEA from collecting social
OIG Observations on the method used to                 security numbers of secondary students.
collect the data - While not without                   However, a social security number is
deficiencies, data obtained from searches of           generally needed to track students to
records is generally more reliable than self-          unemployment compensation records.
reported data obtained from surveys of
former students. In addition, searches of              When SEAs have the social security numbers
records are generally less expensive than              of students, SEAs need to maintain security
surveys of former students and do not                  over the system so that the information is
burden the former students with paperwork.             protected. NCES has issued guidance on
However, there are complex legal issues                security over systems: Safeguarding Your
                                                       Technology: Practical Guidelines for
associated with record searches including
                                                       Electronic Information Security.
SEAs having students’ social security
numbers. If SEAs have student social
security numbers, they must ensure the
security of that information.

OIG Observations on controls over the collection of placement data – While we did not
verify the effectiveness of the following controls, we believe the following controls could
provide greater assurance concerning the reliability of the data:


March 2000                                    - 32 -                             ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



    •   Setting a minimum response or match rate;

    •   Reviewing the data to determine the effect of any systematic bias in the results
        caused by being unable to obtain data on any particular subgroup (e.g. students
        attending schools out of state);

    •   Reviewing the records of surveys or matches conducted by LEAs; and

    •   Disclosing the process used to collect the data and the percentage of students for
        whom data was obtained.




March 2000                                    - 33 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL



ED Comments on Report

OESE and OVAE provided written comments on the draft of this report, which are
reprinted in appendices G and H, respectfully.

OESE noted that to improve data quality, in addition to the consolidated performance
report and the development of IPBS mentioned in the report, provisions related to
performance data have been included in the proposal for reauthorization of ESEA.

OVAE stated that the results presented in this report are consistent with what OVAE has
learned during the course of working with states to implement Perkins III. In their
comments, OVAE provided more detailed information on the requirements of the Perkins
Act, efforts to improve data quality and comparability among states, and plans for future
GPRA reporting. OVAE noted that the new requirements substantially increased the
complexity of data collection. OVAE’s goal is to have a vocational and technical
education data system that is reliable, comparable among states, consistent over time, and
timely. To build that system, OVAE is working closing with the states.

In addition, department officials noted that our work was limited to the SEA level and did
not include a review of the consistency of data collection within states. Department
officials also noted that by their nature state standards and assessment systems change
and, thus, consistency of data will always be an issue.

OESE and OVAE also provided technical comments that we incorporated where
appropriate throughout the text.




March 2000                                    - 34 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                        FINAL



                     OBJECTIVES, SCOPE, AND METHODOLOGY


In December 1998, ED’s Planning and Evaluation Service requested that we review the
processes used by SEAs to collect and report data to ED. We conducted this review to
meet that request and to follow-up on concerns that GAO raised about elementary and
secondary performance information in its review of ED’s fiscal year 1999 Annual Plan.

Objectives

The objectives of our review were to identify the:

     (1) processes used by SEAs to accumulate and report data to ED;

     (2) controls that ensure the reliability of data submitted by SEAs to ED;

     (3) limitations or weaknesses in the data submitted by SEAs to ED; and

     (4) barriers or obstacles to improving the quality of data submitted by SEAs to ED.

Scope

Our review focused on two major state formula grant programs:

     •   Grants for Schools Serving At-Risk Children authorized by Title I/Part A of the
         Elementary and Secondary Education Act, which has an annual appropriation of
         over $7 billion; and

     •   Vocational and Technical Education Assistance to the States authorized by Title I
         of the Carl D. Perkins Vocational and Technology Education Act of 1998, which
         has an annual appropriation of over $1 billion.

Our work focused on the SEAs and not on the local or federal level. For the Title I
program, we focused on indicator 1.1 State and Local Assessments (academic
assessments) and indicator 2.1 Recognition for Quality (count of distinguished schools)
from the fiscal year 1999 Title I program performance plan. 25 For the Perkins program,
we focused on the secondary education program because the postsecondary program in
some of the states we visited was administered by a different state agency. Appendix A


25
  For fiscal years 1999 and 2000, ED submitted an “Annual Plan” to Congress to meet the requirements of
the Results Act. Those Annual plans contain “Program Performance Plans” for each of ED’s programs
reported individually or grouped by related program purpose. ED’s Annual Plans include “Program
Performance Plans” for both the Title I and Perkins programs. Throughout this document, the phrase
“program performance plan” refers to those documents.


March 2000                                       - 35 -                             ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                     FINAL



contains background information on the Title I and Perkins programs and on the program
performance plans for both programs.

Methodology

To achieve our objectives, we interviewed SEA officials and reviewed documents at five
SEAs: California, Georgia, Massachusetts, Minnesota, and New Jersey. Appendix B
provides a statistical profile for each of the five SEAs. Although the five SEAs selected
are not statistically representative and thus the results cannot be projected, they provide
insight into the processes used by SEAs in providing data to ED for the Title I and
Perkins programs.

We limited our time on-site to one or two weeks at each SEA. The SEA officials that we
spoke with during our visits included program directors for Title I and Perkins, and
managers and staff responsible for statewide assessments and for data management. We
did not visit LEAs or interview staff from LEAs. We did not independently verify the
information provided by SEA officials. We did not conduct tests of controls or perform
substantive tests on any data.

We reviewed the logs maintained by ED on the receipt of performance reports for both
programs. We did not perform tests to determine the accuracy of those logs. We
reviewed the program year 1996/1997 performance reports on both programs for the five
states we visited. We also reviewed the program year 1997/1998 Perkins performance
report for New Jersey. The program year 1997/1998 Perkins performance reports for the
other four states and the Title I reports for all five states were not available at the time of
our review.

We reviewed research about performance measurement and educational assessments to
obtain a technical understanding of the concepts and identify current practices. We
reviewed the FY 1999 and FY 2000 program performance plans for the Title I and
Perkins programs. Appendix E contains definitions of the technical terms used in this
report.

Our work did not include a review of the validity26 of the indicators in ED’s annual plans.
In addition, our work did not include a review of the validity27 of any of the assessment
instruments used by the states to measure academic achievement.

Developing educational assessment and performance measurement systems are dynamic
and evolving processes that can be affected by statutory and regulatory changes.


26
   In this context, validity is defined as the extent to which performance is adequately measured.
A measured value is valid if it adequately represents actual performance.
27
   In this context, validity refers to the precision with which the assessment measures what it is
suppose to measure.


March 2000                                     - 36 -                            ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                             FINAL



Therefore, the results we have reached in this report reflect a “snapshot” of the processes
of data accumulation and reporting by these SEAs.

We conducted our review between January and June 1999. We visited four of the SEAs
in January and February 1999, and the fifth, Georgia, in April, May, and June of 1999.
To update our report, in March 2000, we reviewed the sections on the Title I and Perkins
programs in the pre-publication copy of ED’s combined FY 1999 report and FY 2001
plan. In performing this review, we followed the President’s Council on Integrity and
Efficiency (PCIE) Quality Standards for Inspections dated March 1993.




March 2000                                    - 37 -                      ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                               FINAL



Appendix A – Background on Title I and Perkins Programs


The Title I Program Requires Assessments of Academic Achievement

The Elementary and Secondary Education Act (ESEA) amended by the Improving
America’s Schools Act (IASA) in 1994, requires that SEAs have “high-quality, yearly
student assessments” by school year 2000/2001. Those assessments are a data source for
indicators in the fiscal year 1999 and 2000 Title I program performance plans. 28 ESEA
requires that these assessments:

        (1) include at least mathematics and reading or language arts;
        (2) be the same assessments used for all children, if the State measures the
            performance of all children;
        (3) be administered at least once in grades 3 to 5, once in grades 6 to 9, and once
            in grades 10 to 12; and
        (4) allow for disaggregation of results by gender, by each major racial and ethnic
            group, by English proficiency status, by migrant status, by students with
            disabilities, and by economic status.

One purpose of these assessments is to identify schools and districts in need of
improvement.

SEAs will report educational assessment data on Title I participants to ED in the
Consolidated State Performance Report for State Formula Grant Programs Under the
Elementary and Secondary Education Act and Goals 2000: Educate America Act.

Indicator 1.1 State and Local Assessments in the fiscal year 1999 Title I program
performance plan states:

         “Increasing percentages of students in Title I schools will meet or exceed the
        basic and proficient levels in state and local assessments (where in place).”

This indicator is intended to measure progress in achieving the first objective in the
program performance plan:

         “Student achievement in Title I schools and high-poverty schools generally will
        show significant improvement in core subjects.”


28
  For fiscal years 1999 and 2000, ED submitted an “Annual Plan” to Congress to meet the
requirements of the Results Act. Those Annual Plans contain “Program Performance Plans” for
each of ED’s programs reported individually or grouped by related program purpose. ED’s
Annual Plans include “Program Performance Plans” for both the Title I and Perkins programs.
Throughout this report, the phrase “program performance plan” refers to those documents.


March 2000                                    - 38 -                       ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                      FINAL



                                                                             Appendix A (continued)

For the fiscal year 2000 program performance plan, the assessment indicator (1.2 Meeting
or Exceeding State Performance Standards) was revised to:

           “By 2002, 32 states with 2 years of assessment data and aligned standards and
          assessments will report an increase in the percentage of students in schools with at
          least 50% poverty who meet proficient and advanced performance levels in
          reading and math on their state assessment systems.”

Also for fiscal year 2000, the first objective was revised to:

          “Performance of the lowest achieving students and students in the highest poverty
          public schools will increase substantially in reading and math.”

These revisions increase the focus on progress of the most at-risk students and schools.
The revisions to the indicator change the calculation from the “percentage of students” to
the “number of states that report.”

     Subsequent to our work, ED issued a pre-publication copy of its combined FY 1999 report
     and FY 2001 plan. In that document, ED used an assessment indicator (1.2 Meeting or
     Exceeding State Performance Standards) that is essentially the same as the indicator used in
     the FY 2000 plan. Data on FY 1999 was not included because FY 1999 data will not be
     available until Fall 2000.



The Title I Program Requires That States Designate Distinguished Title I Schools

IASA requires that states designate as distinguished any Title I school which for three
consecutive years has exceeded the state’s definition of adequate progress. 29 For fiscal
year 1999, ED plans to use the count of distinguished Title I schools to measure the
performance of the Title I program for GPRA reporting.

Indicator 2.1 Recognition for Quality in the fiscal year 1999 Title I program performance
plan is:

          “Increasing numbers of high-poverty schools will be designated as distinguished
          schools by their states.”

This indicator is intended to measure progress in achieving Objective Two of the Title I
program performance plan:



29
     Improving America Schools Act (IASA), Section 1117(c )(2)(A).


March 2000                                       - 39 -                           ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



                                                                           Appendix A (continued)

            “Increase the number of Title I schools actively working to enable students to
            reach high standards each year.”

In the fiscal year 2000 program performance plan, instead of the count of distinguished
schools being used to measure progress, the measure (2.2 Improving Schools) was
changed to:

            “By the year 2000, an increasing percentage of Title I participating schools will
            report that they have met or exceeded state or district standards for progress for
            two consecutive years.”

Objective Two was also revised in the fiscal year 2000 Title I program performance plan
to:

            “Increase the number of Title I schools using standards-based reform and
            effective strategies to enable all students to reach state and local performance
            standards.”

     Subsequent to our work, ED issued a pre-publication copy of its combined FY 1999 report
     and FY 2001 plan. In that document, ED moved indicator 2.2 (Improving Schools) from
     Objective 2 (Reform Strategies) to Objective 1 (Student Performance) because the indicator
     was more closely related to student performance. In addition, ED deleted the phrase “for two
     consecutive years.” Data on FY 1999 was not included because FY 1999 data will not be
     available until Fall 2000.


Perkins III Requires Performance Measurement Systems

Amendments to the authorizing legislation of the Perkins Vocational Education Program
were signed into law on October 31, 1998. Those amendments, referred to as Perkins III,
require the states to identify core indicators of performance in a minimum of four areas :30

      (1)   attainment of academic and vocational and technical proficiencies;
      (2)   attainment of secondary degree;
      (3)   placement in postsecondary education or employment; and
      (4)   outcomes of non-traditional programs.

The law also requires states to establish levels of performance for those indicators “in a
percentage or numerical form so as to be objective, quantifiable and measurable.”

30
  The prior law required a measure of academic attainment and an additional measure from
another area such as attainment of secondary degree or placement in postsecondary education or
employment.


March 2000                                      - 40 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                     FINAL



                                                                            Appendix A (continued)

These areas of performance are included in the fiscal year 1999 and 2000 program
performance plans for Perkins. For the fiscal year 1999 plan, the data source for many of
these indicators will be studies from PES and from NCES. Our review does not cover the
data in the PES or NCES studies.

For the fiscal year 2000 program performance plan, one of the data sources for indicator
3.1 Secondary Student Outcomes is state performance reports. The data sources for the
other indicators in the fiscal year 2000 program performance plan will be studies from
PES and NCES. Indicator 3.1 states:

          “By 2002, an increasing proportion of vocational concentrators31 , including
          special populations, will attain high school diplomas, enter postsecondary
          programs, or attain employment.”

This indicator is intended to measure progress in achieving Objective Three of the
program performance plan:

          “Ensure that concentrators, including special populations, make transitions to
          continuing education, work, or other career options.”

     Subsequent to our work, ED issued a pre-publication copy of its combined FY 1999 plan
     and FY 2001 report. In that document, ED noted that data from the various studies and
     evaluations will be phased out and replaced with data from the state performance reporting
     that is required by Perkins III.


Perkins III requires states to include in their plan for the Perkins program how the state
“will ensure locally-reported data and data reported to the Secretary are complete,
accurate, and reliable.” States are required to report disaggregated data on the
performance indicators. ED is to “make state reports available to the public and
Congress and shall disseminate state-by-state comparisons of information.” Additional
information on the requirements of Perkins III is included in OVAE’s response to this
report, which is in Appendix H.




31
  Concentrators are defined in the Perkins program performance plan as “students who complete
3 or more Carnegie units in a single specific labor market preparation program area.” A Carnegie
unit is a standardized measure of class time equivalent to one fifty-minute course, five times a
week for an entire school year.


March 2000                                      - 41 -                           ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



Appendix B – Statistical Profiles of the Five SEAs Visited

                          California       Georgia     Massachusetts       Minnesota   New Jersey

Public enrollment:
         K-8              3,903,505         943,086             653,183      577,612       809,874
        9-12              1,465,241         345,419             240,419      248,094       296,831
        Total             5,368,746       1,288,505             893,602      825,706     1,106,705

# of districts                1,006            180                  353          419          608

% of students with             9.5%           9.5%               16.0%         10.6%        16.3%
disabilities

Poverty rate                   25%            20%                  16%          14%          14%

The enrollment figures, number of districts, and percentage of students with disabilities
are based on 1996 information and were obtained from the Council of Chief State School
Officers’ (CCSSO’s) 1997 report State Education Indicators with a Focus on Title I. The
poverty rate was obtained from the Education Week’s report Quality Counts ‘99. That
report did not indicate the year of the data.


                             California     Georgia        Massachusetts   Minnesota   New Jersey
Expenditures per pupil          $4,299       $5,183               $5,785      $5,738       $7,966

% of average district
funding that is federal            9.5%         7.4%               5.4%         4.4%         3.3%

The expenditures per pupil and the percentage of federal funding are based on 1995
information and were obtained from the CCSSO’s 1997 report State Education Indicators
with a Focus on Title I. The percentage of average district funding that is federal is not
equivalent to the percentage of funding that is federal for the state or SEA.


                            California      Georgia        Massachusetts   Minnesota   New Jersey
% of Perkins for
secondary program                 62%          50%                  73%          35%         65%

Perkins funds are divided by the states between secondary and postsecondary programs.
The percentage of Perkins funding for secondary programs was obtained from ED’s
Office of Vocational and Adult Education and is based on fiscal year 1998 state budgets.




March 2000                                        - 42 -                          ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                 FINAL



Appendix C – Descriptions of the Systems for Assessment of Academic Achievement


California

Senate Bill 376, passed in 1997, required all California school districts to use a single,
national norm-referenced, standardized test to test each pupil in grades 2 to 11 by May 15
of each fiscal year, beginning with the 1997-1998 school year.

System name                          Standardized Testing and Reporting Program (STAR
                                     program)
Assessment instrument                Stanford Achievement Test, Ninth Edition, Form T
Assessment type                      Norm-referenced test
Proficiency levels                   Four (below basic, basic, proficient, and advanced)
Grade levels tested                  Grades 2-11
Assessment coverage for              Reading, written expression, spelling, math
grades 2-8
Assessment coverage for              Reading, writing, math, history/social science, science
grades 9-11
Disaggregation categories            Gender, ethnicity, limited English proficient (LEP)
                                     students, Title I students, migrant students, students in
                                     special education, and gifted and talented students
Administration of the                By May 15 of each fiscal year
assessment
Initial assessment                   Spring 1998 (1997-98 school year)

California plans to augment the STAR test with additional questions specifically based on
the state’s content standards (criterion-referenced). In addition, the state plans to
implement additional testing called the "Assessment of Applied Academic Skills” for all
students in grades 4, 5, 8 and 10 in reading, writing, math, history/social science and
science. This assessment, which will employ matrix sampling, is designed to show how
well students can apply their knowledge.




March 2000                                    - 43 -                         ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                            FINAL



                                                                               Appendix C (continued)

Georgia

Georgia State Code requires the State Board of Education to perform an assessment of
the effectiveness of education programs and supervise the development of reports on the
comprehensive evaluation of public schools, local systems, and regional agencies. Below
is a summary of the current and planned assessments:

Assessment       Writing           High School    Iowa Tests of      High School         Criterion-
instruments      Assessment        Writing Test    Basic Skills      Graduation          Referenced
                 (WA)              (HSWT)         (ITBS)             Tests (HSGT)        Competency
                                                                                         Tests (CRCT)
Assessment       --                --             Norm-              --                  Criterion-
type                                              Referenced                             referenced
Proficiency      --                Two            Three (less than   Two (pass/fail)     --
levels                             (pass/fail)    proficient,
                                                  proficient,
                                                  advanced)
Grade levels     3, 5, 8           11             3, 5, 8            11, 12              1-8
tested
Assessment       3rd(imaginative   Persuasive     English            English             English
coverage         writing),         writing        language arts,     language arts,      language arts,
                 5th (assigned     prompt         mathematics,       writing,            reading,
                 prompt),                         science, social    mathematics,        mathematics
                 8th (assigned                    studies            social studies,
                 narrative                                           and science
                 prompt),
Administration   Spring            Fall/Spring    Spring             Fall/Spring         Spring
of assessment
Initial          Mid 1990’s        Mid 1990’s      --                Mid 1990’s          Planned for
assessment                                                                               Spring 2000

The State Board of Education is currently working on revisions to the core curriculum.
Following the adoption of the revised core curriculum, the Board is to contract for the
development of criterion-referenced tests to measure the adopted curriculum.




March 2000                                        - 44 -                               ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                               FINAL



                                                                       Appendix C (continued)

Massachusetts

The Educational Reform Act of 1993 required the Massachusetts Board of Education to
adopt a system for evaluating the performance of public school districts and the
individual schools within them on an annual basis. The system must include a
mechanism for measuring whether students’ performance is or is not improving from
year to year.

Assessment instrument             Massachusetts Comprehensive Assessment System
                                  (MCAS)
Assessment type                   Criterion-referenced
Proficiency levels                Four (advanced, proficient, needs improvement, and
                                  failing)
Current grade levels tested       4, 8, and 10

Future grade levels tested        2, 4, 6, 8 and 10
Assessment coverage               English language arts, mathematics, and science and
                                  technology
Disaggregation categories         District, school, students attending the district for more
                                  than three years, regular education students, students with
                                  disabilities, LEP students, and migrant students
Administration of the             Spring
assessment
Initial assessment                May 1998 (1997-98 school year)




March 2000                                    - 45 -                        ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                       FINAL



                                                                              Appendix C (continued)

Minnesota

The Minnesota Comprehensive Assessment is the result of a 1997 statewide testing law,
referred to as Minnesota Statutes 120B.35 Student Achievement Levels, that called for the
creation of a comprehensive assessment system and use of statewide tests.

Assessment instruments          Minnesota Comprehensive              Basic Standards Test (BST)
                                Assessment (MCA)
Assessment type                 Criterion-referenced                 Criterion-referenced
Proficiency levels              Four (novice, partially              Percentile, no levels
                                proficient, proficient, and
                                advanced)
Current grade levels tested     3, 5                                 8
Future grade levels tested      3, 5, 8, and 11                      *(see below)
Assessment coverage             Math (3rd and 5th ), reading         Math, reading, writing
                                (3rd and 5th ) and writing (5th )
Disaggregation categories       District, school, gender,            --
                                ethnicity, LEP students,
                                students in special education,
                                economically disadvantaged
                                students
Administration of the           Spring                               Spring
assessment
Initial assessment              Spring 1998 (1997-98 school          Since Spring 1996;
                                year)                                however, districts were
                                                                     allowed to use alternative
                                                                     tests until July 1998.

* The Minnesota SEA is considering replacing the Basic Standards Test (BST) with the
new Minnesota Comprehensive Assessment (MCA) system.




March 2000                                    - 46 -                              ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                               FINAL



                                                                                     Appendix C (continued)

New Jersey

The New Jersey Administrative Code requires a statewide assessment system.

Current assessment       --                              Early Warning Test          High School Proficiency
instruments                                              (EWT 8) -                   Test (HSPT 11) -
                                                         Grade Eight Proficiency     High School Proficiency
                                                         Assessment                  Assessment
Future assessment        Early Warning Test              Same                        Same
instruments              (EWT 4) Elementary
                         School Proficiency
                         Assessment
Assessment type          Criterion-referenced            Criterion-referenced        Criterion-referenced
Current proficiency      Three (advanced,                Three (advanced,            Two (pass or fail)
levels                   proficient, and partially       proficient, and partially
                         proficient)                     proficient)
Future proficiency       Three (advanced,                Three (advanced,            Three (advanced,
levels                   proficient, and partially       proficient, and partially   proficient, and partially
                         proficient)                     proficient)                 proficient)
Grade levels tested      Grade 4                         Grade 8                     Grade 11 and 12 (if a
                                                                                     student fails to pass
                                                                                     HSPT during grade 11)
Current assessment       Literacy and language           Reading, writing, and       Reading, writing, and
coverage                 arts, math, and science         math                        math
Planned future           Science, math, literacy         Science, math, literacy     Science, math, literacy
assessment coverage      and language arts, social       and language arts, social   and language arts, social
                         studies, health/PE,             studies, health/PE,         studies, health/PE,
                         world languages, and            world languages, and        world languages, and
                         workplace readiness             workplace readiness         workplace readiness
                         (school year 2001-02)           (school year 2002-03)       (school year 2004-05)
Disaggregation           District, school, Title I       District, school, gender,   District, school, gender,
categories               students, gender,               ethnicity, Title I          ethnicity, Title I
                         ethnicity, LEP students,        students, LEP students,     students, LEP students,
                         and students with               and students with           and students with
                         disabilities                    disabilities                disabilities
Additional               Low-income and                  Low-income and              Low-income and
disaggregation           migrant students                migrant students            migrant students
categories on future
assessments
Administration of the    May                             March                       October (grade 11) and
assessment                                                                           April (grade 12)
Initial assessment       --                              Early 1990’s                Early 1990’s




March 2000                                           - 47 -                              ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                          FINAL



Appendix D – Acronyms

CCSSO             Council of Chief State School Officers

CRESST            National Center for Research on Evaluation, Standards, and Student Testing

ED                U.S. Department of Education

EIAC              Educational Information Advisory Committee of CCSSO

ESEA              Elementary and Secondary Education Act

GAO               General Accounting Office

GPRA              Government Performance and Results Act of 1993 (also known as the Results Act)

IASA              Improving America’s Schools Act, amendment to ESEA in 1993

IDEA              Individuals with Disabilities Education Act

IPBS              Integrated Performance and Benchmarking System

LEA               Local Education Agency

LEP               Limited English Proficiency

NCES              National Center for Education Statistics in ED

NFES              National Forum on Education Statistics (An appointed group that works with NCES)

OESE              Office of Elementary and Secondary Education in ED

OIG               Office of Inspector General

OMB               Office of Management and Budget

OUS               Office of the Under Secretary in ED

OVAE              Office of Vocational and Adult Education in ED

PCIE              President’s Council on Integrity and Efficiency

Perkins program Vocational and Technical Education Assistance to the States, authorized by Perkins III

Perkins II        Carl D. Perkins Vocational and Applied Technology Education Amendments of 1990

Perkins III       Carl D. Perkins Vocational and Technology Education Amendments of 1998

PES               Planning and Evaluation Services in OUS in ED

Results Act       Government Performance and Results Act of 1993 (a lso known as GPRA)

SEA               State Educational Agency

Title I program   Grants for Schools Serving At Risk Children, authorized by ESEA




March 2000                                         - 48 -                            ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                                    FINAL



Appendix E – Terminology

The following definitions are used in this report:

Assessment – An exercise, such as a written test, portfolio or experiment, that seeks to measure a
student’s skills or knowledge in a subject area. (EW)

Carnegie unit – A standardized measure of class time equivalent to one fifty-minute course, five
times a week for an entire school year.

Concentrators – For the Perkins program, students who complete 3 or more Carnegie units in a
single specific labor market preparation program area.

Controls – What an entity does to provide reasonable assurance that what should happen happens.

Criterion-referenced test – A test designed to determine whether each student has achieved
specific skills or concepts. Each individual is compared with a preset standard for acceptable
achievement. The performance of other examinees is irrelevant. (CRESST)

Disaggregated results / Disaggregation - Providing results for subgroups, for example, by gender
or student economic status.

Norm-referenced test – A test designed to rank each student with respect to the achievement of
others in broad areas of knowledge. Each individual is compared with other examinees and
assigned a score. (CRESST)

Reliability – The precision with which a phenomena is measured. A measured value is
considered reliable if it is accurate for its intended use. (OIG)

Validity - (1) The extent to which performance is adequately measured. A measured value is
valid if it adequately represents actual performance. (OIG) (2) Refers to the precision with which
an assessment measures what it is suppose to measure. (CRESST)


CRESST – Based on a definition from the CRESST Assessment Glossary.

EW – Definition from Education Week on the Web’s Glossary of Terms.

OIG – Definition developed by OIG.




March 2000                                     - 49 -                           ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                         FINAL



Appendix F – Publications Cited in this Report

CRESST Assessment Glossary. National Center for Research on Evaluation, Standards,
and Student Testing (CRESST). (Obtained from the Internet at www.cse.ucla.edu/
CRESST /pages /glosssary.htm)

Fiscal Year 1999 Annual Plan. ED, February 1998.

Fiscal Year 2000 Annual Plan. ED, February 1999.

Glossary of Terms. Education Week on the Web, 1999. (Obtained from the Internet at
www.edweek.org/context/glossary)

Moving Towards A Results-Oriented Organization: A Report on the Status of ED’s
Implementation of the Results Act. ED-OIG, 1998. (ED-OIG/A17-70007)

Protecting the Privacy of Student Records: Guidelines for Education Agencies. National
Center for Education Statistics (NCES) and National Forum on Education Statistics
(NFES), 1997.

Quality Counts ’99. Education Week on the Web, 1999.

Study of State Data on Vocational-Technical Education, 1993-1994. OVAE.

State Education Indicators with a Focus on Title I. CCSSO, 1997.

THE RESULTS ACT: Observations on the Department of Education’s Fiscal Year 1999
Annual Performance Plan. GAO, 1998. (GAO/HEHS-98-172R)




March 2000                                    - 50 -                  ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                       FINAL



Appendix G – OESE’s Comments




March 2000                                    - 51 -                 ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                       FINAL



Appendix H – OVAE’s Comments




March 2000                                    - 52 -                 ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL


                                                                     Appendix H (continued)




March 2000                                    - 53 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL


                                                                     Appendix H (continued)




March 2000                                    - 54 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL


                                                                     Appendix H (continued)




March 2000                                    - 55 -                     ED-OIG/S17-90009
Information Report on Data Accumulated by SEAs and Reported to ED:
ESEA/Title I and Perkins Vocational Education Programs                            FINAL


                                                                     Appendix H (continued)




March 2000                                    - 56 -                     ED-OIG/S17-90009