oversight

Audit of Perkins III Performance Data at OVAE.

Published by the Department of Education, Office of Inspector General on 2004-05-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                      UNITED STATES DEPARTMENT OF EDUCATION
                                      OFFICE OF THE INSPECTOR GENERAL




                                                    MAY 24 2004
Memorandum

TO:              Susan Sclafani
                 Assistant Secretary for Vocational and Adult Education


FROM:            Helen Lew          ;{{/u. W
                 Assistant Inspector General for Audit Services

SUBJECT:         Final Audit Report
                 Audit of Perkins III Performance Data at OVAE
                 Control Number ED-OIGIA03-DOO 13

Attached is the subject final audit report that covers the results of our review of the
management controls at OV AE to ensure that performance data collected and reported to
Congress for Perkins III, during the period July 1,2000, through June 30,2001, were
complete, accurate, and reliable. An electronic copy has been provided to your Audit
Liaison Officer. We received your comments generally concurring with the findings and
recommendations in our draft report

Corrective actions proposed (resolution phase) and implemented (closure phase) by your
office will be monitored and tracked through the Department's Audit Accountability and
Resolution Tracking System (AARTS). ED policy requires that you develop a final
corrective action plan (CAP) for our review in the automated system within 30 days of
the: i~suance ufthis report. The CAP should set forth the specific action items, and
targeted completion dates, necessary to implement final corrective actions on the findings
and recommendations contained in this final audit report.

In accordance with the Inspector General Act of 1978, as amended, the Office of
Inspector General is required to report to Congress twice a year on the audits that remain
unresolved after six months from the date of issuance.

In accordance with thc Freedom ofInformation Act (5 U.S.C, §552), reports issued by
the Office of Inspector General are available to members of the press and general public
to the extent information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have any questions,
please call Bernard Tadley, Regional Inspector General for Audit at 215-656-6279.

Enclosure


                                400 MARYLAND AVE., S.W., WASHINGTON, DC 20202·)510
                                                          www.ed.gov

        Our mission is to ensure equa.l access to education and to promote educational excellence throughout the nation.
                          Audit of Perkins III Performance 

                                   Data at OVAE





                                   FINAL AUDIT REPORT

                              Control Number ED-OIG/A03-D0013

                                         MAY 2004





Our mission is to promote the efficiency,               U.S. Department of Education
effectiveness, and integrity of the                     Office of Inspector General
Department’s programs and operations.                   Philadelphia, Pennsylvania
                                    NOTICE

Statements that management practices need improvement, as well as other
conclusions and recommendations in this report, represent the opinions of the Office
of Inspector General. Determinations of corrective action to be taken will be made
by appropriate Department of Education officials.

In accordance with the Freedom of Information Act (5 U.S.C. §552), reports issued
by the Office of Inspector General are available, if requested, to members of the
press and general public to the extent information contained therein is not subject to
exemptions in the Act.




                                                                                    II
                                  TABLE OF CONTENTS


                                                                                  Page


Executive Summary                                                                   1



Audit Results                                                                       2



Finding No. 1 – States Were Not Submitting Complete Perkins III 

Performance Data to ED                                                              2



Finding No. 2 – OVAE Needs to Work With States to Assure That Performance

Measures Are Valid                                                                  4



Finding No. 3 – OVAE’s Monitoring Process Did Not Include WIA Incentive Award 

Recipients for Monitoring Visits                                                    6



Finding No. 4 – Not All States Complied With ED’s CAR Reporting Instructions        7



Other Matters                                                                       8



Background                                                                          8



Objective, Scope, and Methodology                                                   10



Attachment: OVAE’s Written Response to the Draft Report                             11

Audit of Perkins III
Performance Data at OVAE                       Final Report            Control Number ED-OIG/A03-D0013



                                           Executive Summary

The purpose of this audit was to assess management controls at the U.S. Department of
Education (ED), Office of Vocational and Adult Education (OVAE) to ensure that
performance data collected and reported to Congress for the Carl D. Perkins Vocational and
Technical Education Act of 1998, (Perkins III) Public Law 105-332, for the 2000-2001
program year (PY) (July 1, 2000, through June 30, 2001) were complete, accurate, and
reliable.

While OVAE had a process to collect and review the Perkins III performance data, it needs to
strengthen controls over its review and reporting processes to ensure that the data are
complete, accurate, and reliable. Specifically, our audit disclosed that:

    •	 States were not submitting complete Perkins III performance data to ED. A review of
       performance data in the Perkins III Performance Report to Congress for the 2000-2001
       PY disclosed that 57 percent (30 of 53 states and U.S. territories) of states did not
       provide complete data to ED.
    •	 OVAE needs to work with states to assure that performance measures are valid. We
       found that 34 percent (18 of 53 states and U.S. territories) of states use the same
       performance measures for several sub-indicators.
    •	 OVAE’s monitoring process did not include Workforce Investment Act (WIA)
       incentive award recipients for monitoring visits. 1 During our audit work at three state
       agencies (Florida, Kentucky, and Indiana), we found that each of the three states
       audited received an incentive award for the 2000-2001 PY and had problems with the
       quality of the data reported.
    •	 Not all states complied with ED’s Consolidated Annual Report (CAR) reporting
       instructions. We found that some state’s Improvement Strategies were not specific to
       the sub- indicators for which the state did not meet its adjusted level.

We recommend that OVAE develop additional procedures and controls to ensure that data
submitted from the states meet the criteria of each sub- indicator, in addition to ensuring that
the Perkins III data reported are complete, accurate, and reliable.

We provided a copy of our draft report to OVAE. In its response to the draft report, a copy of
which is included as an attachment, OVAE indicated general concurrence with the findings
and recommendations contained in the report. For each recommendation, OVAE provided
details of actions taken to implement the recommendations. We appreciate OVAE’s efforts to
implement the recommendations.




1
  The Department of Labor, in collaboration with the Department of Education, nominates states that are eligible
to apply for WIA incentive awards under WIA regulations. These funds are available to support innovative
workforce development and education activities.


                                                       1

Audit of Perkins III
Performance Data at OVAE                         Final Report             Control Number ED-OIG/A03-D0013



                                                 Audit Results

The purpose of this audit was to assess management controls at OVAE to ensure that
performance data collected and reported to Congress for Perkins III for the 2000-2001
program year (July 1, 2000, through June 30, 2001) were complete, accurate, and reliable.
For the purpose of this audit, we assessed and classified the significant management controls
into the following categories:

     �    Date collection and analysis;
     �    Monitoring of states; compliance with Perkins III requirements; and
     �    Reporting.

We found that OVAE’s management controls were not sufficient to provide reasonable
assurance that the Perkins III performance data received and reported were complete,
accurate, and reliable. Specifically, our audit disclosed that (1) performance data were not
complete for many states, (2) OVAE needs to work with states to assure that performance
measures are valid, (3) monitoring visits were not performed at states that received WIA
incentive awards, and (4) not all states complied with ED’s CAR reporting instructions.

Finding No. 1 – States Were Not Submitting Complete Perkins III Performance Data To
                ED

A review of the performance data in the Perkins III Performance Report to Congress for the
2000-2001 PY disclosed that 57 percent (30 of 53 states and U.S. territories) of states did not
provide complete data. Specifically, we found that:
    •	 One U.S. territory did not report any data for Perkins III;
    •	 Three states did not report complete data on the progress achieved on their adjusted
        levels of performance; 2 and
    •	 Thirty states3 did not report complete data on the level of performance for gender,
        ethnicity, and special populations of students participating in the vocational and
        technical educational programs.

As required under Perkins III, sec. 113 (c):

    (1) In General. - Each eligible agency that receives an allotment under section 111 shall
        annually prepare and submit to the Secretary a report regarding –
        (A)	 the progress of the state in achieving the state adjusted level of performance on
              the core indicators of performance; and
        (B)	 information on the levels of performance achieved by the state with respect to the
              additional indicators of performance, including the level of performance for
              special populations.

2
  The adjusted level of performance is the negotiated final performance level that a state is held accountable for 

meeting or exceeding. 

3
  The 30 states include the U.S. territory and the three states that did not report complete data.



                                                         2

Audit of Perkins III
Performance Data at OVAE                 Final Report        Control Number ED-OIG/A03-D0013

   (2) Special Populations. - The report submitted by the eligible agency in accordance with
       paragraph (1) shall include a quantifiable description of the progress special
       populations participating in vocational and technical educational program have made in
       meeting the state adjusted levels of performance established by the eligible agency.

Under Perkins III, Part B, § 122(c)(20), states are required to include in their state plan
information that “describes how the eligible agency will ensure that the data reported to the
eligible agency from local educational agencies and eligible institutions under this title and
the data the eligible agency reports to the Secretary are complete, accurate, and reliable.”

To give states more time to gather the Ethnicity and Special Population data, OVAE waived
the requirement for the first year of reporting the performance data; however, it was required
and should have been reported for the 2000-2001 PY (the second year). Although OVAE had
provided workshops and technical assistance to states during that period, it appears that some
states required additional assistance to ensure compliance with the requirement. We found
that OVAE generally contacted the states by email and telephone to request the missing data,
but did not perform follow-up actions to obtain the missing data. Since OVAE was aware that
states were having problems with the collection and reporting of the data, additional technical
assistance and follow-up, in addition to e- mail and telephone contact was needed. OVAE
may have assisted states in determining what needed to be done to collect and report complete
Perkins III data if it had made on-site monitoring visits.

In addition, to providing on-site monitoring, OVAE can use the results of the audit work
performed by the states’ independent public accountants (IPAs) to determine which states did
not submit complete performance data. The Office of Management and Budget (OMB)
Circular A-133 Compliance Supplement (Compliance Supplement) requires IPAs to review
the Perkins III performance data.

Since OVAE did not receive the data, ED was unable to report complete Perkins III
performance data to Congress.

Recommendation: The Assistant Secretary for the Office of Vocational and Adult Education
should:

1.1	 In the future perform additional one on one assistance activities to those states that are
     having problems reporting complete data in meeting the requirements of the law.
1.2	 Review the Compliance Supplement to determine if changes are needed to ensure that
     the state’s Perkins III performance data are reviewed for completeness.

OVAE Response:

OVAE concurred with the finding and recommendations. In its response, OVAE stated it
implemented enhancement efforts for technical assistance, put in place additional procedures
to address the problems identified, and revised the reporting instrument to require states to
provide additional information to support the states’ attestation that the data are reliable,




                                                3

Audit of Perkins III
Performance Data at OVAE                       Final Report            Control Number ED-OIG/A03-D0013

accurate, and complete. In addition, OVAE will review the OMB Compliance Supplement to
determine if changes are needed.

OVAE also provided information to show improvements in the states reporting of complete
data. Although OVAE concurred with the finding, the response indicated that the auditors
marked the states incomplete if they did not have students in any of these groups without
considering the states’ written explanations.

OIG Response:

To make our determinations we relied on the report to Congress, prepared by OVAE, which
did not provide state explanations. The missing data was either identified as NP (meaning not
provided, this data was not collected by the state from the districts or entity) or with an
asterisk (meaning there were no concentrators, completers or participants reported in this
category). However, since the related sub- indicator had a performance result reported for the
sub- indicator then the states should have been able to report gender, ethnicity, or special
populations data.

Finding No. 2 – OVAE Needs to Work with States to Assure That Performance
                Measures Are Valid

OVAE needs to work with states to assure that the performance measures used by states are
valid representations of what the sub- indicators are supposed to measure. We found that 34
percent (18 of 53 states and territories) of states were using the same performance measure in
the report to Congress for several different sub- indicators. Perkins III requires states to
establish rigorous levels and standards for performance indicators, while at the same time
giving states the flexibility to define their own performance measures. States were instructed
to use the OVAE Core Indicator Framework to develop performance measures for each sub-
indicator. This framework defines the goals for each sub- indicator measurement approach.
These measurement approaches overlap and can result in states using the same approach for
more than one sub- indicator. For example, one state used the same performance measure
(percentage of concentrators completing vocational programs)4 for three secondary sub-
indicators that measured academic attainment, skills attainment, and secondary completion.
In contrast, another state used separate performance measures to measure the same sub-
indicators. This state defined academic attainment by using the state’s academic assessment
instrument, skills attainment by using national/state vocational standards and secondary
completion by using high school graduation. Using separate performance measures more
adequately represents what the sub- indicators were intending to measure and will provide
more valid data.

The Core Indicator Framework defines the validity of the data as the degree to which the
performance measurement approach directly and fully measures the students’ outcome at an
appropriate time interval. OVAE must ensure that states are aware that, in order to develop

4
 Vocational concentrators are students who enrolled or have completed a threshold level of vocational
education (in a program/sequence) courses or instructional units that prepares the individual for employment
and/or advanced education as determined by a State.


                                                       4

Audit of Perkins III
Performance Data at OVAE                 Final Report         Control Number ED-OIG/A03-D0013

meaningful and valid performance measures, the definitions need to adequately represent
what the indicator is intending to measure.

Discussions with OVAE officials disclosed that OVAE does not believe it has the statutory
authority to regulate how states chose to measure performance. While we realize that Perkins
III gives states the flexibility to develop their own measure and it is silent with regard to the
authority OVAE has when it comes to working with states to develop performance measures,
the law does give OVAE the authority to review and approve the state plan, which contains
the performance measures.

Under Perkins III, section 122. (e) (1) The secretary shall approve a state plan, or a revision to
an approved State plan, unless the secretary determines that –
       (A) the state’s plan, or revision, respectively, does not meet the requirement of this
           section; or
       (B) the state’s levels of performance on the core indicators of performance consistent
           with section 113 are not sufficiently rigorous to meet the purpose of this Act.

Approval of the plan infers that OVAE has some consultation into the state’s development of
the performance measure. Therefore, OVAE could provide guidance to the states to assure
that the performance measures are valid measures of each sub- indicator. We recognize that
Perkins III will be under reauthorization in the near future, and changes to the current state
plans may not be possible. However, in the future, OVAE should request that states make a
full disclosure in the CAR in instances where the measure does not fully represent the sub-
indicator it is intended to measure.

Recommendations: The Assistant Secretary for the Office of Vocational and Adult
Education should:

2.1	 Consider requesting clarification of the law in the current reauthorization to specify that
     OVAE has authority for reviewing and approving states performance definitions.
2.2	 In approving future measures, in those cases where the performance measure do not
     fully represent the sub-indicator, provide guidance to the states for developing other
     measures.
2.3	 In those cases where the definition does not fully represent the performance measure,
     request that states make a full disclosure in the CAR stating that the data is the best the
     state has available to measure the sub- indicator.

OVAE Comments:

OVAE generally concurred with the finding and recommendations. OVAE’s response stated
that because of the Perkins III law, OVAE has no role in the states’ establishment of its
performance indicators. OVAE also stated that the annual performance reporting instrument
has been modified to require states to make a full disclosure and that the President’s
reauthorization proposal will address these issues by streamlining the program’s performance
measures and adopting certain OMB common measures.




                                                5

Audit of Perkins III
Performance Data at OVAE                         Final Report            Control Number ED-OIG/A03-D0013

OIG Response:

We believe, as stated in the finding, that OVAE does have some role and could provide
guidance to states during the plan approval process.

Finding No. 3 – OVAE’s Monitoring Process Did Not Include WIA Incentive Award
                Recipients for Monitoring Visits

A review of OVAE’s monitoring process disclosed that OVAE had a process in place for
identifying high-risk 5 states and selecting those states for a monitoring site visit. However,
only the high-risk states are being scheduled for the full monitoring visit. States that
exceeded their performance levels and were eligible for the WIA incentive awards were not
visited. During our audits of the Perkins III performance data of three states that received the
WIA incentive awards for the 2000-2001 PY, we found serious problems with the quality of
the performance data that were used to obtain the incentive awards. Two of the states we
audited were also selected for an incentive award for the 2001-2002 PY. 6 OVAE officials
told us that they relied on states attesting to the accuracy of the reported data. However, as
we found during our audits of states’ data, the states had not ensured that the data were
complete, accurate, and reliable.

The OVAE monitoring process was designed to ensure that states that were at risk were
scheduled for full monitoring visits. States that exceeded their performance levels and
qualified for a WIA incentive award would not be selected for monitoring. Based on our
audits, these states should not be excluded from full monitoring visits because they qualify for
an incentive award, as these states may be reporting data that are not complete, accurate, and
reliable. While these states were included in the high-risk evaluation process, the fact that
they were considered exceptional lowered their chances of being selected for full monitoring
visits. To ensure the quality of the data for states that receive incentive awards, OVAE needs
to modify its monitoring selection process to ensure that these states are given the same risk
factors as the other states.

Recommendation: The Assistant Secretary for the Office of Vocational and Adult Education
should:

3.1	 Revise its monitoring procedures to include states that qualify for the WIA incentive
     awards for possible selection.

OVAE Response:

OVAE concurred with the finding and recommendation. OVAE provided a schedule of
completed and proposed monitoring visits to states that received WIA incentive awards.


5
  A high-risk state is one that received a high score based on OVAE’s weighted risk analysis criteria. The 

weighted risk analysis criteria was based on accountability, last time monitored, state allocations and audit 

results.

6
  We notified OVAE of our concerns over the states’ eligibility for these incentive awards. 



                                                         6

Audit of Perkins III
Performance Data at OVAE                         Final Report            Control Number ED-OIG/A03-D0013



Finding No. 4 – Not All States Complied With ED’s CAR Reporting Instructions

Some state Improvement Strategies do not address the specific sub- indicators for which the
state did not meet its adjusted performance level. Currently states include an Improvement
Strategy in the CAR to ED. The Improvement Strategy should detail what strategies the state
is going to take to meet the adjusted levels of performance 7 it failed.

Our review of the CAR for three states we had previously audited disclosed that two of the
three Improvement Strategies did not specifically address what the state was going to do to
meet the adjusted level of performance it failed. 8 While these states provided an
Improvement Strategy at the program level, the adjusted level of performance that failed was
at the sub- indicator level. Therefore, the Improvement Strategies should also be reported at
the sub- indicator level.

OVAE instructed states to provide Improvement Strategies at the sub- indicator level in the
CAR guidance entitled, Accountability, and Financial Status Report for the State Basic Grant
and Tech-Prep Grant Programs under Perkins III, OMB NO: 1830-0503. Under Section B.
II. (e), the guidance requires states to provide a brief narrative for each sub-indicator on the
proposed improvement strategies for the next program year. The narrative should be based on
the State Performance Summary (IIa) and the Effectiveness of Improvement Strategies (IId)
in the previous years.

Since some states’ Improvement Strategies did not address the sub- indicators for which the
state failed to meet its adjusted performance levels, OVAE was unable to determine if the
states had developed adequate strategies for sub-indicators in need of improvement. OVAE
needs to follow-up on its original guidance to the states to provide Improvement Strategies
specific to the sub- indicators for which the state failed to meet its adjusted performance
levels.

Recommendations: The Assistant Secretary for the Office of Vocational and Adult
Education should:

4.1	 Provide additional guidance and instruction to states to provide Improvement Strategies
     that address each sub-indicator for which the state did not meet its adjusted performance
     levels.
4.2	 Follow- up with states that do not comply with the reporting instructions.

OVAE response:

OVAE concurred with the finding and recommendations. OVAE stated that it revised the
annual performance reporting instrument to require states to describe how they will improve

7
  The adjusted level of performance is the negotiated final performance level. This becomes the adjusted 

performance level for the subsequent years.

8
  States do not have to exceed their adjusted performance level for all sub-indicators to be eligible for a WIA 

award. States must exceed the adjusted level of performance for their aggregated (bundled) sub-indicators. 



                                                         7

Audit of Perkins III
Performance Data at OVAE                  Final Report         Control Number ED-OIG/A03-D0013

their performance on any sub-indicators for which they failed to meet their performance
targets.


                                          Other Matters

In addition to the findings discussed in the Audit Results section of this report, we noted the
following minor deficiency.

Report to Congress was not Timely

OVAE was not timely in submitting the Perkins III performance data report to Congress for
the 2000-2001 PY. Perkins III legislation does not include a date by which the report must be
submitted; however, it does state that it should be issued to Congress annually. From the time
this data was collected from the states in December 2001, OVAE took 18 months to issue the
report to Congress.

While OVAE has concurred that its report was not timely and stated that future reports will be
more timely, we suggest that OVAE consider requesting clarification of the due date for the
Perkins III report to be issued to Congress in the upcoming reauthorization. Until the
Congress provides clarification, OVAE should establish a date by which to issue the report to
Congress.


                                           Background

The Carl D. Perkins Vocational and Technical Education Act of 1998 (Perkins III), Public
Law 105-332, was signed into law on October 31, 1998, and is administered by OVAE. The
purpose of Perkins III is to develop more fully the academic, vocational, and technical skills
of secondary and postsecondary students who elect to enroll in vocational and technical
education programs. All states receive Perkins III funds for secondary and postsecondary
education. The funds are allotted to the states based on a formula which takes into account
the state’s population in certain age groups and the groups’ per capita income. Only State
Boards for Vocational Education are eligible to apply to ED for State Basic Grants. The
distribution of grant funds within a state is directed to priority items established by the state in
accordance with an approved state plan for vocational-technical education. Local educational
agencies and postsecondary institutions are eligible for the grants. Local agencies can be
school districts, technical institutions, and community colleges.

The central goals of Perkins III are improving student achievement and preparing students for
postsecondary education, further learning, and careers in current or emerging occupatio ns
requiring other than a baccalaureate or advanced degree. Perkins III gives states, school
districts, and postsecondary institutions greater flexibility to design services and activities that
meet the needs of their students and communities. In return for greater flexibility, states must
establish more progressive performance standards and goals, and are held to a higher degree
of accountability than the previous Perkins Act of 1990.


                                                 8

Audit of Perkins III
Performance Data at OVAE                Final Report         Control Number ED-OIG/A03-D0013



Perkins III establishes a rigorous state performance accountability system to assess the
effectiveness of the state in achieving progress in vocational and technical education and to
optimize the return of investment on Federal funds in vocational and technical education
activities. Perkins III requires states to identify specific measures and adjusted levels of
performance in its state plan, and to track its achievements in four specific outcome areas:

    •   academic and technical skill attainment;
    •   completion;
    •   placement and retention; and
    •   non-traditional participation and completion.

There are specific sub- indicators under each outcome area. Performance data must be
provided for vocational concentrators and broken down by special population categories.
States may also add additional performance indicators to its plan. Perkins III requires states
to reach agreement with the Secretary on adjusted levels of performance. States that exceed
agreed upon performance levels for Perkins III, the Adult Education and Family Literacy Act,
and Title I of the WIA are eligible for incentive grant awards. The actual level of a state’s
Perkins III performance is defined as a percentage, based on the number of students in the
numerator and denominator for each sub- indicator. The state reports the numerator,
denominator, and level of performance in the Perkins III CAR.

Perkins III took effect in PY 1999-2000, which began on July 1, 1999. By April 12, 1999,
states needed to submit a revised state Plan for Perkins III, taking into account the adjusted
performance level and measurement of core indicators of performance for Perkins III. Our
audit covered data from PY 2000-2001, the second year governed under the new law.
Congress appropriated more than $1 billion for Perkins III in PY 2000-2001. States needed to
submit their Perkins III performance reports to ED for PY 2000-2001 by December 31, 2001.

OVAE administration of the program included the collection, review, monitoring, and
reporting of the annual performance data reported by states and U.S. territories. The data
were transmitted either through ED’s website or email, and were reviewed by OVAE for
completeness and aggregated through several processes to determine if the data were
reasonable. OVAE consolidated the states’ data into a report to be issued to Congress. This
report presented a state-by-state review of the results for secondary and postsecondary
programs using Perkins III funds and the states’ status in meeting their adjusted performance
levels.




                                               9

Audit of Perkins III
Performance Data at OVAE                    Final Report          Control Number ED-OIG/A03-D0013



                               Objective, Scope, and Methodology

The purpose of this audit was to assess management controls in place at OVAE to ensure that
Perkins III performance data submitted to OVAE by the states were complete, accurate, and
reliable. The review focused on OVAE processes related to collecting and reviewing the data,
assessing state performance, and reporting annually to Congress on the status of states
achieving Perkins III performance goals. Our audit covered the performance data for Perkins
III funds awarded during PY 2000-2001.

To accomplish our audit objective, we interviewed OVAE officials responsible for collecting
and reporting Perkins III performance data. We evaluated OVAE’s procedures to determine
whether its management controls ensure that Perkins III performance data were complete,
accurate, and reliable by reviewing laws, regulations, program guidance and the 2001 Perkins
III report to Congress. We also tested the accuracy, timeliness, and completeness of the data
submitted by all states. We used the results of our audits in three states – Florida, Indiana,
and Kentucky. 9

During the audit, we relied on computer-processed Perkins III-related performance data. To
assess the reliability and accuracy of the data, we traced the data from the CAR, through
OVAE’s data review processes, and finally to the draft report to Congress. Accuracy tests
were limited as the data were verified twice, once during OVAE’s review of the CAR and
again during the attestation process. We tested the data for completeness by verifying that all
53 states and territories submitted data for the required sub-indicators. State submitted data
were also reviewed for reasonableness. Timeliness was reviewed by checking data
submission dates. Based on these tests, we concluded that the data were sufficiently reliable
to be used in meeting the audit objective.

We performed our fieldwork at OVAE from March 18, 2003, to June 30, 2003. We held an
exit conference on July 22, 2003. We conducted the audit in accordance with generally
accepted government auditing standards appropriate to the scope of the review described
above.




9
  The audits are: Audit of Florida’s Management Controls Over Perkins III Performance Data (ED - OIG/A03­
C0019); Audit of Indiana’s Management Control Over Perkins III Performance Data (ED-OIG/A05-D0012); and
Kentucky Department of Technical Education’s Management Controls Over Perkins III Performance Data Needs
Strengthening (ED -OIG/A04-D0007).


                                                   10

Audit of Perkins III
Performance Data at OVAE                           Final Report                    Control Number ED-OIG/A03-D0013

Attachment




                                UNITED STATES DEPARTMENT OF EDUCATION
                                       OFFICE OF VOCATIONAL AND ADULT EDUCATION

                                                                                          THE ASSISTANT SECRETARY


           MEMORANDUM


           To:            Bernard TadJey
                          Regional Inspector General for Audit

           From:          Susan Sclafani   .~ ~;--­
                          Assistant Secretary for Vocatiotal      a~d A~~lt Education
           Subject:       Response to Draft Audit Report: "Audit of Perkins III Performance Data at
                          OVAE, Control Number ED-OIG A03-D0013"


           Thank you for providing the Office of Vocational and Adult Education (OVAE) with the
           opportunity to comment on the draft "Audit of Perkins III Performance Data at OVAE, Control
           Number ED-OIG A03-D0013." . Overall, we believe this draft report provides a fair and
           accurate assessment of the challenges of implementing the Carl D. Perkins Vocational and
           Technical Education (Perkins III) accountability system during the first two years after the law's
           enactment. This draft report also offers useful recommendations. Many of these
           recommendations may be implemented administratively; others will require legislative action.

           Per your request, our written response to your draft is organized into three categories:

                   Ifwe concur with a specific finding and recommendation, we describe the corrective
                   actions that we have taken or plan to take.

                   Ifwe do not concur with a finding, we give reasons for our disagreement and provide
                   data to support our position.

                   Ifwe do not concur with a recommendation, we explain our disagreement and describe
                   alternative corrective action that we have taken or plan to take.




                                      400 MARYLAND AVE .• S.W.   WASHINGTON. D.C. 20202·7100




                                                             11

Audit of Perkins III
Performance Data at OVAE                          Final Report                Control Number ED-OIG/A03-D0013




                                    OVAE Response to Draft Audit Report:
           "Audit of Perkins III Performance Data at OVAE, Control Number ED-OIG A03-D0013"


          Finding No.1 - States Were Not Submitting Complete Perkins III Performance Data to ED

          Recommendations: The Assistant Secretary for the Office of Vocational and Adult Education
                           should:

          1.1   In the future, perfonn additional one-on-one assistance activities to those states that are
                having problems reporting complete data in meeting the requirements ofthe law.

          1.2   Review the Compliance Supplement to detennine if changes are needed to ensure that the
                state's Perkins III performance data are reviewed for completeness.

          OV AE concurs that some states did not report complete perfonnance data for the year in which
          this audit occurred (PY 2000-2001). In response, we have enhanced our technical assistance
          efforts, as you recommend, as well as put in place additional procedures to address the problems
          you identified. These include:

                 Sponsoring technical assistance conferences for state directors and their
                 accountability staff. Attachment A provides a list of the conferences that OVAE has
                 sponsored since 2001 to improve the ability of states to collect and report complete data.
                 We set aside time during each of these conferences for us to provide one-on-one
                 assistance to state directors and their staff.

                 Holding bi-weekly conference calls with state directors and their accountability staff.
                 We sponsor bi-weekly conference calls to discuss various accountability issues with state
                 staff. During these calls, we provide infonnation, but we particularly encourage state
                 staff to share with their peers their strategies for addressing difficult implementation
                 issues, including problems in collecting complete performance data.

                 Increasing our focus on accountability issues during our monitoring visits to states.
                 Accountability issues are now a major focus of our on-site monitoring. States now must
                 document and discuss in detail with the monitoring team each step of their data collection
                 procedures and the extent to which they are reporting to OV AE accurate, valid, and
                 complete accountability data on each recipient of Perkins III funds.

                 Requiring states to address data quality issues in their annual performance reports.
                 As in years past, we continue to require states to attest to the reliability, accuracy, and
                 completeness of the data they report at the time it is submitted. We recently have revised
                 our reporting instrument to require states also to provide additional information to
                 support this attestation.




                                                           2




                                                           12

Audit of Perkins III
Performance Data at OVAE                          Final Report                Control Number ED-OIG/A03-D0013




          As a result of these steps, during PY 2002-2003 (the last year for which data was submitted):

             •   One territory did not report any data for Perkins III;

             •   Only one state did not report complete data on the extent to which it met its adjusted
                 levels of perfonnance; and

             •   Twenty (20) states or 37 percent did not report complete data on the level ofperfonnance
                 for gender, ethnicity, and special populations of students participating in vocational and
                 technical education programs. This is due in part to the difficulty of collecting certain
                 demographic data that could not be identified through administrative record exchanges or
                 other collection methodologies. For example, many postsecondary students with
                 handicaps do not self identify. With no other means of identifying those students,
                 community colleges are unable to report certain disaggregated data. Other examples of
                 states reporting incomplete data include secondary displaced homemakers or single
                 parents. Many states categorically state there is no way of identifying if a high school
                 student is a homemaker or displaced or if they were or are a parent, married or single.
                 Some states did not report "Other Educational Barriers" as a disaggregated group. In this
                 audit the auditors automatically marked states incomplete if they did not have students in
                 any ofthese groups without considering the states' written explanations.

                 States with identified gaps where either "not provided" (NP) or a "0" were possible have
                 been discussing other alternatives for reporting data that is both usable and accurate in
                 spite of completeness if a data element is not usable. OV AE is currently providing states
                 with technical assistance in working with local entities on how best to provide complete
                 data or indicate that a particular data element is not appropriate for a particular instance.
                 The 20 states reported above represent a decrease of ten states since this audit was
                 conducted in PY 2000-2001.

          The territory that failed to report any Perkins III data has entered into a compliance agreement
          with the department to address a number of issues related to its administration of department
          programs. We are following up with the remaining states to identify how we can assist them in
          meeting the requirements of the law.

          We also concur in your recommendation to review how the Office of Management and Budget
          (OMB) Compliance Supplement might be revised to address this issue. We look forward to
          working with the Office ofInspector General (OIG) staff to identify appropriate revisions.




                                                           3




                                                          13

Audit of Perkins III
Performance Data at OVAE                           Final Report                Control Number ED-OIG/A03-D0013




          Finding No.2 - OV AE Needs to Work with States to Assure That Performance Measures
          Are Valid

          Recommendations: The Assistant Secretary for the Office of Vocational and Adult Education
                           should:

          2.1   Consider requesting clarification of the law in the current reauthorization to specify that
                OVAE has authority for reviewing and approving states performance definitions.

          2.2   In approving future measures, in those cases where the performance measure do not fully
                represent the sub-indicator, provide guidance to the states for developing other measures.

          2.3   In those cases where the definition does not fully represent the performance measure,
                request that states make a full disclosure in the CAR stating that the data is the best the
                state has available to measure the sub-indicator.

          We generally concur with this finding and the recommendations. Section 113(b)(2)(D) of
          Perkins III plainly states "indicators of performance ... shall be established solely by each eligible
          agency with input from eligible recipients." OVAE has no role in these decisions. The
          advantage of this approach is that it enables states to utilize existing accountability systems and
          to choose measures they consider most appropriate for each indicator. The disadvantage, as the
          draft audit indicates, is that not all states have selected valid and reliable measures of the
          indicators. The law also does not preclude states from using the same measure for multiple
          indicators, and, in some cases, a single measure may be valid for more than one indicator.

          Community and technical colleges, for example, do not have discrete measures of "student
          attainment of challenging state established academic, and vocational and technical, skill
          proficiencies." They award credentials to students who complete both the academic and technical
          coursework in a degree program. In this instance, when states use "degree completion" we would
          consider this approach a valid measure of both academic and technical skill attainment.

          The president's reauthorization proposal will address these issues by streamlining the program's
          performance indicators and adopting the OMB common measures for job training and related
          education programs. We will focus on a smaller set of indicators that are the most meaningful
          and useful to federal and state policy-makers, as well as local programs, in measuring the return
          on the federal investment.

          In the interim, we have modified our annual performance-reporting instrument to address this
          recommendation. We currently require states to make a full disclosure that the reported data are
          the best the state has available to measure the sub-indicator. Through our review process of the
          state's consolidated annual report, the request for specific state plan revisions, the negotiation of
          future performance targets, and the review and approval of state plan revisions prior to
          recommending the issuing of the annual grant award allocations, we are continuing to clarify
          with individual states what are appropriate "performance definitions", "performance measures"
          and "performance targets". The law establishes the" performance indicators". The indicators
          cannot be changed without an act of the Congress.



                                                            4




                                                            14

Audit of Perkins III
Performance Data at OVAE                           Final Report                Control Number ED-OIG/A03-D0013




          Finding No.3 - OVAE's Monitoring Process Did Not Include WIA Incentive Award
          Recipients for Monitoring Visits

          Recommendation: The Assistant Secretary for the Office of Vocational and Adult Education
          should:

          3.1   Revise its monitoring procedures to include states that qualify for the WIA incentive
                awards for possible selection.

          We concur with this finding and have implemented the recommendation. During PY 2000-2001
          (the year in which this audit was conducted), we conducted only five monitoring visits. Ofthose
          visits, only one visit was to a state that qualified for an incentive grant. Since that time, however,
          we have dispatched monitoring teams to all states that received WIA incentive grants during the
          past two years. The state monitoring visits we have conducted since FY 2001 and plan to
          conduct in FY 2004 are outlined in Attachment B.

          Finding No.4 - Not All States Complied With ED's CAR Reporting Instructions

          Recommendations: The Assistant Secretary for the Office of Vocational and Adult Education
          should:

          4.1   Provide additional guidance and instruction to states to provide Improvement Strategies
                that address each sub-indicator for which the state did not meet its adjusted performance
                levels.
          4.2   Follow-up with states that do not comply with the reporting instructions.

          OVAE concurs with this finding and recommendations. We have revised the annual
          performance-reporting instrument to require states to describe how they will improve their
          performance on any sub-indicators for which they failed to meet their performance targets.

          Other Matters - State-by-State Comparison was not Performed

          We do not concur in this finding. The annual reports we have provided to the Congress and the
          public have included state-by-state comparisons of performance, as required by the statute. Each
          report has included detailed information about the extent to which each state achieved its agreed­
          upon performance targets. OIG contends that these are not "state-by-state" comparisons because
          the states use different performance measures. The law, however, gives each state discretion to
          choose its own performance measures, while, at the same time, requiring OVAE to prepare
          "state-by-state comparisons" of performance results. The direction to issue "state-by-state
          comparisons" must be considered in the context of the statute as a whole. The reports OVAE has
          issued meet the requirements of the law.


          Other Matters - Report to Congress was not Timely

          OV AE concurs in this finding. Future reports will be issued in a more timely fashion.



                                                            5




                                                           15

Audit of Perkins III
Performance Data at OVAE                         Final Report              Control Number ED-OIG/A03-D0013




                                                                                              Attachment A

           Meetings and Conferences Sponsored by the Office of Vocational and Adult Education to
                Improve the Accuracy. Validity, and Completeness of Perkins Data Since 200]



          Data Quality Initiative National Institute: Strategies For Improving Data Quality
          February 1-2,2001; New Orleans, Louisiana

          Improving Validity and Reliability: State Technical Assistance Meeting
          May 13-15, 2001; Reno, Nevada

          Train-the-Trainer Workshop: Working With Local Educators To Collect Quality Data
          August 23-24, 2001; Chicago, Illinois

          National Career and Technical Education Leadership Conference
          November 27-29,2001; Washington, DC

          Program Quality Institute: Strategies For Improving Program Quality
          May 13-14, 2002; Jacksonville, Florida

          Program Quality Institute: Strategies For Improving Program Quality
          August 8-9, 2002; Atlanta, Georgia

          National Career and Technical Education Leadership Conference
          May 7-8,2003; Washington, DC




                                                         6




                                                         16

Audit of Perkins III
Performance Data at OVAE                        Final Report              Control Number ED-OIG/A03-D0013




                                                                                         Attachment B

                         States Monitored by the Office of Vocational and Adult Education

          Note: Shading indicates a State that has received an incentive grant under the Workforce
          Investment Act for FY 2000 and/or 2001-the only years for which incentive grants have been
          awarded to date. Full visits span a week, while targeted visits are two days.

                           State                       Year Monitored (Type of Visit)

           Alabama                            FY 03 (Full)
           Alaska                             FY 04 (Proposed; Full)
           Arizona                            FY 03 (Targeted)
           Arkansas                           FY 02 (Targeted)
           California                         FY 02 (Full)
           Colorado                          FY 03 (Full)
           Connecticut                       FY 04 (Proposed; Targeted)
           District of Columbia               FY 03 (Targeted); FY 02 (Full)
           Delaware                           FY 03 (Regional)
           Florida                            FY 02 (Targeted)
           Georgia                            FY 02 (Targeted)
           Guam                               FY 04 (Proposed; Full)
           Hawaii                             FY 02 (Full)
           Idaho                             FY 04 (Proposed; Full)
           Illinois                          FY 02 (Full)
           Indiana                           FY 04 (Proposed; Full)
           Iowa                              FY 02 (Targeted); FY 02 (Regional)
           Kansas                            FY 04 _(Proposed; Full)
           Kentucky                          FY 04 (Proposed; Targeted)
           Louisiana                         FY 03 (Targeted); FY 01 (Full)
           Maine                             FY 02 (Targeted)
           Maryland                          FY 04 (Proposed; Full)
           Massachusetts                     FY 02 (Full)
           Michigan                          FY 02 (Full)
           Minnesota                         FY 01 (Full)


                                                        7




                                                        17

Audit of Perkins III
Performance Data at OVAE      Final Report               Control Number ED-OIG/A03-D0013




           Mississippi      FY 04 (Targeted)
           Missouri         FY 03 (Targeted); FY 02 (Regional)
           Montana          FY 04 (Targeted)
           Nebraska         FY 03 (Targeted); FY 02 (Regional)
           Nevada           FY 03 (Targeted)
           New Hampshire    FY 03 (Targeted)
           New Jersey       FY 03 (Regional)
           New Mexico       FY 03 (Targeted)
           New York         FY 02 (Targeted)
           North Carolina   FY 04 (Targeted)
           North Dakota     FY 04 (Proposed; Targeted)
           Ohio             FY 02 (Targeted)
           Oklahoma         FY 02 (Targeted)
           Oregon           FY 04 (Proposed; Targeted)
           Pennsylvania     FY 03 (Regional)
           Puerto Rico      FY 03 (Full)
                            FY 02 (Targeted)
           Rhode Island     FY 02 (Full)
           South Carolina   FY 04 (Full)
           South Dakota     FY 04 (Proposed; Targeted)
           Tennessee        FY 02 (Full)
           Texas            FY 02 (Full)
           Utah             FY 04 (Proposed; Targeted)
           Virginia         FY 03 (Regional); FY 02 (Targeted)
           Virgin Islands   FY 03 (Targeted)
           Vermont          FY 02 (Targeted)
           Washington       FY 02 (Targeted)
           West Virginia    FY 03 (Full)
           Wisconsin        FY 04 (Proposed; Targeted)
           Wyoming          FY 04 (Proposed; Targeted)




                                      8




                                      18