oversight

Opportunities for Ohioans with Disabilities' Case Service Report Data Quality

Published by the Department of Education, Office of Inspector General on 2016-03-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                     UNITED STATES DEPARTMENT OF EDUCATION
                                OFFICE OF INSPECTOR GENERAL

                                                                                  AUDIT SERVICES
                                                                            Philadelphia Audit Region




                                        March 01, 2016

                                                                      Control Number
                                                                      ED-OIG/A03P0001

Kevin Miller
Executive Director
Opportunities for Ohioans with Disabilities
150 E. Campus View Boulevard
Columbus, Ohio 43235


Dear Mr. Miller:

This final audit report, “Opportunities for Ohioans with Disabilities’ Case Service Report Data
Quality,” presents the results of our audit. The objectives of our audit were to determine whether
Opportunities for Ohioans with Disabilities (1) had adequate internal controls to provide
reasonable assurance that reported “Case Service Report” (RSA-911 report) data were accurate
and complete and (2) reported RSA-911 performance indicator data that were accurate,
complete, and adequately supported. Our audit covered Opportunities for Ohioans with
Disabilities’ (OOD) 2012 RSA-911 report for the reporting period October 1, 2011, through
September 30, 2012 (2012 reporting period).

OOD had adequate internal controls to ensure that the data it reported to the Rehabilitation
Services Administration (RSA) were complete. However, OOD did not have adequate internal
controls to ensure that its 2012 RSA-911 report data were accurate and adequately supported.
Specifically, we found that OOD (1) lacked policies and procedures to require verification of the
data entered into participants’ case files and (2) lacked an adequate monitoring process to ensure
that data were accurate and required documentation was maintained in participant case files.

Our testing of the data that OOD reported to RSA found a significant number of incorrect and
unverifiable data entries for data elements that RSA used to calculate OOD’s 2012 performance
indicator results. Consequently, we have no assurance that the performance indicator results that
RSA calculated were reliable. RSA uses the performance indicator results to determine whether
OOD meets RSA’s established evaluation standards. As a result, RSA may have improperly
determined OOD’s successful performance on the evaluation standards for the 2012 reporting
period.
Final Report
ED-OIG/A03P0001                                                                                    Page 2 of 39

We made several recommendations to the Commissioner of RSA that would require OOD to
establish and implement enhanced data quality controls. OOD partially concurred with Finding
No.1 regarding weaknesses in its internal controls over data quality for the 2012 reported RSA-
911 data. OOD did not agree that (1) the employer name and employment start date data
elements, listed in Table 6 of this report, were required or included in the 2012 RSA-911 file and
(2) these two data elements had a direct correlation to the inaccuracy of all of OOD’s evaluation
standards and performance indicators. OOD concurred with the recommendations for
Finding No. 1 and noted several planned corrective actions. OOD partially concurred with
Finding No. 2 regarding incorrect and unverifiable performance indicator data reported in its
2012 RSA-911 report. OOD did not agree that all of its evaluation standards and performance
indicator data were unreliable. Specifically, OOD contends that counselors’ data entry of
participants’ self-reported employment information is adequate source documentation. OOD
concurred with the recommendation for Finding No. 2, again noting several planned corrective
actions, but also requested guidance from RSA about what is sufficient documentation for
participant employment information. We did not change our findings and recommendations
based on OOD’s comments to the draft audit report.




                                            BACKGROUND


The U.S. Department of Education’s Office of Special Education and Rehabilitative Services
supports programs that serve millions of children, youth, and adults with disabilities. The Office
of Special Education and Rehabilitative Services’ RSA oversees grant programs that help people
with physical or mental disabilities to obtain employment and live more independently through
the provision of counseling, medical and psychological services, job training, and other
individualized services. RSA provides Vocational Rehabilitation Grants to States to assist them
in operating vocational rehabilitation (VR) programs. The VR program grants are provided to
support a wide range of services designed to help people with disabilities prepare for and engage
in gainful employment consistent with their strengths, resources, priorities, concerns, abilities,
capabilities, interests, and informed choice.

Each State designates a State agency to administer the VR program. Some States have more
than one VR agency (a general agency and an agency for the blind). General agencies serve all
people with disabilities except those who are blind or visually impaired and State agencies for
the blind provide services only for people who are blind or visually impaired. The remaining
States use a combined agency which serves all people with disabilities in the State. In Ohio,
OOD is the State agency designated to administer the VR program. OOD is a combined agency
that is composed of two bureaus: the Bureau of Vocational Rehabilitation and the Bureau of
Services for the Visually Impaired. OOD also includes 14 field offices and 8 administrative
offices.1 The OOD executive director reports directly to the governor of Ohio. A seven-member

1
 The eight offices are: OOD Administration, the Office of Communications and Legislation, the Division of Fiscal
Services, the Division of Human Resources, the Division of Information Technology, the Division of Legal
Services, the Division of Performance and Innovation, and the Division of Disability Determination.
Final Report
ED-OIG/A03P0001                                                                                      Page 3 of 39

commission, appointed by the governor, oversees OOD and approves its VR State plan. During
our audit period, OOD received a VR program grant award of $96,889,776.

People eligible for VR program services (referred to as participants in this report) are those who
have a physical or mental impairment that results in a substantial impediment to employment,
who can benefit from VR services for employment, and who require VR services. When OOD
cannot serve all eligible participants with disabilities due to limited resources, it uses an order of
selection. Under an order of selection, eligible participants are assigned to priority categories
based on the significance of their disability. OOD prioritizes serving participants with the most
significant disabilities. Since 1991, OOD has operated under an order of selection that assigns
participants with disabilities into three categories: most significantly disabled, significant
disability, and disability. Eligible participants who are not designated as most significantly
disabled are placed on a waiting list. According to RSA’s “FY 2012 Ohio Rehabilitation
Services Commission2 Annual Review Report,” 2,997 people were on the waiting list as of
September 30, 2012.

Each year, State VR agencies must use the RSA-911 report to report to RSA case data pertaining
to all participants whose case records were closed in a given fiscal year. The RSA-911 report
must be submitted by November 30 (60 days after the end of the fiscal year). OOD uses its
Accessible Web-Based Activity and Reporting Environment (AWARE) case management
database to store data about its participants’ VR cases and to manage the case flow. OOD
implemented AWARE in October 2011. Before implementing AWARE, OOD used its Online
System for Computer Assisted Rehabilitation (OSCAR) case management system. The case data
reported on OOD’s 2012 RSA-911 report were extracted from its AWARE database. In its
2012 RSA-911 report, OOD reported a total of 21,554 closed participant cases,3 of which
3,510 cases (about 16 percent) were reported closed with an employment outcome.4 Cases are
coded in the RSA-911 report by type of closure to indicate when in the VR process a participant
exited the program, as shown below and in the diagram in Attachment 2:

        exited as an applicant (code 1),
        exited during or after a trial work experience/extended evaluation5 (code 2),
        exited from an order of selection waiting list (code 6),
        exited without an employment outcome after eligibility was determined but before an
         individualized plan for employment (IPE)6 was signed (code 7),
        exited without an employment outcome after an IPE was signed but before receiving
         services (code 5),


2
  In October 2013, the Ohio Rehabilitation Services Commission officially became OOD.
3
  OOD’s 2012 RSA-911 report included 21,559 closed cases; however, we identified 5 participants whose case
records were reported under two different Social Security numbers. The case records for the five participants were
subsequently consolidated in the AWARE database under their correct Social Security numbers.
4
  Employment outcome means obtaining or retaining full-time or part-time competitive employment.
5
  Participants complete trial work experiences or extended evaluations to determine whether they can benefit from
VR services if existing evidence indicates that the participant is incapable of benefiting from the services.
6
  The IPE is a written plan outlining a participant’s vocational employment goal and the services to be provided to
assist the participant in reaching the goal.
Final Report
ED-OIG/A03P0001                                                                                    Page 4 of 39

       exited without an employment outcome after receiving services (code 4), and
       exited with an employment outcome (code 3).

Section 106 of the Rehabilitation Act of 1973, as amended, requires RSA to establish evaluation
standards and performance indicators for the VR program that include outcome and related
measures of program performance. Two evaluation standards were established in June 2000
(34 Code of Federal Regulations [C.F.R.] Part 361). RSA has established minimum levels of
performance for each performance indicator. RSA uses data from the RSA-911 report to
monitor State agencies’ VR program performance, including calculating State agencies’ results
on the performance indicators and determining whether they have met the evaluation standards.

The evaluation standards and performance indicators are as follows.

Evaluation Standard 1—Employment Outcomes

Standard 1 includes six performance indicators, three of which are primary indicators. The
primary indicators (1.3, 1.4, and 1.5) measure the quality of the employment outcomes achieved
by participants served by the program.

       Performance Indicator 1.1—The number of participants exiting the VR program who
        achieved an employment outcome during the current performance period compared to the
        number of participants who exited the VR program after achieving an employment
        outcome during the previous performance period.

       Performance Indicator 1.2—Of all participants who exit the VR program after receiving
        services, the percentage who are determined to have achieved an employment outcome.

       Performance Indicator 1.3—Of all participants determined to have achieved an
        employment outcome, the percentage who exit the VR program in competitive, self- or
        business enterprise program7 employment with earnings equivalent to at least the
        minimum wage.

       Performance Indicator 1.4—Of all participants who exit the VR program in competitive,
        self- or business enterprise program employment with earnings equivalent to at least the
        minimum wage, the percentage who are participants with significant disabilities.

       Performance Indicator 1.5—The average hourly earnings of all participants who exit the
        VR program in competitive, self- or business enterprise program employment with
        earnings equivalent to at least the minimum wage as a ratio to the State’s average hourly
        earnings for all people in the State who are employed (as derived from the Bureau of
        Labor Statistics report “State Average Annual Pay” for the most recent available year).



7
 A business enterprise program means a participant who obtains employment as an operator of a vending facility or
other small business under the management and supervision of a State VR agency.
Final Report
ED-OIG/A03P0001                                                                                Page 5 of 39

      Performance Indicator 1.6—Of all participants who exit the VR program in competitive
       employment, self- or business enterprise program employment with earnings equivalent
       to at least the minimum wage, the difference between the percentage who report their
       own income as the largest single source of economic support at the time they exit the
       VR program and the percentage who report their own income as the largest single source
       of support at the time they apply for VR services.

To achieve successful performance on standard 1, State VR agencies must meet or exceed the
minimum level of performance for four of the six performance indicators in the evaluation
standard, including meeting or exceeding the performance levels for two of the three primary
indicators.

Evaluation Standard 2—Equal Access to Services

Standard 2 includes one performance indicator.

      Performance Indicator 2.1—The service rate for all participants with disabilities from
       minority backgrounds as a ratio to the service rate for all participants with disabilities
       from nonminority backgrounds.

To achieve successful performance on standard 2, State VR agencies must meet or exceed the
performance level established for performance indicator 2.1.

State agencies that fail to meet these performance levels must develop a program improvement
plan outlining specific actions to be taken to improve program performance. For the 2012
reporting period, OOD achieved successful performance on the evaluation standards; however, it
did not meet the performance levels for performance indicators 1.2 and 1.5. Table 1 shows the
performance levels required for the performance indicators and OOD’s performance levels for
the 2012 reporting period.

  Table 1. Performance Levels for the Performance Indicators
   Performance    Performance Level Required of a General/Combined VR Agency                 OOD’s 2012
    Indicator                                                                             Performance Level
        1.1       Number of employment outcomes equals or exceeds previous                      +137
                  performance period                                                             Met
                                                   (3,373)
        1.2       Percent with employment outcomes after services                              48.95%
                                                    55.8%                                   Did Not Meet
        1.3       Percent of employment outcomes that were competitive employment              96.38%
                                                    72.6%                                       Met
        1.4       Percent of participants with competitive employment outcomes who            100.00%
                  had a significant disability                                                  Met
                                                    62.4%
        1.5       Ratio of average hourly VR wage to average State wage                         0.502
                                                     0.52                                   Did Not Meet
        1.6       Difference between percent self-supporting at closure and application         66.36
                                                     53.0                                        Met
        2.1       Ratio of minority service rate to nonminority service rate                    0.812
                                                     0.80                                        Met
Final Report
ED-OIG/A03P0001                                                                                        Page 6 of 39


                                            AUDIT RESULTS


We found that OOD had adequate internal controls to provide reasonable assurance that its
RSA-911 report data were complete. To ensure the data were complete OOD analyzed its
2012 RSA-911 report for data quality using the edit check program, “RSA Errors,
Reasonableness Checks and Anomalies Program.” The program identifies possible problems
with the RSA-911 report data, including data omissions. In addition, OOD’s AWARE database
had controls that required that select data elements were entered into the case file throughout the
VR process.8 For example, the required data elements for a participant’s personal information
included gender, birthdate, and race and ethnicity. OOD VR counselors could not close a
VR case in the AWARE database if all of the required data elements were not completed. The
required data elements included elements that were reported on the RSA-911 report.

However, we found that OOD did not have adequate internal controls to provide reasonable
assurance that its RSA-911 report data were accurate and adequately supported. Specifically, we
found that (1) OOD did not have policies and procedures that required VR counselors or
supervisors to verify that the data entered in participants’ case files were correct and adequately
supported by documentation prior to closing the case file, and (2) OOD’s monitoring process did
not ensure that data entered into the AWARE database were correct and required documentation
was maintained in participant case files.

As a result, we found that VR case service data maintained in OOD’s AWARE database and
reported on its 2012 RSA-911 report, including performance indicator data, were not correct and
adequately supported, and therefore were not reliable.

State VR agencies must have a system of internal control that provides reasonable assurance that
RSA-911 report data are accurate, complete, and supported so that RSA can rely on the data to
reflect the VR agency’s true performance when it calculates performance levels and determines
whether the agency is meeting standards. Reliable data are also important to ensure that RSA’s
annual reports submitted to the President and Congress and made available to the public,
accurately report the VR agency’s performance. The extent that the State VR agency is meeting
performance standards could influence the amount of oversight and monitoring that RSA may
need to conduct at that agency.

We based our conclusions, in part, on the results of our review of a stratified statistical sample of
163 cases OOD closed during the 2012 reporting period. For each sampled case closure, we
determined whether select data elements were correctly reported and adequately supported
according to source documentation maintained in the participants’ case file in OOD’s AWARE
database.



8
 Required data elements (referred to as “ToDo” lists) included personal information, application, disability type,
disability priority, eligibility determination, planned services, employment and closure.
Final Report
ED-OIG/A03P0001                                                                                    Page 7 of 39

We provided a draft of this report to OOD for review and comment on December 2, 2015. We
received OOD’s comments on December 22, 2015. OOD partially concurred with Finding No. 1
regarding weaknesses in its internal controls over data quality for the 2012 reported RSA-911
data and concurred with our recommendations. OOD also partially concurred with
Finding No. 2 regarding incorrect and unverifiable performance indicator data reported in its
2012 RSA-911 report and concurred with our recommendation.

We did not change our findings and recommendations based on OOD’s comments to the draft
audit report. We summarized OOD’s comments on the draft audit report at the end of each
finding and included the comments in their entirety in Attachment 7 of this report.


Finding No. 1 – OOD Did Not Have Adequate Internal Controls to Assure the
                Accuracy of and Support Behind Its Case Service Report Data

OOD did not have adequate internal controls to provide reasonable assurance that VR case
service data reported in its 2012 RSA-911 report were accurate and adequately supported.
Specifically, OOD did not have policies and procedures that required VR counselors or
supervisors to verify that the data entered in participants’ case files were correct and adequately
supported by documentation prior to closing the case. This includes verifying that required
documents were properly completed and maintained in the participant’s case file.

OOD’s controls to ensure staff properly completed, maintained, and recorded VR case file
documents and data included participant case reviews9 conducted by VR supervisors and OOD’s
quality assurance staff. However, we found that OOD reported unverifiable and incorrect data
on its 2012 RSA-911 report. OOD’s controls were not adequate to ensure that staff members
(1) properly completed and maintained documents supporting VR case data in its AWARE
database and (2) detected and corrected participant VR case data recorded in the AWARE
database that were incorrect and inadequately supported by source documentation (unverifiable)
before reporting those data on the RSA-911 report.

Also, as a control, both OOD and RSA analyzed OOD’s 2012 RSA-911 report data for quality
using the edit check program “RSA Errors, Reasonableness Checks and Anomalies Program.”
The edit check program analyzes the RSA-911 report data and identifies “unreasonable” data,
data errors, data anomalies, data duplications, and data omissions.10 However, the edit check
program would not detect the unverifiable and incorrect data entries that we identified because it
cannot compare the data with the actual source documentation that the RSA-911 report was built
on.11


9
  The participant case reviews are discussed in more detail in the section “OOD’s Quality Assurance Program.”
10
   Before submitting its final RSA-911 report, OOD resolved the data problems that the edit check program
identified.
11
   We classified a data entry as unverifiable when required source documents were not present in the AWARE
database or when source documents present in the AWARE database did not include the information (that is,
missing signatures, dates, or other information) needed to verify the entry. We considered a data entry to be
incorrect when source document information did not agree to the data entry.
Final Report
ED-OIG/A03P0001                                                                                          Page 8 of 39

Based on our review of OOD’s VR process policies and procedures, staff interviews, and the
results of our review of a sample of participant case files, we concluded that the incorrect data
and the missing and incomplete source documentation for the data entries recorded in AWARE
and reported on OOD’s 2012 RSA-911 report were the result of (1) a lack of adequate controls to
ensure all required VR case documents were properly completed and maintained and that
VR case data agreed to source documents, (2) human error on the part of OOD staff, and
(3) OOD not using its quality assurance case review process to improve the procedures to ensure
that the VR case data were adequately supported by and agreed to source documents. We
discuss these issues in the sections “Completeness of Participant Case Files,” “Accuracy of
Participant Case File Data Entries,” and “OOD’s Quality Assurance Program” of this finding.

Recipients of Federal awards are required to maintain internal control over Federal programs that
provides reasonable assurance that the Federal awards are managed in compliance with laws,
regulations, and the provisions of contracts or grant agreements that could have a material effect
on each of its Federal programs (2 C.F.R § 200.303).12 Control procedures must be adequately
documented. The lack of documented policies and procedures may result in inconsistencies,
processing, or procedural errors, and noncompliance with laws and regulations.

Completeness of Participant Case Files

We reviewed the participant case files for our stratified statistical sample of 163 participants who
were included on OOD’s 2012 RSA-911 report from the universe of 21,554 participants and
determined whether OOD properly completed and maintained the required case service
documents in the participants’ case files. The sample included 94 participants who exited the
VR process with an employment outcome and 69 participants who exited without an
employment outcome. We found that

        at least one required case service document was missing for 45 of the 163 sampled
         participants,
        at least one required case service document was not completed properly for 16 of the
         163 sampled participants, and
        source documentation for required employment data was missing for 20 of the
         94 sampled participants.

OOD’s 2012 RSA-911 report data were extracted from information in its AWARE database.
OOD staff entered the participant VR case data into AWARE and scanned source documents that
were the basis for the data into the participants’ case files.13 Federal regulations and OOD
policies and procedures require the following source documents to be maintained in a
participant’s case file:14



12
   Although this criterion was not in effect during our audit period, it represents a very important foundational
requirement for OOD going forward.
13
   The AWARE database file is the participants’ official file.
14
   Only those documents applicable to when a participant exited the VR program during the VR process were
required to be maintained in the participant’s file (see the diagram in Attachment 2).
Final Report
ED-OIG/A03P0001                                                                                             Page 9 of 39

        application for services,
        certificate of eligibility,
        order of selection form,15
        IPE, and
        closure letter.

According to 34 C.F.R. § 76.731, a grantee should maintain records to show compliance with
program requirements. Additionally, Federal regulations require OOD to maintain the following
for participants who exited the VR program with an employment outcome: source documentation
showing the employment start date, hours worked in a week at case closure, weekly earnings at
case closure, and documentation that the participant maintained employment for 90 days.

OOD’s missing and incomplete VR case documents caused it to be in noncompliance with
Federal regulations, Ohio laws, and its own policies and procedures as discussed below. In
addition, the missing and incomplete documents resulted in unverifiable data being reported on
OOD’s 2012 RSA-911 report, as discussed briefly in this finding and in detail in Finding No. 2
of this report.

Required Case File Documents Were Missing

We estimate that 27 percent16 of the case files for the 21,554 participants reported on OOD’s
2012 RSA-911 report were missing at least one required document. The documents required to
be maintained (application for services, certificate of eligibility, order of selection form,
eligibility/order of selection letter, IPE, and closure letter) for a case file depended on when
during the VR process the participant exited the VR program (see the diagram in Attachment 2).
For each sampled participant file, we determined the required forms based on closure type and
verified the presence or absence of the required forms. The most frequently missing document
was the closure letter.17

OOD is required to maintain a record of services for each participant and determine the type of
documentation that it will maintain for the services provided (34 C.F.R. § 361.47(a) and (b)).
The required documents are discussed in the following:
   1. Federal regulations 34 C.F.R. § 361.41(b) and 34 C.F.R. § 361.47(a)(6);
   2. Ohio laws
       a) 3304-2-54(A)(6), “Eligibility for services, assessment, and trial work experiences,”
       b) 3304-2-65(F), “Order of selection,”
       c) 3304-2-56, “Conditions for providing services; and the individualized plan for
           employment,” and
       d) 3304-2-61(B)(1) and (D), “Closure;” and
   3. OOD policies
       a) 80-VR-01, “Vocational Rehabilitation Application and Intake Policy,” April 2, 2012,

15
   During our audit period the eligibility/order of selection letter replaced the certificate of eligibility and order of
selection form.
16
   We are 95 percent confident the rate of case files missing at least one form is between 18 and 35 percent.
17
   Table 3 in Attachment 3 shows the number and percent of required documents that were missing for the
163 participants in our sample.
Final Report
ED-OIG/A03P0001                                                                                    Page 10 of 39

        b)   VRP-0500, “Eligibility,” October 31, 2008,
        c)   VR Manual Chapter 6, “Order of Selection,” October 10, 2010,
        d)   VRP-0900, “Individualized Plan for Employment,” January 5, 2007, and
        e)   VRP-0300, “Case Closure,” April 3, 2007.

In addition, 34 C.F.R. § 80.42 discusses the retention and access requirements for grantee
records, supporting documents, programmatic and statistical records, and other records required
to be maintained by program regulations or the grant agreement or are otherwise reasonably
considered pertinent to program regulations or the grant agreement.

Required Case File Documents Were Not Properly Completed

We estimate that 4 percent18 of the case files for the 21,554 participants reported on OOD’s
2012 RSA-911 report had at least one required case service document that was not properly
completed. The application for services, certificate of eligibility, and IPE were required to be
signed and dated. For each sampled participant file, we determined whether the participant and
the VR counselor signed and dated the required forms, as applicable. The IPE was the document
most frequently missing the required signatures or dates or both.

According to 34 C.F.R. § 361.41(b)(2)(i)(A), a participant is considered to have submitted an
application when either the participant or their representative has completed and signed an
agency application. When the application is complete, the participant and the VR agency staff
must sign and date it (OOD Policy 80-VR-01).

Ohio law 3304-2-54(A)(6) requires a VR agency counselor to sign and date all certificates of
eligibility. Also, OOD policy VRP-0500 sets forth this same requirement.

The participant or their representative must sign an IPE, and a VR counselor employed by the
VR agency must approve and sign it (34 C.F.R. § 361.45(d)(3)(i) and (ii)). Similarly, Ohio law
3304-2-56, (B), “Conditions for providing services and the individualized plan for employment,”
and OOD Policy VRP-0900, “Individualized Plan for Employment,” January 5, 2007, set forth
this same requirement. According to OOD Policy VRP-0320, “Casework Development and
Service Delivery Timeline,” April 28, 2008, the counselor has 120 days to develop the IPE with
the participant. Therefore, OOD must document the IPE date in the participant’s file to
determine whether the counselor meets the 120-day requirement. In addition, VR agencies are
required to report the participant’s IPE date on the RSA-911 report (RSA Policy Directive 12-05,
February 8, 2012). The instructions for the RSA-911 report state that if the date the participant
and counselor signed the IPE are different, the later date should be used. Consequently, the IPE
must be signed and dated.

Source Documentation for Required Employment Data Were Missing

For the 94 participants in our sample with an employment outcome, we determined whether the
data entries in AWARE for the employment start date, weekly earnings amount, number of hours
18
  We are 95 percent confident the rate of case files having at least one incomplete required document is between
1 and 8 percent.
Final Report
ED-OIG/A03P0001                                                                                     Page 11 of 39

worked, and the employment outcome were adequately supported. Although employment data
were entered in the case files, we could not verify whether the data were correct because
supporting documentation for at least one of the employment data elements was missing for
20 of the 94 participants sampled. Because employment data has a significant impact on the
performance indicator calculations, we discuss this issue in detail in Finding No. 2 of the report.

A VR counselor explained that it was the responsibility of each counselor to ensure the VR case
file documents were properly completed and maintained in participants’ files and that the data
recorded in AWARE agreed with the source documents. A VR supervisor and a VR counselor
informed us that (1) required documents may not be properly completed and maintained in
participants’ case files due to human error, and (2) some documents may be missing because
staff did not complete the final step needed to attach a document in the AWARE database. After
selecting “Attach Document” staff must then select “Finish” to complete the process of attaching
documents. On occasion, staff may close out of AWARE before selecting “Finish.”

Accuracy of Participant Case File Data Entries

For the 163 participant case files, we also determined the reliability of selected data elements
maintained in OOD’s AWARE database and reported on its 2012 RSA-911 report.19 We
determined whether the selected data elements were accurate and adequately supported
according to source documentation maintained in OOD’s AWARE database. We identified data
quality problems that included (1) unverifiable and incorrect application dates, eligibility
determination dates, IPE dates, employer names, employment start dates, weekly earnings, and
hours worked, (2) unverifiable disability priority and employment outcomes, and (3) incorrect
birth dates, race and ethnicity, closure date, and type of closure codes.20 A VR supervisor and
VR counselor explained that incorrect data entries in AWARE were probably the result of human
error. VR counselors and supervisors were not required to verify the accuracy of the data
elements recorded in AWARE to source documents. Consequently, a VR participant’s case data
recorded in AWARE was not always the correct data according to source documents. We
discuss the results of this review in detail in Finding No. 2 of the report.

OOD’s Quality Assurance Program

OOD did not use its monitoring process to improve procedures and to provide reasonable
assurance that data were accurate and supporting documentation was maintained in participant
case files. Specifically, OOD’s case review policy 30-QA-01, “Quality Assurance Reviews for
Vocational Rehabilitation,” January 15, 2012, did not require that OOD officials use the results
of the case reviews to determine whether OOD’s internal controls over the VR process were
working effectively or to evaluate whether revisions to its internal controls or policies and




19
   The data elements reviewed, according to the participant’s type of closure data element code, are shown in
Table 4 in Attachment 4 to the report.
20
   Table 6 in Attachment 6 shows the number of incorrect and unverifiable data entries for the 163 participants
according to the data element and closure type.
Final Report
ED-OIG/A03P0001                                                                                      Page 12 of 39

procedures were necessary.21 In addition, OOD’s quality assurance process did not include
specific steps to verify numerous data elements in the case file.

OOD updated the quality assurance process that was used to evaluate the VR program and to
determine whether case documentation complied with Federal and State regulations and OOD
policy during our audit period. According to OOD’s Program Integrity and Evaluation Manager,
during the 2012 reporting period, quality assurance staff completed 36822 reviews based on the
universe of 6,467 cases that had an IPE written in the 2011 reporting period.23 At the beginning
of each month OOD’s Division of Information Technology staff generated a list of randomly
selected VR cases for review. The list contained random VR cases from all open and closed
cases where participants were applicants in the previous 24 months and had at least been
determined eligible for VR services (with an IPE created). The closed cases included on the list
were only those cases closed within the current Federal fiscal year. OOD’s quality assurance
staff reviewed cases selected from the random list using a “Case Review for VR” form to
evaluate the quality of case work in the following areas: application, eligibility, service and
employment, closure and after, and financial.24 The results of the case reviews were sent to the
responsible VR supervisors for review and comment. Corrections could be made to open and
closed cases reviewed on an ongoing basis. The quality assurance program manager then
worked with the area managers to develop training updates related to any findings. Quality
assurance staff participated in monthly video conference trainings and provided updates related
to case review results to field staff.

Quality and compliance issues identified in the quality assurance process were summarized in
quarterly and annual “Case Review Summary” reports. The report identified the areas reviewed
on the “Case Review for VR” form as strengths (all areas with 90–100 percent compliance),
opportunities for improvement (all areas with 80–89 percent compliance), and threats (all areas
with 79 percent compliance and below). According to OOD’s case review policy, the report
results were used to identify patterns and trends of staff documentation issues and areas of
training needs. However, OOD did not use the findings to consider changes to its procedures
used to ensure accurate data and supporting documentation. The annual “Case Review
Summary” report for the 2012 reporting period showed two areas as opportunities for
improvement that are similar to what we found during our audit: maintaining the eligibility/order
of selection letter in the case file and application for services signed and dated by an OOD
representative. Although the report noted the application being signed and dated by the
participant as a strength, the “Case Review for VR” form did not include a specific review step
to ensure the accuracy of the application date.

Also, during our audit period, VR supervisors conducted targeted case reviews throughout the year
as-needed on cases that could be either open or closed. Reviews conducted by VR supervisors
21
   Similarly OOD’s revised policy 30-QA-01, “Case Reviews for Vocational Rehabilitation Policy,”
October 1, 2013, did not contain this requirement.
22
   According to OOD’s program integrity and evaluation manager, the sample size of 368 case reviews performed
was intended to achieve estimates having a margin of error of 5 percent or less at the 95 percent confidence level.
23
   The 2011 reporting period is from October 1, 2010, through September 30, 2011.
24
   According to OOD’s program integrity and evaluation manager, over 200 of the case reviews were completed
using this single form. The remaining reviews were conducted using an older version of separate quality and
compliance review forms.
Final Report
ED-OIG/A03P0001                                                                                Page 13 of 39

were used for performance evaluation and to monitor complex cases, including when there is
concern about sufficient documentation being present in the case file to support program decisions.
According to a VR supervisor, targeted case reviews (1) focused on only a specific area of the
VR case work, such as the IPE or order of selection, and (2) were performed by completing the
section of the “Case Review for VR” form applicable to the scope of the review.

The Committee of Sponsoring Organizations of the Treadway Commission’s “Internal Control-
Integrated Framework” (COSO Report) provides a framework for organizations to design,
implement, and evaluate internal controls that will facilitate compliance with Federal laws,
regulations, and program compliance requirements. According to the COSO Report, one of the
five components of internal control is monitoring. Monitoring is a process that assesses the
quality of internal control over time. One of the activities that serve to monitor the effectiveness
of internal control is conducting internal quality control reviews. Quality control reviews should
provide reasonable assurance with respect to the stated objectives of the review.

Furthermore, we found that OOD’s VR quality assurance case review process was not adequate
to monitor whether the data recorded in AWARE were supported by and agreed to source
documents and were therefore reliable. Specifically, the “Case Review for VR” form that was
used for both quality assurance and supervisory reviews did not include specific checks to verify
that the following critical data elements in source documents agreed with information recorded
in AWARE:

        the race and ethnicity of the participant,
        the application date (OOD policy states that it must be signed by the applicant and
         initialed and dated by an OOD VR staff),
        the IPE date (when the IPE is signed on different dates by the participant and
         VR counselor, the later date should be used),
        the name of the employer of the participant,25
        the start date of employment,25
        the hours worked in a week, and
        the weekly earnings at closure.

Additionally, for cases closed with an employment outcome, the case reviews did not include a
specific step to determine whether source documents supported that the participant maintained
90 days of employment.

Without a VR quality assurance case review form that includes steps to ensure the AWARE
VR case data are accurate and verifiable, OOD management will be unable to use the results of
the case reviews to identify all patterns and trends of staff documentation and areas of potential
training needs.

Further, without (1) policies and procedures to require staff to verify that the data entered in
participants’ case files are correct and adequately supported by documentation, and (2) an

25
  The employer name and employment start date were not required to be reported on the 2012 RSA-911 report.
Beginning with the 2014 RSA-911 report, the employment start date is a required data element.
Final Report
ED-OIG/A03P0001                                                                     Page 14 of 39

assessment of OOD’s internal controls and policies and procedures over the VR process, OOD
may continue to report incorrect and unverifiable data, such as the data issues discussed
previously, on its RSA-911 report.

As a result of the lack of adequate internal controls over the VR program documentation
requirements for a participant’s record of service, OOD did not comply with Federal regulations,
Ohio law, and OOD policies. Consequently, OOD reported unverifiable case service data on its
2012 RSA-911 report. Therefore, RSA cannot assure that OOD met the performance indicator
requirements. It is important that OOD have internal controls that provide reasonable assurance
that data are accurate, complete, and supported since a number of data elements reported in the
RSA-911 report are used by RSA to monitor States’ compliance with mandated timelines for
delivering VR services to participants.

Recommendations

We recommend that the Commissioner of RSA require OOD to—

1.1    Establish and implement policies and procedures to ensure that

       (1) all required VR case documents are completed and maintained in participants’ case
           files; and
       (2) all required VR case data recorded in the AWARE database agree to and are
           supported by adequate source documentation.

1.2    Consider adding the specific checks listed above to the “Case Review for VR” form and
       to the quarterly and annual summary reports as part of the “scoring” to improve the
       quality assurance process and to better identify patterns and trends of staff documentation
       issues and areas for staff training.

1.3    Revise its quality assurance process to use the results of its case reviews to assess
       whether its internal controls are working effectively and determine whether it should
       revise its VR policies and procedures.

OOD Comments and OIG Response

OOD partially concurred with this finding and concurred with its recommendations. OOD
discussed corrective actions that it has taken, are in process, or it plans to take.

OOD Comments
OOD concurred that the controls in place during the 2012 RSA-911 reporting period did not
ensure that the cases reviewed had no instances of missing documentation or inaccurate data.
OOD did not concur that the employer name and employment start date data elements, as listed
in Table 6 of the report, were required in the 2012 RSA-911 report file. OOD also did not
concur that including these two data elements should be interpreted as having a direct correlation
to the inaccuracy of all of OOD’s evaluation standards and performance indicator results.
Final Report
ED-OIG/A03P0001                                                                     Page 15 of 39

OOD concurred with the recommendations and discussed corrective actions that it plans to take
related to the recommendations. Examples of these corrective actions include the following.

      Reorganize VR staff to create a new VR Data and Reporting Unit that will be responsible
       for performance and reporting management.
      Reorganize VR staff to increase resources in the VR Policy and Training Unit, with a
       focus on policy revisions due to Workforce Innovation and Opportunity Act
       implementation and staff compliance with case management data requirements and
       quality assurance monitoring findings.
      Update all VR policies and procedures to align with new requirements of the Workforce
       Innovation and Opportunity Act.
      Revise its quality assurance case review process.

OIG Response
We agree that the employer name and employment start date were not required to be reported in
the 2012 RSA-911 report, as we state in footnote 25 of our report. However, we reviewed these
data elements to determine whether the participants’ reported closure codes, which are reported
on the RSA-911 report, were accurate. Table 6 of the report shows the results (the number of
incorrect and unverifiable data entries) of all of the data elements we reviewed. We did not state
that these two data elements have a direct correlation to the inaccuracy of all of OOD’s
evaluation standards and performance indicators results.

We commend OOD for designing corrective actions intended to improve its internal controls and
its ability to ensure compliance with Federal regulations and RSA-911 reporting requirements.
We did not make any changes to the finding or the related recommendations as a result of
OOD’s comments.


Finding No. 2 – Performance Indicator Data Reported on OOD’s Fiscal Year 2012
                Case Service Report Were Not Reliable

We found that VR performance indicator data maintained in OOD’s AWARE database and
reported on its 2012 RSA-911 report were not correct and adequately supported and therefore
were not reliable. In its 2012 RSA-911 report, OOD provided incorrect and unverifiable data for
data elements that RSA used to calculate OOD’s 2012 performance indicator results. We
determined that

      performance indicators 1.1 and 1.2 were calculated using an unverifiable count of
       participant cases closed with an employment outcome;
      performance indicators 1.3 through 1.6 were calculated using incorrect and unverifiable
       weekly earnings at closure amounts; and
      performance indicator 2.1 was calculated using incorrect closure counts and
       race/ethnicity data element codes.
Final Report
ED-OIG/A03P0001                                                                                       Page 16 of 39

As a result, we have no assurance that the OOD performance indicators results that RSA
calculated for the 2012 reporting period are reliable. Federal regulations require that data
reported by a VR agency be valid, accurate, and in a consistent format (34 C.F.R. § 361.88(c)).
We estimate that 44 percent of the 21,554 reported participant case closures included on OOD’s
2012 RSA-911 report included at least one incorrect or one unverifiable data entry as follows:26

        7 percent of reported closures should not have been reported,27
        24 percent of reported closures included at least one incorrect data entry,28
        7 percent of reported closures included at least one unverifiable data entry,29 and
        6 percent of reported closures included at least one incorrect and one unverifiable data
         entry.30

We reviewed selected data elements to

        verify a participant’s identification (Social Security number and date of birth);
        determine whether reported data elements used in performance indicator calculations
         were correct and verifiable (race and ethnicity, weekly earnings at closure, hours worked
         in a week at closure, disability priority, and type of closure (codes 3 and 4); 31 and
        determine whether both reported and unreported32 data elements were correct and
         verifiable (application date, eligibility determination date, IPE date, services provided,
         employer name, employment start date, type of closure (codes 1, 5, 6, and 7), and closure
         date).

Performance Indicators

In its 2012 RSA-911 report, OOD provided incorrect and unverifiable data entries for data
elements used in performance indicator calculations (see the sections “Participants Who Exited
the VR Process With Employment” and “Participants Who Exited the VR Process Without
Employment” below) and RSA used these data entries to calculate OOD’s 2012 performance
indicator results.

The calculations for performance indicators 1.1 and 1.2 rely primarily on an accurate count of
employment outcomes. To determine whether the calculations for these performance indicators

26
   Table 5 in Attachment 5 shows the number of participant case files with incorrect and unverifiable data entries for
the 163 participants sampled according to their closure type.
27
   We are 95 percent confident that the percent of reported closures that should not have been reported ranges
between 3 and 16 percent.
28
   We are 95 percent confident that the percent of reported closures that included at least one incorrect data entry
ranges between 16 and 34 percent.
29
   We are 95 percent confident that the percent of reported closures that included at least one unverifiable data entry
ranges between 3 and 13 percent.
30
   We are 95 percent confident that the percent of reported closures that included at least one incorrect and one
unverifiable data entry ranges between 3 and 10 percent.
31
   We did not review the primary support at application, primary support at closure, and employment status at
closure data elements used in performance indicator calculations because we limited what we verified to the most
used and more critical data elements. Table 4 in Attachment 4 shows the data elements that we reviewed.
32
   These data elements were used in the verification of the performance indicator data.
Final Report
ED-OIG/A03P0001                                                                                     Page 17 of 39

were reliable, we reviewed participants’ case file documents in the AWARE database to
determine whether we could verify that the 94 participants sampled who were reported as
achieving an employment outcome maintained employment for 90 days. According to
34 C.F.R. § 361.56(b), a participant’s case may be closed as employed only if the participant has
maintained employment for at least 90 days to ensure the employment is stable. Based on the
results of our sample, we estimate that 7 percent of the 3,510 participants reported as achieving
an employment outcome on OOD’s 2012 RSA-911 report lacked documentation in their case
files to support the outcome reported.33 Therefore, performance indicators 1.1 and 1.2 were
calculated using data that we were unable to verify for accuracy.

Performance indicators 1.3 through 1.6 rely substantially on the weekly earnings at closure data
element. To calculate these performance indicators, for each participant that exited the
VR process with an employment outcome, the weekly earnings at closure data element is divided
by the hours worked in a week at closure data element to obtain an hourly wage. To determine
whether the calculations for these performance indicators were reliable, for the 94 participants
sampled, we reviewed the participants’ case files to determine whether the weekly earnings at
closure amounts reported to RSA were correct and supported by source documents. Based on
the results of our sample, we estimate that 14 percent of the 3,510 participants included in each
of these performance indicator calculations had incorrect weekly earnings at closure amounts and
17 percent had unverifiable weekly earnings at closure amounts.34 Based on the prevalence of
incorrect and unverifiable weekly earnings at closure amounts, we conclude that performance
indicators 1.3 through 1.6 were calculated using incorrect or unverifiable data.

The calculation for performance indicator 2.1 relies on accurate closure counts and race and
ethnicity data element codes across all types of closures. To determine whether the data used for
the calculation for this performance indicator were reliable, we verified whether all
163 participants sampled (1) represented actual case closures and (2) had correct race and
ethnicity data element codes. Based on the results of our sample, we estimate that 8 percent of
the 21,554 case closure records used to calculate performance indicator 2.1 either were
incorrectly included in the calculation or included an incorrect race and ethnicity code.35 Based
on the prevalence of incorrect race and ethnicity data element codes, and participants erroneously
being included on OOD’s 2012 RSA-911 report, we conclude that performance indicator 2.1 was
calculated using incorrect data.

Participants Who Exited the VR Process With Employment (Type of Closure Code 3)

Our review found unreliable (incorrect and unverifiable) data element entries for 73 (78 percent)
of the 94 participants reported as employed (see Table 5 in Attachment 5). This included
unreliable data elements for 36 participants (38 percent) that were used in one or more of OOD’s
33
   We are 95 percent confident that the percent of employment outcome cases reported on OOD’s 2012
RSA-911 report that lack supporting documentation needed to assess the accuracy of the employment outcome
ranges between 3 and 15 percent.
34
   We are 95 percent confident that the percent of employment outcome cases reported on OOD’s 2012
RSA-911 report with (1) an incorrect weekly earnings amount at closure ranges between 8 and 22 percent and
(2) an unverifiable weekly earnings amount at closure ranges between 10 and 26 percent.
35
   We are 95 percent confident that the percent of cases reported on OOD’s 2012 RSA-911 report that either were
incorrectly included on the report or had an incorrect race and ethnicity data code ranges between 3 and 15 percent.
Final Report
ED-OIG/A03P0001                                                                                      Page 18 of 39

performance indicator calculations for the 2012 reporting period. Specifically, we found the
following:36

        The type of closure code was unverifiable for seven participants because the file
         contained no documentation to verify that the participant was employed for 90 days.
        The race and ethnicity code was incorrect for two participants.
        The disability priority code was unverifiable for five participants.
        The amount of weekly earnings at closure was incorrect for 13 participants and
         unverifiable for 16 participants.
        The number of hours worked in a week at closure was incorrect for 5 participants and
         unverifiable for 15 participants.

Additional data quality problems for the data elements reviewed included (1) incorrect data
entries for date of birth and closure date and (2) incorrect and unverifiable data entries for
application date, eligibility determination date, IPE date, employer name, and employment start
date. Table 6 in Attachment 6 shows the number of incorrect and unverifiable data entries
according to the data element and closure type.

Based on the results of our sample, we estimate that of the 3,510 participants with employment
reported on OOD’s 2012 RSA-911 report:

        38 percent of the closures included at least one incorrect data entry,
        12 percent of the closures included at least one unverifiable data entry,
        28 percent of the closures included at least one incorrect and one unverifiable data entry,
         and
        22 percent of the closures were fully supported by source documentation.37

Employment Data Elements Were Unverifiable

We estimate that 21 percent38 of the 3,510 cases reported as closed with an employment outcome
on OOD’s 2012 RSA-911 report had at least one employment data element (employment start
date, weekly earnings amount, number of hours worked, and the employment outcome) that was
unverifiable because required supporting documentation was missing.39 Supporting
documentation for the weekly earnings amount was the most frequently missing.

If a participant obtains or maintains employment as a result of the VR services provided, Federal
regulations, Ohio law, and OOD policy require that employment data be maintained in the
participant’s case file. To successfully close and report a participant’s case as having exited the
VR program with an employment outcome, VR agencies must document (1) the participant’s
36
   A participant could have more than one unreliable data entry.
37
   The estimates have a margin of error of at most plus or minus 10 percentage points at the 95 percent confidence
level.
38
   We are 95 percent confident that for participants with an employment outcome the rate of case files missing
supporting documentation for an employment data entry is between 14 and 31 percent.
39
   The case files for 20 of the 94 participants sampled were missing supporting documentation for at least one of the
employment data elements.
Final Report
ED-OIG/A03P0001                                                                                       Page 19 of 39

employment start date and (2) that the participant maintained employment for 90 days. Also,
Ohio law 3304-2-61(D) and OOD policy VRP-0300 require that when a participant’s case is
closed as employed, a closure letter must be provided to the participant and the letter must
include the participant’s employment start date.

RSA requires VR agencies to provide the number of weekly hours worked and the weekly
earnings of the participant on the RSA-911 report (RSA Policy Directive 12-05,
February 8, 2012). RSA uses participants’ weekly hours worked and weekly earnings data to
calculate VR agencies’ compliance with performance indicators 1.3, 1.4, 1.5, and 1.6.
According to 34 C.F.R. § 361.84(b), the performance indicators require VR agencies to provide
information that will enable the Secretary to determine an agency’s compliance with the
VR program evaluation standards. Consequently, the number of weekly hours worked and
weekly earnings must be documented in the participant’s case file.

For participants who obtain employment, State VR agencies must maintain verification that the
participant is paid at or above the minimum wage and that the wage and level of benefits are not
less than that normally paid by the employer for the same or similar work performed by
participants who are not disabled (34 C.F.R. 361.47(a)(9)).

A condition for closing the case of a participant as employed is that the employment has been
maintained for a period of not less than 90 days (34 C.F.R. § 361.56(b)). In addition,
34 C.F.R. § 361.47(a)(15) requires State VR agencies to maintain documentation verifying that
the provisions of 34 C.F.R. § 361.56 have been met when the record of services for a participant
who has achieved an employment outcome is closed.

Consequently, it is RSA’s position that the requirements in 34 C.F.R. §§ 361.47 and 361.56
taken together require State VR agencies to maintain verifying documentation in the participant
case file related to the employment outcome including the employment start date, that the
participant maintained employment for 90 days, the hours worked, and the amount of earnings.
Although the requirements do not specify the type of verifying documentation that the agency
must maintain, the regulations show that VR agencies need to have some type of supporting
documentation for the employment data in the case file.

Participants Who Exited the VR Process Without Employment (Type of Closure Codes 1,
4, 5, 6, and 7)

We found unreliable data element entries for 26 (38 percent) of the 69 people reported to have
exited the VR process without employment (see Table 5 in Attachment 5). This included six
people who never applied to OOD for VR services and should not have been reported on OOD’s
2012 RSA-911 report.40 OOD’s Performance and Reporting Manager, Division of Performance
and Innovation, stated that on October 4, 2011, OOD began using AWARE as the case
management system, and everyone interested in VR services was entered into AWARE as a
participant. This included people who contacted OOD with interest or were referred for services
but who had not yet applied. In the OSCAR system these people were entered as referrals.
40
  For the 26 reported participants, these were the only unreliable data entries identified that RSA used in a
performance indicator calculation.
Final Report
ED-OIG/A03P0001                                                                               Page 20 of 39

AWARE’s referral module was unable to hold case attachments, so these peoples’ documents
were housed in the participant’s case file in the participant module.41

Because the people referred for services were entered into the AWARE participant module, the
system identified them as applicants. OOD’s Performance and Reporting Manager also stated
that “OOD issued guidance in April 2012 that instructed staff to only enter individuals who had
completed an application in the participant module.”

We found additional data quality problems for the data element entries reviewed that included
(1) an incorrect date of birth and Social Security number, (2) incorrect IPE, eligibility
determination, and closure dates, (3) an unverifiable disability priority designation; and
(4) incorrect and unverifiable application dates. Table 6 in Attachment 6 shows the number of
incorrect and unverifiable data entries according to the data element and closure type.

As explained in Finding No. 1, OOD did not have adequate internal controls to ensure that

        records that were the basis for VR case service data, including performance indicator
         data, were properly completed and maintained in participants’ case files; and
        all incorrect and unverifiable participant VR case file data entered into the AWARE
         database were detected and corrected before being reported on the RSA-911 report.

Also as stated in Finding No. 1, the edit check program “RSA Errors, Reasonableness Checks
and Anomalies Program” would not have detected the unverifiable and incorrect data entries that
we identified that were the result of missing and incomplete source documentation and
misreported source data.

As a result of OOD reporting unreliable (unverifiable and incorrect) data on its 2012 RSA-911
report, including performance indicator data, all of the performance indicators were calculated
using inaccurate or unsupported data elements, or both. Consequently, RSA may have
improperly determined OOD’s successful performance on the evaluation standards, and OOD
may have continued to participate in the VR program without entering into a required program
improvement plan.

In addition, unreliable data entries for the application date, eligibility determination date, and IPE
date hinder OOD management’s ability to monitor whether its staff timely serves participants in
compliance with Federal regulations and OOD policy. Those unreliable data entries also prevent
RSA from effectively monitoring OOD’s compliance with the required VR program timelines
for determining a participant’s eligibility for services and developing the participant’s IPE. RSA
conducts monitoring reviews of VR agencies on a 5-year cycle, and findings concerning meeting
the timelines for determining eligibility and developing the IPE are based on a review of the
RSA-911 report data for the 5 years prior to the fiscal year in which the monitoring review is
conducted.


41
  On January 17, 2013, OOD upgraded AWARE to version 5.12 which gave OOD the ability to attach files in the
Referral module.
Final Report
ED-OIG/A03P0001                                                                                  Page 21 of 39

Federal regulation 34 C.F.R. § 361.41(b) and OOD Policy VRP-0320, “Casework Development
and Service Delivery Timeline,” April 28, 2008, require that a participant’s eligibility
determination must be made within 60 days of application.42

Federal regulations require the IPE to be developed in a timely manner, and the VR agency must
establish and implement standards for the prompt development of the IPE, including timelines
that take into consideration the needs of the individuals (34 C.F.R. § 361.45(a)(1) and (e)).
OOD Policy VRP-0320, “Casework Development and Service Delivery Timeline,”
April 28, 2008, requires the counselor to develop the IPE with the participant within 120 days
from the eligibility determination date.

Recommendation

We recommend that the Commissioner of RSA require OOD to—

2.1       Establish and implement controls to ensure that OOD staff obtain and maintain source
          documentation supporting participants employment information including the
          employment start date, weekly earnings at closure, hours worked in a week at closure,
          and that the participant maintained employment for 90 days.

OOD Comments and OIG Response

OOD partially concurred with this finding and concurred with its recommendation. OOD stated
that it concurs with the importance of establishing controls to ensure that staff obtain and
maintain source documentation supporting participant employment information and discussed
corrective actions that it has taken and plans to take. However, OOD also requested that RSA
provide clear guidance and expectations regarding supporting employment and wage
documentation.

OOD Comments
OOD stated that not all performance indicators and evaluation standards data were unreliable.
OOD stated that the processes in place during our audit period met RSA standards and were
adequate to ensure accurate reporting. Specifically, OOD asserted that the VR counselors’ entry
of participant-reported employment information into the AWARE database would be adequate
source documentation. OOD also stated that the data elements that we did not review (primary
support at application and at closure) are critical in calculating performance indicator 1.6.

OOD concurred with the recommendation and discussed corrective actions that it has taken or
plans to take related to the recommendation. Examples of corrective actions discussed include
the following.

         Use standard templates from vendors that will include wage and hour data when
          purchasing job placement services.

42
 Exceptions are allowed if exceptional and unforeseen circumstances prevent this determination and the
VR agency and the participant agree to an extension.
Final Report
ED-OIG/A03P0001                                                                     Page 22 of 39

      Review the RSA-911 text file and run calculations to ensure the data are consistent with
       information in the AWARE database.
      Obtain the employment and wage verification information necessary to validate
       participants’ employment data that will be required by the final Workforce Innovation
       and Opportunity Act regulations and any related guidance provided by RSA.

OIG Response
We identified certain data element entries used to calculate performance data that were not
correct or adequately supported and, therefore, were not reliable. We subsequently concluded
that we have no assurance that the performance indicator results that RSA calculated for the
2012 reporting period were reliable and, as a result, RSA may have improperly determined
OOD’s successful performance on the evaluation standards.

We disagree that the VR counselors’ entry of participant-reported employment information into
the AWARE database meets RSA standards and is adequate source documentation. Although
the case files in the AWARE database contained entries for employment data elements, the files
did not contain source documentation to verify whether these data elements were reliable.
Entering data into the AWARE database, including the employment data screen, does not create
corroborating or supporting documentation. It is instead merely a data entry process. As stated
in the finding, RSA has taken the position that the requirements in 34 C.F.R. §§ 361.47 and
361.56, taken together, require State VR agencies to maintain verifying documentation in the
participant case file related to the participant’s employment outcome, including the employment
start date, the weekly earnings at closure, the hours worked in a week at closure, and that the
participant maintained employment for 90 days.

Regarding the calculation of performance indicator 1.6, we agree that the primary support at
application and closure data elements are used in the calculation of the performance
indicator. As stated in the report, we limited our review to the most used and more critical data
elements, such as weekly earnings and hours worked in a week at closure, which are used to
calculate performance indicators 1.3, 1.4, 1.5, and 1.6. The primary support at application and
closure data elements are only used in the calculation of performance indicator 1.6. In addition,
not reviewing these data elements would not change the results reported in the finding.

We commend OOD for designing corrective actions intended to improve its controls to ensure
that OOD staff obtain and maintain source documentation supporting participants’ employment
information. We did not make any changes to the finding or the related recommendation as a
result of OOD’s comments.




                 OBJECTIVES, SCOPE, AND METHODOLOGY


The audit objectives were to determine whether OOD (1) had adequate internal controls to
provide reasonable assurance that reported RSA-911 report data were accurate and complete and
Final Report
ED-OIG/A03P0001                                                                                      Page 23 of 39

(2) reported RSA-911 report performance indicator data that were accurate, complete, and
adequately supported. Our audit covered OOD’s 2012 RSA-911 report. To achieve our audit
objectives we performed the following procedures.

     1. Reviewed relevant laws, regulations, and guidance including the Rehabilitation Act of
        1973, Title I, Parts A and B, Sections 100-111; Federal regulations at 34 C.F.R. Part 361
        and 34 C.F.R. Parts 76 and 80; and the Ohio Administrative Code Chapter 3304-2 to gain
        an understanding of the requirements that OOD was required to follow when
        administering the VR program.

     2. Reviewed RSA’s “VR Program FY [Fiscal Year] 2013 Monitoring and Technical
        Assistance Guide;” RSA’s fiscal year 2008 and 2013 monitoring reports on OOD;
        RSA’s “FY 2012 Ohio Rehabilitation Services Commission Annual Review Report,”
        September 5, 2013; the State of Ohio’s single audit reports for fiscal years 2011 and
        2012;43 and the Ohio Auditor of State’s “Ohio Rehabilitation Services Commission
        Performance Audit,” July 3, 2008.

     3. Interviewed OOD officials to gain an understanding of OOD’s AWARE system and its
        procedures for capturing, reviewing, verifying, and submitting the RSA-911 report data.
        We interviewed a VR supervisor, four VR counselors, three program administrators, a
        rehabilitation program specialist, an AWARE business issues coordinator, a contracts
        manager for third-party arrangements, two database administration specialists, the
        Deputy Director for the Bureau of Services for the Visually Impaired, and the Program
        Integrity and Evaluation Manager.

     4. Reviewed OOD’s policies and procedures to gain an understanding of

             a. the processes for preparing and submitting the RSA-911 report,
             b. the processes for providing eligible participants with services to help them obtain
                or retain employment, and
             c. the processes for ensuring VR case file documents were properly completed and
                maintained in participants’ files and that the VR case data were properly recorded
                in the AWARE system.

     5. We performed limited testing of the AWARE system controls including a review of the
        system security levels.

     6. We reviewed the case files for a statistical sample of 163 case closures reported on
        OOD’s 2012 RSA-911 report to determine whether OOD properly completed and
        maintained the required case service documents (application for services, certificate of
        eligibility, order of selection form, IPE, and closure letter) in the participants’ case files.
        See the diagram in Attachment 2 for the required documents according to the
        participant’s closure type. Also, for 94 of the 163 selected cases that were reported as
        closed with an employment outcome, we determined whether source documentation was
43
  The State of Ohio’s fiscal year 2011 and 2012 single audit reports are for the periods July 1, 2010, through
June 30, 2011, and July 1, 2011, through June 30, 2012, respectively.
Final Report
ED-OIG/A03P0001                                                                              Page 24 of 39

       maintained in the participants’ case files to show the employment start date, hours
       worked in a week at case closure, weekly earnings at case closure, and that the participant
       maintained employment for 90 days.

We used the COSO Report and Office of Management and Budget Circular A-133 as criteria for
evaluating OOD’s internal controls over its VR case management process and the reporting
process used to report its RSA-911 report data. We concluded that OOD did not have adequate
internal controls to provide reasonable assurance that the data reported on its RSA-911 report
were correct and adequately supported (see Finding No. 1).

Sampling Methodology

We verified the completeness of the data reported in OOD’s final 2012 RSA-911 report that was
submitted on February 22, 2013. To verify the completeness of the data, we obtained a data
extract in October 2013 from OOD’s AWARE system for the universe of closed VR cases for
the 2012 reporting period and reconciled the universe to the universe of closed cases reported in
OOD’s final 2012 RSA-911 report submission.

We stratified the universe of 21,554 closed cases reported on OOD’s 2012 RSA-911 report into
two strata based on whether or not the case closure was designated as an employment outcome,
as shown in Table 2 below.

Table 2. Sampling Stratum by Closure Type

Stratum                              Type of Closure                              Universe     Sample Size
   1       Cases Closed with an employment outcome (closure type 3)                 3,510          94
   2       Cases closed without an employment outcome (all other closure types)    18,044          69
           Total                                                                   21,554         163

We selected a stratified random sample of 163 cases for review: from Stratum 1 we randomly
selected 94 cases, and from Stratum 2 we randomly selected 69 cases. We established the
sample sizes so that the estimated prevalence of data element attributes would have at most a
margin of error of plus or minus 10 percent at the 95 percent confidence level for estimates both
at the employment outcome stratum and across the universe of total case closures.

We calculated all estimates using sampling weights so that estimates reflect the intended
population. Some attributes that we tested were applicable only to cases closed with an
employment outcome, and those estimates are projected to the employment outcome case
closures. Because we followed a probability procedure based on random selections, our sample
is only one of a large number of samples that we might have drawn. Because each sample could
have provided different estimates, we express our confidence in the precision of our particular
sample’s results as a 95 percent confidence interval or a margin of error. This is the interval that
would contain the actual population value for 95 percent of samples we could have drawn. All
percentage estimates from the audit have margins of error of plus or minus 10 percentage points
or less at the 95 percent confidence level.
Final Report
ED-OIG/A03P0001                                                                     Page 25 of 39

The 163 participant case files included

      20 cases closed as a closure type 1,
      94 cases closed as a closure type 3,
      10 cases closed as a closure type 4,
      2 cases closed as a closure type 5,
      1 case closed as a closure type 6, and
      36 cases closed as a closure type 7.

Data Reliability

We verified the reliability of select data reported on OOD’s 2012 RSA-911 report and
maintained in OOD’s AWARE database as follows.

   1. For the 94 cases with an employment outcome, we determined whether the following
      data elements were correct and adequately supported according to source documents
      maintained in OOD’s AWARE database: Social Security number, date of birth, race and
      ethnicity, application date, eligibility determination date, disability priority, IPE date,
      services provided, employer name, employment start date, weekly earnings at closure,
      hours worked in a week at closure, type of closure, and closure date.

   2. For the 69 cases without an employment outcome, we determined whether the following
      data elements were correct and adequately supported according to source documents
      maintained in OOD’s AWARE database: Social Security number, date of birth, race and
      ethnicity, application date, eligibility determination date, disability priority, IPE date,
      services provided, type of closure, and closure date. Not all of the data elements were
      applicable to all 69 cases. See Table 4 in Attachment 4 for the data elements reviewed
      according to the participant’s type of closure code.

For the 163 cases sampled, we did not review the primary support at application, primary support
at closure, and employment status at closure data elements used in performance indicator
calculations because we limited what we verified to the most used and more critical data
elements.

We calculated OOD’s performance indicator scores using the data from the October 2013 data
extract of closed cases for the 2012 reporting period and compared the performance indicator
scores to the performance indicator scores RSA calculated by using the data from OOD’s final
submission of its 2012 RSA-911 report.

We found that VR case service data maintained in OOD’s AWARE database and reported on its
2012 RSA-911 report, including performance indicator data, were not correct and adequately
supported, and therefore were not reliable. Therefore, we were unable to determine the
reliability of the performance indicator calculations used by RSA to assess OOD’s performance
against the evaluation standards’ requirements (see Findings No. 1 and 2).
Final Report
ED-OIG/A03P0001                                                                      Page 26 of 39

We conducted site work at OOD’s offices in Columbus, Ohio, from September 16, 2013, through
September 20, 2013. We conducted additional audit work at our offices from
September 2013 through May 2015. We held an exit conference with OOD officials on
April 13, 2015.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objectives.




                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

This report incorporates the comments that you provided in response to the draft audit report. If
you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following U.S. Department of
Education official, who will consider them before taking final Departmental action on this audit:

                                          Janet LaBreck
                                         Commissioner
                             Rehabilitation Services Administration
                                U.S. Department of Education
                               550 12th Street, SW, Room 5086
                                   Washington, D.C. 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 calendar days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.
Final Report
ED-OIG/A03P0001                                                                   Page 27 of 39

We appreciate the cooperation and assistance extended by your staff during our audit. If you
have any questions or require additional information, you may contact me at (215) 656-6279 or
Teri L. Lewis, Assistant Regional Inspector General for Audit, at (215) 656-6276.




                                           Sincerely,

                                           /s/

                                           Bernard Tadley
                                           Regional Inspector General for Audit

Attachments
Final Report
ED-OIG/A03P0001                                                                Page 28 of 39

                                                                               Attachment 1

           Abbreviations, Acronyms, and Short Forms Used in This Report

  2012 reporting period   October 1, 2011, through September 30, 2012

  AWARE                   Accessible Web-based Activity and Reporting Environment

  C.F.R.                  Code of Federal Regulations

  COSO Report             The Committee of Sponsoring Organizations of the Treadway
                          Commission’s, “Internal Control Framework”

  IPE                     Individualized Plan for Employment

  OOD                     Opportunities for Ohioans with Disabilities

  OSCAR                   Online System for Computer Assisted Rehabilitation

  RSA                     Rehabilitation Services Administration

  RSA-911 report          Case Service Report

  VR                      Vocational Rehabilitation
Final Report
ED-OIG/A03P0001                                                                       Page 29 of 39

                                                                                                                                        Attachment 2

  Diagram:      The VR Process and its Related Participant Outcomes, Required Case File Documents, and Performance
                Indicators


        Step in VR Process                 Case Closure Type                     Required Documents in                            Performance
                                                                                       Case File                               Indicators Affected

        Application is completed           Exit VR as applicant (code 1)         Application and closure letter                         2.1




        Trial work experience              Exit VR during or after a trial       Application and closure letter                         2.1
                                           work experience (code 2)


                                           Exit VR from an order of
        Eligibility is determined          selection waiting list (code 6)
                                                                                  Application, certificate of eligibility,
        and participant is assigned                                                                                                     2.1
                                                                                  order of selection form and closure letter
        to a disability priority           Exit VR without employment
        category                           after eligibility but before an IPE
                                           was signed (code 7)


                                            Exit VR without employment,            Application, certificate of eligibility,
        IPE is signed                       after a signed IPE, but before         order of selection form, IPE and closure             2.1
                                            receiving services (code 5)            letter


                                                                                   Application, certificate of eligibility,
        Services provided to participant     Exit VR without employment,                                                               1.2 and 2.1
                                                                                   order of selection form, IPE and closure
                                             after receiving services (code 4)
                                                                                   letter


                                                                                   Application, certificate of eligibility,             1.1, 1.2, 1.3,
        Participant is employed for 90       Exit VR as employed (code 3)          order of selection form, IPE, closure letter,        1.4, 1.5, 1.6,
        days and exits the VR program                                              and related employment records                       and 2.1
Final Report
ED-OIG/A03P0001                                                                                        Page 30 of 39

                                                                                                        Attachment 3

Table 3. Missing Documentation for Our Sample of 163 Participants

                                                                                                        (D)
                                                                            (C)
                                                      (B)                                           Percent of
                                                                         Number of
                     (A)                           Number of                                         Required
                                                                          Required
                  Document                          Required                                        Documents
                                                                         Documents
                                                   Documents                                         Missing44
                                                                          Missing
                                                                                                      (C/B)*

 Application for services                               163                    8                          5%
 Certificate of eligibility                             130                    2                          2%
 Order of selection form                                130                    6                          5%
 Eligibility/order of selection letter                   13                    3                         23%
 IPE                                                    106                    2                          2%
 Closure letter                                         163                   30                         18%
* Rounded to the nearest percent.




44
   The percentages missing for the application and closure letters were based on the total number of 163 participants
in the sample. Of the 163 participants 143 (163 -20 closure type 1 participants) were required to have a certificate of
eligibility and order of selection form or an eligibility/order of selection letter in the file. Of the 143 participants,
130 were required to have a certificate of eligibility and order of selection form in the file. The percentages missing
for certificate of eligibility and order of selection forms are based on the total of 130 participants. The remaining 13
participants (143-130) were required to have an eligibility/order of selection letter in the file. The percent of
eligibility/order of selection letters missing is based on the total of 13 participants. The percent missing for the IPE
is based on 106 participants in the sample requiring an IPE to be maintained in the file (94 closure type 3, 10 closure
type 4, and 2 closure type 5).
 Final Report
 ED-OIG/A03P0001                                                                                                Page 31 of 39

                                                                                                                                                                Attachment 4

      Table 4. Data Elements Reviewed According to the Participant’s Type of Closure Code
                      Social    Birth     Race      Application    Eligibility      Disability   IPE Date    Services    Employer      Employment    Weekly     Hours     Closure   Closure
Closure Type         Security   Date      and         Date        Determination      Priority                Provided     Name          Start Date   Earnings   Worked     Type      Date
                     Number             Ethnicity                     Date                                                                              at        in a
                                                                                                                                                     Closure    Week at
                                                                                                                                                                Closure
                                                                  N/A, Verified
1: Exited as an
                        X        X         X            X         Eligibility Not     N/A          N/A         N/A         N/A            N/A          N/A       N/A        X         X
applicant
                                                                   Determined

3: Exited with an
employment              X        X         X            X               X               X           X           X            X             X            X         X         X         X
outcome
                                                                                                                            N/A
4: Exited without
                                                                                                                          Verified
an employment
                                                                                                                         Participant
outcome, after          X        X         X            X               X               X           X           X                         N/A          N/A       N/A        X         X
                                                                                                                          Did Not
receiving IPE
                                                                                                                           Obtain
services
                                                                                                                        Employment
5: Exited without
                                                                                                               N/A
an employment
                                                                                                             Verified
outcome, after
                        X        X         X            X               X               X           X           No         N/A            N/A          N/A       N/A        X         X
signed IPE, but
                                                                                                             Services
before receiving
                                                                                                             Provided
IPE services
                                                                                                   N/A
6: Exited from an
                                                                                                  Verified
Order of Selection      X        X         X            X               X               X                      N/A         N/A            N/A          N/A       N/A        X         X
                                                                                                 No Signed
Wait List
                                                                                                    IPE

7: Exited without
an employment                                                                                      N/A
outcome, after                                                                                    Verified
                        X        X         X            X               X               X                      N/A         N/A            N/A          N/A       N/A        X         X
eligibility, but                                                                                 No Signed
before an IPE was                                                                                   IPE
signed
Final Report
ED-OIG/A03P0001                                                                          Page 32 of 39

                                                                                         Attachment 5

Table 5.   Summary of Incorrect and Unverifiable Data Entries for Participants Sampled
           According to the Closure Type

                                                                             Total
                                                                          Participant          Data
                              Included Both                  Included      Files That        Correctly
               Number of      Incorrect and    Included      Incorrect     Included          Reflected
               Participants    Unverifiable   Unverifiable     Data      Incorrect and      the Source
Closure Type    Sampled        Data Entries   Data Entries    Entries     Unverifiable      Documents
                                   (A)            (B)           (C)       Data Entries
                                                                           (A+B+C)
Employment
 Obtained          94              26             11            36            73               21
  (Type 3)
Employment
Not Obtained       69              1               4            21            26               43
 (All Other
   Types)
   Totals          163             27             15            57            99               64
 Final Report
 ED-OIG/A03P0001                                                                                        Page 33 of 39

                                                                                                                                                                Attachment 6

 Table 6.       Number of Incorrect and Unverifiable Data Entries According to the Data Element and Participant’s Closure Type

                             Closure    Closure      Closure      Closure     All        All          All          All       All       All        All          All          All       All
                             Type 3     Type 3       Type 3       Type 3     Other      Other        Other        Other     Other    Closure    Closure     Closure       Closure   Closure
                             Correct   Incorrect   Unverifiable   Totals    Closure    Closure      Closure      Closure   Closure    Types      Types       Types         Types     Types
                                                                             Types      Types        Types        Types     Types    Correct   Incorrect   Unverifiable     N/A     Totals
       Data Element                                                         Correct   Incorrect   Unverifiable     N/A     Totals
Social Security Number         94          0           0            94        68         1             0           0         69       162         1             0           0        163
Birth Date                     92          2           0            94        68         1             0           0         69       160         3             0           0        163
Race Ethnicity                 92          2           0            94        69         0             0           0         69       161         2             0           0        163
Application Date               74         11           9            94        50         15            4           0         69       124         26           13           0        163
Eligibility Date               74         13           7            94        45         4             0           20        69       119         17            7           20       163
Disability Priority            89          0           5            94        48         0             1           20        69       137         0             6           20       163
IPE Date                       56         28           10           94        9          3             0           57        69        65         31           10           57       163
Employer Name                  85          8           1            94        0          0             0           69        69        85         8             1           69       163
Employment Start Date          65         21           8            94        0          0             0           69        69        65         21            8           69       163
Weekly Earnings at Closure     65         13           16           94        0          0             0           69        69        65         13           16           69       163
Weekly Hours Worked at         74          5           15           94        0          0             0           69        69        74         5            15           69       163
Closure
Closure Date                   93         1             0           94        63         6             0           0         69       156         7             0           0        163
Closure Type                   87         0             7           94        63         6             0           0         69       150         6             7           0        163

      N/A – The data element was not applicable because it was not required for the participant’s closure type.
Final Report
ED-OIG/A03P0001                                                                            Page 34 of 39

                                                                                            Attachment 7

OOD’s Response to the Draft Audit Report




December 22, 2015                                                Control Number: ED-OIG/A03P0001


Bernard Tadley
U.S. Department of Education, Office of Inspector General
The Wanamaker Building
100 Penn Square East, Room 502
Philadelphia, PA 19107


Dear Mr. Tadley,

Opportunities for Ohioans with Disabilities (OOD) is in receipt of the U.S. Department of Education, Office
of Inspector General (OIG) communication from December 2, 2015. This correspondence included a draft
audit report that covered the OOD 2012 RSA-911 report for the reporting period October 1, 2011, through
September 30, 2012 (2012 reporting period).

We appreciate the opportunity to provide feedback to this draft audit report. OOD shares the U.S.
Department of Education’s commitment to the integrity of the data included in federal reports provided by
the Vocational Rehabilitation (VR) program. Per your instruction, OOD submits the following written
comments on the findings and recommendations contained in the draft audit report. This response includes
information about continuous improvement efforts that OOD has implemented in the areas below to
demonstrate increased controls implemented during and subsequent to Federal Fiscal Year (FFY) 2012.

Finding No. 1 – OOD Did Not Have Adequate Internal Controls to Assure the Accuracy of and
Support Behind Its Case Service Report Data

OOD partially concurs with this finding and offers the following input:

OOD concurs with the finding that the controls in place during FFY 2012 did not ensure that the cases
reviewed by the OIG had no instances of missing and/or inaccurate data as outlined in Table 3 and Table
6. It is important to note that OOD implemented a new case management system (AWARE) on 10/1/2012.
One of the reasons for the selection of the AWARE case management system was the increased capability
for improved management controls to address data integrity.
                          400 East Campus View Boulevard                  614 | 438.1200
                          Columbus, Ohio 43235-4604 U.S.A.                800 | 282.4536
                          www.ood.ohio.gov
Final Report
ED-OIG/A03P0001                                                                             Page 35 of 39

OOD does not concur that all of the elements in Table 6 were required or included in the 2012 RSA-911
file. Specifically, the employer name and the employment start date are not defined in RSA-PD-12-05 as
data elements. The draft audit report acknowledges this in footnote 25: “The employer name and
employment start date were not required to be reported on the 2012 RSA-911 report. Beginning with the
2014 RSA-911 report, the employment start date is a required data element.”

Consequently, OOD does not agree that including these two (2) data elements should be interpreted as
having a direct correlation to the inaccuracy of all of OOD’s Standards and Indicators.

Recommendation: We recommend that the Commissioner of RSA require OOD to —

       1.1 Establish and implement policies and procedures to ensure that
               (1) all required VR case documents are completed and maintained in participants’ case
               files; and
               (2) all required VR case data recorded in the AWARE database agree to and are
               supported by adequate source documentation.

OOD concurs with this recommendation and offers the following input:

       1.1 OOD concurs and is in the process of reviewing and updating Vocational Rehabilitation (VR)
           policies and procedures with the intent of aligning with new requirements under the Workforce
           Innovation and Opportunity Act (WIOA) and adding a level of detail that will ensure the
           consistency and accuracy of recorded data. It is worth noting that OOD has significant concern
           about guidance contained in OMB Control Number 1820-0508 which indicates more than forty
           RSA-911 data elements may not be modified or updated due to the nature of the elements
           during the life of the case. The social security number, birthdate, race/ethnicity, application
           date, eligibility date, and start date of employment, all of which were cited in the OIG’s audit,
           are included in this list of forty. This directive would create a significant barrier for OOD and
           other States to maintain the standards of data integrity that are being evaluated in this audit.
           Therefore, OOD requests that RSA consider the input previously provided during the comment
           period for this policy directive change.

           Corrective Action(s) Taken:

              OOD conducted a Lean Six-Sigma Kaizen improvement process in April 2012 to
               restructure and simplify the VR application and intake procedure, which set in motion
               significant reductions in the timeframe from application to eligibility for services. This
               revised process clarified the manner in which OOD uses AWARE to track individuals who
               had not yet applied for services in the referral module rather than the participant module.
               This prevents reporting individuals who have not yet applied for services in the RSA-911
               report. The application and intake procedure was also subsequently revised to specifically
               define the application date in AWARE as the date the counselor signs the application.

              In May 2014, OOD developed an exceptions report to identify case records where the
               Eligibility letter may be missing. This report was made available to supervisors to monitor
               and ensure compliance, which also successfully addressed a similar State of Ohio audit
               finding. The quality assurance case review validates that this process has resulted in
               increased compliance in the area of having the Certificate of Eligibility/Eligibility letter
Final Report
ED-OIG/A03P0001                                                                               Page 36 of 39

               present in the case file at the following rates: FFY 2013 = 86.37%; FFY 2014 = 92.99%;
               and FFY 2015 = 97.85%.

              In October 2014, OOD began to use the State Verification Exchange System (SVES) to
               obtain information from the Social Security Administration for increased accuracy of SSNs,
               dates of birth, and SSI/SSDI benefit data. OOD validates AWARE data against the SVES
               database on a weekly basis and has processes in place to resolve any discrepancies in
               the data.

           Corrective Action(s) Planned:

              OOD has reorganized VR staff to create a new VR Data and Reporting Unit. This Unit will
               be fully operational as of January 11, 2016, and will be responsible for performance
               reporting and management. This Unit will develop and distribute reports, no later than
               September 30, 2016, to verify the presence of the Closure letter document consistent with
               the strategy previously implemented for the Eligibility letter.

              OOD also has reorganized VR staff to increase resources in the VR Policy & Training Unit.
               They will focus on policy revisions due to WIOA implementation and staff compliance with
               case management data requirements and quality assurance monitoring findings.

Recommendation: We recommend that the Commissioner of RSA require OOD to —

       1.2 Consider adding the specific checks listed above to the “Case Review for VR” form and to the
           quarterly and annual summary reports as part of the “scoring” to improve the quality assurance
           process and to better identify patterns and trends of staff documentation issues and areas for
           staff training.

OOD concurs with this recommendation and offers the following input:

       1.2 OOD concurs with this recommendation, and will expand the quality assurance process to
           include additional items related to data integrity. This will allow OOD to better identify patterns
           and trends of staff documentation issues and areas for staff training. OOD is also focusing on
           more proactive solutions to this type of verification process throughout the life of the case.
           These solutions will be evaluated as part of the quality assurance case review process.

           Corrective Action(s) Taken:

              In April 2012, OOD revised the quality assurance case review process and added areas of
               evaluation related to review and authentication of casework files. This additional review
               includes validating the source document against the pre-populated dates from the AWARE
               case system date for application, eligibility and IPE dates.
                    o OOD has updated the case review procedure to reflect the revised process that
                        has been put in place since the audit period. If the source data and the system do
                        not match, the reviewer indicates this on the case review form and provides this
                        feedback to the supervisor who works with the staff to correct the data.
Final Report
ED-OIG/A03P0001                                                                              Page 37 of 39

            Corrective Action(s) Planned:

               Upon issuance of the final WIOA regulations, OOD will be updating all VR policies and
                procedures to ensure alignment with new requirements under the law. As a part of these
                revisions, the Monitoring and Compliance Unit (MCU) will work in tandem with the VR
                program to update the quality assurance case review policy, procedure and associated
                forms. This will include the addition of elements related to data integrity and proper source
                documentation and is expected to be completed no later than twelve months following the
                final WIOA regulations.

               OOD is evaluating the case review procedure to determine changes related to this
                recommendation.

Recommendation: We recommend that the Commissioner of RSA require OOD to —

        1.3 Revise its quality assurance process to use the results of the case reviews to assess whether
            its internal controls are working effectively and determine whether it should revise its VR
            policies and procedures.

OOD concurs with this recommendation and offers the following input:

        1.3 OOD concurs and the MCU is currently involved in the policy and procedure development
            process and routinely provides training for VR staff about quality assurance standards and
            results of reviews.

            Corrective Action(s) Planned:

               OOD is evaluating the case review procedure to incorporate changes that are in
                compliance with the new WIOA regulations and plans to implement no later than twelve
                months following the final regulations.

               OOD also has reorganized VR staff to increase resources in the VR Policy & Training Unit.

               They will focus on policy revisions due to WIOA implementation and staff compliance with
                case management data requirements and quality assurance monitoring findings.

Finding No. 2 – Performance Indicator Data Reported on OOD’s Fiscal Year 2012 Case Service
Report Were Not Reliable

OOD partially concurs with this finding and offers the following input:

OOD does not concur that all Standards and Indicator data were unreliable. OOD contends that the
processes in place during the period of this review met the standards put forth by the Rehabilitation
Services Administration and were therefore adequate to ensure accurate reporting.

Specifically, OOD contends that the counselors’ entry into the AWARE case management system of the
self-reported information from the individual would constitute adequate source documentation . Neither the
Final Report
ED-OIG/A03P0001                                                                              Page 38 of 39

OIG nor OOD found any evidence as a part of this review that an individual reported as working was not in
fact employed. This aligns with CFR §34.361.47(a)(9) which requires if an individual obtains competitive
employment, verification that the individual is compensated at or above the minimum wage and that the
wage and level of benefits are not less than that customarily paid by the employer for the same or similar
work performed by nondisabled individuals.

Per footnote 31 of the draft audit report, the OIG did not review the primary support at application, primary
support at closure, and employment status at closure data elements used in performance indicator
calculations because they limited what was verified to the most used and more critical data elements. The
omitted elements (primary support at application and at closure) are critical in calculating Standard 1.6.

Recommendation: We recommend that the Commissioner of RSA require OOD to—

        2.1 Establish and implement controls to ensure that OOD staff obtain and maintain source
            documentation supporting participants employment information including the employment start
            date, weekly earnings at closure, hours worked in a week at closure, and that the participant
            maintained employment for 90 days.

OOD concurs with the recommendation and offers the following input:

        2.1 OOD concurs with the importance of establishing and implementing controls to ensure that staff
            obtain and maintain source documentation supporting participant employment information
            including the employment start date, weekly earnings at closure, hours worked in a week at
            closure, and that the participant maintained employment for 90 days. OOD requests guidance
            from the Rehabilitation Services Administration about what constitutes sufficient source
            documentation for these data elements. As noted above, OOD contends that the counselors’
            entry into the AWARE case management system of the self-reported information from the
            individual would constitute adequate source documentation.

            Corrective Action(s) Taken:

               In December 2012, OOD required the use of standard templates from vendors which are
                to include wage and hour information when purchasing job placement services.

               OOD runs the edit checker provided by the Rehabilitation Services Administration in order
                to resolve errors and anomalies in the RSA-911 data file prior to submission of this annual
                report. Beginning with the FFY 2014 RSA-911 file, OOD reviews the RSA-911 text file and
                runs calculations to ensure the data is consistent with information in the case management
                system. This is a further means to ensure accuracy of the data in the RSA-911 file.

            Corrective Action(s) Planned:

               Based on final WIOA regulations scheduled to be released in early 2016, OOD requests
                RSA to provide clear guidance and expectation regarding supporting wage documentation,
                which is required consistently for all participating WIOA programs. Based on this guidance,
                OOD will continue partnering with the Ohio Department of Job and Family Services and
                the Ohio Department of Higher Education to obtain the necessary verification of
                employment and wage information to validate employment of individuals served by the VR
Final Report
ED-OIG/A03P0001                                                                            Page 39 of 39

                program. This is a requirement under WIOA and new processes should be in place no
                later than June 30, 2016.

Thank you for the opportunity to provide comment on this draft report. Please let us know if you have any
questions about the contents of this response.


Sincerely,



Kevin Miller
Executive Director
Opportunities for Ohioans with Disabilities




Cc:     OOD Commissioners
        OOD Executive Staff