oversight

Pennsylvania's Department of Labor and Industry, Office of Vocational Rehabilitation's Case Service Report Data Quality

Published by the Department of Education, Office of Inspector General on 2016-03-02.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                      UNITED STATES DEPARTMENT OF EDUCATION
                                 OFFICE OF INSPECTOR GENERAL

                                                                                   AUDIT SERVICES
                                                                             Philadelphia Audit Region


                                      March 02, 2016
                                                                                 Control Number
                                                                               ED-OIG/A03P0002

Ms. Kathy Manderino
Secretary
Pennsylvania Department of Labor and Industry
1700 Labor and Industry Building
7th and Forster Streets
Harrisburg, PA 17120


Dear Ms. Manderino:

This final audit report, “Pennsylvania’s Department of Labor and Industry, Office of Vocational
Rehabilitation’s Case Service Report Data Quality,” presents the results of our audit. The
objectives of our audit were to determine whether Pennsylvania’s Department of Labor and
Industry, Office of Vocational Rehabilitation (1) had adequate internal controls to provide
reasonable assurance that reported “Case Service Report” (RSA-911 report) data were accurate
and complete and (2) reported RSA-911 performance indicator data that were accurate,
complete, and adequately supported. Our audit covered Pennsylvania’s Department of Labor
and Industry, Office of Vocational Rehabilitation (PA OVR) 2013 RSA-911 report for the
reporting period October 1, 2012, through September 30, 2013 (2013 reporting period).

PA OVR had adequate internal controls to ensure that the data it reported to the Rehabilitation
Services Administration (RSA) were complete. However, PA OVR did not have adequate
internal controls to ensure that its 2013 RSA-911 report data were accurate and adequately
supported. Specifically, we found that PA OVR (1) lacked policies and procedures to require
verification of the data entered into participants’ case files and for its RSA-911 reporting process
and (2) lacked an adequate monitoring process to ensure that data were accurate and required
documentation was maintained in participant case files.

Our testing of data that PA OVR reported to RSA found a significant number of unverifiable
data entries for data elements that RSA used to calculate PA OVR’s 2013 performance indicator
results. Consequently, we have no assurance that the performance indicator results that RSA
calculated were reliable. RSA uses the performance indicator results to determine whether
PA OVR meets RSA’s established evaluation standards. As a result, RSA may have improperly
determined PA OVR’s successful performance on the evaluation standards for the 2013 reporting
period.
Final Report
ED-OIG/A03P0002                                                                                          Page 2 of 35

We made several recommendations to the Commissioner of RSA that would require PA OVR to
establish and implement enhanced data quality controls. PA OVR did not explicitly agree or
disagree with Finding No. 1 and the associated recommendations regarding weaknesses in its
internal controls over data quality for the 2013 reported RSA-911 data. However, PA OVR
discussed corrective actions that it plans to take to make improvements, including enhancements
to its case management system, the Commonwealth Workforce Development System (CWDS).
PA OVR disagreed with Finding No. 2 regarding unverifiable performance indicator data
reported in its 2013 RSA-911 report and disagreed with our recommendation. PA OVR
disagreed with Finding No. 2 because it believes that since its current process requires that
counselors validate closure data at case closure, additional documentation would be duplicative.
PA OVR stated that it would provide further training on the closure process and continue to
reinforce the requirement through its case review process and existing training. We did not
change our findings and recommendations based on PA OVR’s comments to the draft audit
report.




                                               BACKGROUND


The U.S. Department of Education’s Office of Special Education and Rehabilitative Services
supports programs that serve millions of children, youth, and adults with disabilities. The Office
of Special Education and Rehabilitative Services’ Rehabilitation Services Administration
oversees grant programs that help people with physical or mental disabilities to obtain
employment and live more independently through the provision of counseling, medical and
psychological services, job training, and other individualized services. RSA provides Vocational
Rehabilitation Grants to States to assist them in operating vocational rehabilitation (VR)
programs. The VR program grants are provided to support a wide range of services designed to
help people with disabilities prepare for and engage in gainful employment consistent with their
strengths, resources, priorities, concerns, abilities, capabilities, interests, and informed choice.

Each State designates a State agency to administer the VR program. Some States have more
than one VR agency (a general agency and an agency for the blind). General agencies serve all
people with disabilities except those who are blind or visually impaired and State agencies for
the blind provide services only for people who are blind or visually impaired. The remaining
States use a combined agency which serves all people with disabilities in the State.

In Pennsylvania, PA OVR is the State agency designated to administer the VR program.
PA OVR is a combined agency that is composed of two bureaus: the Bureau of Blindness and
Visual Services and the Bureau of Vocational Rehabilitation Services. 1 PA OVR is overseen by
an executive director, who reports directly to the Secretary of the Department of Labor and
Industry. The executive director approves PA OVR’s VR State plan. During our audit period,
PA OVR received a VR program grant award of $111,450,404.

1
 PA OVR includes 6 district offices that serve the blind or visually impaired and 15 district offices that serve all
other people with disabilities for the 67 counties in the commonwealth.
Final Report
ED-OIG/A03P0002                                                                                       Page 3 of 35

People eligible for VR program services (referred to as participants in this report) are those who
have a physical or mental impairment that results in a substantial impediment to employment,
who can benefit from VR services for employment, and who require VR services. When
PA OVR cannot serve all eligible participants with disabilities due to limited resources, it uses an
order of selection. Under an order of selection, eligible participants are assigned to priority
categories based on the significance of their disability. PA OVR prioritizes serving participants
with the most significant disabilities. Since 1994, PA OVR has operated under an order of
selection that assigns participants with disabilities into three categories: most significantly
disabled, significant disability, and disability. Eligible participants who are not designated as
most significantly disabled are placed on a waiting list. However, while receiving American
Recovery and Reinvestment Act funding, PA OVR also provided services to those who were
significantly disabled. It ended those services in October 2012. According to RSA’s
“FY 2013 Pennsylvania Office of Vocational Rehabilitation Annual Review Report,” 534 people
were on the waiting list as of September 30, 2013.

Each year, State VR agencies must use the RSA-911 report to report to RSA case data pertaining
to all participants whose case records were closed in a given fiscal year. The RSA-911 report
must be submitted by November 30 (60 days after the end of the fiscal year). PA OVR uses its
CWDS case management database, along with a hard copy case file, to store data about its
participants’ VR cases and to manage the case flow. PA OVR developed and implemented
CWDS in 2007. The case data reported on PA OVR’s 2013 RSA-911 report were extracted
from its CWDS database. In its 2013 RSA-911 report, PA OVR reported a total of 25,709 closed
participant cases, of which 9,950 cases (about 39 percent) were reported closed with an
employment outcome. 2 Cases are coded in the RSA-911 report by type of closure to indicate
when in the VR process a participant exited the program, as shown below and in the diagram in
Attachment 2:

    •    exited as an applicant (code 1),
    •    exited during or after a trial work experience/extended evaluation 3 (code 2),
    •    exited from an order of selection waiting list (code 6),
    •    exited without an employment outcome after eligibility was determined but before an
         individualized plan for employment (IPE) 4 was signed (code 7),
    •    exited without an employment outcome after an IPE was signed but before receiving
         services (code 5),
    •    exited without an employment outcome after receiving services (code 4), and
    •    exited with an employment outcome (code 3).

Section 106 of the Rehabilitation Act of 1973, as amended, requires RSA to establish evaluation
standards and performance indicators for the VR program that include outcome and related
measures of program performance. Two evaluation standards were established in June 2000
(34 Code of Federal Regulations [C.F.R.] Part 361). RSA has established minimum levels of

2
  Employment outcome means obtaining or retaining full-time or part-time competitive employment.
3
  Participants complete trial work experiences or extended evaluations to determine whether they can benefit from
VR services if existing evidence indicates that the participant is incapable of benefiting from the services.
4
  The IPE is a written plan outlining a participant’s vocational employment goal and the services to be provided to
assist the participant in reaching the goal.
Final Report
ED-OIG/A03P0002                                                                                    Page 4 of 35

performance for each performance indicator. RSA uses data from the RSA-911 report to
monitor State agencies’ VR program performance, including calculating State agencies’ results
on the performance indicators and determining whether they have met the evaluation standards.

The evaluation standards and performance indicators are as follows.

Evaluation Standard 1—Employment Outcomes
Standard 1 includes six performance indicators, three of which are primary indicators. The
primary indicators (1.3, 1.4, and 1.5) measure the quality of the employment outcomes achieved
by participants served by the program.

    •   Performance Indicator 1.1—The number of participants exiting the VR program who
        achieved an employment outcome during the current performance period compared to the
        number of participants who exited the VR program after achieving an employment
        outcome during the previous performance period.

    •   Performance Indicator 1.2—Of all participants who exit the VR program after receiving
        services, the percentage who are determined to have achieved an employment outcome.

    •   Performance Indicator 1.3—Of all participants determined to have achieved an
        employment outcome, the percentage who exit the VR program in competitive, self-or
        business enterprise program 5 employment with earnings equivalent to at least the
        minimum wage.

    •   Performance Indicator 1.4—Of all participants who exit the VR program in competitive,
        self- or business enterprise program employment with earnings equivalent to at least the
        minimum wage, the percentage who are participants with significant disabilities.

    •   Performance Indicator 1.5—The average hourly earnings of all participants who exit the
        VR program in competitive, self- or business enterprise program employment with
        earnings equivalent to at least the minimum wage as a ratio to the State’s average hourly
        earnings for all participants in the State who are employed (as derived from the Bureau of
        Labor Statistics report “State Average Annual Pay” for the most recent available year).

    •   Performance Indicator 1.6—Of all participants who exit the VR program in competitive
        employment, self- or business enterprise program employment with earnings equivalent
        to at least the minimum wage, the difference between the percentage who report their
        own income as the largest single source of economic support at the time they exit the
        VR program and the percentage who report their own income as the largest single source
        of support at the time they apply for VR services.

To achieve successful performance on standard 1, State VR agencies must meet or exceed the
minimum level of performance for four of the six performance indicators in the evaluation

5
 A business enterprise program means a participant who obtains employment as an operator of a vending facility or
other small business under the management and supervision of a State VR agency.
Final Report
ED-OIG/A03P0002                                                                                 Page 5 of 35

standard, including meeting or exceeding the performance levels for two of the three primary
indicators.

Evaluation Standard 2—Equal Access to Services

Standard 2 includes one performance indicator.

   •    Performance Indicator 2.1—The service rate for all participants with disabilities from
        minority backgrounds as a ratio to the service rate for all participants with disabilities
        from nonminority backgrounds.

To achieve successful performance on standard 2, State VR agencies must meet or exceed the
performance level established for performance indicator 2.1.

State agencies that fail to meet these performance levels must develop a program improvement
plan outlining specific actions to be taken to improve program performance. For the 2013
reporting period, PA OVR achieved successful performance on the evaluation standards;
however, it did not meet the performance levels for performance indicators 1.2 and 1.5. Table 1
shows the performance levels required for the performance indicators and PA OVR’s
performance levels for the 2013 reporting period.

          Table 1.     Performance Levels for the Performance Indicators

       Performance   Performance Level Required of a General/Combined VR          PA OVR’s 2013
        Indicator                               Agency                            Performance Level
                     Number of employment outcomes equals or exceed previous             +11
           1.1       performance period                                                  Met
                                                 (9,939)
                     Percent with employment outcomes after services                   55.43%
           1.2
                                                  55.8%                              Did Not Meet
                     Percent of employment outcomes that were competitive              95.12%
           1.3       employment                                                          Met
                                                72.6%
                     Percent of participants with competitive employment               99.99%
           1.4       outcomes who had a significant disability                          Met
                                                62.4%
                     Ratio of average hourly VR wage to average State wage               0.519
           1.5
                                                  0.52                               Did Not Meet
                     Difference between percent self-supporting at closure and          55.35
           1.6       application                                                         Met
                                                  53.0
                     Ratio of minority service rate to nonminority service rate         0.823
           2.1
                                                  0.80                                   Met
Final Report
ED-OIG/A03P0002                                                                                    Page 6 of 35


                                           AUDIT RESULTS


We found that PA OVR had adequate internal controls to provide reasonable assurance that its
RSA-911 report data were complete. To ensure the data were complete, PA OVR analyzed its
2013 RSA-911 report for data quality using the edit check program, “RSA Errors,
Reasonableness Checks and Anomalies Program.” The program identifies possible problems
with the RSA-911 report data, including data omissions. In addition, PA OVR’s CWDS
database had controls that required that select data elements were entered into the case file
throughout the VR process. 6 For example, the required data elements for order of selection
included eligibility determination date, disability significance (priority), and whether the
VR counselor made an order of selection determination. PA OVR counselors could not move a
case forward to the next status if all of the required data elements were not completed. The
required data elements included elements that were reported on the RSA-911 report.

However, we found that PA OVR did not have adequate internal controls to provide reasonable
assurance that its RSA-911 report data were accurate and adequately supported. Specifically, we
found that (1) PA OVR did not have policies and procedures that required VR counselors or
supervisors to verify that the data entered in participants’ case files were correct and adequately
supported by documentation prior to closing the case file, and (2) PA OVR’s monitoring process
did not ensure that data entered into the CWDS database were correct and required
documentation was maintained in participant case files. In addition, PA OVR did not have
written policies and procedures for its RSA-911 reporting process.

As a result, we found that VR case service data maintained in PA OVR’s CWDS database and
reported on its 2013 RSA-911 report, including performance indicator data, were not correct and
adequately supported, and therefore were not reliable.

State VR agencies must have a system of internal control that provides reasonable assurance that
RSA-911 report data are accurate, complete, and supported so that RSA can rely on the data to
reflect the VR agency’s true performance when it calculates performance levels and determines
whether the agency is meeting standards. Reliable data are also important to ensure that RSA’s
annual reports submitted to the President and Congress, and made available to the public,
accurately report the VR agency’s performance. The extent that the State VR agency is meeting
performance standards could influence the amount of oversight and monitoring that RSA may
need to conduct at that agency.

We based our conclusions, in part, on the results of our review of a stratified statistical sample of
139 cases PA OVR closed during the 2013 reporting period. For each sampled case closure, we
determined whether select data elements were correctly reported and adequately supported


6
 Required data elements included personal information, medical information, order of selection, placement,
employment, and closure.
Final Report
ED-OIG/A03P0002                                                                                   Page 7 of 35

according to source documentation maintained in the participants’ case file and in PA OVR’s
CWDS database.

We provided a draft of this report to PA OVR for review and comment on November 24, 2015.
We received PA OVR’s comments on December 21, 2015. PA OVR did not explicitly agree or
disagree with Finding No. 1 regarding weaknesses in its internal controls over data quality for
the 2013 reported RSA-911 data or its recommendations, but stated it is designing corrective
actions intended to make improvements. PA OVR disagreed with Finding No. 2 regarding
unverifiable performance indicator data reported in its 2013 RSA-911 report and disagreed with
our recommendation.

We did not change our findings and recommendations based on PA OVR’s comments to the
draft audit report. In response to Finding No. 2 and its recommendation, PA OVR stated that
additional case closure documentation would be duplicative. We disagree. Although PA OVR’s
case closure process requires counselors to validate case closure data, counselors should
maintain documentation of that validation in the case file. We summarized PA OVR’s
comments on the draft audit report at the end of each finding and included the comments in their
entirety as Attachment 7 of this report.


Finding No. 1 – PA OVR Did Not Have Adequate Internal Controls to Assure the
                Accuracy of and Support Behind Its Case Service Report Data

PA OVR did not have adequate internal controls in place to provide reasonable assurance that
VR case service data reported on its 2013 RSA-911 report were accurate and adequately
supported. Specifically, PA OVR did not have policies and procedures that required
VR counselors or supervisors to verify that the data entered in participants’ case files were
correct and adequately supported by documentation prior to closing the case. This includes
verifying that required documents were properly completed and maintained in the participant’s
case file.

PA OVR’s controls to ensure staff completed, maintained, and recorded VR case file documents
and data included participant case reviews conducted by PA OVR staff. 7 However, we found
that PA OVR reported unverifiable and incorrect data on its 2013 RSA-911 report. PA OVR’s
controls were not adequate to ensure that staff members (1) properly completed and maintained
documents supporting VR case data in the participants’ case files and (2) detected and corrected
participant VR case data recorded in the CWDS database that were incorrect and inadequately
supported by source documentation (unverifiable) before reporting those data on the RSA-
911 report.

Also, as a control, both PA OVR and RSA analyzed PA OVR’s 2013 RSA-911 report data for
data quality using the edit check program “RSA Errors, Reasonableness Checks and Anomalies
Program.” The edit check program analyzes and identifies the RSA-911 report data for

7
 The participant case reviews are discussed in more detail in the section “PA OVR’s Quality Assurance Case
Review Process.”
Final Report
ED-OIG/A03P0002                                                                                       Page 8 of 35

“unreasonable” data, data errors, data anomalies, data duplications, and data omissions. 8
PA OVR also used an edit check program that mimicked the “RSA Errors, Reasonableness
Checks and Anomalies Program.” A batch file 9 that was built into the CWDS database
automatically ran the edit check program daily and produced a “RPT 105 - RSA-911 Error
Listing” report, which listed the possible errors and anomalies that the program detected. At the
end of each quarter and the end of the fiscal year, district offices were required to review the
report for their office and correct or verify the errors or anomalies identified. However, neither
edit check program could detect the unverifiable and incorrect data entries that we identified
because the programs could not inspect the actual source documentation that the RSA-911 report
was built on. 10

Based on our review of PA OVR’s VR process policies and procedures, staff interviews, and the
results of our review of a sample of participant case files, we concluded that the incorrect data
and the missing and incomplete source documentation for the data entries recorded in CWDS
and reported on PA OVR’s 2013 RSA-911 report were the result of (1) a lack of adequate
controls to ensure all required VR case documents were properly completed and maintained and
that VR case data agreed to source documents, (2) human error on the part of PA OVR staff, and
(3) PA OVR not using its quality assurance case review process to improve the procedures to
ensure that the VR case data were adequately supported by and agreed to source documents. We
discuss these issues in the sections “Completeness of Participant Case Files,” “Accuracy of
Participant Case File Data Entries,” and “PA OVR’s Quality Assurance Case Review Process” of
this finding.

We also found that PA OVR did not have written policies and procedures for its RSA-911
reporting process. According to the division chief for Systems and Evaluation, PA OVR used
the RSA-911 reporting instructions and guidance the RSA provided. However, RSA’s
instructions and guidance showed VR agencies only how to complete the RSA-911 report.
The Committee of Sponsoring Organizations of the Treadway Commission “Internal Control-
Integrated Framework” (COSO Report) provides a framework for organizations to design,
implement, and evaluate internal controls that will facilitate compliance with Federal laws,
regulations, and program compliance requirements. According to the COSO Report, a
component of internal control is control activities. One of the principles of control activities is
that an organization uses control activities through policies that establish what is expected and in
procedures that put policies into action. Also, documentation of policies and procedures
facilitates the training of new employees and ensures continuity of operations. PA OVR’s
program analyst who was responsible for reporting PA OVR’s RSA-911 report data retired in
July 2015. Not having written policies and procedures could result in PA OVR reporting
inaccurate or incomplete data on its RSA-911 report because the new program analyst may not
know how to properly process and report the data. 11
8
  Before submitting its final RSA-911 report, PA OVR resolved the data problems that the edit check program
identified.
9
  A batch file is a single text file that runs a set of system commands.
10
   We classified a data entry as unverifiable when required source documents were not present in the participant’s
case file or when source documents present in the case file did not include the information (that is, missing
signatures, dates, or other information) needed to verify the entry. We considered a data entry to be incorrect when
source document information did not agree to the data entry.
11
   PA OVR had not hired a new program analyst as of September 2015.
Final Report
ED-OIG/A03P0002                                                                                         Page 9 of 35

Recipients of Federal awards are required to maintain internal control over Federal programs that
provides reasonable assurance that the Federal awards are in compliance with laws, regulations,
and the provisions of contracts or grant agreements that could have a material effect on each of
its Federal programs (2 C.F.R § 200.303). 12 PA OVR must adequately document control
procedures. The lack of documented policies and procedures may result in inconsistencies,
processing, or procedural errors, and noncompliance with laws and regulations.

Completeness of Participant Case Files

We reviewed the participant case files for our stratified statistical sample of 139 participants who
were included on PA OVR’s 2013 RSA-911 report from the universe of 25,709 participants and
determined whether PA OVR properly completed and maintained the required case service
documents in the participants’ case files. The sample included 95 participants who exited the
VR process with an employment outcome and 44 participants who exited without an
employment outcome. We found that

     •   at least one required case service document was missing for 29 of the 139 sampled
         participants,
     •   at least one required case service document was incomplete for 15 of the 139 sampled
         participants,
     •   source documentation for required employment data elements was missing for 52 of
         the 95 participants who exited the program with an employment outcome, and
     •   PA OVR was not able to provide us with the hardcopy participant case file for one
         closure type 7 case.

PA OVR’s 2013 RSA-911 report data were extracted from information in its CWDS database.
PA OVR staff entered the participant VR case data into CWDS and placed source documents
that were the basis for the data into the participants’ hard copy case files. Federal regulations
and PA OVR policies and procedures require the following source documents to be maintained
in a participant’s case file: 13

     •   employment planning application,
     •   certificate of eligibility,
     •   order of selection form,
     •   IPE, and
     •   closure letter.

According to 34 C.F.R. § 76.731, a grantee should maintain records to show compliance with
program requirements. Additionally, Federal regulations and PA OVR policies and procedures
required PA OVR to maintain the following for participants who exited the VR program with an
employment outcome: source documentation showing the employment start date, hours worked

12
   Although this criterion was not in effect during our audit period, it represents a very important foundational
requirement for PA OVR going forward.
13
   Only those documents applicable to when a participant exited the VR program during the VR process were
required to be maintained in the participant’s file (see the diagram in Attachment 2).
Final Report
ED-OIG/A03P0002                                                                                      Page 10 of 35

in a week at case closure, weekly earnings at case closure, and documentation that the participant
maintained employment for 90 days.

PA OVR’s missing and incomplete VR case documents caused it to be in noncompliance with
Federal regulations and its own policies and procedures as discussed below. In addition, the
missing and incomplete documents resulted in unverifiable data being reported on PA OVR’s
2013 RSA-911 report, as discussed briefly in this finding and in detail in Finding No. 2 of this
report.

Required Case File Documents Were Missing

We estimate that 21 percent 14 of the case files for the 25,709 participants reported on PA OVR’s
2013 RSA-911 report were missing at least one required document. The documents required to
be maintained for a case file (employment planning application (including Certification of
Customer Rights and Responsibilities [Rights and Responsibilities] form), 15 certificate of
eligibility, order of selection, IPE, and closure letter) depended on when during the VR process
the participant exited the VR program. For each sampled participant file, we determined the
required forms based on closure type and verified the presence or absence of the required forms.
The most frequently missing document was the closure letter. 16

PA OVR is required to maintain a record of services for each participant and determine the type
of documentation that it will maintain for the services provided (34 C.F.R. § 361.47(a) and (b)).
The required documents are discussed in the following:

     1. Federal regulations 34 C.F.R. § 361.41(b) and 34 C.F.R. § 361.47(a)(6) and
     2. PA OVR policy, “Program Policies, Procedures and Guideline, No. 1010000400, OVR
        Case Filing System,” June 15, 2010.

In addition, 34 C.F.R. § 80.42 discusses the retention and access requirements for grantee
records, supporting documents, programmatic and statistical records, and other records required
to be maintained by program regulations or the grant agreement or are otherwise reasonably
considered pertinent to program regulations or the grant agreement.

Required Case File Documents Were Not Properly Completed

We estimate that 10 percent 17 of the case files for the 25,709 participants reported on PA OVR’s
2013 RSA-911 report had at least one required case service document that was not properly
completed. The employment planning application (including the Rights and Responsibilities
form), certificate of eligibility, and IPE were required to be signed and dated. For each sampled

14
   We are 95 percent confident the rate of case files missing at least one form is between 13 and 30 percent.
15
   PA OVR required participants to sign the Rights and Responsibilities form as a part of the application process.
The date signed was used as the application date when the application did not have a place for signature.
16
   Table 3 in Attachment 3 shows the number and percent of required documents that were missing for the
139 participants in our sample.
17
   We are 95 percent confident the rate of case files having at least one incomplete required document is between
5 and 18 percent.
Final Report
ED-OIG/A03P0002                                                                                   Page 11 of 35

participant file, we determined whether the participant and the VR counselor signed and dated
the required forms, as applicable. The employment planning application, including the Rights
and Responsibilities form, was the document most frequently missing the required signatures or
dates or both.

According to 34 C.F.R. § 361.41(b)(2)(i)(A), a participant is considered to have submitted an
application when either the participant or their representative has completed and signed an
agency application. The RSA Policy Directive RSA-PD-12-05, February 8, 2012, provides
guidance to designated State agencies for collecting data for the RSA-911 report. It states that a
participant is considered to have submitted an application when a participant has completed and
signed an agency application or has otherwise requested services, has provided information
necessary to initiate an assessment to determine eligibility and priority for services, and is
available to complete the assessment process.

“PA OVR Program Policies, Procedures and Guideline, No. 1010000400, OVR Case Filing
System,” June 15, 2010, states that the employment planning application (including the Rights
and Responsibilities form) and IPE require handwritten signatures. It further states that any form
with a handwritten signature must be stored in the participant case file.

The participant or their representative must sign an IPE, and a VR counselor employed by the
VR agency must approve and sign it (34 C.F.R. § 361.45(d)(3)(i) and (ii)). In addition,
RSA Policy Directive 12-05, February 8, 2012, requires VR agencies to report the participant’s
IPE date on the RSA-911 report. The instructions for the RSA-911 report state that if the date
the participant and counselor signed the IPE are different, the later date should be used.
Consequently, the IPE must be signed and dated.

Source Documentation for Required Employment Data Were Missing

For the 95 participants in our sample with an employment outcome, we determined whether the
data entries in CWDS for employment start date, weekly earnings amount, number of hours
worked, and the employment outcome were adequately supported. Although employment data
were entered in the case files, we could not verify whether the data were correct because
supporting documentation for the data was missing. For example, a case progress note 18 in the
participant’s file may have shown the participant’s employment start date; however, there was
nothing in the file to show that the participant was still employed after 90 days. The case files
for 52 of the 95 participants sampled were missing supporting documentation for at least one of
the employment data elements. Because employment data has a significant impact on the
performance indicator calculations, we discuss this issue in detail in Finding No. 2 of the report.




18
 VR counselors entered case progress notes on any developments in the progress of the participant’s case in the
CWDS database.
Final Report
ED-OIG/A03P0002                                                                                  Page 12 of 35

Accuracy of Participant Case File Data Entries

For the 139 participant case files, we also determined the reliability of selected data elements
maintained in CWDS and reported on its 2013 RSA-911 report. 19 We determined whether the
selected data elements were accurate and adequately supported according to source
documentation maintained in the case file. We identified data quality problems that included
(1) an incorrect race and ethnicity code, (2) unverifiable dates of birth and closure dates,
(3) unverifiable and incorrect Social Security numbers, and (4) incorrect and unverifiable
application and IPE dates. 20 A VR specialist stated that the incorrect data entries in the
CWDS database may have been the result of human error. For example, VR staff may have
input the date the employment planning application was sent to the participant instead of the date
the application was signed. VR staff could manually enter the application date, the IPE date,
eligibility determination date, employment start date, and closure date into the CWDS database.
VR counselors or supervisors were not required to verify the accuracy of the data elements
recorded in CWDS to source documents. Consequently, a VR participant’s case data recorded in
CWDS was not always the correct data according to source documents. We discuss the results
of this review in detail in Finding No. 2 of the report.

PA OVR’s Quality Assurance Case Review Process

PA OVR did not use its monitoring process to improve procedures and to provide reasonable
assurance that data were accurate and supporting documentation was maintained in participant
case files. Specifically, PA OVR’s case review policy “Performance Evaluations for Vocational
Rehabilitation Counselors and Supervisors,” October 1, 2012, did not require that PA OVR
officials use the results of the case reviews to determine whether PA OVR’s internal controls
over the VR process were working effectively or to evaluate whether revisions to its internal
controls or policies and procedures were necessary. In addition, the PA OVR quality assurance
process did not include steps to verify numerous data elements in the case file.

During our audit period, PA OVR had a quality assurance process to evaluate the program and to
determine whether case file documentation complied with Federal regulations and PA OVR
policy. The quality assurance process included the following three levels of review of
participant case files.

     •   Level One. VR supervisors reviewed participant case files for each counselor in their
         unit. They performed these reviews quarterly21 and included three open and two closed
         participant cases.
     •   Level Two. District administrators reviewed participant case files for each
         VR supervisor’s unit 22 in their district office. They performed these reviews quarterly
         and included three open and two closed participant cases.

19
   The data elements reviewed, according to the participant’s type of closure data element code, are shown in
Table 4 in Attachment 4 to the report.
20
   Table 6 in Attachment 6 shows the number of incorrect and unverifiable data entries for the 139 participants
according to the data element and closure type.
21
   The quarterly case reviews were performed during the calendar year quarters of March 31, June 30, September 30,
and December 31.
Final Report
ED-OIG/A03P0002                                                                                      Page 13 of 35

     •   Level Three. Rehabilitation specialists reviewed participant case files in each district
         office. They performed these reviews every spring and fall and included six open and
         four closed participant cases.

Cases selected for review were randomly generated by CWDS. All three levels of review were
scored and counted toward each employee’s performance review (job knowledge).
VR counselors and supervisors were provided the results of the level one and two reviews during
meetings with their supervisors or during employee performance reviews. Any issues noted
were corrected. Because the majority of the reviews occurred before the RSA-911 report
submission deadline (November 30), data issues found were corrected before submitting the data
to RSA.

District administrators were provided with the results of the level three reviews using report
cards. Each district office was given a report card for all 10 cases (6 open and 4 closed) that
were reviewed in that scoring cycle (fall or spring) and annually (fall and spring scores
combined). The district administrators also received the scoring sheets for each case reviewed so
that results could be used for performance ratings or to train the VR counselors and
VR supervisors. According to a VR specialist, district offices were required to conduct
mandatory training with their staff if the office received a review score below 70 percent in any
review area. For example, one district administrator conducted staff training for her office in
January 2015, after her office received a fiscal year 2014 level 3 review score below 70 percent
in the area of eligibility. VR supervisors in this office were also required to discuss the issues
with staff during their unit meetings. Central office officials plan to provide each district office
with more quantitative analysis and feedback once more longitudinal data is obtained to show
any cause and effect from PA OVR’s current case review process.

PA OVR staff used three different case review forms to perform case reviews during our audit
period. Before May 2013, the “Department of Labor and Industry, Office of Vocational
Rehabilitation Case Record Review Form” was used for both open and closed cases. Reviewers
were required to indicate either a “yes” or “no” answer for each question on the form in the areas
of eligibility, IPE, financial accountability, and case closure. In May 2013, a workgroup
developed two new OVR case review forms: one for open cases, and one for closed cases.
According to a rehabilitation specialist, the workgroup revised the forms to make the case
reviews a better learning tool. Staff used the new case review forms to determine whether
participants’ case files contained the appropriate documentation to show that the VR counselor
followed Federal regulations and PA OVR policies in the review areas. 23 The review of open
cases evaluated the quality of the case work in the following areas: eligibility and order of
selection determination, IPE, and financial accountability. The closed case review evaluated the
quality of the case work in the following areas: employment outcome, case closure, services
provided, financial accountability, appeal policies, and post-employment services.
According to 2 C.F.R. §200.303, PA OVR must establish and maintain effective internal control
over its vocational rehabilitation grant award that provides reasonable assurance that it is

22
   A supervisor’s unit consists of the VR staff that they supervise, generally five to eight VR counselors and one or
two clerks.
23
   The closed case review form determined whether the documentation was present, partially present, or not present.
Final Report
ED-OIG/A03P0002                                                                                Page 14 of 35

managing the award in compliance with Federal statutes, regulations, and the terms and
conditions of the award. These internal controls should be in compliance with guidance in
“Standards for Internal Control in the Federal Government” issued by the Comptroller General
of the United States or the COSO Report. According to the COSO Report, one of the five
components of internal control is monitoring. Monitoring is a process that assesses the quality of
internal control over time. One of the activities that serve to monitor the effectiveness of internal
control is conducting internal quality control reviews. Quality control reviews should provide
reasonable assurance with respect to the stated objectives of the review.

We found that PA OVR’s VR quality assurance case review process was not adequate to monitor
whether the data recorded in CWDS were supported by and agreed to source documents, and
were therefore reliable. Specifically, the case review forms that PA OVR used for the quality
assurance reviews did not include specific checks to verify that the following critical data
elements in source documents agreed with information recorded in the CWDS database:

     •   the race and ethnicity of the participant,
     •   the application date,
     •   the IPE date (when the IPE is signed on different dates by the participant and
         VR counselor, the later date should be used),
     •   the name of the employer of the participant, 24
     •   the start date of employment,24
     •   the hours worked in a week, and
     •   the weekly earnings at closure.

Additionally, for cases closed with an employment outcome, the case reviews did not determine
whether the source documents supported that the participant maintained 90 days of
employment. 25

Without a VR quality assurance case review form that includes steps to ensure the VR case file
data are accurate and verifiable, PA OVR management will be unable to use the results of the
case reviews to identify all areas that can be used for the quantitative analysis of PA OVR’s
VR case review process and all areas for potential staff training.

Further, without (1) policies and procedures to require staff to verify that the data entered in
participants’ case files are correct and adequately supported by documentation, and (2) an
assessment of PA OVR’s internal controls and policies and procedures over the VR process,
PA OVR may continue to report incorrect and unverifiable data, such as the data issues discussed
previously, on its RSA-911 report.

As a result of the lack of adequate internal controls over the VR program documentation
requirements for a participant’s record of service, PA OVR did not comply with Federal
24
   The employer name and employment start date were not required to be reported on the 2013 RSA-911 report. We
reviewed these data elements to determine whether the closure code was correct. Beginning with the 2014 RSA-911
report, the employment start date is a required data element.
25
   According to PA OVR’s current “Case Record Review Manual,” reviewers were required to determine only
whether the participant showed as being in employment status in CWDS for more than 90 days.
Final Report
ED-OIG/A03P0002                                                                       Page 15 of 35

regulations and PA OVR policies. Consequently PA OVR reported unverifiable case service
data on its 2013 RSA-911 report and cannot assure that it met its performance indicators. It is
important that PA OVR have internal controls that provide reasonable assurance that data are
accurate, complete, and supported since a number of data elements reported in the RSA-911
report are used by RSA to monitor States’ compliance with mandated timelines for delivering
VR services to participants.

Recommendations

We recommend that the Commissioner of RSA require PA OVR to––

1.1    Establish and implement policies and procedures to ensure that
       (1) all required VR case documents are completed and maintained in participants’ case
            files; and
       (2) all required VR case data recorded in the CWDS database agrees to source
            documentation.

1.2    Consider adding the specific checks listed above to PA OVR’s case review forms as part
       of the “scoring” to improve the quality assurance process and to better identify patterns
       and trends for quantitative analysis and areas for staff training.

1.3    Revise its quality assurance process to use the results of its case reviews to assess
       whether its internal controls are working effectively and determine whether it should
       revise its VR policies and procedures.

1.4    Develop written policies and procedures for its RSA-911 reporting process.


PA OVR Comments and OIG Response

PA OVR did not explicitly agree or disagree with this finding or its recommendations but
discussed corrective actions it plans to take regarding each recommendation.

PA OVR Comments
PA OVR stated that it agrees that data integrity and internal controls are an essential part of any
VR program and case management system. PA OVR further stated that it is committed to
reviewing areas identified in the draft report to determine how it can further ensure data integrity
and that it will continue to provide training to staff to ensure compliance with regulations and
reporting requirements. PA OVR explained that it plans to update CWDS so that staff can
upload documents directly into a participant’s case file and noted that it will automate the case
closure letter process and consider adding additional edit checks to further refine and determine
data anomalies. PA OVR will also enhance its case review process and review it annually to
determine whether any changes are needed to better adhere to regulations, data integrity, and
case file maintenance. Finally, PA OVR will develop instructions and procedures for the
completion of its RSA-911 report.
Final Report
ED-OIG/A03P0002                                                                                       Page 16 of 35

OIG Response
We commend PA OVR for designing corrective actions intended to improve its internal controls
and its ability to ensure compliance with Federal regulations and RSA-911 reporting
requirements. We did not change our finding and recommendations based on PA OVR’s
comments.


Finding No. 2 – Performance Indicator Data Reported on PA OVR’s Fiscal Year
                2013 Case Service Report Were Not Reliable

We found that VR performance indicator data maintained in PA OVR’s CWDS database and
reported on its 2013 RSA-911 report were not correct and adequately supported and therefore
were not reliable. In its 2013 RSA-911 report, PA OVR provided incorrect and unverifiable data
for data elements that RSA used to calculate PA OVR’s 2013 performance indicator results. We
determined that

     •   performance indicators 1.1 and 1.2 were calculated using an unverifiable count of
         participants cases closed with an employment outcome;
     •   performance indicators 1.3 through 1.6 were calculated using incorrect and unverifiable
         weekly earnings at closure amounts; and
     •   performance indicator 2.1 was calculated using an unverifiable closure count.

As a result, we have no assurance that the PA OVR performance indicator results that RSA
calculated for the 2013 reporting period are reliable. Federal regulations require that data
reported by a VR agency be valid, accurate, and in a consistent format (34 C.F.R. § 361.88(c)).
We estimate that almost half of the 25,709 reported participant case closures included on
PA OVR’s 2013 RSA-911 report included at least one incorrect or one unverifiable data entry as
follows: 26

     •   10 percent of reported closures included at least one incorrect data entry, 27
     •   25 percent of reported closures included at least one unverifiable data entry, 28 and
     •   13 percent of reported closures included at least one incorrect and one unverifiable data
         entry. 29

We reviewed selected data elements to

     •   verify a participant’s identification (Social Security number and date of birth);


26
   Table 5 in Attachment 5 shows the number of participant case files with incorrect and unverifiable data entries for
the 139 participants sampled according to their closure type.
27
   We are 95 percent confident that the percent of reported closures that included at least one incorrect data entry
ranges between 5 and 17 percent.
28
   We are 95 percent confident that the percent of reported closures that included at least one unverifiable data entry
ranges between 17 and 33 percent.
29
   We are 95 percent confident that the percent of reported closures that included at least one incorrect and one
unverifiable data entry ranges between 8 and 20 percent.
Final Report
ED-OIG/A03P0002                                                                                    Page 17 of 35

     •   determine whether reported data elements used in performance indicator calculations
         were correct and verifiable (race and ethnicity, weekly earnings at closure, hours worked
         in a week at closure, disability priority, and type of closure (codes 3 and 4); 30 and
     •   determine whether both reported and unreported 31 data elements were correct and
         verifiable (application date, eligibility determination date, IPE date, employer name,
         employment start date, type of closure (codes 1, 5, 6, and 7), and closure date).

Performance Indicators

In its 2013 RSA-911 report, PA OVR provided incorrect and unverifiable data entries for data
elements used in performance indicator calculations (see the sections “Participants Who Exited
the VR Process With Employment” and “Participants Who Exited the VR Process Without
Employment” below) and RSA used these data entries to calculate PA OVR’s 2013 performance
indicator results.

The calculations for performance indicators 1.1 and 1.2 rely primarily on an accurate count of
employment outcomes. To determine whether the calculations for these performance indicators
were reliable, we reviewed participants’ case files to determine whether we could verify that the
95 participants sampled who were reported as achieving an employment outcome maintained
employment for 90 days. According to 34 C.F.R. § 361.56(b), a participant’s case may be closed
as employed only if the participant has maintained employment for at least 90 days to ensure the
employment is stable. Based on the results of our sample, we estimate that 25 percent of the
9,950 participants reported as achieving an employment outcome on PA OVR’s 2013 RSA-911
report lacked documentation in their case files to support the outcome reported. 32 Therefore,
performance indicators 1.1 and 1.2 were calculated using data that we were unable to verify for
accuracy.

Performance indicators 1.3 through 1.6 rely substantially on the weekly earnings at closure data
element. To calculate these performance indicators, for each participant that exited the
VR process with an employment outcome, the weekly earnings at closure data element is divided
by the hours worked in a week at closure data element to obtain an hourly wage. To determine
whether the calculations for these performance indicators were reliable, for the 95 participants
sampled, we reviewed the participants’ case files to determine whether the weekly earnings at
closure amounts reported to RSA were correct and supported by source documents. Based on
the results of our sample, we estimate that 5 percent of the 9,950 participants included in each of
these performance indicator calculations had incorrect weekly earnings at closure amounts and
41 percent had unverifiable weekly earnings at closure amounts. 33 Based on the prevalence of

30
   We did not review the primary support at application, primary support at closure, and employment status at
closure data elements used in performance indicator calculations because we limited what we verified to the most
used and more critical data elements. Table 4 in Attachment 4 shows the data elements that we reviewed.
31
   These data elements were used in the verification of the performance indicator data.
32
   We are 95 percent confident that the percent of employment outcome cases reported on PA OVR’s 2013
RSA-911 report that lack supporting documentation needed to assess the accuracy of the employment outcome
ranges between 16 and 34 percent.
33
   We are 95 percent confident that the percent of employment outcome cases reported on PA OVR’s 2013
RSA-911 report with (1) an incorrect weekly earnings amount at closure ranges between 2 and 12 percent and
(2) an unverifiable weekly earnings amount at closure ranges between 31 and 51 percent.
Final Report
ED-OIG/A03P0002                                                                                     Page 18 of 35

incorrect and unverifiable weekly earnings at closure amounts, we conclude that performance
indicators 1.3 through 1.6 were calculated using incorrect or unverifiable data.

The calculation for performance indicator 2.1 relies on accurate closure counts and race and
ethnicity data element codes across all types of closures. To determine whether the data used for
the calculation for this performance indicator were reliable, we verified whether all
139 participants sampled (1) represented case closures during the reporting period and (2) had
correct race and ethnicity data element codes. Based on the results of our sample, we estimate
that 10 percent of the 25,709 case closure records used to calculate performance indicator 2.1
lacked documentation in their case files to support the employment outcome reported. 34
Therefore, because of the missing supporting documentation we could not gauge the extent of
errors for performance indicator 2.1.

Participants Who Exited the VR Process With Employment (Type of Closure Code 3)

Our review found unreliable (incorrect and unverifiable) data element entries for 70 (74 percent)
of the 95 participants reported as employed (see Table 5 in Attachment 5). This included
unreliable data elements for 51 participants (54 percent) that were used in one or more of
PA OVR’s performance indicator calculations for the 2013 reporting period. Specifically, we
found the following: 35

     •   The type of closure code was unverifiable for 24 participants because the file contained
         no documentation to verify that the participant was employed for 90 days.
     •   The amount of weekly earnings at closure was unverifiable for 39 participants and
         incorrect for 5 participants.
     •   The number of hours worked in a week at closure was unverifiable for 38 participants
         and incorrect for 4 participants.

Additional data quality problems for the data elements reviewed included (1) an incorrect
eligibility determination date, (2) unverifiable data entries for employer name and date of birth,
and (3) incorrect and unverifiable data entries for Social Security number, closure date,
application date, IPE date, and employment start date. Table 6 (see attached) shows the number
of incorrect and unverifiable data entries according to the data element and closure type.
Based on the results of our sample, we estimate that of the 9,950 participants with employment
reported on PA OVR’s 2013 RSA-911 report:

     •   11 percent of the closures included at least one incorrect data entry,
     •   37 percent of the closures included at least one unverifiable data entry,
     •   26 percent of the closures included at least one incorrect and one unverifiable data entry,
         and
     •   26 percent of the closures were fully supported by source documentation. 36
34
   We are 95 percent confident that the percent of cases reported on PA OVR’s 2012 RSA-911 report that were
unverifiable ranges between 5 and 16 percent.
35
   A participant could have more than one unreliable data entry.
36
   The estimates have a margin of error of at most plus or minus 10 percentage points at the 95 percent confidence
level.
Final Report
ED-OIG/A03P0002                                                                                      Page 19 of 35

Employment Data Elements Were Unverifiable

We estimate that 55 percent 37 of the 9,950 cases reported as closed with an employment outcome
on PA OVR’s 2013 RSA-911 report had at least one employment data element (employment
start date, weekly earnings amount, number of hours worked, and the employment outcome) that
was unverifiable because required supporting documentation was missing. 38 Supporting
documentation for the weekly earnings amount was the most frequently missing.

If a participant obtains or maintains employment as a result of the VR services provided, Federal
regulations require that employment data be maintained in the participant’s case file. To
successfully close and report a participant’s case as having exited the VR program with an
employment outcome, VR agencies must document (1) the participant’s employment start date
and (2) that the participant maintained employment for 90 days.

RSA requires VR agencies to provide the number of weekly hours worked and the weekly
earnings of the participant on the RSA-911 report (RSA Policy Directive 12-05,
February 8, 2012). RSA uses participants’ weekly hours worked and weekly earnings data to
calculate VR agencies’ compliance with performance indicators 1.3, 1.4, 1.5, and 1.6.

According to 34 C.F.R. § 361.84(b), the performance indicators require VR agencies to provide
information that will enable the Secretary to determine an agency’s compliance with the
VR program evaluation standards. Consequently, the number of weekly hours worked and
weekly earnings must be documented in the participant’s case file.

For participants who obtain employment, State VR agencies must maintain verification that the
participant is paid at or above the minimum wage and that the wage and level of benefits are not
less than that normally paid by the employer for the same or similar work performed by
participants who are not disabled (34 C.F.R. 361.47(a)(9)).

A condition for closing the case of a participant as employed is that the employment has been
maintained for a period of not less than 90 days (34 C.F.R. § 361.56(b)). In addition,
34 C.F.R. § 361.47(a)(15) requires State VR agencies to maintain documentation verifying that
the provisions of 34 C.F.R. § 361.56 have been met when the record of services for a participant
who has achieved an employment outcome is closed.

Consequently, it is RSA’s position that the requirements in 34 C.F.R. §§ 361.47 and 361.56
taken together require State VR agencies to maintain verifying documentation in the participant
case file related to the employment outcome including the employment start date, that the
participant maintained employment for 90 days, the hours worked, and the amount of earnings.
Although the requirements do not specify the type of verifying documentation that the agency
must maintain, the regulations show that VR agencies need to have some type of supporting
documentation for the employment data in the case file.

37
   We are 95 percent confident that for participants with an employment outcome the rate of case files missing
supporting documentation for an employment data entry is between 45 and 65 percent.
38
   The case files for 52 of the 95 participants sampled were missing supporting documentation for at least one of the
employment data elements.
Final Report
ED-OIG/A03P0002                                                                                 Page 20 of 35

Participants Who Exited the VR Process Without Employment (Type of Closure Codes 1,
4, 5, 6, and 7)

We found unreliable data element entries for 14 (32 percent) of the 44 participants reported to
have exited the VR process without employment (see Table 5 in Attachment 5). We found data
quality problems for the data element entries reviewed that included (1) unverifiable dates of
birth and (2) incorrect and unverifiable Social Security numbers, application dates and IPE dates.
Table 6 in Attachment 6 shows the number of incorrect and unverifiable data entries according to
the data element and closure type.

As described in Finding No. 1, PA OVR did not have adequate internal controls to ensure that

     •   records that were the basis for VR case service data, including performance indicator
         data, were properly completed and maintained participants’ case files; and
     •   all incorrect and unverifiable participant VR case file data entered into the CWDS
         database were detected and corrected before being reported on the RSA-911 report.

Also as explained in Finding No. 1, the edit check programs used by PA OVR and RSA would
not have detected the unverifiable and incorrect data entries that we identified that were the
result of missing and incomplete source documentation and misreported source data.

As a result of PA OVR reporting unreliable (unverifiable and incorrect) data on its 2013 RSA-
911 report, including performance indicator data, all of the performance indicators were
calculated using inaccurate or unsupported data elements, or both. Consequently, RSA may have
improperly determined PA OVR’s successful performance on the evaluation standards, and
PA OVR may have continued to participate in the VR program without entering into a required
program improvement plan.

In addition, unreliable data entries for the application date, eligibility determination date, and IPE
date hinder PA OVR management’s ability to monitor whether its staff timely serves participants
in compliance with Federal regulations. Those unreliable data entries also prevent RSA from
effectively monitoring PA OVR’s compliance with the required VR program timelines for
determining a participant’s eligibility for services and developing the participant’s IPE. RSA
conducts monitoring reviews of VR agencies on a 5-year cycle, and findings concerning meeting
the timelines for determining eligibility and developing the IPE are based on a review of the
RSA-911 report data for the 5 years prior to the fiscal year in which the monitoring review is
conducted. 39

Federal regulation 34 C.F.R. § 361.41(b) requires that a participant’s eligibility determination
must be made within 60 days of application. 40



39
   In November 2015 RSA officials reconfirmed that RSA temporarily suspended its monitoring activities because
of other work related to the Workforce Innovation and Opportunity Act of 2014.
40
   Exceptions are allowed if exceptional and unforeseen circumstances prevent this determination and the
VR agency and the participant agree to an extension.
Final Report
ED-OIG/A03P0002                                                                    Page 21 of 35

Federal regulations require the IPE to be developed in a timely manner, and the VR agency must
establish and implement standards for the prompt development of the IPE, including timelines
that take into consideration the needs of the individuals (34 C.F.R. § 361.45(a)(1) and (e)).
PA OVR policy requires the counselor to develop the IPE with the participant within 90 days
from the eligibility determination date.

Recommendation

We recommend that the Commissioner of RSA require PA OVR to—

2.1    Establish and implement controls to ensure that PA OVR staff obtain and maintain source
       documentation supporting participants employment information including the
       employment start date, weekly earnings at closure, hours worked in a week at closure,
       and that the participant maintained employment for 90 days.

PA OVR Comments and OIG Response

PA OVR disagreed with this finding and its recommendation.

PA OVR Comments
PA OVR stated that the audit report recommended creation of case progress notes to show that
the VR counselor validated case closure data. However, PA OVR believes that because its
current process already requires the VR counselor to validate the case closure data elements at
the time of closure, additional documentation of the data elements would be duplicative.

OIG Response
Although PA OVR’s current process requires the VR counselor to validate the closure data
elements before closing the case, which often entails the counselor having a conversation with
the participant, it does not require the counselor to obtain source documentation supporting this
validation. The audit report does not recommend the creation of case progress notes to document
the case closure data. As stated in footnote 18, PA OVR’s policy requires VR counselors to
enter case progress notes into the CWDS case file on any developments in the progress of the
participant’s case, which includes case closure data. As stated in the finding, RSA has taken the
position that the requirements in 34 C.F.R. §§ 361.47 and 361.56, taken together, require State
VR agencies to maintain verifying documentation in the participant case file related to the
participant’s employment outcome, including the employment start date, the weekly earnings at
closure, the hours worked in a week at closure, and that the participant maintained employment
for 90 days.

As stated in the finding, we estimate that 55 percent of the 9,950 cases reported as closed with
employment on PA OVR’s 2013 RSA-911 report had at least one employment data element that
was missing supporting documentation to validate the data reported. Based on this estimate and
the significant number of other unverifiable and incorrect data elements we found, PA OVR staff
should maintain documentation to support their data entry in the CWDS database so that the data
can be verified to help ensure the reliability of data that PA OVR reports annually to RSA. We
did not change our finding and recommendation.
Final Report
ED-OIG/A03P0002                                                                                  Page 22 of 35


                    OBJECTIVES, SCOPE, AND METHODOLOGY


The audit objectives were to determine whether PA OVR (1) had adequate internal controls to
provide reasonable assurance that reported RSA-911 report data were accurate and complete and
(2) reported RSA-911 report performance indicator data that were accurate, complete, and
adequately supported. Our audit covered PA OVR’s 2013 RSA-911 report. To achieve our
audit objectives we performed the following procedures.

     1. Reviewed relevant laws, regulations and guidance including the Rehabilitation Act of
        1973, Title I, Parts A and B, Sections 100-111; and Federal regulations at
        34 C.F.R. Part 361 and 34 C.F.R. Parts 76 and 80 to gain an understanding of the
        requirements that PA OVR was required to follow when administering the VR program.

     2. Reviewed RSA’s “VR Program FY [Fiscal Year] 2013 Monitoring and Technical
        Assistance Guide;” RSA’s “FY 2011 Monitoring Report on the PA OVR Program;”
        RSA’s “FY 2013 Pennsylvania Office of Vocational Rehabilitation Annual Review
        Report,” July 7, 2014; and the Commonwealth of Pennsylvania’s single audit reports for
        fiscal years 2012 and 2013. 41

     3. Interviewed PA OVR officials to gain an understanding of PA OVR’s CWDS database
        and its procedures for capturing, reviewing, verifying, and submitting the RSA-911 report
        data. We interviewed the Director of the Bureau of Central Operations, the division chief
        for Systems and Evaluation, one program analyst, two rehabilitation specialists, three
        district administrators, one assistant district administrator, five VR supervisors, one
        training supervisor, nine VR counselors, four clerk typists, and one employment
        facilitator.

     4. Reviewed PA OVR’s policies and procedures to gain an understanding of

                 a. the processes for preparing and submitting the RSA-911 report,
                 b. the processes for providing eligible people with services to help them obtain
                    or retain employment, and
                 c. the processes for ensuring VR case file documents were properly completed
                    and maintained in participants’ files and that the VR case data were properly
                    recorded in the CWDS database.

     5. We performed limited testing of the CWDS database controls including a review of the
        database security levels.



41
  The Commonwealth of Pennsylvania’s fiscal year 2012 and 2013 single audit reports are for the periods
July 1, 2011, through June 30, 2012, and July 1, 2012, through June 30, 2013, respectively.
Final Report
ED-OIG/A03P0002                                                                     Page 23 of 35

   6. We reviewed the case files for a statistical sample of 139 case closures reported on
      PA OVR’s 2013 RSA-911 report to determine whether PA OVR properly completed and
      maintained the required case service documents (employment planning application,
      certificate of eligibility, order of selection form, IPE, and closure letter) in the
      participants’ case files. See the diagram in Attachment 2 for the required documents
      according to the participants’ closure type. Also, for 95 of the 139 selected cases that
      were reported as closed with an employment outcome, we determined whether source
      documentation was maintained in the participant’s case file to show the employment start
      date, hours worked in a week at case closure, weekly earnings at case closure, and that
      the participant maintained employment for 90 days.

We used the COSO report and the Office of Management and Budget Circular A-133 as criteria
for evaluating PA OVR’s internal controls over its VR case management process and the
reporting process used to report its RSA-911 report data. We concluded that PA OVR did not
have adequate internal controls to provide reasonable assurance that the data reported on its
2013 RSA-911 report were correct and adequately supported (see Finding No. 1).

Sampling Methodology

We verified the completeness of the data reported in PA OVR’s final 2013 RSA-911 report that
was submitted on February 26, 2014. To verify the completeness of the data, we obtained a data
extract in December 2014 from PA OVR’s CWDS database for the universe of closed VR cases
for the 2013 reporting period and reconciled the universe to the universe of closed cases reported
in PA OVR’s final 2013 RSA-911 report submission.

We stratified the universe of 25,709 closed cases reported on PA OVR’S 2013 RSA-911 report
based on whether or not the case closure was designated as an employment outcome, as shown in
Table 2.

Table 2. Sampling Stratum by Closure Type

                                                                                            Sample
 Stratum                          Type of Closure                            Universe        Size
    1        Case Closed with an employment outcome (closure type 3)           9,950          95
    2        Cases closed without an employment outcome (all other            15,759          44
             closure types)
             Total                                                            25,709         139

We selected a stratified random sample of 139 cases for review: 95 cases from Stratum 1 and
44 cases from Stratum 2. We established the sample size so that the estimate prevalence of data
elements attributes would have at most a margin of error of plus or minus 10 percent at the
95 percent confidence level for estimates both at the employment outcome stratum and across the
universe of total case closures.

We calculated all estimates using sampling weights so that estimates reflect the intended
population. Some attributes that we tested were applicable only to cases closed with an
Final Report
ED-OIG/A03P0002                                                                          Page 24 of 35

employment outcome, and those estimates are projected to the employment outcome case
closures. Because we followed a probability procedures based on random selections, our sample
is only one of a large number of samples that we might have drawn. Because each sample could
have provided different estimates, we express our confidence in the precision of our particular
sample’s results as a 95 percent confidence interval or a margin of error. This is the interval that
would contain the actual population value for 95 percent of samples we could have drawn. All
percentage estimates from the audit have margins of error of plus or minus 10 percentage points
or less at the 95 percent confidence level.

The 139 participant case files included 42

       •    11 cases closed as a closure type 1,
       •    1 case closed as a closure type 2,
       •    95 cases closed as a closure type 3,
       •    21 cases closed as a closure type 4,
       •    1 case closed as a closure type 6, and
       •    10 cases closed as a closure type 7.

Data Reliability

We verified the reliability of select data reported on PA OVR’s 2013 RSA-911 report and
maintained in PA OVR’s CWDS database as follows.

       1.   For the 95 cases with an employment outcome, we determined whether the following
            data elements were correct and adequately supported according to source documents
            maintained in PA OVR’s CWDS database and participants’ hard copy case files:
            Social Security number, date of birth, race and ethnicity, application date, eligibility
            determination date, disability priority, IPE date, service provided, employer name,
            employment start date, weekly earnings at closure, hours worked in a week at closure
            type of closure, and closure date.

       2. For the 44 cases without an employment outcome, we determined whether the following
          data elements were correct and adequately supported according to source documents
          maintained in PA OVR’s CWDS database and participants’ hard copy case files:
          Social Security number, date of birth, race and ethnicity, application date, eligibility
          determination date, disability priority, IPE date, service provided, type of closure, and
          closure date. Not all of the data elements were applicable to all 44 cases. See Table 4
          in Attachment 4 for the data elements reviewed according to the participant’s type of
          closure code.

For the 139 cases sampled, we did not review the primary support at application, primary support
at closure, and employment status at closure data elements used in performance indicator
calculations because we limited what we verified to the most used and more critical data
elements.

42
     There were no closure type 5 cases in the universe of closed cases.
Final Report
ED-OIG/A03P0002                                                                     Page 25 of 35

We calculated PA OVR’s performance indicator scores using the data from PA OVR’s final
submission of the 2013 RSA-911 report and compared the performance indicator scores to the
performance indicator scores RSA calculated and published for PA OVR’s 2013 RSA-911
report.

We found that VR case service data maintained in PA OVR’s participant case files and reported
on its 2013 RSA-911 report, including performance indicator data, were not correct and
adequately supported and therefore were not reliable. Therefore, we were unable to determine
the reliability of the underlying data or the performance indicators used by RSA to assess the
VR agency’s performance against the required evaluation standards (see Findings No. 1 and 2).

We conducted site visits at PA OVR’s offices in Harrisburg, Pennsylvania, from
November 24, 2014, through November 25, 2014, and from June 29, 2015, through July 1, 2015.
We performed onsite visits to PA OVR’s Philadelphia district office in Philadelphia,
Pennsylvania, from December 16, 2014, through December 18, 2014, and from April 1, 2015,
through April 2, 2015; the Norristown district office in Norristown, Pennsylvania, from
April 28, 2015, through April 29, 2015, and on May 26, 2015; and to the Pittsburgh district
office in Pittsburgh, Pennsylvania, from June 15, 2015, through June 18, 2015. We conducted
additional audit work at our offices from November 2014 through September 2015. We held an
exit conference with PA OVR officials on September 23, 2015.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objectives.




                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

This report incorporates the comments that you provided in response to the draft audit report. If
you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following U.S. Department of
Education official, who will consider them before taking final Departmental action on this audit:
Final Report
ED-OIG/A03P0002                                                                      Page 26 of 35

                                          Janet LaBreck
                                         Commissioner
                             Rehabilitation Services Administration
                                U.S. Department of Education
                               550 12th Street, SW, Room 5086
                                   Washington, D.C. 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 calendar days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation and assistance extended by your staff during our audit. If you
have any questions or require additional information, you may contact me at (215) 656-6279 or
Teri L. Lewis, Assistant Regional Inspector General for Audit, at (215) 656-6276.


                                            Sincerely,

                                                /s/

                                             Bernard Tadley
                                             Regional Inspector General for Audit


Attachments
Final Report
ED-OIG/A03P0002                                                                  Page 27 of 35

                                                                                Attachment 1

                 Abbreviations, Acronyms, and Short Forms Used in This Report


2013 reporting period        October 1, 2012, through September 30, 2013

CWDS                         Commonwealth Workforce Development System

C.F.R.                       Code of Federal Regulations

COSO Report                  The Committee of Sponsoring Organizations of the Treadway
                             Commission’s, “Internal Control Integrated Framework”

IPE                          Individualized Plan for Employment

PA OVR                       Pennsylvania’s Department of Labor and Industry, Office of
                             Vocational Rehabilitation

Rights & Responsibilities    Certification of Customer Rights and Responsibilities Form
Form

RSA                          Rehabilitation Services Administration

RSA-911 report               Case Service Report

VR                           Vocational Rehabilitation
Final Report
ED-OIG/A03P0002                                                                                                                         Page 28 of 35

                                                                                                                                        Attachment 2

  Diagram:      The VR Process and its Related Participant Outcomes, Required Case File Documents, and Performance
                Indicators


        Step in VR Process                 Case Closure Type                     Required Documents in                             Performance
                                                                                       Case File                                Indicators Affected

        Application is completed           Exit VR as applicant (code 1)         Application and closure letter                          2.1




        Trial work experience              Exit VR during or after a trial       Application and closure letter                          2.1
                                           work experience (code 2)


                                           Exit VR from an order of
        Eligibility is determined          selection waiting list (code 6)
                                                                                  Application, certificate of eligibility,
        and participant is assigned                                                                                                      2.1
                                                                                  order of selection and closure letter
        to a disability priority           Exit VR without employment
        category                           after eligibility but before an IPE
                                           was signed (code 7)


                                            Exit VR without employment,            Application, certificate of eligibility,
        IPE is signed                       after a signed IPE, but before         order of selection, IPE and closure letter            2.1
                                            receiving services (code 5)


                                                                                   Application, certificate of eligibility,
        Services provided to participant     Exit VR without employment,                                                                1.2 and 2.1
                                                                                   order of selection, IPE and closure letter
                                             after receiving services (code 4)



                                                                                   Application, certificate of eligibility,              1.1, 1.2, 1.3,
        Participant is employed for 90       Exit VR as employed (code 3)          order of selection, IPE, closure letter, and          1.4, 1.5, 1.6,
        days and exits the VR program                                              related employment records                            and 2.1
Final Report
ED-OIG/A03P0002                                                                                   Page 29 of 35

                                                                                                  Attachment 3

Table 3. Missing Documentation for Our Sample of 139 Participants

                                                                        (C)                      (D)
                                                   (B)               Number of                Percent of
                     (A)
                                                Number of             Required                Required
                  Document
                                                 Required            Documents            Documents Missing
                                                Documents             Missing                  (C/B)* 43
                          44
 Application Package                               139                   13                      9%
 IPE                                               116                    6                      5%
 Closure letter                                      139                  15                       11%

* Rounded to the nearest percent.




43
   The percent missing for the IPE is based on the 116 participants in the sample that required an IPE to be
maintained in the file (95 closure type 3, and 21 closure type 4).
44
   The application package consists of the employment planning application and the Rights and Responsibilities
form.
    Final Report
    ED-OIG/A03P0002                                                                                                                                                  Page 30 of 35

                                                                                                                                                                     Attachment 4
          Table 4. Data Elements Reviewed According to the Participant’s Type of Closure Code

                           Social    Birth     Race      Application    Eligibility      Disability   IPE Date    Services    Employer      Employment    Weekly     Hours     Closure   Closure
  Closure Type            Security   Date      and         Date        Determination      Priority                Provided     Name          Start Date   Earnings   Worked     Type      Date
                          Number             Ethnicity                     Date                                                                              at        in a
                                                                                                                                                          Closure    Week at
                                                                                                                                                                     Closure
                                                                       N/A, Verified
1: Exited as an
                             X        X         X            X         Eligibility Not     N/A          N/A         N/A         N/A            N/A          N/A        N/A       X         X
applicant
                                                                        Determined
2: Exited during or
                                                                       N/A, Verified
after a trial work
                             X        X         X            X         Eligibility Not     N/A          N/A         N/A         N/A            N/A          N/A        N/A       X         X
experience/extended
                                                                        Determined
evaluation

3: Exited with an
                             X        X         X            X               X               X           X           X            X             X            X         X         X         X
employment outcome

                                                                                                                                 N/A
4: Exited without an
                                                                                                                               Verified
employment
                                                                                                                              Participant
outcome, after               X        X         X            X               X               X           X           X                         N/A          N/A        N/A       X         X
                                                                                                                               Did Not
receiving IPE
                                                                                                                                Obtain
services
                                                                                                                             Employment
                                                                                                        N/A
6: Exited from an
                                                                                                       Verified
Order of Selection           X        X         X            X               X               X                      N/A         N/A            N/A          N/A        N/A       X         X
                                                                                                      No Signed
Wait List
                                                                                                         IPE


7: Exited without an
                                                                                                        N/A
employment
                                                                                                       Verified
outcome, after               X        X         X            X               X               X                      N/A         N/A            N/A          N/A        N/A       X         X
                                                                                                      No Signed
eligibility, but before
                                                                                                         IPE
an IPE was signed
Final Report
ED-OIG/A03P0002                                                                          Page 31 of 35


                                                                                     Attachment 5

Table 5.    Summary of Incorrect and Unverifiable Data Entries for Participants Sampled
            According to the Closure Type

                                                                             Total
                                                                          Participant          Data
                              Included Both                  Included      Files That        Correctly
               Number of      Incorrect and    Included      Incorrect     Included          Reflected
               Participants    Unverifiable   Unverifiable     Data      Incorrect and      the Source
Closure Type    Sampled        Data Entries   Data Entries    Entries     Unverifiable      Documents
                                   (A)            (B)           (C)       Data Entries
                                                                           (A+B+C)
Employment
 Obtained          95              25             35            10            70               25
  (Type 3)
Employment
Not Obtained       44              2               8            4             14               30
 (All Other
   Types)

   Totals          139             27             43            14            84               55
Final Report
ED-OIG/A03P0002                                                                                                                                 Page 32 of 35

                                                                                                                                                              Attachment 6

Table 6.       Number of Incorrect and Unverifiable Data Entries According to the Data Element and Participant’s Closure Type

                             Closure    Closure      Closure      Closure     All        All          All          All       All       All        All          All          All       All
                             Type 3     Type 3       Type 3       Type 3     Other      Other        Other        Other     Other    Closure    Closure     Closure       Closure   Closure
                             Correct   Incorrect   Unverifiable   Totals    Closure    Closure      Closure      Closure   Closure    Types      Types       Types         Types     Types
                                                                             Types      Types        Types        Types     Types    Correct   Incorrect   Unverifiable     N/A     Totals
       Data Element                                                         Correct   Incorrect   Unverifiable     N/A     Totals
Social Security Number         91         2            2            95        40         2             2           0         44       131         4             4           0        139
Birth Date                     89         0            6            95        42         0             2           0         44       131         0             8           0        139
Race Ethnicity                 87         1            7            95        42         0             2           0         44       129         1             9           0        139
Application Date               71         16           8            95        34         3             7           0         44       105         19           15           0        139
Eligibility Date               94         1            0            95        32         0             0           12        44       126         1             0           12       139
Disability Priority            95         0            0            95        32         0             0           12        44       127         0             0           12       139
IPE Date                       75         12           8            95        15         2             4           23        44        90         14           12           23       139
Employer Name                  77         0            18           95        0          0             0           44        44        77         0            18           44       139
Employment Start Date          56         7            32           95        0          0             0           44        44        56         7            32           44       139
Weekly Earnings at Closure     51         5            39           95        0          0             0           44        44        51         5            39           44       139
Weekly Hours Worked at         53         4            38           95        0          0             0           44        44        53         4            38           44       139
Closure
Closure Date                   91         2            2            95        44         0             0           0         44       135         2            2            0        139
Closure Type                   71         0            24           95        44         0             0           0         44       115         0            24           0        139

            N/A – The data element was not applicable because it was not required for the participant’s closure type.
Final Report
ED-OIG/ A03 P0002                                                                                                        Page 33 of 35

                                                                                                                         Attachment 7




            ·~      pennsylvania
                    DEPAflTMENT Of LAIJOR & INDUSTRY 

                    OITICE Of VOCAllONAL RL HABJLITATION 



       12/2112015


       1.3cmard Tadlcy, Regional Inspector General for Audit
       United States Depa1tment of Education
       Office of Inspector General
       The Wanamaker Building
       I00 Penn Square East, Room 502
       Philadelphia, PA 19107
                                                                           Reference: Control Number ED-OIG/A03P0002

       IJear Mr. Tadley, 


       This correspondence serves as the Department of Labor & Industry, Office of Vocational Rehabilitation's 

       (OVR's) response to the U.S. Department of Education, Office of Inspector General's (OIG's) audit report 

       issued Novc::mber 24, 2015 and addresses the:: OIG's Findings and Recommendations. 


       Finding No. I- PA O VR Did Not Have Adegunte Internal Controls to Assure the Accuracy of and 

       Support Behind Its Case Sen •ice Report Data 


       OVR Response to Finding No. 1: 

       OVR agrees that data integrity and internal controls are an essential part of any vocational rehabilitation 

       program and case management system. OVR will continue to review policy and procedure and provide training 

       to staff to ensure compliance with federal regulations and reporting requirements. 


       Recommendation 1.1 - Establish and implement policies and procedures to ensure that 


      (1) All required VR case documents nrc completed and maintained in p:u·ti cipant~' case Iiles; a nd
      (2) All requi•·cd VR case da ta recorded in the CWDS database agrees to source documentation.

      Reco mmendation 1.1 Implementation: OVR's policy "OVR Case Filing System" govems the storage and
      maintenance of documents used in the vocationaJ rehabilitation program. Tlus document will be reviewed and
      updated, and staff will be u·ained on new elements as part of the WIOA implementation. During the update,
      OVR is committed to reviewing areas that have been identified within the OIG's report to dctcm1inc how OVR
      can further assure data integrity. As far as maintaining documentation in the casefile, OVR is working on an IT
      solution that would allow documents to be uploaded from a copier/scanner directly into the OVR case
      management system, U1e Commonwealth Workforce Development System (CWDS). Phase I ofthis solution
      will commence in early 2016 with phase 2 being completed by early 20 17. This will enable staff to upload
      documents more easily and will help to ensure that all appropriate documents arc stored for the life of the case.
               OVR's t1isslon: To assist Pennsylvanians with disabilities to secure and maintain employment and Independence. 

                             Department or Labor & Industry 1 Office of Vocational Rehabilitation I Executive Office 

                                                152 1 North Sixth Street 1 Harrisburg, PA 17102 

                                                       717.787.7312 1 Fax 717.772.1629 

                                                              www.dll state oil.US 

                             Auxiliary akfs and services are available upon request to Individuals wit/1 disabilities. 

                                                     Equal Opportunity Employer/Program 

Fi nal Report
ED-OIG/A03P0002                                                                                                           Page 34 of 35

                                                                                                                          Attachment 7


      Mr. Tadlcy 

      December 2 1, 2015 

      l'age 2 


     OVR is also committed to further ensuring the accuracy of participant case file data entries and will explore
     adding additional business rules and edit checks to further refine and determine data anomalies. In order to
     further instill the necessity of validating information that is entered and reponed through CWDS to RSA, OVR
     will provide staiT training on the importance of maintenance of records and data integrity on a yearly basis.

     'l11c OIG also reported thnt thc most common missing element in the cases reviewed wns the OVR Case C losure
     Letter. OVR wi ll address this by the end of20 16 by maki ng the Case C losure Leiter process automated so that
     when a case moves to a closure status, a clos ure leite r is automatically generated for printing and will also be
     emailed to the customer (when an email address is available), s tored in CWDS and uvailablc l(u· the customer to
     review if they log into their case through CWDS.

     The OlG reported that RSA may be unable to tell ifOVR was strictly ad hering to the 60- and 90-day time
      fi-amcs for Eligibility and Individual Plan for Employment (lPE) development due to a lack of consistency of
     application date entry. OVR recognizes that meeting these deadlines has been an issue. Accordingly. Central
     Office generates monthly reports that contain information regarding time in status, if 11 Time Extension is
     currently signed and when the case will reach the deadline. These reports arc provided to assist sta ff in
     managing their cascloads to meet the timeliness requirements. OVR has and continues to provide tmining to
     staffon service timeliness. Furthermore, this i~sue is addressed in OVR's Oack to Oasics program and wi ll
     continue to be reinfo rced . OVR is committed to increasing compliance wi th th is requi rement and will continue
     to provide repo rts and training to the field on an ongoing basis.

     Reco mmendation 1.2 - Cons ider addin g the specific checks listed a bove to PA OVR 's case r eview forms
     as part of the "scoring" to improve th e quali ty assurnnce proce~s a nd to b etter identify patterns and
     trends for quantitati ve nnalys is and areas foa· staff u ·ninin g.

     Recommendation 1.2 Implem entation: OVR acknowledges that the existing case review process docs not
     necessarily focus on data integrity. As a result, OVR will review this process, identify the key documen ts that
     sho uld be stored and add va lidations into the case review process to ensure tha t cascfilcs arc being properly
     ma intained. OVR will a lso update the case review process for the spring 2016 reviews to include a validation of
     signed documents in each revi ew~:d case (open and c losed) for the fo llowing clements: the dates of the
     a pplication signature, 0 0 11, IPE final sigmtture date and case closure data to ensure YO days of employment.

     Recommendation 1.3- Revise its quality assurnn cc process to usc the results of th e case r eviews to assess
     whethea· its intern ul controls are worl<ing effectively and determine wheth er it s hould r evise its VR
     policies nnd proccdua·es.

     Recommendation 1.3 Implementation: OVR is comrnilted to reviewing the OVR case review process on a
     yearly basis to determine if the case review, other policies/procedures or changes to CWOS are needed to
     better adhere to regulations, data integrity and cascfile mai ntenance. OVR will revise the current case rev iew
     process and require a yearly review following the completion of the annuallevcltlu·ce case reviews.


              OVR's Mission: To assist Pennsylvanians wi th disabilities to secure and maintain employment and Independence. 

                           Department or Labor & Industry I Office or Vocational Rehabilitation) Executive orr,ce 

                                             1521 North S•xth Street I Harrisburg, PA 17102 

                                                    717.787.7312 I Fax 717.772.1629 

                                                              ~I!..S~
                           Auxllt.Jry alris and services are ava/laO/e upon request to Individuals with dlsaOIIItles. 

                                                     Equal Opportunity Employer/Program 

Final Report
ED-01G/ A03P0002                                                                                                       Page 35 of 35


                                                                                                                       Attachment 7


      Mr. Tadlcy 

      December 21. 2015 

      Page 3 


      OVR case reviewers and designees from the field offices will discuss issues that have arisen during the review
      process and make recommendations to senior management for policy, procedure or CWDS changes.

      Recommendations 1.4- Develop written policies nnd procedures for its HSA-911 repnrting procc.~s.

      Recommcndntion 1.4 Implementation: OYR is committed to developing instructions nnd procedures
      associated with the completion of the RSI\ 9 11. Since the 911 will be changing for the upcoming year, OYR
      will focus on developing procedures for the new 911 requirements and will have a draft completed by July 2016
      with finalized instructions completed by December 31, 2016.

      Finding No.2- Performance Indicator Datu Reported on PA OYR's Fiscnl Yenr 2013 Case Service
      Report Were Not Reliable

     OVR Response Finding No.2
     OVR disagree.~ with this finding. The OIG's report indicated that OVR case closure data was not validated nnd
     recommends the creation of case progress notes evidencing validation of the dnta. However, the current
     process requires th e OVR counselor to validate the case closure information at the time of closure, confim1ing
     that the person has worked more than 90 days above minimum wage in an integrated setting as per our current
     procedure and as indicated in the !Jack to Uasics training series. Therefore, additional documentation oflhe case
     closure would be duplicative.

     Recommendation 2. 1 -Establish and implement controls to ensure th a t PA OVR staff obtain and
     maint:nin source documentation supporting participants' employmenl informnlion including rhe
     employment start date, weekly earnings at closure, hours worked in a week at closure, and that the
     participant maintained employment for 90 days.

     Reco mmendation 12. 1 Implementation: OVR is committed to providing further training on this process and
     will continue to reinforce this requirement through the case review process and Back to Basics training series.

     OVR would like to thnnk tJ1e Office of Inspector General for their professionalism during the review and looks
     forward to discussing HllY necessary cmTcctivc action, recommendations or suggestions.

     Thank you for your time.




             OVR's Mission: To assist Pennsylvanians with dlsllbllltles to secure and maintain employment and Independence. 

                          Department or Labor & Industry I ornce of Vocational Rehabilitation) Executive Office 

                                             1521 North Sixth Streett Harrisburg, PA 17 102 

                                                    717.787.73 12 1 FaK 717 .772.1629 

                                                           www.dh.state oa,us 

                          Auxlfliuy aids and services are available upon request to indiVIduals with d/sab/1/lles. 

                                                  Equal Opportunity Employer/Program