oversight

Rehabilitation Services Administration's Internal Controls Over Case Service Report Data Quality

Published by the Department of Education, Office of Inspector General on 2016-12-08.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                              UNITED STATES DEPARTMENT OF EDUCATION
                                              OFFICE OF INSPECTOR GENERAL

                                                                                                                   AUDIT SERVICES
                                                                                                             Philadelphia Audit Region



                                                        December 08, 2016
                                                                                                    Control Number
                                                                                                    ED-OIG/A03N0006
Sue Swenson
Acting Assistant Secretary
U.S. Department of Education
Office of Special Education and Rehabilitative Services
400 Maryland Ave., SW
Washington, D.C. 20202-7100


Dear Ms. Swenson:

This final audit report, “Rehabilitation Services Administration’s Internal Controls Over Case
Service Report Data Quality,” presents the results of our audit. The objective of our audit was to
determine whether the Rehabilitation Services Administration (RSA) had adequate internal
controls to provide reasonable assurance that “Case Service Report” (RSA-911 report) data
submitted by State vocational rehabilitation agencies were accurate and complete. Our audit
covered October 1, 2011, through September 30, 2012. This audit is a follow on to audits of
State vocational rehabilitation agency RSA-911 report data quality at three State agencies.




                                                      BACKGROUND


The U.S. Department of Education’s Office of Special Education and Rehabilitative Services
supports programs that serve millions of children, youth, and adults with disabilities. RSA, an
office within the Office of Special Education and Rehabilitative Services, oversees grant
programs that help people with physical or mental disabilities obtain employment and live more
independently by providing counseling, medical and psychological services, job training, and
other individualized services. RSA provides Vocational Rehabilitation Grants to States to assist
them in operating vocational rehabilitation (VR) programs. VR program grants are provided to
support a wide range of services designed to help people with disabilities prepare for and engage
in gainful employment consistent with their strengths, resources, priorities, concerns, abilities,
capabilities, interests, and informed choice. RSA has awarded about $3 billion to State VR
agencies in each of the past 4 fiscal years.

RSA’s State Monitoring and Program Improvement Division (SMPID) oversees State VR
agencies, which includes reviewing and approving State plans, developing program performance
measures, providing technical assistance, performing monitoring activities, and developing
program policy and guidance. The SMPID has 39 full-time employees and consists of four
functional units and State teams. The Data Collection and Analysis Unit collects and verifies
 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
Final Report
ED-OIG/A03N0006                                                                                       Page 2 of 31

RSA-911 report data, as discussed in further detail below. The VR Program Unit administers the
VR State grant program, maintains a program performance profile on each State VR agency,
develops program performance measures, and supports the State teams in their responsibilities
(see below). The Fiscal Unit supports monitoring reviews and oversees the resolution of audit
findings. The Technical Assistance Unit develops and maintains a database of technical
assistance resources and identifies resources that will assist grantees in their program
improvement efforts.

Each State is assigned a State team that monitors all of SMPID’s formula grantees that operate in
that State. Each State team is made up of one or more people from each of the functional units.
State teams review and approve State VR plans, perform program and fiscal monitoring, provide
technical assistance and policy guidance, and resolve audit findings.

Each State designates a State agency to administer the VR program. Some States have more
than one VR agency (an agency for those who are blind or visually impaired and a general
agency for all other people with disabilities). Other States use a combined agency that serves all
people with disabilities in the State.

People eligible for VR program services (referred to as participants in this report) are those who
have a physical or mental impairment that results in a substantial impediment to employment,
who can benefit from VR services for employment, and who require VR services. When State
agencies cannot serve all eligible participants with disabilities due to limited resources, they use
an order of selection. Under an order of selection, eligible participants are assigned to priority
categories based on the significance of their disability. Eligible participants with the most
significant disabilities are required to be selected first for the provision of VR services. 1 Eligible
participants who are not designated as the most significantly disabled are placed on a waiting list.

Section 106 of the Rehabilitation Act of 1973, as amended (Rehabilitation Act), requires RSA to
establish evaluation standards and performance indicators for the VR program that include
outcome and related measures of program performance. RSA established two evaluation
standards and seven performance indicators in June 2000 (34 Code of Federal Regulations
[C.F.R.] Parts 361.82 and 361.84). RSA also established minimum levels of performance for
each performance indicator (34 C.F.R. § 361.86), as shown in Table 1 in Attachment 2. RSA
uses data from the RSA-911 report to monitor the performance of State VR agencies, including
calculating their results on the performance indicators and determining whether they have met
the evaluation standards. RSA required agencies that failed to achieve successful performance
on the evaluation standards to develop a program improvement plan outlining specific actions
the agency would take to improve program performance.

State VR agencies use the current version of the RSA-911 report, implemented for fiscal year
(FY) 2014, to submit to RSA case data on all VR program participants whose case records were
closed in a given fiscal year. State VR agencies must submit the RSA-911 report by
November 30 (60 days after the end of the fiscal year). Agencies code cases in the RSA-911


1
  Under the new Workforce Innovation and Opportunity Act, VR agencies using an order of selection are permitted
to serve eligible participants regardless of the order of selection if these participants have an immediate need for
specific services or equipment to maintain employment.
Final Report
ED-OIG/A03N0006                                                                                        Page 3 of 31

report by type of closure to indicate when in the VR process a participant exited the program, as
shown below:

    •    exited as an applicant (code 1),
    •    exited during or after a trial work experience/extended evaluation 2 (code 2),
    •    exited from an order of selection waiting list (code 6),
    •    exited without an employment outcome after eligibility was determined but before an
         individualized plan for employment 3 was signed (code 7),
    •    exited without an employment outcome after an individualized plan for employment was
         signed but before receiving services (code 5),
    •    exited without an employment outcome after receiving services (code 4), and
    •    exited with an employment outcome 4 (code 3).

RSA analyzes the RSA-911 reports for accuracy and completeness of the required data fields
using two edit check programs: (1) the “RSA Errors, Reasonableness Checks, and Anomalies
Program” (RSA ERA program), and (2) a program written in the Statistical Package for the
Social Sciences, a statistical software program. The RSA ERA program identifies data errors,
data anomalies, unreasonable data, 5 data omissions, and data duplications and generates four
output logs: an error log, anomaly log, reasonableness log, and an omission/duplication log. RSA
also requests that State VR agencies (1) examine their data for reliability using the RSA ERA
program; (2) either make corrections to or prepare narrative justifications to verify that any data
issues identified in the error, anomaly, and reasonableness logs were correct; and (3) submit the
justifications with their RSA-911 report file. The statistical software program identifies missing
data, invalid data, unreasonable data, data anomalies, and data contradictions. 6 Many of the edit
checks included in the statistical software program are also included in the RSA ERA program.
Also, to ensure the accuracy of the count of participants included on the RSA-911 report, RSA
compares the participant counts reported on the RSA-911 report to another report the agency
submits.

Workforce Innovation and Opportunity Act
On July 22, 2014, the Workforce Innovation and Opportunity Act (WIOA) was signed into law
and amended the Rehabilitation Act. The WIOA added new RSA-911 reporting requirements.
The new requirements under section 101(a)(10) of the Rehabilitation Act, as amended by the
WIOA, include reporting data on students with disabilities who are receiving pre-employment



2
  Participants completed trial work experiences or extended evaluations to determine whether they could benefit
from VR services if existing evidence indicated that the participant was incapable of benefiting from the services.
Under the WIOA, the option to use extended evaluations was removed.
3
  The individualized plan for employment is a written plan outlining a participant’s vocational employment goal and
the services to be provided to assist the participant in reaching this goal.
4
  Employment outcome means obtaining or retaining full-time or part-time competitive employment. Changes to
the definition of employment outcome require that all employment outcomes achieved through the VR program be
in competitive integrated employment or supported employment (34 C.F.R. § 361.5(c)(15)).
5
  Reasonableness edit checks examine the characteristics of the reported data for results that are not within expected
values. Data error edit checks examine data elements for invalid entries and valid entries that are not permitted.
Data anomaly edit checks examine data elements for deviations from the normally expected values.
6
  Data contradiction edit checks examine two or more data elements for conflicting values.
Final Report
ED-OIG/A03N0006                                                                                       Page 4 of 31

transition services; 7 participants with open cases and the types of services they are receiving;
participants referred to the VR program by one-stop operators; 8 and participants referred to these
one-stop operators by VR agencies.

The WIOA also amended section 106 of the Rehabilitation Act by eliminating the current VR
evaluation standards and indicators and requiring that standards and indicators used to assess the
performance of the VR program be consistent with the common performance accountability
measures established under section 116 of the WIOA for six core programs. 9 Section 116(2)(A)
includes the following new primary indicators. 10

    •    Employment in Second Quarter After Exit—the percentage of program participants
         who are in unsubsidized employment during the second quarter after exit from the
         program.
    •    Employment in Fourth Quarter After Exit—the percentage of program participants
         who are in unsubsidized employment during the fourth quarter after exit from the
         program.
    •    Earnings—the median earnings of program participants who are in unsubsidized
         employment during the second quarter after exit from the program.
    •    Skills Gains in Education or Training Programs—the percentage of program
         participants who obtain a recognized postsecondary credential, or a secondary school
         diploma or its recognized equivalent, during participation in or within 1 year after exit
         from the program.
    •    Attainment of a Credential or Diploma—the percentage of program participants who,
         during a program year, are in an education or training program that leads to a recognized
         postsecondary credential or employment and who are achieving measurable skill gains
         toward such a credential or employment.
    •    Effectiveness in Serving Employers—indicators of effectiveness in serving employers
         to be developed by the Secretaries of the Departments of Education and Labor. 11

Because the new performance indicator requirements necessitate the reporting of data for both
current program participants as well as participants who have exited the program on a quarterly
basis, State VR agencies will be required to submit the RSA-911 report on a quarterly basis.


7
   Pre-employment transition services are VR services available to students and youth with disabilities, such as job
exploration counseling and workplace readiness training, in order to maximize their potential to transition from
school to postsecondary education and employment.
8
  One-stop operators are locations where participants can access all of the WIOA program services or receive
referrals for those services.
9
  The six WIOA designated core programs are administered by the Departments of Education and Labor. The
Department of Education programs are Vocational Rehabilitation and Adult Education and Family Literacy. The
Department of Labor programs are Title I Adult, Title I Dislocated Worker, Title I Youth, and Wagner-Peyser
Employment Services of the Workforce Investment Act of 1998.
10
   Unlike the prior performance indicators, the new performance indicators have no standard minimum levels of
performance. State agencies negotiate targets for each of the performance indicators.
11
   The agencies were required to develop indicators of effectiveness in serving employers by July 2016. According
to the chief of SMPID’s Data Collection and Analysis Unit, these measures were developed and were being sent to
the Office of Management and Budget’s Office of Information and Regulatory Affairs for approval in
November 2016.
Final Report
ED-OIG/A03N0006                                                                              Page 5 of 31

As a result of the changes to the data reporting requirements and the standards and indicators
brought about by the WIOA, RSA requested and received approval of a revised RSA-911 report
(RSA-911-16 report) from the Office of Management and Budget’s (OMB) Office of
Information and Regulatory Affairs. RSA issued a new “Reporting Manual for the Case Service
Report” in June 2016 (Case Service Manual). State VR agencies will begin to report data
required by the RSA-911-16 report in program year 2017 (July 1, 2017, through June 30, 2018)
on a quarterly basis. The first RSA-911-16 report data collection will cover the reporting period
from July 1, 2017, through September 30, 2017, and is due by November 15, 2017.

Related Reports
We conducted audits of State VR agency RSA-911 report data quality at three State agencies to
determine whether they (1) had adequate internal controls to provide reasonable assurance that
reported RSA-911 report data were accurate and complete and (2) reported RSA-911
performance indicator data that were accurate, complete, and adequately supported. We
reviewed RSA-911 data reported for the FY 2012 reporting period by Opportunities for Ohioans
with Disabilities (OOD) and for the FY 2013 reporting period by the California Health and
Human Services Agency, Department of Rehabilitation (CA DOR) and the Pennsylvania
Department of Labor and Industry, Office of Vocational Rehabilitation (PA OVR). 12

For purposes of this audit, we are describing data quality by using the terms defined in the
Government Accountability Office’s guidance, “Assessing the Reliability of Computer-
Processed Data,” July 2009. Data reliability refers to the accuracy and completeness of data,
given the uses for which they are intended. Accuracy refers to the extent that recorded data
reflect the actual underlying information. Completeness refers to the extent that relevant records
are present and the fields in each record are populated appropriately.




                                        AUDIT RESULTS


We found that RSA did not have adequate internal controls to provide reasonable assurance
that RSA-911 report data State VR agencies submitted were accurate and complete.
Specifically, we found that RSA’s monitoring procedures did not require program staff to
determine whether State VR agencies had established and implemented adequate internal
controls that provided reasonable assurance that their RSA-911 report data were accurate and
complete, nor did the procedures require program staff to perform any testing of the data
during monitoring visits. In addition, RSA did not require State VR agencies to certify that the
RSA-911 report data submitted were accurate and complete. Lastly, we found that although
RSA’s edit check programs provided some level of assurance regarding the completeness of
RSA-911 report data these agencies submitted, RSA had not properly documented its
procedures on the use of these programs.

12
 “Opportunities for Ohioans with Disabilities' Case Service Report Data Quality,” March 1, 2016,
ED-OIG/A03P0001; “California Department of Rehabilitation Case Service Report Data Quality,”
December 10, 2015, ED-OIG/A09O0008; and “Pennsylvania’s Department of Labor and Industry, Office of
Vocational Rehabilitation’s Case Service Report Data Quality,” March 2, 2016, ED-OIG/A03P0002.
Final Report
ED-OIG/A03N0006                                                                                       Page 6 of 31

We conducted a series of audits of State VR agencies (OOD, CA DOR, and PA OVR) and
found that the agencies’ lack of adequate internal controls over data quality contributed to their
reporting of unverifiable and incorrect data on RSA-911 reports. 13

Without a system of internal control that provides reasonable assurance that RSA-911 report data
are accurate and complete, RSA cannot rely on the data to reflect the agencies’ true performance
when it calculates performance levels and determines compliance with mandated timelines for
delivering VR services to participants. The extent to which an agency is meeting its performance
levels and timely serving VR participants should influence the amount of oversight and
monitoring that RSA may need to conduct at that agency. Reliable data are also important to
ensure that the annual reports RSA submits to Congress and the President and makes available to
the public accurately report the VR agency’s performance.

We provided a draft of this report to RSA for review and comment on August 30, 2016. We
received RSA’s comments on October 6, 2016. RSA did not explicitly agree with the finding or
its recommendations; however, RSA provided a corrective action plan (CAP) stating the
corrective actions that it has taken and that it is considering taking for each recommendation.

We did not change our finding and recommendations as a result of the comments provided by
RSA. We summarize the comments at the end of the finding and include the full text of RSA’s
response as Attachment 4 to this report. Because of the voluminous size of the exhibits RSA
included in its comments on the draft report, we have not included them in this report. Copies of
the exhibits are available on request.


FINDING – RSA Should Improve Its Internal Controls to Provide Reasonable
          Assurance That State Vocational Rehabilitation Agencies Submit
          Reliable RSA-911 Report Data

RSA did not have adequate internal controls to provide reasonable assurance that RSA-911
report data submitted by State VR agencies were accurate and complete. Specifically we
found that RSA’s monitoring procedures did not include determining whether State VR
agencies established and implemented data quality controls that ensure accurate and complete
RSA-911 report data are provided or include steps to determine whether the report data were
reliable. RSA also did not have State VR agencies certify to the accuracy and completeness of
RSA-911 report data that they submitted.

We found that RSA had some controls that helped ensure the completeness of State VR
agencies’ RSA-911 report data. As noted above, RSA analyzed each agency’s RSA-911 report
data for quality using the RSA ERA and Statistical Package for the Social Sciences edit check
programs. A management and program analyst in SMPID’s Data Collection and Analysis Unit
retrieved each agency’s RSA-911 report submission, ran the RSA ERA program against the
RSA-911 report file, and provided the error, anomaly, and reasonableness logs and the agency’s

13
  We classified a data entry as unverifiable when required source documents were not present in the participant’s
case file or when source documents present in the case file did not include the information (that is, missing
signatures, dates, or other information) needed to verify the entry. We considered a data entry to be incorrect when
source document information did not agree to the data entry.
Final Report
ED-OIG/A03N0006                                                                                             Page 7 of 31

justifications to the statistician in SMPID’s Data Collection and Analysis Unit. The statistician
determined whether the agency’s narrative justifications adequately addressed all of the issues
identified in their error, anomaly, and reasonableness logs. The statistician also ran the Statistical
Package for the Social Sciences program against the agency’s RSA-911 report submission.
When the RSA ERA and statistical software program outputs included a data quality issue that
the agency did not previously address or did not adequately address, the statistician contacted the
agency to resolve the deficiency. RSA required the agency to fix or explain any data quality
issues found and resubmit the RSA-911 report until RSA identified no unexplained data
issues. This process could require agencies to provide multiple submissions of their RSA-911
report. RSA’s management and program analyst also compared the number of participants,
according to their reported closure type, to the number of participants exiting the VR program,
according to their closure type, reported on the fourth quarter of the “Quarterly Cumulative
Caseload Report” 14 to verify whether agencies had any discrepancies. If an agency had two or
more discrepancies between the numbers of participants reported, by closure type, the
management and program analyst contacted the agency to resolve the issue. 15

We found that the RSA ERA and the Statistical Package for the Social Sciences programs are
adequate for what they are intended to do, such as identifying possible data errors and outliers,
and ensuring the completeness of the required data record fields. However, the programs are not
designed to review the source documentation on which State VR agencies build their RSA-911
reports. Unlike program monitoring reviews, the software programs cannot identify unverifiable
and incorrect data entries that are the result of missing source documentation and misreported
source data. We also noted that RSA’s procedures for processing State VR agencies’ RSA-911
report data submissions using these programs were not properly documented.

RSA Should Improve Its Monitoring Procedures to Ensure States Submit Accurate and
Complete RSA-911 Report Data

Past Monitoring Activities

RSA used a “Vocational Rehabilitation Program Federal Monitoring and Technical Assistance
Guide” (MTAG) instrument to monitor State VR agencies’ VR program performance and
compliance. The MTAGs that RSA used in FYs 2011 through 2014 did not include procedures
for determining whether State VR agencies established and implemented internal controls that
ensure the agencies submit accurate and complete RSA-911 report data. The MTAGs also did
not include procedures for reviewing the underlying support for data the agencies reported in
their RSA-911 reports.

The FY 2011 through 2014 MTAGs focused on transition services and employment outcomes
for youth with disabilities 16 and the fiscal integrity of the VR and supported employment 17

14
   The “Quarterly Cumulative Caseload Report” provides for the quarterly collection of information on the case status
of participants in the rehabilitation process at State VR agencies. Data reported includes persons applying for VR
services, determinations of eligibility made by State VR agencies, identification of persons with significant disabilities,
IPE development, service implementation, and program outcomes.
15
   RSA’s current procedures now require the management and program analyst to contact the State agency when the
agency has 10 or more discrepancies between the number of participants reported, by closure type.
16
   The FY 2014 MTAG also included a more in-depth review of improvements in performance resulting from
innovative practices.
Final Report
ED-OIG/A03N0006                                                                                      Page 8 of 31

programs. The FY 2011 MTAG also focused on the organizational structure of the State agency
and designated State unit, while the FY 2014 MTAG also focused on the performance of the VR
program, including improvements resulting from innovative practices. RSA has not updated its
MTAG or performed State VR agency program monitoring since FY 2014. The chief of
SMPID’s VR Program Unit stated that all monitoring activities were suspended in the beginning
of FY 2014, after just four monitoring visits were performed, because RSA needed staff to work
on developing WIOA regulations.

Our audits at three State VR agencies (OOD, PA OVR, and CA DOR) found that these
agencies did not have adequate internal controls to provide reasonable assurance that their
RSA-911 report data were accurate and adequately documented. 18 Specifically, we found that
the State VR agencies did not have (1) policies and procedures that required VR staff to verify
that the data entered into participants’ case files were correct and adequately supported by
documentation before closing the case files and (2) monitoring processes to ensure that data
entered into the agencies’ respective VR case management databases were correct and that
required documentation was maintained.

We found that as a result of the inadequate internal controls, the State VR agencies did not
submit accurate RSA-911 report data. For example, we found that OOD’s controls were not
adequate to ensure staff members properly completed and maintained required VR documents
in participants’ case files, which contributed to OOD reporting inaccurate and unverifiable data
on its RSA-911 report. We estimated that 44 percent 19 of the 21,554 reported participant case
closures included on OOD’s 2012 RSA-911 report included at least one incorrect or one
unverifiable data element entry. The data elements we reviewed, according to the participant’s
type of closure data element code, are shown in Table 2 in Attachment 3.

Our audits at OOD, PA OVR, and CA DOR also found that these agencies did not maintain
adequate source documentation in participants’ case files for employment data elements
(employment start date, 20 weekly earnings amount, number of hours worked, and the
employment outcome) reported on their RSA-911 reports. For example, we estimated that
55 percent 21 of the 9,950 cases PA OVR reported as closed with an employment outcome on its
2013 RSA-911 report had at least one employment data element that was unverifiable because
required supporting documentation was missing. For CA DOR’s 2013 RSA-911 report, we
estimated that 50 percent of participants’ employment start dates, 56 percent of the weekly

17
   The supported employment program provides grants for time-limited services leading to supported employment
for people with the most significant disabilities.
18
   We selected the three states based on the following factors: (1) the amount of VR program funding in FYs 2011
and 2012, (2) changes in funding amounts between FYs 2007 and 2011, (3) the number of State agencies involved in
the administration of the program, (4) performance indicator results in FYs 2007 and 2011, and (5) the changes in
performance indicator results between FYs 2007 and 2011. Because we did not use statistical sampling to select
these States, review results cannot be projected to other State VR agencies.
19
   We are 95 percent confident that for the participants reported, the rate of case files that included at least one
incorrect or unverifiable data element entry is between 34 and 54 percent.
20
   The employer start date was not required to be reported on the FYs 2012 and 2013 RSA-911 reports and was
not specifically used to determine performance. However, a reliable employment start date is necessary to
determine a participant’s employment outcome. Beginning with the 2014 RSA-911 report, the employment start
date was a required data element.
21
   We are 95 percent confident that for participants with an employment outcome, the rate of case files missing
supporting documentation for an employment data entry is between 45 and 65 percent.
Final Report
ED-OIG/A03N0006                                                                                     Page 9 of 31

earnings at closure, and 53 percent of the hours worked in a week at closure were
unverifiable. 22 RSA used these data to determine State VR agencies’ performance related to
employment and earnings.

Although the Federal regulations in effect during the scope of our audit required State VR
agencies to maintain documentation verifying participants’ employment outcomes
(34 C.F.R. §§ 361.47(a) and 361.56), the requirements did not specify the type of verifying
documentation that agencies must maintain. Each State could decide which type of verifying
documentation it would maintain. The absence of source documentation for these data
elements may be due, in part, to a lack of clear guidance regarding the necessary source
documentation needed to validate participants’ employment and earnings information. Further,
the lack of clear guidance may lead to inconsistencies in the type of employment
documentation State agencies maintain. Although the old performance indicators are no longer
in use, ensuring States have adequate employment outcome documentation is essential going
forward because the WIOA establishes three performance indicators that will be calculated
using VR program participant employment outcome data.

RSA added a statement in the Case Service Manual for the RSA-911-16 report clarifying that
Federal regulations require State VR agencies to maintain supporting documentation in
participants’ case files, particularly regarding eligibility determinations, development of the
individualized plan for employment, services provided, and case closure. RSA also included a
statement noting that the use of an electronic case management system does not remove the
requirement for the agency to maintain either hard copies or scanned copies of required
supporting documentation in the participant’s case file. Further, it states that RSA will
maintain a table that lists the RSA-911 supporting documentation requirements on its Web site
for State agencies to reference. As of November 2016, RSA was preparing the table for
management approval.

Planned Monitoring Activities

The chief of SMPID’s VR Program Unit stated that RSA is revising its monitoring guide, which
will be used in the fourth quarter of FY 2016. She explained that SMPID intends to conduct a
pilot of the revised monitoring guide following issuance of final regulations implementing
amendments to the Rehabilitation Act made by the WIOA. She stated that the pilot will be
designed to provide significant technical assistance to VR agencies and to gather information
that RSA will use to develop a more expansive monitoring guide for use in the monitoring cycle
beginning in FY 2017. The pilot will cover the following three areas of significant change to the
VR and supported employment programs:

        1 . the achievement of competitive integrated employment by people with disabilities in
            light of changes to the definitions of “competitive integrated employment” and
            “employment outcome” for purposes of the VR program;
        2 . the provision of preemployment transition services to students with disabilities (a new
            category of service under the VR program) and the use of the required 15 percent

22
  We are 95 percent confident that the estimates for cases with unverifiable employment start dates, weekly
earnings at closure, and hours worked in a week at closure are between 40 and 60 percent, 45 and 66 percent, and
43 and 64 percent, respectively.
Final Report
ED-OIG/A03N0006                                                                                    Page 10 of 31

            reservation of State allotments for this purpose; and
        3 . the provision of supported employment services, including extended services, and the
            use of the required 50 percent reservation of supported employment allotments to
            serve youth with disabilities.

Although the pilot will cover significant changes to the VR program, the acting chief of
SMPID’s Data Collection and Analysis Unit 23 stated that the monitoring guide will not include
procedures to review State VR agencies’ internal controls for ensuring the reliability of their
RSA-911 report data. Consequently, RSA will not determine whether State VR agencies have
internal controls to provide reasonable assurance that RSA-911 data the agencies report are
reliable.

The acting chief of SMPID’s Data Collection and Analysis Unit stated that RSA understands that
it is critical that the RSA-911 report data are accurate and complete. He stated that RSA did not
plan to include procedures for reviewing State agencies controls over RSA-911 report data
quality in the revised MTAG because not all agencies may have data quality issues. He further
stated that RSA was considering what the best approach would be for determining whether State
agencies have controls to provide reasonable assurance that RSA-911 report data the agencies
submit are reliable. He added that RSA will consider adding a step to the MTAG or using Data
Collection and Analysis Unit staff to perform a separate review of State agency controls over the
RSA-911 report data quality.

Sections 107(a)(1)(A) and (B) of the Rehabilitation Act state that the Commissioner of RSA is
required to conduct annual reviews and periodic onsite monitoring of programs authorized under
Title I of the Rehabilitation Act to determine whether each State VR agency is complying
substantially with the provisions of its State Plan under section 101 of the Rehabilitation Act and
with the performance indicators established under section 106. In addition, section 107(a)(3)(D)
of the Rehabilitation Act states that, in conducting monitoring, RSA must review individual case
files.

According to OMB Circular A-123, “Management’s Responsibility for Internal Control,”
(OMB Circular A-123), it is management’s responsibility to develop and maintain effective
internal controls and to design management structures that help ensure accountability for
results. OMB Circular A-123 also requires Federal agencies and managers to develop and
implement management controls that reasonably ensure that laws and regulations are followed.

The U.S. Department of Education’s Grant Bulletin, “Policy, Guidance, and Responsibilities for
Principal Office Monitoring Frameworks for Formula Grant Programs,” December 26, 2014,
requires each principal office that administers formula grant programs to monitor grantees’
success in meeting performance standards and compliance with the legal requirements of those
programs, and create a process to evaluate and improve the efficiency and effectiveness of its
monitoring procedures.

RSA should strengthen its VR program monitoring procedures to ensure that State VR agencies
have adequate internal controls to ensure the RSA-911 report data the agencies submit are

23
  This person was the acting chief of the Data Collection and Analysis Unit during part of our audit period. A new
chief of the unit was hired in July 2016.
Final Report
ED-OIG/A03N0006                                                                                Page 11 of 31

accurate and complete. Without an MTAG or other process that includes procedures to
determine whether agencies have a system of internal control that provides reasonable assurance
that RSA-911 report data are accurate and complete, RSA cannot rely on the data to reflect the
agencies’ true performance when it calculates performance levels and determines whether the
agencies are meeting mandated timelines for delivery of VR services to participants.

RSA-911 Report Data Certifications

RSA did not have State VR agencies certify to the accuracy and completeness of the RSA-911
report data submitted or certify that the agency had maintained documentation of the procedures
it performed to ensure that the data were accurate and complete. RSA’s former chief of
SMPID’s Data Collection and Analysis Unit stated that RSA did not require agencies to certify
the data because Federal regulations required that the data submitted by a State agency must be
valid, accurate, and in a consistent format (34 C.F.R. § 361.88(c)).

The need for annual data certifications is reinforced by the fact that RSA cannot conduct onsite
monitoring at all 80 State VR agencies annually. 24 RSA needs reliable data to conduct accurate
performance assessments and determine whether State VR agencies provide services to
participants in compliance with timelines that Federal regulations require. Management
certifications of data reliability, including full disclosure of any known issues with controls over
the data or the quality of the reported data, can help in this regard. OMB’s “Uniform
Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards,”
2 C.F.R. § 200.208, states that, unless prohibited by Federal statutes or regulations, each Federal
awarding agency or pass-through entity is authorized to require non-Federal entities to submit
certifications and representations annually, and that submission may be required more frequently
if a non-Federal entity fails to meet a requirement of a Federal award.

Adding a requirement for a management certification from an authorized State VR official as to
the accuracy and completeness of the submitted RSA-911 data would promote accountability
and further highlight the importance of the data at all levels. RSA’s acting chief of SMPID’s
Data Collection and Analysis Unit told us that RSA plans to require management certifications
of RSA-911 report data beginning with the FY 2016 data collection. Further, he stated that a
management certification will be required with each quarterly RSA-911-16 report submission
beginning in program year 2017.

We reviewed the Case Service Manual for the RSA-911-16 report, which includes the
management certification statement that will be required with each quarterly RSA-911-16
report submission. States will be required to certify that the report is true, complete, and
accurate and that the expenditures reported are for the purposes and objectives set forth in the
terms and conditions of the Federal award. Submitters must also certify that they are aware
that any false, fictitious, or fraudulent information, or the omission of any material fact, may
subject them to criminal, civil, or administrative penalties for fraud, false statements, false
claims, or otherwise. The RSA-911-16 report itself includes statements on State VR agency
internal controls and maintaining supporting documentation for the report data; however, the
management certification does not cover these areas. RSA should amend the data certification

24
  Based on RSA’s previous monitoring schedule, all 80 State VR agencies would have been reviewed over 6 years
(10 during FY 2011 and the remaining 70 over the next 5 years, resulting in 14 reviews annually).
Final Report
ED-OIG/A03N0006                                                                     Page 12 of 31

to require each State VR agency to include an assertion on its internal controls over RSA-911
data and an assertion about maintaining supporting documentation of the procedures performed
to ensure that the data were accurate and complete. These certifications could improve RSA’s
ability to monitor agencies’ controls over the RSA-911 report process.

The management certification could also be used to inform agencies that section 116(d)(5) of
WIOA requires RSA-911 report data to be valid and reliable, and if the data do not achieve the
accuracy standards, the State may be required to develop and implement corrective actions
(34 C.F.R. § 361.240). A management certification that includes all of these elements could
result in more reliable data by making State VR agency officials explicitly accountable for the
accuracy of the data or misreporting of the data and could also help ensure that State VR
agencies take steps to improve the processes used to validate the accuracy and completeness of
the data.

Review of Single Audit Reports

RSA had a process to review States’ single audit reports, required under 2 C.F.R. § 200.501, and
to use the results of these reviews to inform State monitoring activities. The June 2015
Compliance Supplement (2 C.F.R. Part 200, Appendix XI) requires external auditors to review
States’ VR program internal controls over performance reporting and perform compliance testing
of RSA-911 report data. Before 2015, the Compliance Supplement did not require external
auditors to perform these procedures. Suggested compliance audit procedures cover testing a
sample of the reports for accuracy and completeness and include (1) tracing reported data to
supporting records and (2) testing underlying data to verify that the data were accumulated and
summarized properly. Key data elements that should be reviewed include

   •   date of application,
   •   significance of disability,
   •   date of eligibility determination,
   •   date of individualized plan for employment,
   •   employment start date,
   •   employment status at closure,
   •   weekly earnings at closure,
   •   hours worked in a week at closure,
   •   type of case closure, and
   •   date of case closure.

RSA’s acting chief of SMPID’s Data Collection and Analysis Unit stated that RSA plans to
provide guidance, separate from the Compliance Supplement, for external auditors to refer to
regarding performing compliance testing of the RSA-911 report data. RSA could use single
audit report results to determine technical assistance needs of State VR agencies and areas of
concern, including internal control and data quality issues, for follow-up review activities.

The U.S. Department of Education’s Departmental Directive, “Guide for Managing State
Administered Programs,” December 18, 2014, requires principal offices to assess the quality of
performance under a State’s existing State plan by reviewing performance reports, monitoring
Final Report
ED-OIG/A03N0006                                                                            Page 13 of 31

results, audit findings (including those in single audits), and other relevant information available
to the Department.

In addition, the Government Accountability Office’s “Standards for Internal Control in the
Federal Government” states that reliable internal and external data sources should provide data
that are reasonably free from error and bias and faithfully represent what they purport to
represent; thus, RSA should evaluate both internal and external sources of data for reliability.

RSA Did Not Properly Document Its Procedures for Processing the RSA-911 Report

During our audit period, RSA did not have proper documentation of procedures for processing
the RSA-911 report data that State VR agencies submitted. Although the RSA ERA and the
Statistical Package for the Social Sciences edit check programs were working as intended, RSA
had not properly documented the procedures it used to run these programs. Specifically, we
found that the written procedures covering the process for running the RSA ERA program were
incomplete and that there were no written procedures for running the statistical software
program. We determined RSA did not have complete written procedures because of an
oversight on the part of RSA management. Policies and procedure should be properly
documented. The Government Accountability Office’s “Standards for Internal Control in the
Federal Government” require that internal control procedures be clearly documented in
management directives, policies, or operating manuals.

In December 2013, 25 we informed the former chief of SMPID’s Data Collection and Analysis
Unit that RSA needed to improve its written procedures for processing the RSA-911 report by
describing all of the steps required to run the RSA ERA program and writing procedures for
running the statistical software program to ensure these activities are completely, consistently,
and accurately performed. In response, in March 2014, the former chief provided us with
enhanced written procedures that captured all of the key steps for running the RSA ERA and
the Statistical Package for the Social Sciences programs.

The acting chief of SMPID’s Data Collection and Analysis Unit stated that RSA plans to revise
the RSA-911 report editing process. Beginning in FY 2017, RSA will no longer use the RSA
ERA program and has not determined whether it will continue to use the Statistical Package for
the Social Sciences program. He further explained that RSA will maintain a comprehensive edit
check table on its Web site that will detail, by data element, the edit checks required to ensure the
integrity of data submissions. Each data submission will be analyzed to determine whether the
data is consistent with the edit checks. Data submissions that fail to pass the edit checks will be
returned to the VR agency for correction and resubmission.

RSA should ensure that the new programs and procedures, including any developed as a result of
implementation of the WIOA, are documented and that staff are aware of and trained on any new
procedures for processing RSA-911 reports. Incomplete and undocumented written procedures
reduce the assurance that staff correctly and consistently perform processes and activities,
increase the risk of disruption and errors when staff absences and changes occur, and could affect
the reliability of data in RSA-911 reports.

25
 Our audit at RSA was put on hold until we completed our audits of OOD, PA OVR, and CA DOR. The last report
was issued on March 2, 2016.
Final Report
ED-OIG/A03N0006                                                                       Page 14 of 31

Recommendations

We recommend that the Acting Assistant Secretary for the Office of Special Education and
Rehabilitative Services require the Commissioner of RSA to—

1.1    Develop procedures for periodically reviewing the processes States VR agencies use to
       capture, review, verify, and submit their RSA-911 report data, including a review of, on
       at least a sample basis, the supporting documentation for the data.

1.2    Provide guidance to State VR agencies on the importance of developing and
       implementing adequate internal controls over VR data to ensure that the data State VR
       agencies report are accurate and complete.

1.3    Provide guidance, training, and technical assistance to State VR agencies on the type of
       source documentation needed to validate employment and earnings data.

1.4    Require an authorized State VR official to certify that (1) the State VR agency has
       internal controls over RSA-911 report data that have been tested and are working as
       intended, or that the State VR agency has fully disclosed any data limitations and
       corrective actions; (2) the required RSA-911 report data submitted are accurate and
       complete; (3) documentation of the procedures performed to ensure the required RSA-
       911 report data submitted are accurate and complete has been maintained; and (4) they
       understand submitting inaccurate or incomplete data may result in having to develop
       and implement corrective actions, or other consequences as determined by the
       Department.

1.5    Review the results of States’ single audit reports for findings related to the RSA-911
       report to determine technical assistance needs of State agencies and areas of concern
       and perform follow-up review activities as necessary.

1.6    Provide guidance to external auditors regarding performing compliance testing of RSA-
       911 report data.

1.7    Ensure that any updates to its process for performing edit checks of State VR agencies’
       RSA-911 report data are clearly and adequately documented and disseminated to staff,
       and that staff are sufficiently trained on the new procedures.

RSA Comments and OIG Response

RSA did not explicitly agree or disagree with the finding and its recommendations; however, it
provided a CAP that discussed corrective actions that RSA has taken and is considering taking
regarding each recommendation.

RSA Comments
In response to the finding, RSA stated that it will ensure that any internal control procedures it
develops will be consistent with Federal statutory and regulatory requirements. RSA further
stated that it intends to include a monitoring component consistent with the requirements of
section 107 of the Rehabilitation Act and the U.S. Department of Education’s policy on
Final Report
ED-OIG/A03N0006                                                                      Page 15 of 31

administering formula grants. It also stated that monitoring will include a sample review of
supporting documentation, as needed, consistent with section 107(a)(3) of the Rehabilitation Act,
to ensure the validity of each State VR agency’s internal control process.

In response to Recommendation 1.1, RSA stated that it has (1) reviewed past monitoring guides
that it used when staff conducted case service record reviews; (2) spoken with Department of
Labor, Employment and Training Administration staff about their internal controls, quality
assurance, and data validation methods; and (3) included regulatory citations and requirements
for submission of accurate data in the RSA-911-16 Case Record Service Manual. RSA is
considering developing and implementing internal procedures for reviewing State VR agency
RSA-911 report internal control procedures to ensure they result in the submission of accurate
and reliable data. It is also considering providing technical assistance and training to State VR
agencies in developing and implementing RSA-911 internal controls and quality assurance
procedures, that could include developing a sample case record review instrument.

In response to Recommendation 1.2, RSA stated that it has included language in the
RSA-911-16 Case Record Service Manual informing State VR agencies that they must develop
and implement policies and procedures to ensure accurate data collection. It has also begun
developing a comprehensive list of the types of supporting documentation that may be used to
verify core RSA-911 data elements, and it has begun developing more comprehensive edit
checks to improve the accuracy of data submitted by State VR agencies. One example of a
corrective action RSA is considering is providing technical assistance to States regarding the
importance of implementing and adhering to policies and procedures that will promote accurate
and reliable data collection and submission.

In response to Recommendation 1.3, RSA stated that, in coordination with the Department of
Labor, Employment and Training Administration and the Department of Education’s Office of
Career, Technical, and Adult Education, it is developing joint guidance for States on what
documentation is required to validate employment and earnings data. RSA is considering
providing joint technical assistance on this guidance with these offices once it has been issued.

In response to Recommendation 1.4, RSA stated that it has already included language in the
RSA-911-16 report to require States to certify that the report being submitted is true, complete,
and accurate, and the expenditures reported are for the purposes and objectives set forth in the
terms and conditions of the Federal award, beginning with data submitted in program year 2017.
RSA also explained that the certification statement is adapted directly from OMB’s “Uniform
Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards,” at
2 C.F.R. § 200.415. RSA is considering requiring State VR agencies to use this language to
certify their data, via email, prior to the acceptance of FYs 2016 and 2017 RSA-911 data
collections.

In response to Recommendation 1.5, RSA stated that it responds to State single audit findings
related to the RSA-911 report by providing technical assistance to the grantee and requiring that
grantees submit the correct data to RSA. RSA is considering requiring grantees to determine the
root cause of inaccurate RSA-911 reporting and to review and revise their internal control
processes, as necessary, to ensure they correct any systemic issues they identify.
Final Report
ED-OIG/A03N0006                                                                      Page 16 of 31

In response to Recommendation 1.6, RSA stated that it has (1) performed an RSA-911 report
compliance testing needs assessment and (2) included the core RSA-911 data elements for
compliance testing in the Compliance Supplement. One example of a corrective action RSA is
considering is developing a fact sheet or quick reference guide in calendar year 2017 to instruct
external auditors on compliance testing of the RSA-911 data.

In response to Recommendation 1.7, RSA stated that it has documented the current ERA
program edit check system, trained Data Unit staff on running the edit check program, and stored
all documentation on a server that is accessible to all Data Unit staff. RSA is considering
developing documentation for the edit check program being developed for the new RSA-911
data collection to be implemented in July 2017. It is also considering training staff as backups in
running and reviewing the outputs of the edit check program and developing and implementing
internal control processes for ensuring consistency and reliability among staff using the edit
check program.

OIG Response
The corrective actions that RSA stated it has already performed and those it is considering
performing for Recommendations 1.1 through 1.3 and 1.5 through 1.7 will address our
recommendations if RSA completes all of the actions stated. However, the corrective actions
that RSA stated it has taken for Recommendation 1.4 do not adequately address the
recommendation. We acknowledge in the report that the Case Service Manual for the RSA-911-
16 report includes a management certification statement that will be required with each quarterly
RSA-911-16 report submission. We also understand that the management certification statement
that RSA plans to use essentially mirrors the language from OMB’s “Uniform Administrative
Requirements, Cost Principles, and Audit Requirements for Federal Awards,” at
2 C.F.R. § 200.415. However, as stated in the report, we believe that including additional
assertions in this management certification statement about the documentation and testing of
internal control procedures over RSA-911 report data reliability, as well as the disclosure of any
data limitations and corrective actions, could help ensure that State VR agencies take steps to
improve the processes used to validate the accuracy and completeness of their data.

We commend RSA for designing corrective actions intended to ensure that State VR agencies
submit accurate and complete RSA-911 report data. We did not make any changes to the finding
or the related recommendations as a result of RSA’s comments.




                 OBJECTIVES, SCOPE, AND METHODOLOGY


The audit objective was to determine whether RSA had adequate internal controls to provide
reasonable assurance that RSA-911 report data submitted by State VR agencies were accurate
and complete. Our audit covered October 1, 2011, through September 30, 2012. To achieve our
audit objectives, we performed the following procedures:

1. Reviewed relevant laws, regulations, proposed regulations, and guidance, including the
   Rehabilitation Act of 1973, Title I, Parts A and B, sections 100-111; Federal regulations at
Final Report
ED-OIG/A03N0006                                                                                 Page 17 of 31

     2 C.F.R. Part 200, 34 C.F.R. Part 361 and 34 C.F.R. Parts 76 and 80; the Workforce
     Innovation and Opportunity Act, including the proposed rules to implement the Act; 26 the
     U.S. Department of Education’s Grant Bulletin, “Policy, Guidance, and Responsibilities for
     Principal Office Monitoring Frameworks for Formula Grant Programs,” December 26, 2014;
     and the U.S. Department of Education’s Departmental Directive, “Guide for Managing State
     Administered Programs,” December 18, 2014, to gain an understanding of the requirements
     for administering the VR program.

2. Reviewed RSA’s FYs 2011 through 2014 “Vocational Rehabilitation Program Federal
   Monitoring and Technical Assistance Guide.”

3. Reviewed ED OIG audits of State VR agency RSA-911 report data quality at three State
   agencies: (1) Opportunities for Ohioans with Disabilities' Case Service Report Data Quality,”
   March 1, 2016, ED-OIG/A03P0001; (2) “California Department of Rehabilitation Case
   Service Report Data Quality,” December 10, 2015, ED-OIG/A09O0008; and
   (3) “Pennsylvania’s Department of Labor and Industry, Office of Vocational Rehabilitation’s
   Case Service Report Data Quality,” March 2, 2016, ED-OIG/A03P0002.

4. Interviewed RSA officials to gain an understanding of RSA’s procedures for obtaining,
   reviewing, and verifying RSA-911 report data. We interviewed the chief of SMPID’s VR
   Program Unit, the acting chief of SMPID’s Data Collection and Analysis Unit, the former
   chief of SMPID’s Data Collection and Analysis Unit, a management and program analyst, a
   statistician, an information technology management specialist, and a program specialist.

5. Reviewed RSA’s written procedures for processing the RSA-911 report and calculating State
   VR agencies’ performance indicator scores.

6. Reviewed the RSA ERA and Statistical Package for the Social Sciences edit check programs
   to determine whether the programs were performing the edit checks as intended and the types
   of data quality issues that would be identified. We also reviewed the edit logs for OOD’s
   2012 reporting period RSA-911 report submissions (the first State agency we reviewed) to
   determine whether the edit check programs were actually performing the edit checks as
   intended.

7. Reviewed OMB’s “2 C.F.R. Part 200, Appendix XI, Compliance Supplement” to determine
   the compliance requirements relevant to an audit of the VR program, as well as the audit
   objectives and suggested audit procedures for determining compliance with the requirements.

8. Reviewed the (1) RSA Policy Directive RSA-PD-12-05, February 8, 2012, Approved Use of
   the Case Service Report (RSA-911) through November 30,2014,” and the related “Reporting
   Manual for the Case Service Report (RSA-911),” (2) RSA Policy Directive RSA-PD-14-01,
   October 25, 2013, “Revision of PD-13-05 Vocational Rehabilitation Program Case Service
   Report (RSA-911) Data Elements” and the related “Reporting Manual for the Case Service

26
  We reviewed the following April 16, 2015, Notices of Proposed Rulemaking: (1) the Workforce Innovation and
Opportunity Act; Joint Rule for Unified and Combined State Plans, Performance Accountability, and the One-Stop
System Joint Provisions, and (2) the State Vocational Rehabilitation Services Program; State Supported
Employment Services Program; Limitations on Use of Subminimum Wage.
Final Report
ED-OIG/A03N0006                                                                       Page 18 of 31

   Report (RSA-911),” and the “Reporting Manual for the Case Service Report (RSA-911),”
   June 2016.

We used the Government Accountability Office’s “Standards for Internal Control in the Federal
Government and OMB Circular A-123, “Management’s Responsibility for Internal Control,” for
evaluating RSA’s internal controls over RSA-911 report data submitted by State VR agencies.
We concluded that RSA did not have adequate internal controls to provide reasonable assurance
that RSA-911 data reported by State VR agencies were accurate and complete
(see Finding No. 1).

Sampling

We did not perform any sampling in this audit. However, we considered the results of the
sampling performed in the ED OIG audits of State VR agency RSA-911 report data quality at
three State agencies in our assessment of RSA’s internal controls over RSA-911 report data
quality: (1) Opportunities for Ohioans with Disabilities' Case Service Report Data Quality,”
March 1, 2016, ED-OIG/A03P0001; (2) “California Department of Rehabilitation Case Service
Report Data Quality,” December 10, 2015, ED-OIG/A09O0008; and (3) “Pennsylvania’s
Department of Labor and Industry, Office of Vocational Rehabilitation’s Case Service Report
Data Quality,” March 2, 2016, ED-OIG/A03P0002.

Data Reliability

Our use of computer-processed data for the audit was limited to data included in the ED-OIG
audits of State VR agency RSA-911 report data quality at the three State agencies discussed
above: OOD, CA DOR, and PA OVR. For these audits, we determined the reliability of select
data reported on (1) OOD’s FY 2012 RSA-911 report and (2) PA OVR and CA DOR’s FY 2013
RSA-911 reports. We used the results of these audits in our assessment of RSA’s internal
controls over RSA-911 report data quality. Therefore, we believe that the data used from the
results of these audits are sufficiently reliable for purposes of this report. We did not assess the
reliability of computer-processed data for this audit.

We conducted initial site work at RSA’s offices in Washington, D.C., in August 2013. We
placed the audit on hold until we completed our related audits at the three State VR agencies
discussed above. We conducted additional audit work at our offices from February 2016 through
August 2016. We held an exit conference with RSA officials on June 10, 2016.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objectives.
Final Report
ED-OIG/A03N0006                                                                      Page 19 of 31



                            ADMINISTRATIVE MATTERS


Corrective actions proposed (resolution phase) and implemented (closure phase) by your offices
will be monitored and tracked through the Department’s Audit Accountability and Resolution
Tracking System. The Department’s policy requires that you develop a final CAP for our review
in the automated system within 30 calendar days of the issuance of this report. The CAP should
set forth the specific action items, and targeted completion dates necessary to implement final
corrective actions on the findings and recommendations contained in this final audit report. An
electronic copy of this report has been provided to your Audit Liaison Officers.

In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector
General is required to report to Congress twice a year on the audits that remain unresolved after
6 months from the date of issuance.

Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation and assistance extended by your staff during the audit. If you
have any questions, please call Bernard Tadley at 215-656-6279.

                                             Sincerely,

                                             /s/
                                             Patrick J. Howard
                                             Assistant Inspector General for Audit


Attachments
Final Report
ED-OIG/A03N0006                                                              Page 20 of 31

                                                                              Attachment 1

             Abbreviations, Acronyms, and Short Forms Used in This Report


CA DOR                    California Health and Human Services Agency, Department of
                          Rehabilitation

CAP                       Corrective Action Plan

Case Service Manual       Reporting Manual for the Case Service Report

C.F.R.                    Code of Federal Regulations

Compliance Supplement     2 C.F.R. Part 200, Appendix XI

FY                        Fiscal Year

MTAG                      Vocational Rehabilitation Program Federal Monitoring and
                          Technical Assistance Guide

OMB                       Office of Management and Budget

OMB Circular A-123        OMB Circular A-123, Management’s Responsibility for Internal
                          Control

OOD                       Opportunities for Ohioans with Disabilities

PA OVR                    Pennsylvania Department of Labor and Industry, Office of
                          Vocational Rehabilitation

Rehabilitation Act        Rehabilitation Act of 1973, as amended

RSA                       Rehabilitation Services Administration

RSA-911-16 report         Revised RSA-911 report

RSA-911 report            Case Service Report

RSA ERA program           RSA Errors, Reasonableness Checks, and Anomalies Program

SMPID                     State Monitoring and Program Improvement Division

VR                        Vocational Rehabilitation

WIOA                      Workforce Innovation and Opportunity Act
Final Report
ED-OIG/A03N0006                                                                          Page 21 of 31

                                                                                          Attachment 2

Table 1. Performance Levels for the Performance Indicators
          Performance Indicator                    Performance Level Required       Performance Level
                                                of a General/Combined VR Agency   Required of VR Agency
                                                                                       for the Blind
 1.1 - Number of employment outcomes.               Equals or exceeds previous
                                                       performance period.                Same

 1.2 - Percent with employment outcomes
       after services.                                       55.8%                       68.9%

 1.3 - Percent of employment outcomes that
       were competitive employment.                          72.6%                       35.4%


 1.4 - Percent of participants with
       competitive employment outcomes                       62.4%                       89.0%
       who had a significant disability.

 1.5 - Ratio of average hourly VR wage to
       average State wage.                                    0.52                        0.59

 1.6 - Difference between percent self-
       supporting at closure and application.                 53.0                        30.4

 2.1 - Ratio of minority service rate to
       nonminority service rate.                              0.80                        Same
 Final Report
 ED-OIG/A03N0006                                                                                                                                                          Page 22 of 31

                                                                                                                                                                           Attachment 3

      Table 2. Data Elements Reviewed According to the Participant’s Type of Closure Code
                      Social    Birth     Race      Application    Eligibility      Disability   IPE* Date   Services    Employer      Employment    Weekly     Hours      Closure   Closure
Closure Type         Security   Date      and         Date        Determination      Priority                Provided     Name          Start Date   Earnings   Worked      Type      Date
                     Number             Ethnicity                     Date                                                                              at        in a
                                                                                                                                                     Closure    Week at
                                                                                                                                                                Closure
                                                                  N/A, Verified
1: Exited as an
                        X        X         X            X         Eligibility Not     N/A          N/A         N/A         N/A            N/A          N/A       N/A         X         X
applicant
                                                                   Determined

3: Exited with an
employment              X        X         X            X               X               X           X           X            X             X            X         X          X         X
outcome
                                                                                                                            N/A
4: Exited without
                                                                                                                          Verified
an employment
                                                                                                                         Participant
outcome after           X        X         X            X               X               X           X           X                         N/A          N/A       N/A         X         X
                                                                                                                          Did Not
receiving IPE
                                                                                                                           Obtain
services
                                                                                                                        Employment
5: Exited without
                                                                                                               N/A
an employment
                                                                                                             Verified
outcome after
                        X        X         X            X               X               X           X           No         N/A            N/A          N/A       N/A         X         X
signed IPE but
                                                                                                             Services
before receiving
                                                                                                             Provided
IPE services
                                                                                                   N/A
6: Exited from an
                                                                                                  Verified
Order of Selection      X        X         X            X               X               X                      N/A         N/A            N/A          N/A       N/A         X         X
                                                                                                 No Signed
Wait List
                                                                                                    IPE

7: Exited without
an employment                                                                                      N/A
outcome after                                                                                     Verified
                        X        X         X            X               X               X                      N/A         N/A            N/A          N/A       N/A         X         X
eligibility but                                                                                  No Signed
before an IPE was                                                                                   IPE
signed


 * Individualized plan for employment
Final Report
ED-OIG/A03N0006                                                                                       Page 23 of 31

                                                                                                      Attachment 4


                             UNITED STATES DEPARTMENT OF EDUCATION
                         OFFICE OF SPECIAL EDUCATION AND REHABILITATIVE SERVICES




MEMORANDUM

DATE:            October 5, 2016

TO:              Patrick J. Howard
                 Assistant Inspector General for Audit

FROM:            Sue Swenson,
                 Acting Assistant Secretary

                 Janet L. LaBreck
                 Commissioner
                 Rehabilitation Services Administration

SUBJECT:         OSERS and RSA Response: Rehabilitation Services Administration's Internal
                 Controls Over Case Service Report Data Quality, Control Number ED-
                 OIG/A03N0006

The Office of Special Education and Rehabilitative Services (OSERS) and the Rehabilitation
Services Administration (RSA) have reviewed the Office of Inspector General (OIG) draft
audit report, dated August 30, 2016, entitled "Rehabilitation Services Administration's Internal
Controls Over Case Service Report Data Quality," Control Number ED-OIG/A03N0006.

OSERS and RSA acknowledge the sole finding identified in the draft audit report. In our
response, we have included a corrective action plan (CAP) that explains some of the specific
actions we are taking, consistent with the seven recommendations in the draft audit. In the
CAP, we also included actions we are considering that are appropriate to the Federal role and
that are raised in the recommendations in the draft report.

Enclosure




                                   400 MARYLAND AVE. S.W., WASHI NGTON DC 20202-2500
                                                           www.ed.gov
 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by
                             fostering educational excellence and ensuring equal access.
Final Report
ED-OIG/A03N0006                                  Page 24 of 31



              RESPONSE:
        DRAFT OIG AUDIT REPORT
          ED-OIG/A03N0006
        REHABILITATION SERVICES
       ADMINISTRATION’S INTERNAL
      CONTROLS OVER CASE SERVICE
         REPORT DATA QUALITY




                  U.S. DEPARTMENT OF EDUCATION
              OFFICE OF SPECIAL EDUCATION AND
                  REHABILITATIVE SERVICES
          REHABILITATION SERVICES ADMINISTRATION
                        OCTOBER 5, 2016



                       TABLE OF CONTENTS
Final Report
ED-OIG/A03N0006                                         Page 25 of 31

                                                                Page

OSERS/RSA Response to Draft Findings……………………………………………………...…1

RSA Corrective Action Plan: ED-OIG/A03N0006…………………………………………...….3

Exhibit 1 – RSA-911 Case Service Report

Exhibit 2 – RSA-911 Policy Directive

Exhibit 3 – ERA Program Instructions
Final Report
ED-OIG/A03N0006                                                                     Page 26 of 31

                        Response to Draft Findings
The Office of Special Education and Rehabilitative Services (OSERS) and the Rehabilitation
Services Administration (RSA) acknowledge the sole finding identified in the Office of Inspector
General (OIG) draft audit report, dated August 30, 2016, entitled “Rehabilitation Services
Administration’s Internal Controls Over Case Service Report Data Quality,” Control Number
ED-OIG/A03N0006. As explained below, OSERS and RSA have attached a corrective action
plan (CAP) that explains some of the specific actions we are taking, consistent with the seven
recommendations below, such as those we described during the OIG’s exit conference with RSA
on June 20, 2016. In the CAP, we also include actions we are considering that are appropriate to
the Federal role and that are consistent with the recommendations in the draft report.

Finding and Response

Finding No. 1

RSA Should Improve Its Internal Controls to Provide Reasonable Assurance that State
Vocational Rehabilitation Agencies Submit Reliable RSA-911 Report Data.

In its draft report in Recommendation 1.1, the OIG recommends that RSA develop procedures
for periodically reviewing the processes State vocational rehabilitation (VR) agencies use to
capture, review, verify, and submit their RSA-911 report data, including a review of, on at least a
sample basis, the supporting documentation for the data.

The OSERS Acting Assistant Secretary appreciates the OIG’s recommendation to develop
internal control procedures necessary to ensure that VR agencies submit accurate RSA-911 data.
RSA will ensure that any internal control procedures it develops will be consistent with Federal
statutory and regulatory requirements. RSA intends to include, in its internal control procedures,
a monitoring component consistent with the requirements of section 107 of the Rehabilitation
Act of 1973, as amended (Rehabilitation Act) and the Department of Education’s (Department’s)
policy on administering formula grants. In addition, RSA is considering developing more
comprehensive edit checks of the data submitted by State VR agencies to improve the validity of
that data.

As part of its monitoring process, RSA also plans to provide technical assistance to VR agencies
regarding the development and implementation of internal controls for ensuring the accurate and
timely reporting of RSA-911 data. In providing technical assistance, RSA plans to recommend
that VR agencies include, as part of their internal controls, a monitoring component through
which VR agencies periodically compare and reconcile the data entered into the agency’s
electronic reporting system against the supporting documentation for the data.

By enhancing monitoring of VR agency’s internal control processes, RSA will be able to ensure
that VR agencies are implementing the internal control processes necessary for reporting
accurate RSA-911 data. RSA’s monitoring of the VR agencies will include a sample review of
supporting documentation, as needed, and as consistent with section 107(a)(3) of the
Rehabilitation Act, to ensure the validity of the VR agency’s internal control process.
                                                1
Final Report
ED-OIG/A03N0006                                                                      Page 27 of 31

In its draft report for Recommendation 1.4, the OIG recommends that RSA require an authorized
State VR official to certify the RSA-911 data submission. OSERS Acting Assistant Secretary
agrees with the need for VR agencies to certify RSA-911 data submissions. RSA already has
implemented a requirement for the certification of RSA-911 data submissions for FFY 2016 and
2017, including data to be submitted under the revised RSA-911 that is scheduled to begin on
July 1, 2017 for Program Year 2017. RSA will require the VR director or authorized designee to
certify, via email, that the RSA-911 data submitted for FFYs 2016 and 2017 are correct, prior to
RSA accepting the State agency’s data.

The following certification language was included in the recently revised RSA-911 and is
emphasized in RSA’s Policy Directive issued September 27, 2016 (PD-16-04). The VR director,
or an authorized designee, will have to certify, prior to data submission, the following statement:
By submitting this report, I certify to the best of my knowledge and belief that the report is true,
complete, and accurate, and the expenditures reported are for the purposes and objectives set
forth in the terms and conditions of the Federal award. I am aware that any false, fictitious, or
fraudulent information, or the omission of any material fact, may subject me to criminal, civil or
administrative penalties for fraud, false statements, false claims or otherwise.

This certification statement was adapted directly from the Uniform Guidance at 2 CFR 200.415.
While this regulation is specific to the submission of financial data by grantees, it was adapted
because the RSA-911 also contains financial information regarding the amount of funds
expended for VR consumer services. RSA believes that implementation of this certification, as
adapted from the Uniform Guidance, in conjunction with practices RSA is implementing that are
consistent with Recommendation 1.1 and Recommendation 1.2, are appropriate steps to take,
within the context of its Federal role, to satisfy this recommendation.




                                                 2
Final Report
ED-OIG/A03N0006                                                                       Page 28 of 31

          RSA Corrective Action Plan: ED-OIG/A03N0006
Finding No. 1 - RSA should improve its internal controls to provide reasonable assurance that
State Vocational Rehabilitation (VR) agencies submit reliable RSA-911 report data.

Recommendations: We recommend that the Acting Assistant Secretary for the Office of
Special Education and Rehabilitative Services require RSA to -

1.1       Develop procedures for periodically reviewing the processes State VR agencies use to
          capture, review, verify, and submit their RSA-911 report data, including a review of,
          on at least a sample basis, the supporting documentation for the data.

      Response: The OSERS Acting Assistant Secretary appreciates the OIG’s recommendation.
      RSA has taken steps consistent with this recommendation, as explained below and previously
      in comments provided in RSA’s Response to the Draft Finding. In addition, RSA is
      considering other ways to address this recommendation, appropriate to its Federal role,
      taking into account the changes to the VR program required by the Workforce Innovation
      and Opportunity Act (WIOA), which was signed into law on July 22, 2014. Specifically,
      RSA has:

      •    reviewed monitoring guides that RSA used when conducting case service record reviews
           prior to FFY2007. RSA had discontinued the case record reviews because they resulted in
           few issues of non-compliance being identified;
      •    spoken with staff within the Department of Labor, Employment and Training
           Administration (DOL-ETA) about their internal controls, quality assurance, and data
           validation methods. DOL-ETA collects data similar to the data collected through the
           RSA-911 and has recently revised its internal controls, quality assurance, and data
           validation methods; and
      •    included the regulatory citations and requirements for submission of accurate data in the
           RSA-911 Case Record Manual to emphasize the legal requirement that agencies submit
           accurate data.

      RSA is considering additional steps consistent with Recommendation 1.1, such as:

      •    developing and implementing internal procedures for reviewing State VR agency RSA-
           911 internal control procedures to ensure they result in the submission of accurate and
           reliable data; and
      •    providing technical assistance and training to State VR agencies in developing and
           implementing RSA-911 internal controls and quality assurance procedures, that, e.g.,
           could include developing a sample case record review instrument, similar to the ones
           used by RSA previously, as a tool for VR agencies to use when implementing their
           internal controls and quality assurance practices.




                                                   3
Final Report
ED-OIG/A03N0006                                                                         Page 29 of 31

1.2       Provide guidance to State VR agencies on the importance of developing and
          implementing adequate internal controls over VR data to ensure that the data State
          VR agencies report are accurate and complete.

      Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and
      RSA has already taken steps consistent with it. Specifically, RSA has:

      •    included language in the RSA-911, issued in June 2016, to inform State VR agencies
           that they must develop and implement policies and procedures to ensure accurate data
           collection in accordance with 34 CFR 361.12;
      •    begun developing a comprehensive list of the types of supporting documentation that
           may be used to verify core RSA-911 data elements. RSA has already informed State VR
           agencies, in the RSA-911 issued in June 2016, of the list and the need for them to
           maintain supporting documentation; and
      •    begun developing more comprehensive edit checks to improve accuracy of the data
           submitted by VR agencies.

      In addition, RSA is considering taking additional steps consistent with Recommendation 1.2,
      such as:

      •    reiterating the fact that State VR agencies need to develop internal controls in the RSA-
           911 Policy Directive issued on September 27, 2016;
      •    finalizing a comprehensive list of the types of supporting documentation that may be
           used to verify core RSA-911 data elements; and
      •    providing technical assistance to States regarding the importance of implementing and
           adhering to policies and procedures that will promote accurate and reliable data collection
           and submission.

1.3       Provide guidance, training, and technical assistance to State VR agencies on the type
          of source documentation needed to validate employment and earnings data.

      Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and
      RSA has taken steps consistent with it. Specifically, RSA has:

      •    in coordination with DOL-ETA and the Office of Career, Technical, and Adult Education
           (OCTAE) within the Department of Education, RSA is developing joint guidance for
           States on what documentation is required to validate employment and earnings data. The
           Departments expect to issue this guidance in calendar year 2017. RSA is issuing this
           guidance jointly with OCTAE and DOL because each agency administers core programs
           of the workforce development system, all of which must satisfy certain common
           reporting requirements for performance accountability purposes under the WIOA. The
           VR program is one of those core programs.
      •    RSA is considering taking additional steps to implement Recommendation 1.3, e.g.,
           providing joint technical assistance with DOL-ETA and OCTAE related to the guidance,
           once it has been issued.

                                                    4
Final Report
ED-OIG/A03N0006                                                                        Page 30 of 31

1.4    Require an authorized State VR official to certify that (1) the State VR agency has
       internal controls over RSA-911 report data that have been tested and are working as
       intended, or that the State VR agency has fully disclosed any data limitations and
       corrective actions; (2) the required RSA-911 report data submitted are accurate and
       complete; (3) documentation of the procedures performed to ensure the required
       RSA-911 report data submitted are accurate and complete has been maintained; and
       (4) they understand submitting inaccurate or incomplete data may result in having to
       develop and implement corrective actions, or other consequences as determined by
       the Department.

      Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and is
      taking steps consistent with the comments provided in RSA’s Response to the draft finding.
      RSA will be taking into account the statutory and regulatory requirements, in considering
      actions that our appropriate to its Federal role. To that end, RSA has already included
      language in the RSA-911, issued in June 2016, to require States to certify that the report
      being submitted is true, complete, and accurate, and the expenditures reported are for the
      purposes and objectives set forth in the terms and conditions of the Federal award, beginning
      with data submitted in program year 2017.

      RSA is considering additional steps to implement this certification in RSA-911 data
      collections prior to the implementation of data submissions required by WIOA, e.g.,
      providing the language below to State VR agency directors and require them to certify their
      data set via email prior to acceptance of RSA-911 for Federal Fiscal Year (FFY) 2016 and
      FFY 2017.

             o By submitting this report, I certify to the best of my knowledge and belief that the
               report is true, complete, and accurate, and the expenditures reported are for the
               purposes and objectives set forth in the terms and conditions of the Federal
               award. I am aware that any false, fictitious, or fraudulent information, or the
               omission of any material fact, may subject me to criminal, civil or administrative
               penalties for fraud, false statements, false claims or otherwise.

1.5    Review the results of States’ single audit reports for findings related to the RSA-911
       report to determine technical assistance needs of State agencies and areas of concern
       and perform follow-up review activities as necessary.

      Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and
      RSA has already take steps consistent with Recommendation 1.5. Specifically, RSA
      responds to State single audit findings related to the RSA-911 by providing technical
      assistance to the grantee and requiring that the correct data be submitted to RSA.

      RSA is considering taking additional steps to implement Recommendation 1.5, e.g., in future
      responses to RSA-911 single audit findings, requiring the grantees to determine the root
      cause of the inaccurate reporting and to review and revised their internal control processes, as
      necessary, to ensure identified issues have been corrected on a systemic level.


                                                   5
Final Report
ED-OIG/A03N0006                                                                        Page 31 of 31

1.6       Provide guidance to external auditors regarding performing compliance testing of
          RSA-911 report data.

Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and RSA
has already taken steps consistent with it. Specifically, RSA has:

      •    spoken with external auditors to generate an informal needs assessment to determine
           what information will need to be provided to them in order to adequately perform
           compliance testing on the RSA-911 data; and
      •    included core RSA-911 data elements for compliance testing in the Department’s A-133
           Compliance Supplement.

RSA is considering taking additional steps consistent with Recommendation 1.6, such as:

      •    developing a fact sheet or quick reference guide to instruct external auditors in
           compliance testing RSA-911 data in calendar year 2017; and
      •    reviewing the RSA-911 compliance testing information included in the Department’s A-
           133 Compliance Supplement to determine if more information is required. This will be
           accomplished by February 2017.

1.7       Ensure that any updates to its process for performing edit checks of State VR
          agencies’ RSA-911 report data are clearly and adequately documented and
          disseminated to staff, and that staff are sufficiently trained on the new procedures.

Response: The OSERS Acting Assistant Secretary appreciates this recommendation, and RSA
has already taken steps consistent with it. Specifically, RSA has:

      •    documented the current ERA edit check systems and trained Data Unit staff in running
           the edit check programs; and
      •    stored all documentation on a server that is accessible to all Data Unit staff.

RSA is considering taking additional steps consistent with Recommendation 1.7, such as:

      •    continuing to develop documentation for the edit check programs being developed for the
           new RSA-911 data collection to be implemented in July 2017;
      •    training staff as backups in running and reviewing the outputs of the edit check programs.
      •    develop and implement internal control processes for ensuring consistency and reliability
           among staff utilizing the edit check program.




                                                   6