oversight

California Department of Rehabilitation Case Service Report Data Quality

Published by the Department of Education, Office of Inspector General on 2015-12-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                               UNITED STATES DEPARTMENT OF EDUCATION
                                              OFFICE OF INSPECTOR GENERAL


                                                                                                                   AUDIT SERVICES
                                                                                                              Sacramento Audit Region


                                                       December 10, 2015

                                                                                                            Control Number
                                                                                                            ED-OIG/A09O0008

Joe Xavier
Director
California Department of Rehabilitation
721 Capitol Mall
Sacramento, CA 95814


Dear Mr. Xavier:

This final audit report, “California Department of Rehabilitation Case Service Report Data
Quality,” presents the results of our audit. The objectives of our audit were to determine whether
the California Department of Rehabilitation (1) had adequate internal controls to provide
reasonable assurance that the reported “Case Service Report” (RSA-911 report) data were
accurate and complete and (2) reported RSA-911 performance indicator data that were accurate,
complete, and adequately supported. Our review covered the Department of Rehabilitation’s
(DOR) 2013 RSA-911 report for the reporting period October 1, 2012, through September
30, 2013 (2013 reporting period).

DOR did not have adequate data quality controls to ensure that information it reported to the
Rehabilitation Services Administration (RSA) was accurate, complete, and adequately supported.
Specific control weaknesses we identified were: (1) lack of an adequate control to prevent staff
from changing the date that a participant’s case was closed in its Accessible Web-Based Activity
Reporting Environment (AWARE) case management system; (2) insufficient requirements that
personnel maintain documentation to corroborate key dates for application, eligibility, case
closure, and employment data entered into AWARE; (3) lack of guidance for determining
effective dates for participants’ plans to obtain employment and cost data for purchased services
provided to participants; and (4) insufficient manager oversight to provide assurances that data
were accurate and required documentation was maintained in participant files or in AWARE.

Our testing of data that DOR reported to RSA showed that most of the data elements in our
review contained significant data errors (estimated error rates exceeding 5 percent) that could
undermine RSA’s ability to effectively evaluate DOR’s performance or a significant unverifiable
data rate (estimated unverifiable data rate exceeding 5 percent) that would raise questions about
the reliability of data that DOR reported. RSA uses some of the data elements we tested to
calculate individual performance indicators, which are then used to determine whether DOR is
meeting RSA’s evaluation standards.



 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
Final Report
ED-OIG/A09O0008                                                                       Page 2 of 34

We made a variety of recommendations to the Commissioner of RSA that would require DOR to
establish and implement enhanced controls. DOR disagreed with many parts of Finding No. 1
regarding weaknesses in its internal controls over data quality for the 2013 reported RSA-911
data but agreed with most of the associated recommendations. DOR also disagreed with Finding
No. 2 regarding unverifiable performance indicator data reported in its 2013 RSA-911 report and
disagreed with our recommendation. DOR disagreed with parts of Finding No. 1 and Finding
No. 2 because it considers the information entered in AWARE as the official records for certain
data elements such as application date and employment start date. DOR does not believe that it
needs to maintain documentation outside of AWARE for many data elements. DOR discussed
some corrective actions it has planned or that are underway, including AWARE enhancements
that it plans to fully implement by the end of 2015. DOR also stated that some new requirements
in the Workforce Innovation and Opportunity Act of 2014 (WIOA) would achieve the same or
similar results as our recommendations.




                                      BACKGROUND


The U.S. Department of Education’s Office of Special Education and Rehabilitative Services
supports programs that serve millions of children, youth, and adults with disabilities. The Office
of Special Education and Rehabilitative Services’ Rehabilitation Services Administration
oversees grant programs that help people with physical or mental disabilities to obtain
employment and live more independently through the provision of counseling, medical and
psychological services, job training, and other individualized services. RSA provides
Vocational Rehabilitation Grants (VR) to States to assist them in operating VR programs. The
VR program grants are provided to support a wide range of services designed to help people
with disabilities prepare for and engage in gainful employment consistent with their strengths,
resources, priorities, concerns, abilities, capabilities, interests, and informed choice.

Each State designates a State agency to administer the VR program. The Rehabilitation Act of
1973 provides flexibility for a State to have two State VR agencies – one for individuals who are
blind and one for individuals with other types of disabilities, typically referred to as a general
agency. Combined agencies serve individuals with all types of disabilities. In California, the
Department of Rehabilitation is the State agency designated to administer the VR program.
DOR is part of the California Health and Human Services agency. DOR is composed of several
divisions, including the Specialized Services, Blind and Visually Impaired, and Deaf and Hard of
Hearing division. DOR maintains a central office in Sacramento along with 13 district offices
and 75 branch offices spread across most of California’s 58 counties. For FY 2013, DOR
received a VR program grant award of nearly $290 million.

People eligible for VR program services (referred to as participants in this report) are those who
have a physical or mental impairment that results in a substantial impediment to employment,
who can benefit from VR services for employment, and who require VR services. When all
eligible individuals with disabilities cannot be served due to limited resources, Federal
regulations require that an order of selection must be used. During the 2013 reporting period,
DOR’s policy was to operate under an order of selection that first served participants with the
Final Report
ED-OIG/A09O0008                                                                                         Page 3 of 34

most significant disabilities; followed by participants with significant disabilities; and lastly
participants with lesser disabilities. DOR placed eligible participants that could not be served on
a waiting list. According to RSA’s 2013 “California DOR Annual Review Report,” there were
214 individuals on the waiting list as of September 30, 2013.

Each year, State VR agencies must use the RSA-911report to report to RSA case data pertaining
to all participants whose case records were closed in a given fiscal year. DOR is required to
                                                        th
submit the RSA-911 report each year by November 30 . DOR uses the AWARE case
management system to store participant data and manage its cases. In a written response, DOR
explained it fully implemented AWARE by September 2011. Prior to using AWARE, DOR
used its Field Computer System case management system.1 For the 2013 RSA-911 report, DOR
extracted the case data from the AWARE database and reported a total of 47,356 closed
participant cases. Of this total, 12,239 cases were reported as closed with an employment
outcome (Code 3). Cases are coded in the RSA-911 report by type of closure to indicate when in
the VR process a participant exited the program, as shown below and in the diagram in
Attachment 2:

        exited as an applicant (code 1),
        exited during or after a trial work experience/extended evaluation2 (code 2),
        exited from an order of selection waiting list (code 6),
        exited without an employment outcome after eligibility was determined but before an
         individualized plan for employment (IPE)3 was signed (code 7),
        exited without an employment outcome after an IPE was signed but before receiving
         services (code 5),
        exited without an employment outcome after receiving services (code 4), and
        exited with an employment outcome (code 3).

Section 106 of the Rehabilitation Act of 1973, as amended, requires RSA to establish evaluation
standards and performance indicators for the VR program that include outcome related measures
of program performance. Two evaluation standards were established in June 2000 (34 Code of
Federal Regulations [C.F.R.] Part 361). RSA has established and minimum performance levels
for each performance indicator. RSA uses data from the RSA-911 report to monitor the
performance indicators and determining whether they have met the evaluation standards.




1
 DOR transferred participant data from its Field Computer System to its AWARE system when it converted to
AWARE for its case management system.
2
  Trial work experience/extended evaluation is used to determine eligibility if existing evidence indicates that the
individual is not capable of benefiting from VR services.
3
 The IPE is a written plan outlining a participant’s vocational employment goal and the services to be provided to
assist the participant in reaching the goal.
Final Report
ED-OIG/A09O0008                                                                                      Page 4 of 34

The evaluation standards and performance indicators are as follows:

Evaluation Standard 1 — Assesses VR's Impact on Employment

Standard 1 includes six performance indicators, three of which are primary indicators. The
primary indicators (1.3, 1.4, and 1.5) measure the quality of the employment outcomes achieved
by participants served by the program.

       Performance Indicator 1.1 — The number of participants exiting the VR program who
        achieved an employment outcome during the current performance period compared to the
        number of participants who exited the VR program after achieving an employment
        outcome during the previous performance period.

       Performance Indicator 1.2 — Of all participants who exit the VR program after receiving
        services, the percentage who are determined to have achieved an employment outcome.

       Performance Indicator 1.3 — Of all participants determined to have achieved an
        employment outcome, the percentage who exit the VR program in competitive, self- or
        business enterprise program4 employment with earnings equivalent to at least the
        minimum wage.

       Performance Indicator 1.4 — Of all participants who exit the VR program in
        competitive, self- or business enterprise program employment with earnings equivalent to
        at least the minimum wage, the percentage who are participants with significant
        disabilities.

       Performance Indicator 1.5 — The average hourly earnings of all participants who exit the
        VR program in competitive, self- or business enterprise program employment with
        earnings equivalent to at least the minimum wage as a ratio to the State’s average hourly
        earnings for all people in the State who are employed (as derived from the Bureau of
        Labor Statistics report “State Average Annual Pay” for the most recent available year).

       Performance Indicator 1.6 — Of all participants who exit the VR program in competitive
        employment, self- or business enterprise program employment with earnings equivalent
        to at least the minimum wage, the difference between the percentage who report their
        own income as the largest single source of economic support at the time they exit the VR
        program and the percentage who report their own income as the largest single source of
        support at the time they apply for VR services.

To achieve successful performance on standard 1, State VR agencies must meet or exceed the
minimum performance level for four of the six performance indicators in the evaluation standard,
including meeting or exceeding the performance levels for two of the three primary indicators.



4
 Business enterprise program means an employment outcome in which an individual with a significant disability
operates a vending facility or other small business under the management and supervision of a designated State unit.
Final Report
ED-OIG/A09O0008                                                                          Page 5 of 34

Evaluation Standard 2 — Assesses Equal Access Opportunity for People of All Groups and
Backgrounds

Standard 2 includes one performance indicator:

      Performance Indicator 2.1 — The service rate for all participants with disabilities from
       minority backgrounds as a ratio to the service rate for all participants with disabilities
       from non-minority backgrounds.

To achieve successful performance on standard 2, State VR agencies must meet or exceed the
performance level established for performance indicator 2.1 or must describe the policies it has
adopted or will adopt and the steps it has taken or will take to ensure that individuals with
disabilities from minority backgrounds have equal access to VR services.

State agencies that fail to meet these performance levels must develop a program improvement
plan outlining specific actions to be taken to improve program performance. For the 2013
reporting period, DOR achieved successful performance on the evaluation standards; however, it
did not meet the performance levels for performance indicators 1.2 and 1.5. Table 1 shows the
performance levels required for the performance indicators and DOR’s performance levels for
the 2013 reporting period from the RSA Web site.

Table 1. Performance Levels for the Performance Indicators
                                                                                  DOR’s 2013
  Performance                                                                   Reporting Period
                       Performance Level Required of VR Agency
   Indicator                                                                      Performance
                                                                                     Level
                    Number of employment outcomes equals or exceeds
                                                                                    +1,052
       1.1                      previous performance period
                                                                                     Met
                                           (11,187)
                      Percent with employment outcomes after services               37.11%
       1.2
                                            55.8%                                 Did not meet
                   Percent of employment outcomes that were competitive
                                                                                    89.17%
       1.3                               employment
                                                                                     Met
                                            72.6%
                    Percent of individuals with competitive employment
                                                                                    99.21%
       1.4                outcomes who had a significant disability
                                                                                     Met
                                            62.4%
                   Ratio of average hourly VR wage to average State wage             0.427
       1.5
                                             0.52                                 Did not meet
                  Difference between percent self-supporting at closure and
                                                                                     70.53
       1.6                                application
                                                                                      Met
                                             53.0
                  Ratio of minority service rate to non-minority service rate        0.999
       2.1
                                             0.80                                     Met
Final Report
ED-OIG/A09O0008                                                                                     Page 6 of 34



                                           AUDIT RESULTS


DOR’s internal controls, including information system controls, were not adequate to provide
reasonable assurance that 9 of the 13 data elements we reviewed from DOR’s 2013 RSA-911
report were accurate, complete, or supported by documentation in its participant case files or
participant records in AWARE. Of these nine data elements, four (type of closure, employment
start date,5 weekly earnings at closure, and hours worked in a week at closure) are used to
calculate four or more of the performance indicators. The type of closure data element is used
for all seven of the performance indicators. The data elements related to employment
(employment start date, weekly earnings at closure, and hours worked in a week at closure) are
used in four or more of the performance indicators. The remaining five (date of application, date
of eligibility determination, date of closure, date of IPE, and cost of purchased services) of nine
data elements do not directly impact any performance indicators.

DOR’s internal controls did provide reasonable assurance that four RSA-911 report data
elements included in our review (Social Security number, date of birth, race and ethnicity, and
significant disability) did not contain significant errors. DOR used interviews with participants,
data collection instruments, and third-party documentation to collect participants’ Social Security
numbers, date of birth, and race and ethnicity information. DOR also maintained documentation
in case files to support its determination of whether the participant had a significant disability.
The significant disability element is used for Performance Indicator 1.4 and the race and
ethnicity element is used for Performance Indicator 2.1.

Finding No. 1 discusses weaknesses in DOR’s internal controls that led to significant data error
or unverifiable data rates for the data elements listed below in Table 2. We determined that a
particular data element’s estimated error rate was significant if our estimate exceeded 5 percent.
We considered a data element to be unverifiable if our estimate for data entries not supported by
documentation exceeded 5 percent. DOR’s internal control weaknesses are summarized below:

       DOR did not have system controls over AWARE that would prevent its staff from
        changing a participant’s closure date or type after the case was initially closed.
       DOR did not require staff to maintain documentation to corroborate participants’
        application dates or employment data entered into the AWARE system.
       DOR did not require staff to ensure that notifications to participants for eligibility
        determinations or case closures reflected the same dates as those recorded in the AWARE
        system.
       DOR did not have policies and procedures to ensure that staff recorded the correct
        effective IPE date and total cost of purchased services in AWARE.
       DOR’s monitoring did not provide assurance that data were accurate and required
        documentation was maintained in participant files or in AWARE.

5
  Even though it is not a required RSA data element for assessing a State VR agency’s performance, the accuracy of
the employment start date for individual participants could affect whether their case is closed with an employment
outcome.
Final Report
ED-OIG/A09O0008                                                                                        Page 7 of 34

Table 2 shows the data elements included in our review that had significant error or unverifiable
data rates based on our review, along with the estimated rates of data errors or unverifiable data.
Most of these estimates have a margin of error that does not exceed plus or minus 10 percent at
the 95 percent confidence level. As an example, we estimate that the error rate for the date of
case closure data element was 47 percent as shown in Table 2. However, due to sampling
variation, we are 95 percent confident that the error rate for the date of case closure ranges
between 37 percent and 57 percent. We discuss these error rates and unverifiable data rates in
more detail in Findings No.1 and 2.

Table 2. Data Elements With Estimated Error and Unverifiable Rates Greater Than
5 Percent
                                                             Estimated            Estimated
 Data Elements                                              Error Rate        Unverifiable Rate
 Date of Application                                           N/A*                   17%
 Date of Eligibility Determination                              17%                   10%
 Date of Individualized Plan for Employment                     62%                   13%
 Cost of Purchased Services                                      7%                  N/A*
 Employment Start Dates** (effect on closure types)             12%                   50%
 Weekly Earnings at Closure**                                   11%                   56%
 Hours Worked in a week at Closure**                            11%                   53%
 Date of Closure                                                47%                    9
 Type of Closure**                                               7%                  N/A*
* Not applicable because the estimated error or unverifiable rate was 5 percent or less.
** The high estimated unverifiable rates for employment start date, weekly earnings at closure, and hours
worked in a week at closure may have a direct or indirect effect on four or more of the performance
indicators, as discussed in Finding No. 2.

We estimated the frequency that selected data elements had incorrect or unverifiable entries by
reviewing a statistical stratified sample of closed cases reported by DOR during the 2013
reporting period. The sample consisted of 162 out of 47,356 closed cases that DOR reported in
its 2013 RSA-911 report. For each sampled case, we tested those data elements that were
applicable based on the type of closure to determine accuracy, completeness, and whether the
data were supported by documentation.6

Finding No. 2 discusses the performance indicator data reported in DOR’s 2013 RSA-911 report
that were not supported by underlying documentation and, therefore, unverifiable. DOR did not
always maintain the required documentation in its participant case files or participant records in
AWARE to support important data elements, including employment start dates7 used to

6
  We determined that there was an error when the data in the RSA-911 report did not agree with documentation in
the participant’s case file or the AWARE system. We considered the data unverifiable when the participant’s case
file or AWARE record did not contain documentation to support the data reported in the RSA-911 report. Social
Security numbers, date of birth, and race and ethnicity, did not have any unverifiable data. Significant disability had
a low unverifiable rate.
7
 Employment start dates were not a required data element until the 2014 reporting period, as amended in the
Reporting Manual for the Case Service Record Report through the policy directive RSA- PD-14-01, October 25,
2013.
Final Report
ED-OIG/A09O0008                                                                                       Page 8 of 34

determine closure type for closed cases with employment outcomes, weekly earnings at closure,
or hours worked in a week at closure.

The participants’ closure type is used in the calculations for all seven performance indicators.
Even though the participant’s employment start date was not a required data element for the
RSA-911 report during our audit period, DOR used this date to determine whether a participant
was employed for the required 90 days before a participant’s case could be closed with an
employment outcome. We estimate that 50 percent of the employment start dates reported in
DOR’s 2013 RSA-911 report were unverifiable because DOR did not have documentation to
support the reported start dates. We were unable to determine the effect of the unverifiable start
dates on the closures that DOR reported with employment outcomes in the RSA-911 report and
DOR's reporting period performance levels on the performance indicators shown in Table 1
above.8

The participants’ weekly earnings at closure and hours worked in a week at closure are used in
calculating Performance Indicators 1.3 through 1.6. We estimate that 56 percent of the weekly
earnings at closure and 53 percent of the hours worked in a week at closure reported in the
RSA-911 report were unverifiable because DOR did not have documentation to support these
two data elements for those sampled cases. As with the employment start date, because the
weekly earnings at closure and the hours worked in a week at closure were unverifiable, we were
unable to determine the reliability of Performance Indicators 1.3 through 1.6.

As shown in Table 2 above, the estimated rate of data errors for individual data elements that
may have significantly affected the performance indicators ranged from 7 percent to 12 percent
and the estimated rate of unverifiable data that may have significantly affected the performance
indicators ranged from 50 percent to 56 percent. These error or unverifiable rates have an
unknown impact on data reliability and may have varying effects on the performance indicators
RSA calculated.

State VR agencies must have a system of internal control that provides reasonable assurance that
RSA-911 report data are accurate, complete, and supported so that RSA can rely on the data to
reflect the VR agency’s true performance when it calculates performance levels and determines
whether the agency is meeting standards. Reliable data are also important to ensure that RSA’s
annual reports submitted to the President and Congress, and made available to the public,
accurately report the VR agency’s performance. The extent that the State VR agency is meeting
performance standards could influence the amount of oversight and monitoring that RSA may
need to conduct at that agency.

We provided a draft of this report to DOR for review and comment on August 25, 2015. We
received DOR’s comments and additional documentation on September 25, 2015. DOR
disagreed with many parts of Finding No. 1 regarding weaknesses in its internal controls over
data quality for the 2013 reported RSA-911 data but agreed with most of the associated

8
  As shown in Table 2 above, based on our review of 162 sample closed cases, we estimate that DOR closed about
7 percent of its cases with an incorrect type of closure. Because the type of closure determines whether a record is
included in all performance indicator calculations, the 50 percent of employment start dates that we estimate were
unverifiable may impact the closure type data element significantly. The 7 percent error rate for closure types might
be understated, which could lead to significant errors in the performance indicator calculations.
Final Report
ED-OIG/A09O0008                                                                      Page 9 of 34

recommendations. DOR also disagreed with Finding No. 2 regarding unverifiable performance
indicator data reported in its 2013 RSA-911 Report and disagreed with our recommendation.

We did not change our findings and recommendations based on DOR’s comments on the draft
audit report. Although DOR did not think many of the recommendations were needed, we
believe the recommendations are still applicable because we do not know the effectiveness of
newly implemented controls over data for future RSA-911 reports or the effects of the WIOA on
the program. We summarized DOR’s comments on the draft audit report at the end of each
finding and included the comments in their entirety as Attachment 6 of this report.


Finding No. 1 – DOR’s Internal Controls Did Not Provide Reasonable Assurance
                That All 2013 Reported RSA-911 Data Were Accurate, Complete,
                or Supported

DOR’s system of internal controls did not ensure that data reported to RSA for the 2013
reporting period were accurate, complete, or supported. We identified various internal control
weaknesses that caused or contributed to incorrect or unverifiable data being reported to RSA.
DOR’s controls over information systems did not prevent VR team members from reopening and
editing data in previously closed cases. DOR did not have policies and procedures to ensure that
the data reported in its 2013 RSA-911 report were accurate, supported by documentation in
participant case files or AWARE, or reported in accordance with RSA guidance. DOR did not
require staff to maintain documentation to corroborate data they entered into its case
management system, nor did it require staff to ensure that eligibility determination and closure
notification dates reflected the dates reported in its AWARE system. Finally, DOR’s managerial
review process did not identify weaknesses in the data reported to RSA or ensure that the data
were supported by documentation.

Information System Control Weakness Allowed Changes to Closed Cases

DOR did not have a system control to prevent VR staff from reopening closed cases and editing
participant records, including closure dates and types, in the AWARE system. As a result, DOR
submitted incorrect data for 86 closed cases in its 2013 RSA-911 report because it included cases
that were closed in the previous reporting period. Also, our comparison of the RSA-911 required
data elements maintained in AWARE as of September 2014 to DOR’s 2013 RSA-911 report
showed that the closure dates in AWARE were different (including blank closure fields) for 144
closed cases. DOR’s information technology staff stated that the changes to closure dates
occurred for two reasons: (1) participants challenged case closures after the closure date was
entered in AWARE or (2) cases were reopened to pay bills and then were closed again with the
current activity date overriding the original closure date.

We determined that the 86 closures that were incorrectly reported in the 2013 RSA-911 report
occurred because the participant cases had already been closed and reported in 2012 and then
were reopened during the 2013 reporting period. Incorrect closures will likely occur again
during the 2014 reporting period and beyond because the remaining 144 closures incorrectly
reported in the 2013 reporting period either have new 2014 reporting period closure dates, a date
previous to 2013, or a blank closure date field. The cases that do not have closure dates are now
Final Report
ED-OIG/A09O0008                                                                                         Page 10 of 34

in open status and have the potential to be given a closure date during the 2014 reporting period
or later.

Even though the incorrect closures represent only about 0.5 percent of the total closures during
the 2013 reporting period (230 incorrect closures out of more than 47,000 total closures), the
AWARE system should have controls in place to prevent staff from reopening closed cases
unless a case was closed by mistake. VR staff should also have to provide a justification for
reopening a closed case, and DOR should require supervisory review and approval.

Weak Data Quality Controls Resulted in RSA-911 Reporting Issues

We identified various internal control weaknesses that caused or contributed to DOR reporting
incorrect or unverifiable data to RSA in the 2013 RSA-911 report. DOR did not ensure that the
data reported in its 2013 RSA-911 report were accurate or have policies and procedures for staff
to report data in accordance with RSA guidance. DOR also did not require that staff maintain
documentation to corroborate data entered into its case management system, nor did it ensure
that eligibility determination and closure notification dates in the case files reflected the dates
reported in its AWARE system. Finally, monitoring activities performed by DOR managers did
not provide assurance that the reported data were accurate and complete, or that required
documentation was maintained in participant files.

According to DOR officials, VR counselors and service coordinators, and to a lesser extent,
office technicians, employment coordinators, and team managers entered the required data into
AWARE that would subsequently be included in the RSA-911 report, based on interviews with
participants and various source documents.9 DOR maintained the agency-required
documentation and other documentation such as medical assessments in the participant’s
hardcopy case file.10 DOR extracts from AWARE the data elements required by RSA for its
annual RSA-911 report for cases closed as of September 30. State VR agencies must report
program performance data to RSA by November 30th each year.

Attachment 3 of this report shows the frequency of verified data, incorrect data, and unverifiable
data for each data element for a sample of closures reviewed in the audit and is further
summarized by closures resulting in an employment outcome (Closure Type 3) and all other
closure types (Closure Types 1, 2, 4, 5, 6, and 7). Below, we discuss each of the data elements
that had significant errors or unverifiable data and the associated control weaknesses that caused
or contributed to incorrect or unverifiable data being reported in the 2013 RSA-911 report.




9
  In a written response, DOR explained that starting in October 2012, DOR implemented a team approach to
delivering VR services to participants. A team consists of a team manager, senior vocational rehabilitation
counselor, SVRC, qualified rehabilitation professional (SVRC/QRP), service coordinator, employment coordinator,
and office technician. The previous approach was a one-on-one counselor relationship with the participant. For this
report, we use the term counselor to mean either SVRC or SRVC/QRP.
10
     Attachment 4 of this report shows the documents required to be maintained in the participant’s case file.
Final Report
ED-OIG/A09O0008                                                                                   Page 11 of 34

DOR Did Not Require Staff to Maintain Documentation to
Corroborate Data Entered Into the Case Management System

DOR did not have policies and procedures requiring team members to maintain documentation
(either hardcopy or electronic documents in AWARE or Field Computer System) to corroborate
each participant’s date of application, employment start date, weekly earnings at closure, and
hours worked in a week at closure for closed cases reported in its case management system.
Thus, we were unable to verify that these data elements, as reported in the 2013 RSA-911 report,
were reliable.

Date of Application
The Reporting Manual for the Case Service Report (RSA-911 Reporting Manual) in the RSA
Policy Directive RSA-PD-12-05 (RSA-PD-12-05), dated February 8, 2012, page 7, states that a
participant is considered to have submitted an application when the participant has completed
and signed an agency application or has otherwise requested services, has provided information
necessary to initiate an assessment to determine eligibility and priority for services, and is
available to complete the assessment process. Other than the application date in the AWARE
application input screen, DOR did not maintain documentation corroborating the date when all
three requirements were met for participants.

According to DOR’s Rehabilitation Administrative Manual Chapter 30 – Record of Services,
revised January 2009 (administrative manual), the application date is when the individual meets
all three application requirements (signed application, ability to assess eligibility, and availability
to complete the assessment process). In a written correspondence, DOR told us that the
participant’s application date in AWARE is the date when the counselor has determined that the
participant met all requirements for submitting an application.

Using participant or counselor signature dates from applications or initial interview dates
recorded in case files to corroborate application dates, we estimate that 17 percent of application
dates recorded in AWARE for the sampled closures did not have documentation to support the
date of application.11 Application dates need to be accurate because DOR generally has 60 days
from the date of application to determine a participants’ eligibility for VR services. According
to DOR’s current policies (effective September 2014), staff must confirm that all three
requirements for date of application are met before entering the application date in the AWARE
system. Also, if the signature dates are different than the date recorded in AWARE, staff must
explain the variance in a case note. This extra control may provide the documentation to support
the date of application in AWARE. We did not confirm or test the control because it occurred
after our audit period.

Employment Data
RSA-911 Reporting Manual, pages 34 and 35, states that VR agencies are to provide certain
employment data in the RSA-911 report, including the participant’s weekly earnings at closure
and the hours worked in a week at closure. DOR did not have written procedures requiring
counselors or other team members to maintain underlying documentation supporting

11
  We are 95 percent confident that the estimate for application dates without supporting documentation is between
10 and 26 percent.
Final Report
ED-OIG/A09O0008                                                                                      Page 12 of 34

participants’ employment start dates, weekly earnings at closure, or hours worked in a week at
closure. Because DOR did not maintain adequate records to support reported employment data,
almost half of the employment data in our sample cases were unverifiable. Finding No. 2
discusses this issue in detail, as employment data affects the performance indicators that RSA
uses to determine whether DOR meets the evaluation standards during each reporting period.

DOR Did Not Require Staff to Ensure That Eligibility Determination and
Closure Notification Dates Reflected the Dates Reported in the Case Management System

DOR did not require staff to ensure that notifications to participants associated with eligibility
determinations and case closures reflect the dates that were reported in its case management
system.

Date of Eligibility Determination
RSA-911 Reporting Manual, page 20, states that the VR agency should record the date that an
eligibility determination was made. DOR’s policies and procedures did not contain instructions
for counselors to ensure that the Notice of Eligibility was dated with the same date the
participant became eligible for VR services as recorded in AWARE.

We estimate that 17 percent of DOR case files contained a Notice of Eligibility with a
counselor’s signature date, stamp date notification sent, or determination date (if one was
included) that was different than the one recorded in the AWARE system.12 For Notices of
Eligibility with a counselor’s signature date or a determination date that was different from the
date recorded in AWARE, the eligibility date reported to RSA should have been the earlier date
since that was the date the eligibility determination was made. Eligibility dates need to be
accurate so that RSA can evaluate DOR’s compliance with mandatory requirements for
completing certain actions in a timely manner in support of VR participants.

In written responses to explain why the eligibility date is different in AWARE and the
Notification of Eligibility, DOR stated that as a standard of practice, the eligibility determination
date in AWARE should be entered at the same time the Notification of Eligibility is signed and
dated. As noted in previous DOR responses, eligibility determination in AWARE and the
printing of the Notification of Eligibility is currently a two-step process. Our review of the case
files confirmed that letters were prepared both before and after the eligibility determination date
in AWARE and the variance in the dates ranged from 1 to 55 days.

Date of Closure
RSA-911 Reporting Manual, page 41, states that the VR agency should record the date when it
closed the participant’s service record. To ensure that the date of closure is reported accurately,
the date recorded in AWARE should match the date on the closure letter issued to the
participant. DOR did not have written policies and procedures that would require validation of
the date a participant case was reported as closed in the AWARE system. DOR’s policies and
procedures require managers to close a case in AWARE after they complete their review of the
case file. However, in a written response, DOR stated that the policies did not require managers

12
  We are 95 percent confident that the estimate for cases with a Notice of Eligibility dated differently than what was
recorded in AWARE is between 10 and 27 percent.
Final Report
ED-OIG/A09O0008                                                                                        Page 13 of 34

to review and close a case in a timely manner after the counselor recommended the case for
closure. Lengthy delays in closing cases could result in case data being reported in a subsequent
year, which could lead to inaccurate performance results in two separate years. We estimate that
47 percent of the closure dates reported in the 2013 RSA-911 report were different than the
participants’ actual closure report dates.13

DOR’s written responses explained that the closure letters can be sent before or after the case is
officially closed in AWARE. In a written response, DOR stated that the reason for the date
discrepancy is due to a two-step process in which a manager officially closes a case in AWARE
and the closure report is printed on another day. Before closing the case in AWARE, managers
review the case file, which may not occur until potentially weeks later if a manager is on
vacation or otherwise occupied. Our review of 162 sampled files showed that closure letters
were prepared both before and after the closure dates recorded in the RSA-911 report and that
the variance in the dates ranged from 1 to 117 days. The majority of letters were prepared before
the closure date recorded in the RSA-911 report.

DOR Did Not Have Policies and Procedures for Staff to
Report Data in Accordance With RSA Guidance

DOR did not have policies and procedures requiring staff to report the date the IPE became
effective or to compile and report each participant’s actual total cost of purchased services in
accordance with RSA guidance noted below. Thus, the IPE date and the cost of purchased
services that DOR reported in its 2013 RSA-911 report were not accurate for some participants.

Date of Individualized Plan for Employment
RSA-911 Reporting Manual, page 21, states that the IPE is effective on the date on which both
parties reach agreement. If the two signatures are different, the later date should govern. DOR’s
written policies and procedures did not instruct staff on the proper date to use as the IPE
effective date to conform to the applicable RSA policy directive.

For participants required to have IPEs, we estimate that 62 percent of the IPE dates reported in
the RSA-911 report reflect incorrect dates. DOR counselors either (1) recorded their signature
date on the IPE as the effective date even though the participant signed and dated the IPE later or
(2) the counselor recorded a different date entirely.14

Cost of Purchased Services
RSA-911 Reporting Manual, page 21, instructs the VR agency to enter, to the nearest dollar, the
total amount of money spent by the agency to purchase services for a participant, over the life of
the current service record. DOR did not have sufficient procedures in place to ensure that the
total costs for purchased services for each participant were calculated correctly in AWARE.

For participants who received services, we estimate that 7 percent of the participants’ total costs
for purchased services recorded in AWARE did not reconcile to the amount reported in the 2013

13
     We are 95 percent confident that the estimate for incorrect closure dates is between 37 and 57 percent.
14
     We are 95 percent confident that the estimate for incorrect IPE dates is between 51 and 73 percent.
Final Report
ED-OIG/A09O0008                                                                                      Page 14 of 34

RSA-911 report.15 DOR explained that the total costs for purchased services in AWARE are
calculated correctly, but are not reconciled with the RSA-911 report after the end of the reporting
period and the report’s submission to RSA. As part of our reconciliation of total costs for
purchased services for participants, we reviewed a judgmentally selected high-dollar transaction
for participants’ purchased services and confirmed that DOR had invoices, authorizations to pay,
and checks to support the amounts in AWARE.

DOR’s Monitoring Did Not Provide Assurance
That Data Were Accurate and Required Documentation
Was Maintained in Participant Files

DOR’s procedures for reviewing participant case closures did not ensure that all required
documentation was included in the case files and that the documents were complete and
accurate. In a written response, DOR told us that team managers review case files (both hard
copy and in AWARE) at closure to ensure all required documents are included in a participant’s
records. We found that case files did not always contain the required supporting documents for
the date of eligibility determination, date of IPE, date of closure, or closure type. We also
identified documents that were unsigned or undated even though they should have been, as
discussed in further detail below. After our audit period, DOR implemented a comprehensive
case monitoring program for its rehabilitation counselors with approval authority.16 Title 34
C.F.R. § 80.40(a) states that grantees are responsible for managing the daily operations of grant
and subgrant supported activities to ensure compliance with applicable Federal requirements and
that performance goals are met. Monitoring by grantees must cover each program, function, or
activity.

The characteristics of internal control are presented in the Committee of Sponsoring
Organizations of the Treadway Commission’s “Internal Control-Integrated Framework,” May
2013 (COSO Report).17 As covered in the COSO Report, one of the five components of internal
control is monitoring. Monitoring is a process that assesses the quality of internal control over
time and is implemented to help ensure that internal controls are “present and functioning.” One
of the activities that serve to monitor the effectiveness of internal control is conducting internal
quality control reviews. Quality control reviews should provide reasonable assurance that the
controls management has put into place are working as intended.

Date of Eligibility Determination
DOR’s administrative manual requires counselors to sign and date the Notice of Eligibility and
retain it in the participant file. For participants who had an eligibility determination before


15
  We are 95 percent confident that the estimate for cases with an unreconciled total cost of purchased services is
between 2 and 15 percent.
16
  Rehabilitation counselors with approval authority complete five functions that cannot be delegated: determination
of eligibility, determination of priority for services, development of IPEs and IPE Amendments, review of IPE
progress, and case closure determinations.
17
   The COSO Report provides the framework for organizations to effectively and efficiently develop and maintain
systems of internal control that can enhance the likelihood of achieving the entity’s objectives and adapt to changes
in the business and operating environments.
Final Report
ED-OIG/A09O0008                                                                                      Page 15 of 34

exiting the program, we estimate that 10 percent of case files either did not contain the Notice of
Eligibility or contained an unsigned Notice of Eligibility.18

Date of Individualized Plan for Employment
DOR’s administrative manual requires that the IPE must be fully completed, dated, and signed
by the counselor, the manager, and the participant or the participant’s representative, and filed in
the participant’s case file. For participants who exited the program after an IPE was required, we
estimate that 13 percent of participant case files did not have an IPE signed and dated by all or
any of the required parties.19

Date of Closure
DOR’s administrative manual requires the counselor to sign and date either the “Closure Report-
Rehabilitated” or “Closure Report-Other Than Rehabilitated,” whichever is applicable. We
estimate that 9 percent of closures either did not have copies of the applicable closure report or
had copies that were unsigned or undated.20

Closure Type
We determined that 86 case closures had been incorrectly reported in the 2013 RSA-911 report
because of weak information system controls, as discussed above. Of the closures in our
statistical sample, 10 were from this subpopulation of 86, as discussed in detail in the
“Objectives, Scope, and Methodology” section of this report. In addition to the 10 incorrect case
closures in our sample, DOR manager reviews did not detect that another 6 participants’ cases
were closed with incorrect closure types for reasons other than weak system controls. DOR
closed cases for four participants in our sample who exited without an employment outcome,
after a signed IPE, but before receiving services (closure type 5). However, our review showed
that these four participants did not sign their IPEs. Therefore, DOR should have closed the
participants’ cases with closure type 7, which is, closed without an employment outcome, after
an eligibility determination, but before an IPE was signed. Another participant’s case should
have been closed as a closure type 5 instead of a closure type 7 because the participant’s file
contained a valid IPE. The remaining participant’s case was closed as exited with an
employment outcome (closure type 3), but the file showed that the participant was not employed
for at least 90 days.

RSA uses closure types 5 and 7 to calculate Performance Indicator 2.1. However, due to the way
closure types 5 and 7 are aggregated in the formula for this indicator, correcting these six closure
type errors would not have changed the result for Performance Indicator 2.1.

The 10 participant cases in our sample that were closed with the incorrect type of closure as a
result of weak system controls would affect only 86 closures, which would likely have minimal
impact on the 7 performance indicators. As described in Finding No. 2, we estimate that

18
  We are 95 percent confident that the estimate for cases with missing Notices of Eligibility or that were unsigned is
between 5 and 17 percent.
19
  We are 95 percent confident that the estimate for cases with missing IPEs or that were unsigned or undated is
between 6 and 23 percent.
20
  We are 95 percent confident that the estimate for cases with missing closure reports or reports that were unsigned
or undated is between 4 and 17 percent.
Final Report
ED-OIG/A09O0008                                                                      Page 16 of 34

50 percent of the employment start dates reported in AWARE were unverifiable and, therefore,
unreliable for use when determining whether all participant case closures with an employment
outcome were accurate.

In July 2014, DOR implemented new policies and procedures requiring managers to conduct
reviews of 10 percent of open and closed cases for all counselors with approval authority. Team
managers complete a Record of Service Review/Case File Review to confirm that a completed
notice of eligibility, IPE, and closure report are in the case file. In a written response, DOR
explained that districts are required to submit a “Record of Service Review Summary Report” to
DOR’s customer service unit every other year along with a corrective action plan to remedy
potential regulatory noncompliance. Also, team managers review and approve all case decisions
of newly hired counselors, counselors on probation, or counselors who have not yet
demonstrated proficiency in implementing the policies and procedures. Although we did not
review or test the current monitoring process since it was implemented after our audit scope, if
followed, it should improve the effectiveness of DOR’s case monitoring program provided DOR
also implements our recommendations for Findings No. 1 and 2.

Weaknesses in DOR management’s internal control systems resulted in incorrect data reported in
the RSA-911 report or incomplete or missing participant case file documentation not supporting
data reported in the RSA-911 report. It is important that DOR have internal controls that provide
reasonable assurance that data are accurate, complete, and supported since a number of data
elements reported in the RSA-911 report are used by RSA to monitor States’ compliance with
mandated timelines for delivering VR services to participants. Also, RSA has used the data for
closure type and cost of purchased services from the RSA-911 report for its Agency Report
Cards of Vocational Rehabilitation Performance available on the Department’s Web site and in
annual reports to the President and Congress.

Recommendations

We recommend that the Commissioner of RSA require DOR to—

1.1    Implement system controls that prevent VR staff from altering closure dates in the
       AWARE system unless the case was mistakenly closed.

1.2    Establish and implement controls that ensure that the dates for the notice of eligibility and
       closure reports are accurate in AWARE and that any discrepancies between the dates in
       participant case files and AWARE are fully explained.

1.3    Establish and implement controls to ensure that DOR staff record the correct date of the
       IPE (the latter of the counselor or participant signature date) in the AWARE system and
       account for all participant costs in AWARE.

1.4    Establish and implement controls to ensure that closures are reviewed timely and case
       files contain all required documentation for the closure type.

1.5    Provide evidence demonstrating that its recently implemented policies and procedures
       related to the date of application are effective in ensuring that team members are
Final Report
ED-OIG/A09O0008                                                                       Page 17 of 34

       documenting when all requirements have been met to establish an accurate application
       date.

1.6    Provide evidence demonstrating that its newly implemented policies and procedures
       related to manager monitoring of counselors with approval authority are effective in
       ensuring that managers will routinely identify missing documents and missing signatures
       or dates on documents, including notices of eligibility, IPEs, and closure reports.

DOR Comments and OIG Response

DOR disagreed with many parts of this finding regarding weaknesses in its internal controls over
the 2013 reported RSA-911 data but agreed with many of our recommendations. In the
following sections, we first summarize DOR’s comments (see Attachment 6 for a copy of DOR’s
comments) and then provide our response, if warranted.

Information System Control Weakness Allowed Changes to Closed Cases

DOR Comments
DOR concurred with this part of the finding and the associated recommendation and stated it has
taken actions to address the weakness. DOR explained before it completes the RSA-911 report,
it reviews cases opened and closed in multiple reporting periods. It reviews the cases for
appropriate closure type and to ensure that staff reported the case in the correct reporting period.
Also, DOR issued a memorandum in May 2015, “Invoice Payment After Record of Service
Closure,” to remind staff not to open closed cases solely for administrative reasons.

Weak Data Quality Controls Resulted in RSA-911 Reporting Issues

DOR Did Not Require Staff to Maintain Documentation to
Corroborate Data Entered Into the Case Management System

DOR Comments
DOR disagreed with this part of the finding and the associated recommendation. DOR cited 34
C.F.R 361.47(b), which delegates to the designated State unit, in consultation with the State
Rehabilitation Council, the determination of the type of documentation required for the
participants’ record of services. DOR agrees that it must document the date of application,
employment start date, weekly earnings, and hours worked at closure. However, DOR considers
the AWARE record to be a participant’s record of services and does not believe further
verification or corroboration of this data is required.

OIG Response
Wherever possible, participant data stored in AWARE should be corroborated with supporting
documentation that is either stored electronically in AWARE or in a separate electronic or
hardcopy case file. The lack of corroborating documents prevents third parties such as RSA,
OIG, and others who may need to perform program evaluations or audits from verifying the
accuracy and completeness of participants’ case information or summary data reported to entities
charged with governance. Without supporting documentation, DOR’s own team managers have
no way to verify or corroborate the data entered into AWARE by counselors and others.
Final Report
ED-OIG/A09O0008                                                                      Page 18 of 34

Title 34 C.F.R. § 76.731 requires that States maintain records to demonstrate compliance with
program requirements. We did not evaluate DOR’s September 2014 policy requiring staff to
confirm that all three requirements for date of application are met before entering the application
date in AWARE and explain any date variances in a case note because it was implemented after
our audit period. As a result, we did not confirm that DOR was following the new procedures,
nor did we review case notes that explain variances between dates, such as the date in the
application for services and the corresponding date in AWARE. However, DOR needs
documentation supporting the application date because DOR generally has 60 days from the
application date to determine a participant’s eligibility under Federal requirements.

RSA requires DOR to maintain certain documentation to support data elements associated with
employment outcomes. As stated in Finding No. 2, the requirements in 34 C.F.R. § 361.47 and
§ 361.56, taken together, require State VR agencies to maintain verifying documentation in the
participant case file related to employment outcomes. This includes the employment start date,
evidence that the participant maintained employment for 90 days, weekly earnings at closure,
and hours worked in a week at closure.

DOR Did Not Require Staff to Ensure That Eligibility Determination and
Closure Notification Dates Reflected the Dates Reported in the Case Management System

DOR Comments
DOR disagreed with this part of the finding but agreed with the associated recommendation.
DOR stated that it did and continues to require staff to ensure that eligibility determination and
closure notification dates reflect the dates in AWARE. DOR acknowledged that there were date
inconsistencies due to the two-step process in effect during our audit period. DOR explained that
it implemented system changes in April 2014 that will significantly enhance data integrity for
eligibility and closure report dates by automating these notifications in AWARE. By automating
the eligibility and closure reports, DOR stated that the respective forms will be printed from
AWARE. This change will ensure dates correlate and will eliminate discrepancies between
AWARE and a participant’s case file. DOR noted that it has scheduled the changes to occur at
the end of 2015. DOR will implement and communicate a policy for managers to timely review
closures to ensure the closure type is accurate and case documentation supports the closure.
Also, DOR stated that it continuously examined its processes, and closure policies and
procedures have been revised accordingly, to ensure alignment with the WIOA.

OIG Response
Even with DOR’s two-step process for reporting eligibility and closure dates and sending the
associated notifications, the dates in AWARE and on the notifications should still have matched.
If the notification dates do not match the reported date in AWARE, the notification should
inform the participant of the actual date they were determined eligible for VR services or their
case was closed.

Because DOR informed us of the planned system enhancements and automation of the
notifications to address the date discrepancies after we had completed our fieldwork, we could
not evaluate whether the changes will result in the notification dates matching the dates reported
in AWARE. Also, the WIOA was effective after our audit period so we did not confirm that the
policies and procedures over closure dates were implemented in accordance with the Act.
Final Report
ED-OIG/A09O0008                                                                       Page 19 of 34

DOR Did Not Have Policies and Procedures for Staff to
Report Data in Accordance With RSA Guidance

DOR Comments
DOR agreed that it did not have policies and procedures for determining the effective date of a
participant’s IPE and the associated recommendation. However, DOR disagreed with the finding
section and recommendation related to determining the total cost of purchased services.

DOR stated it will provide written guidance to staff clarifying the IPE effective date as defined
by 34 C.F.R. 361.45 (d)(3) by December 31, 2015. DOR also stated that personnel will verify
that the required signatures have been obtained and AWARE reflects the appropriate IPE
effective date when participant case records are reviewed.

DOR stated that it correctly calculates its total costs for purchased services in AWARE for the
RSA-911 submission. DOR stated that the discrepancies we found were between the costs in
AWARE at a point in time (year-end for RSA-911 report) and the current case record which
reflected costs at the time of our review, including changes attributed to refunds and late invoices
that DOR received and paid after the RSA-911 report submission.

OIG Response
Because DOR is implementing new policies and procedures for determining the effective date of
the IPE after our fieldwork, we cannot verify that DOR has corrected this issue. We recognize
that participant records in AWARE may include additional service costs and refunds that were
not included in the RSA-911 report submission because the reported data reflects a point in time.
Thus, we calculated a participants’ total purchased service costs using only data up to September
30, 2013. We identified eight participants’ whose total purchased service costs in AWARE as of
September 30, 2013 did not reconcile to the amounts reported in the 2013 RSA-911 report. We
provided our totals to DOR for the eight participants but DOR did not explain the differences
between our calculated amounts and the corresponding amounts DOR reported in its 2013 RSA-
911 report.

DOR’s Monitoring Did Not Provide Assurance
That Data Were Accurate and Required Documentation
Was Maintained in Participant Files

DOR Comments
DOR agreed with this part of the finding and the associated recommendation. DOR stated that
in July 2014, it implemented a formal policy requiring team managers to annually perform 10-
percent record of service caseload review for all counselors that have “approval authority.”
Also, team managers are required to review and approve all cases for counselors that do not have
approval authority. The review results are communicated to management for consideration and
determination of appropriate next steps.
Final Report
ED-OIG/A09O0008                                                                      Page 20 of 34

Finding No. 2 – Performance Indicator Data Reported in DOR’s 2013 RSA-911
               Report Were Not Supported
The employment data reported in DOR’s 2013 RSA-911 report for employment start date,
weekly earnings at closure, and hours worked in a week at closure were not, in many instances,
supported by documentation in participant case files or AWARE. In addition, DOR did not have
any procedures requiring supervisors to review employment related data entered into participant
records or any underlying documentation supporting the AWARE employment screens. RSA
uses the employment data as a key component of several performance indicators that measure
DOR’s performance against RSA’s evaluation standards. We found a common issue with much
of the employment data in our sample of closures with employment outcomes—namely, that the
data recorded in AWARE could not be verified because DOR did not require staff to maintain
documentation in its case files or in AWARE. Because significant portions of the employment
data for our sampled cases were unverifiable, we could not calculate error rates and, thus, could
not determine how the associated performance indicators would be affected.

A condition for closing the case of a participant as employed is that the participant maintains the
employment for a period of not less than 90 days (34 C.F.R. § 361.56(b)). Title 34 C.F.R.
§ 361.47(a)(15) requires that the participant case file include documentation for closures with an
employment outcome that verifies that the participant has maintained the employment for not
less than 90 days to ensure the employment is stable. In addition, 34 C.F.R. § 361.47(a)(9)
requires that the case file include documentation that verifies that the participant obtained
competitive employment and is compensated at or above the minimum wage.

Consequently, according to RSA, the requirements in 34 C.F.R. § 361.47 and § 361.56, taken
together, require State VR agencies to maintain verifying documentation in the participant case
file related to the employment outcome including the employment start date, that the participant
maintained employment for 90 days, the hours worked, and the amount of earnings. Although
the requirements do not specify the type of verifying documentation that must be maintained, the
regulations show that VR agencies need to have some type of supporting documentation for the
employment data in the case file.

The employment start date was not a data element collected in the 2013 RSA-911 report.
However, DOR maintains the employment start date in AWARE to compute whether
participants have been employed for at least 90 days. Closures with employment outcomes are
considered in the calculations for all seven performance indicators. The weekly earnings at
closure and hours worked in a week at closure are used to determine DOR’s performance level
for Performance Indicators 1.3 through 1.6. RSA divides each participant’s weekly earnings at
closure by hours worked in a week at closure to determine whether the participant earned an
hourly wage equal to or greater than $8 per hour (California’s minimum hourly wage during our
audit period). RSA uses the number of participants with hourly wages equal to or greater than $8
per hour in the calculations for Performance Indicators 1.3 through 1.6.

Almost half of our sampled cases with employment outcomes did not contain documentation
from third-party entities or DOR staff case notes supporting the participants’ employment start
dates, used to determine whether participants were employed for at least 90 days, or for
determining the participants’ weekly earnings at closure or the hours worked in a week at
closure. DOR staff also did not routinely store such information in AWARE. As a result, we
Final Report
ED-OIG/A09O0008                                                                                    Page 21 of 34

estimate that 50 percent of participants’ employment start dates were unverifiable. We also
estimate that 56 percent of the weekly earnings at closure and 53 percent of the hours worked in
a week at closure that DOR reported in its 2013 RSA-911 report were unverifiable.21

As explained in Finding No. 1, DOR did not have written procedures for counselors or other
team members to obtain and maintain supporting documentation of participants’ employment
start dates, weekly earnings at closure, or hours worked in a week at closure. Instead, DOR
explained that the data entered into the AWARE employment input screen represents the support
for employment-related data elements. However, we found that counselors and other team
members sometimes retained other documentation that led us to identify weaknesses in DOR’s
process.

Some participant case files, for example, contained a letter DOR referred to as the “Are You
Employed Letters,” “Supported Employment-Job Placement Forms,” DOR district-generated
forms, or third-party documents containing employment data. In other cases, we located case
notes in AWARE related to the employment data elements. However, this information did not
always match the recorded employment data in AWARE or the information that DOR reported
in its 2013 RSA-911 report. Federal regulations require that data reported by a VR agency be
valid, accurate, and in a consistent format (34 C.F.R. § 361.88(c)).

We identified 11 participant cases in our sample of 95 case closures with employment outcomes
(cases that were not incorrectly reopened and closed during the 2013 reporting period) that had
contradicting employment start dates in the participants’ case files and AWARE. We ultimately
determined that only one of these participants did not maintain employment for at least 90 days
when using the employment start date shown in employment documentation found in the
participant’s case file rather than that shown in AWARE—an error rate unlikely to have had an
impact on performance indicator results, but indicative of a problem nonetheless. We identified
another 11 participant cases from this same sample that had contradicting weekly earnings at
closure or hours worked in a week at closure data in the participants’ case files and AWARE. Of
these 11 participants, the 2013 RSA-911 report showed 10 had hourly earnings equal to or
greater than California’s minimum wage and 1 earned below it. Again, when we evaluated the
contradicting data found in the case files, we ultimately determined that had the data in the case
files been used to determine whether any of the 11 participants earned more or less than the
minimum wage, the results would have been the same; that is, 10 participants would have earned
at or above the minimum wage and 1 would have earned below it. However, the existence of
contradicting data remains a problem in and of itself.

Because of the issues of unverifiable employment data, we were unable to estimate error rates for
the employment-related data elements and, thus, could not gauge the effect that errors may have
had on case closures with employment outcomes or the performance indicators that are directly
linked to employment-related data elements. When we identified contradicting information
related to start dates, weekly earnings at closure, or hours worked in a week at closure in case
files or case notes in AWARE, we determined that the participant’s status on the data elements
generally remained unchanged. However, DOR could better ensure the accuracy and
21
  We are 95 percent confident that the estimates for cases with unverifiable employment start dates, weekly
earnings at closure, and hours worked in a week at closure is between 40 and 60 percent, 45 and 66 percent, and 43
and 64 percent, respectively.
Final Report
ED-OIG/A09O0008                                                                    Page 22 of 34

completeness of performance indicator data reported in its RSA-911 reports by maintaining
supporting documentation for each participant that shows the information needed to record the
various employment-related data elements in AWARE. Implementing such controls would
provide RSA with higher quality data with which to assess DOR’s performance on the evaluation
standards—which would then help RSA determine whether DOR needs to develop a program
improvement plan and tailor its oversight and monitoring.

Recommendation

We recommend that the Commissioner of RSA require DOR to—

2.1    Develop and implement policies and procedures that require VR team members to
       document the employment start date, weekly earnings at closure, and hours worked in a
       week at closure in applicable participants’ case files or AWARE using third-party
       documentation or DOR staff case notes. The policies and procedures should also require
       supervisory review of the employment related data entered in the AWARE system and
       the supporting documentation.


DOR Comments and OIG Response

DOR disagreed with this finding and the associated recommendation.

DOR Comments
As DOR stated in its response to Finding No. 1, Federal regulations delegate to designated State
units the determination of the type of documentation required for the record of services. DOR
considers the data entered into AWARE, including data entered on the employment screen
verifying employment and the details of that employment, to be a participant’s record of services
and does not believe further verification or corroboration of this data is required. DOR also
noted that it requires staff to record the source of each participant’s employment data that they
enter. In addition, DOR discussed implementing a participant verification process in which
employment data would be confirmed by participants at the time their cases were closed.

OIG Response
As stated in the finding, RSA has taken the position that the requirements in 34 C.F.R. § 361.47
and § 361.56, taken together, require State VR agencies to maintain verifying documentation in
the participant case file related to the participant’s employment outcome, including the
employment start date, that the participant maintained employment for 90 days, weekly earnings
at closure, and hours worked in a week at closure.

Entering data into AWARE, including employment data in the AWARE employment screen,
does not create corroborating or supporting documentation. It is instead merely a data entry
process that is susceptible to data entry errors. Requiring staff to note the source of the
employment data also does not help improve data reliability. Implementing a participant data
verification process for employment data would be effective only if DOR could achieve high
response rates from participants. Also, DOR should try to obtain third-party employment data
for participants, such as documentation from employers, because that is generally the most
Final Report
ED-OIG/A09O0008                                                                     Page 23 of 34

reliable source for verification and assures management and others of the accuracy of
information maintained in the system.

We found that 11 participants’ cases in our sample of 95 case closures with employment
outcomes (cases that were not incorrectly reopened and closed during the 2013 reporting period)
had contradicting employment start dates in the participants’ case files and AWARE. Also, we
identified another 11 participant cases from the same sample that had contradicting weekly
earnings at closure or hours worked in a week at closure data in the participants’ case files and
AWARE.

Based on those instances of contradicting information in participants’ case files and the
significant error rates our testing showed across numerous data elements that RSA uses to
calculate performance indicators, DOR staff should maintain documentation to support their data
entry in the AWARE system so that team managers can use it to verify data and help ensure the
reliability of data that DOR reports annually to RSA.
Final Report
ED-OIG/A09O0008                                                                   Page 24 of 34



                 OBJECTIVES, SCOPE, AND METHODOLOGY


The audit objectives were to determine whether DOR (1) had adequate internal controls to
provide reasonable assurance that the reported “Case Service Report” (RSA-911 report) data
were accurate and complete and (2) reported RSA-911 performance indicator data that were
accurate, complete, and adequately supported. Our review covered DOR’s 2013 RSA-911 report
for the reporting period October 1, 2012, through September 30, 2013.

To achieve our audit objectives, we performed the following procedures:
      Reviewed the Rehabilitation Act of 1973, Federal regulations at 34 C.F.R. Part 361, and
       34 C.F.R. Parts 76 and 80; and California Code of Regulations, Title 9, Division 3,
       Department of Rehabilitation, to gain an understanding of the requirements that DOR
       was to follow when administering the VR program.
      Reviewed multiple RSA documents including “RSA FY 2009 Monitoring: State
       Vocational Rehabilitation and Independent Living Programs Information Guide,” RSA’s
       FY 2009 monitoring report on DOR, and two special condition letters that RSA issued to
       DOR as a result of its on-site monitoring review in June 2014.
      Reviewed the State of California’s single audit reports for FYs 2009–2010 through
       FYs 2012–2013.
      Interviewed appropriate DOR officials to gain an understanding of the AWARE system
       and DOR’s procedures for collecting, reviewing, verifying, and submitting the RSA-911
       report data. We also interviewed team managers; qualified rehabilitation professionals
       (counselors); employment and service coordinators; the technical operations manager;
       and the assistant chief of claims in accounting services.
      Reviewed DOR’s administrative manual for policies and procedures and other guidance
       to gain an understanding of the processes related to: preparing and submitting the RSA-
       911 report, providing eligible participants with services to help them obtain or retain
       employment, and ensuring that VR participant case file documents were properly
       completed and maintained in participants’ files and were properly recorded in the
       AWARE system.
      Performed limited testing of the AWARE system controls including a review of the
       system security levels.
      Reviewed a statistical sample of case closures reported in DOR’s final 2013 RSA-911
       report to evaluate whether selected data elements that DOR reported to RSA were
       accurate, complete, and supported by documentation.
      Reviewed the COSO Report for the components of internal controls, including
       monitoring activities.
Final Report
ED-OIG/A09O0008                                                                        Page 25 of 34

Sampling Methodology

To assess the accuracy of case closure data elements reported to RSA, we tested a stratified
random sample from the universe of case closures. We obtained the universe of case closures
DOR reported to RSA from its 2013 RSA-911 report submission. The universe consisted of case
closures and data elements related to the participants and closure outcomes.

Due to data discrepancies we discovered in DOR’s submission, which we discuss further under
the Data Reliability section, we decided to divide the universe into two subpopulations: those
cases with known data discrepancies and those without known data discrepancies. We further
stratified the subpopulations based on whether the cases were closed with an employment
outcome or without an employment outcome. Ultimately, we selected a stratified random
sample from four strata having universe and sample sizes shown in Table 3, with Strata 1 and 2
representing the subpopulation with known data discrepancies (86 combined records), and Strata
3 and 4 representing the subpopulation without known data discrepancies (47,270 combined
records). We established the sample size so that our estimates would generally have at most a
margin of error of plus or minus 10 percent at the 95 percent confidence level for both closures
with employment outcomes and all closures.

Table 3. Universe and Sample Size of Closed Cases Reported in DOR’s RSA-911 Report
Submission for the 2013 Reporting Period
  Stratum                      Type of Closure                    Universe          Sample
               Cases closed with an employment outcome
      1                                                                 58              5
               (with known data discrepancies)*
               Cases closed without an employment outcome
      2                                                                 28              5
               (with known data discrepancies)*
               Cases closed with an employment outcome
      3                                                             12,181             95
               (without known data discrepancies)
               Cases closed without an employment outcome
      4                                                             35,089             57
               (without known data discrepancies)
               Total                                                47,356            162
* Contained known discrepancies where cases closed during the 2012 reporting period were improperly
reopened and closed again during the 2013 reporting period.

We tested selected case closures by reviewing supporting documentation, including
DOR-generated forms and employment information. We verified the data values reported on the
RSA-911 report by comparing reported values to values in supporting documentation. If
supporting documentation was not available, we considered the data entry to be unverifiable.
When evaluating the accuracy and reliability of the performance indicators, we attempted to
verify key data elements for the performance indicator. If we were unable to verify the key data
element in the performance indicator, we considered the data element not verifiable.

We reviewed the 162 case files to determine whether DOR properly completed and maintained
the required participant case file documents (Application for VR Services, Notice of Eligibility
and Priority for Services, IPE, and Closure Report) in the participants’ case files. See
Attachment 4 for the required documents according to the participant’s closure type. Further, for
the 100 sampled cases with employment outcomes (Strata 1 and 3), we determined whether
source documentation was maintained in the case file to support the employment start date, hours
Final Report
ED-OIG/A09O0008                                                                         Page 26 of 34

worked in a week at case closure, weekly earnings at case closure, and that the participant
maintained employment for 90 days.

For the 100 cases with an employment outcome (Strata 1 and 3), we also determined whether the
following data elements were correct and adequately supported according to source documents
maintained in DOR’s AWARE system and hardcopy case files: Social Security number, date of
birth, race and ethnicity, application date, eligibility determination date, disability priority, IPE
date, services provided, employer name, employment start date, weekly earnings at closure,
hours worked in a week at closure, closure date, and type of closure.

For the 62 cases without an employment outcome (Strata 2 and 4), we determined whether the
following data elements were correct and adequately supported according to source documents
maintained in DOR’s AWARE system and hardcopy case files: Social Security number, date of
birth, race and ethnicity, application date, eligibility determination date, disability priority, IPE
date, services provided, closure date, and type of closure. Not all of the data elements were
required depending on the type of closure for the 62 cases.

For the 162 cases, we did not review primary support at application and primary support at
closure data elements because they affected only one performance indicator (Performance
Indicator 1.6). We limited our work for employment status to verifying that participants with
employment outcomes had a code in the 2013 RSA-911 report identifying the data element as a
record to be used in performance indicators 1.3 to 1.6.

We calculated all estimates using sampling weights so that estimates reflect the intended
population. As an example, some attributes that we tested were applicable only to cases closed
with an employment outcome and those estimates are projected to the employment outcome case
closures. Because we followed a probability procedure based on random selections, our sample
is only one of a large number of samples that we might have drawn. Each sample could have
provided different estimates, so we express our confidence in the precision of our particular
sample’s results as a 95 percent confidence interval. This is the interval that would contain the
actual population value for 95 percent of samples we could have drawn. All percentage
estimates from the review have margins of error of plus or minus 10 percentage points or less at
the 95 percent confidence level unless otherwise noted.

Data Reliability

To verify the completeness of the data in DOR’s final 2013 RSA-911 report submission, we
compared closed case data for the 2013 reporting period from the AWARE system (September
2014 extract) to the universe of closed cases reported in DOR’s final 2013 RSA-911 report
submission. We found fewer records (closed cases) in the 2013 RSA-911 report than in
AWARE. According to DOR staff, the record counts differed because some of the closed cases
from the previous reporting period were reopened and closed again during the 2013 reporting
period. We describe this information system control weakness in Finding No. 1. After further
analysis, we were able to verify that the 2013 RSA-911 report was complete.

Because the reopening and subsequent reclosing of cases resulted in record count discrepancies
between the AWARE data and the RSA-911 report submission we included two additional strata
in our sample—those with known data discrepancies. One additional stratum included only
Final Report
ED-OIG/A09O0008                                                                     Page 27 of 34

closed cases with an employment outcome (58 cases) from the 86 closed cases that were reported
in both the 2012 and 2013 reporting periods. A second additional stratum included only closed
cases without an employment outcome (28 cases) from the same subpopulation. The sample
sizes selected from each group are described above under “Sampling Methodology.”

We performed a recalculation of the performance indicators using data from DOR’s 2013 RSA-
911 report final submission and verified the numbers and percentages RSA had calculated and
published. However, we were not able to recalculate DOR’s performance indicator scores using
the September 2014 data extract of 2013 closed cases due to the issues noted above.

We found that VR case service data maintained in DOR’s AWARE system and reported on its
2013 RSA-911 report final submission, including performance indicator data, were not always
correct and were often unverifiable. Therefore, we were unable to determine the reliability of the
underlying data or the performance indicators used by RSA to assess the VR agency’s
performance against the required evaluation standards (see Findings No. 1 and 2).

We performed our audit procedures at DOR’s offices in Sacramento, Gardena, and Long Beach,
California, from August 26, 2014, through January 28, 2015. We performed additional audit
procedures at our offices from August 2014 through August 2015. We held an exit conference
with DOR officials on June 10, 2015.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objectives.
Final Report
ED-OIG/A09O0008                                                                      Page 28 of 34



                            ADMINISTRATIVE MATTERS


Statements that managerial practices need improvements, as well as other conclusions and
recommendations in this report, represent the opinions of the Office of Inspector General.
Determinations of corrective action to be taken will be made by the appropriate Department of
Education officials.

This report incorporates the comments that you provided in response to the draft audit report. If
you have any additional comments or information that you believe may have a bearing on the
resolution of this audit, you should send them directly to the following U.S. Department of
Education official, who will consider them before taking final Departmental action on this audit:

                                          Janet LaBreck
                                         Commissioner
                             Rehabilitation Services Administration
                                U.S. Department of Education
                               550 12th Street, SW, Room 5086
                                   Washington, D.C. 20202

It is the policy of the U. S. Department of Education to expedite the resolution of audits by
initiating timely action on the findings and recommendations contained therein. Therefore,
receipt of your comments within 30 calendar days would be appreciated.

In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the
Office of Inspector General are available to members of the press and general public to the extent
information contained therein is not subject to exemptions in the Act.

We appreciate the cooperation and assistance extended by your employees during our audit. If
you have any questions or require additional information, please do not hesitate to contact me at
(916) 930-2399 or Beverly Dalman, Assistant Regional Inspector General for Audit, at (916)
930-2393.


                                              Sincerely,

                                              /s/

                                              Raymond Hendren
                                              Regional Inspector General for Audit



Attachments
Final Report
ED-OIG/A09O0008                                                              Page 29 of 34

                                                                             Attachment 1

           Abbreviations, Acronyms, and Short Forms Used in This Report

  AWARE                   Accessible Web-Based Activity Reporting Environment

  Administrative Manual   DOR’s Rehabilitation Administrative Manual

  BEP                     Business Enterprise Program

  C.F.R.                  Code of Federal Regulations

  COSO Report             The Committee of Sponsoring Organizations of the Treadway
                          Commission’s, “Internal Control-Integrated Framework”

  DOR                     California Department of Rehabilitation

  FY                      Fiscal Year

  IPE                     Individualized Plan for Employment

  RSA                     Rehabilitation Services Administration

  RSA-911Reporting        Reporting Manual for the Case Service Report
  Manual

  RSA-911                 Case Service Report

  VR                      Vocational Rehabilitation

  WIOA                    Workforce Innovation and Opportunity Act of 2014
Draft Report
ED-OIG/A09O0008                                                                                                                              Page 30 of 34

                                                                                                                                             Attachment 2

  Diagram:      The VR Process and its Related Participant Outcomes, Required Case File Documents, and Performance
                Indicators


        Step in VR Process                                                       Required Documents in                           Performance
                                           Case Closure Type
                                                                                       Case File                              Indicators Affected


        Application is completed           Exit VR as applicant (code 1)         Application and closure report                        2.1



        Trial work
                                           Exit VR during or after a trial       Application and closure report                        2.1
        experience/extended
                                           work experience or extended
        evaluation
                                           evaluation (code 2)

                                           Exit VR from an order of
        Eligibility is determined          selection waiting list (code 6)
                                                                                  Application, notice of eligibility and
        and participant is assigned                                                                                                    2.1
                                                                                  priority for services, and closure report
        to a disability priority           Exit VR without employment
        category                           after eligibility but before an IPE
                                           was signed (code 7)


                                            Exit VR without employment,            Application, notice of eligibility and
        IPE is signed                       after a signed IPE, but before         priority for services,                              2.1
                                            receiving services (code 5)            IPE, and closure report


                                                                                   Application, notice of eligibility and
        Services provided to participant     Exit VR without employment,                                                              1.2 and 2.1
                                                                                   priority for services,
                                             after receiving services (code 4)
                                                                                   IPE, and closure report


                                                                                   Application, notice of eligibility and              1.1, 1.2, 1.3,
        Participant is employed for 90       Exit VR as employed (code 3)          priority for services, IPE, and                     1.4, 1.5, 1.6,
        days and exits the VR program                                              closure report                                      and 2.1
Draft Report
ED-OIG/A09O0008                                                                                                                                              Page 31 of 34

                                                                                                                                                          Attachment 3

Results of Verification of Sampled 2013 DOR Closures from RSA-911 Submission of Data Elements
                                                                                         Data Elements
                                                                                                                                                   *†
   Closure                                                                                                                             *†       Hours
    Type                                                                                                                              Weekly    Worked
                     Social                                         * Eligibility       *†                 * Cost of       *†        Earnings     in a
                    Security   Date of   † Race &    Application   Determination    Significant   * IPE   Purchased    Employment       at      Week at    Closure   † Closure
                    Number      Birth    Ethnicity     Date            Date         Disability    Date     Services     Start Date   Closure    Closure     Date       Type
     Type 3
Verified                 100        97         95            85               69           100       21          95             40         36        38         42          94
Not Verified               0         0          0            15               15             0        6           5             48         53        51         10           0
Incorrect                  0         3          5             0               16             0       73           0             12         11        11         48           6
            Total        100       100        100           100              100           100      100         100            100        100       100        100         100

 All Other Types
Verified                  61        62         60            50                40           59       12          41              0          0         0         25          52
Not Verified               0         0          0            12                 7            1        6           3              0          0         0          5           0
Incorrect                  1         0          2             0                 9            2       20           0              0          0         0         32          10
* Not Required             0         0          0             0                 6            0       24          18             62         62        62          0           0
            Total         62        62         62            62                62           62       62          62             62         62        62         62          62

    All Types
Verified                 161       159        155           135              109           159       33         136             40         36        38         67         146
Not Verified               0         0          0            27               22             1       12           8             48         53        51         15           0
Incorrect                  1         3          7             0               25             2       93           0             12         11        11         80          16
* Not Required             0         0          0             0                6             0       24          18             62         62        62          0           0
            Total        162       162        162           162              162           162      162         162            162        162       162        162         162
* These data elements may not necessarily be required depending on the closure type.
† These elements may have a direct effect on the performance indicators used to evaluate program performance.
Draft Report
ED-OIG/A09O0008                                                                        Page 32 of 34

                                                                                     Attachment 4

  Documents Required to be Maintained in the Participant’s Case File Based on
  Applicable Type of Closure Code
                                                          Documents Required by DOR to be
                                                       Maintained in the Participant’s Case File
                                                                   Notice of
                                                                  Eligibility
                   Closure Type
                                                                       &
                                                                   Priority
                                                                      for                 Closure
                                                     Application   Services       IPE      Report
   1: Exited as an applicant                            Yes           No           No        Yes
   2: Exited during or after a trial work
                                                        Yes            No            No      Yes
   experience/extended evaluation
   3: Exited with an employment outcome                 Yes            Yes           Yes     Yes
   4: Exited without an employment outcome, after
                                                        Yes            Yes           Yes     Yes
   receiving services
   5: Exited without an employment outcome, after
                                                        Yes            Yes           Yes     Yes
   a signed IPE, but before receiving services
   6: Exited from an order of selection waiting list*   Yes            Yes           No      Yes
   7: Exited without an employment outcome, after
                                                        Yes            Yes           No      Yes
   eligibility, but before an IPE was signed
  * DOR did not have any cases with Closure Type 6 during the 2013 reporting period.
 Draft Report
 ED-OIG/A09O0008                                                                                                                                           Page 33 of 34

                                                                                                                                                           Attachment 5

 Data Elements Required According to the Participant’s Type of Closure Code
                                                                                                                                    Weekly       Hours
                       Social              Race                     Eligibility                              Total                  Earnings   Worked in
   Closure            Security   Birth     and       Application   Determination   Significant              Service   Employment       at      a Week at   Closure   Closure
    Type              Number     Date    Ethnicity     Date            Date        Disability    IPE Date    Cost      Start Date   Closure     Closure     Date      Type
1: Exited as an
                        Yes      Yes       Yes          Yes             No            No           No         No          No          No          No        Yes       Yes
applicant
2: Exited during or
after a trial work
experience/             Yes      Yes       Yes          Yes             No            No           No         No          No          No          No        Yes       Yes
extended
evaluation
3: Exited with an
employment              Yes      Yes       Yes          Yes            Yes            Yes          Yes       Yes         Yes          Yes        Yes        Yes       Yes
outcome
4: Exited without
an employment
                        Yes      Yes       Yes          Yes            Yes            Yes          Yes       Yes          No          No          No        Yes       Yes
outcome, after
receiving services
5: Exited without
an employment
outcome, after
                        Yes      Yes       Yes          Yes            Yes            Yes          Yes        No          No          No          No        Yes       Yes
signed IPE, but
before receiving
IPE services

6: Exited from an
order of selection      Yes      Yes       Yes          Yes            Yes            Yes          No         No          No          No          No        Yes       Yes
waiting list *

7: Exited without
an employment
outcome, after
                        Yes      Yes       Yes          Yes            Yes            Yes          No         No          No          No          No        Yes       Yes
eligibility, but
before an IPE was
signed
 * DOR did not have any cases with Closure Type 6 during the 2013 reporting period.
Draft Report
ED-OIG/A09O0008                             Page 34 of 34

                                           Attachment 6




                   Auditee Comments on
                  the Draft Audit Report
                                                              Edmund G. Brown Jr.,
                                                                  Governor




                                                                  State of California
                                                          Health and Human Services Agency

                                                               Office of the Director
                                                                  721 Capitol Mall
                                                              Sacramento, CA 95814
Mr. Raymond Hendren                                           (916) 558-5802 VOICE
                                                               (916) 558-5806 FAX
Regional Inspector General for Audits                           (916) 558-5807 TTY
United States Department of Education
Office of Inspector General
501 I Street, Suite 9-200                                  September 25, 2015
Sacramento, California 95814-2559
Via E-mail (ray.hendren@ed.gov)

RE: Response to Draft Audit Report (ED OIG/A09O0008)

Dear Mr. Hendren:

The California Department of Rehabilitation (DOR) is in receipt of the draft audit
report, dated August 25, 2015, and provides the below responses to the findings
and recommendations. The DOR’s response addresses the concerns raised in
the draft report by providing detailed information on actions that DOR has or will
be taking to improve its processes. While DOR appreciates the Office of
Inspector General’s review of its vocational rehabilitation program for the RSA-
911 reporting period October 1, 2012 through September 30, 2013, based on our
discussions it appears that there is mutual recognition of the significant changes
to DOR’s processes since 2013, as part of DOR’s continuous improvement
efforts. In addition, DOR also noted that the Workforce Innovation and
Opportunity Act of 2014 (WIOA) has made significant amendments to the federal
laws governing vocational rehabilitation programs, including performance
measures, prompting changes to the collection and reporting of RSA-911 and
other data to the federal government. As such, the recommendation that the
Rehabilitation Services Administration require DOR to take action to address
concerns from the Federal Fiscal Year 2012-13 is not warranted in light of the
actions taken to date, and new requirements under WIOA.

Finding No. 1 – DOR’s Internal Controls Did Not Provide Reasonable
Assurance That All 2013 Reported RSA-911 Data Were Accurate, Complete,
or Supported
Mr. Hendren
September 25, 2015
Page 2 of 11

DOR Response:
The DOR agrees, in part, to Finding No. 1, and disagrees, in part to this Finding
and related recommendations as follows:

1.1   Information System Control Weakness Allowed Changes to Closed Cases

 DOR did not have a system control to prevent VR staff from reopening closed
  cases and editing participant records, including closure dates and types, in the
  AWARE system.

DOR Response:
The DOR agrees with this finding; however, since Federal Fiscal Year 2013,
DOR has taken significant steps to address this system control weakness as
indicated below.

The DOR currently completes a data review, consisting of a data analysis of
closed cases that identifies any case noted to have been open and closed in
more than one Federal Fiscal Year. These cases are reviewed for appropriate
case management and that the appropriate closure is noted in the correct
Federal Fiscal Year. This review occurs prior to completion of the RSA-911
report to ensure case closures are reported in the correct RSA 911 report.

To ensure consumer cases are not opened solely for administrative reasons,
DOR staff received a written reminder in May 2015 instructing them that cases
shall not be reopened to pay outstanding bills (refer to May 2015 memo (see
Attachment 01)) entitled, “Invoice Payment After Record of Service Closure.”

1.2   Weak Data Quality Controls Resulted in RSA-911 Reporting Issues

1. DOR Did Not Require Staff to Maintain Documentation to Corroborate Data
   Entered Into the Case Management System

DOR Response:
The DOR disagrees with this finding. As noted in the Office of Inspector
General’s August 25, 2015 draft audit report, “Although the requirements do not
specify the type of verifying documentation that must be maintained, the
Mr. Hendren
September 25, 2015
Page 3 of 11

regulations show that VR agencies need to have some type of supporting
documentation for the employment data in the case file.”
Pursuant to title 34 Code of Federal Regulation section 361.47(b), the designated
state unit, in consultation with the State Rehabilitation Council, must determine
the type of documentation required for the record of services, including eligibility
determination, plan services, closure and employment outcome. In California,
the Accessible Web-based Activity Reporting Environment (AWARE) case
management system is an individual’s record of services. The DOR’s policies
and procedures, as provided in the Rehabilitation Administrative Manual (RAM)
Chapter 30 and in compliance with title 34 Code of Federal Regulation section
361.47(b), requires staff to document the date of application, employment start
date, weekly earnings and hours worked at closure. This information must be
recorded in the AWARE case management system, and DOR considers reported
data from the participant or other third party source to be valid, sufficient
supporting documentation that does not require further verification or
corroboration. The DOR also notes that there may be documents received from
the participant and other third parties (e.g., job placement report) that are not
uploaded into AWARE but are maintained in a hard copy file as part of the record
of services.

2. DOR Did Not Require Staff to Ensure That Eligibility Determination and
   Closure Notification Dates Reflected the Dates Reported in the Case
   Management System

DOR Response:
The DOR disagrees with this finding. During the audit period, DOR did, and
continues to, require that staff ensure eligibility determination and closure
notification dates are consistent with the dates reported in the AWARE case
management system as noted in the RAM 30002 (01/09), and the California
Code of Regulations, title 9, section 7122.

The DOR acknowledges that during the audit period, the completion and
approval of the requisite AWARE screens for eligibility determination and closure
and the printing and mailing of the notice(s) to consumers was a two-step
process, resulting in noted date inconsistencies.
Mr. Hendren
September 25, 2015
Page 4 of 11

In April 2014, DOR implemented system changes that significantly enhanced
data integrity to address date discrepancies on eligibility and closure report dates
in participant case files and AWARE through the automation of case activity
process and forms. The following forms have been automated in AWARE
version 5.17, which is scheduled to be implemented at the end of 2015:

•   DR 211 (Extension of Eligibility and Priority for Services)
•   DR 212 (Notice of Eligibility and Priority for Services)
•   DR 229A (Closure - Rehabilitated)
•   DR 229B (Closure - Other than Rehabilitated)

With the automation, AWARE requires that eligibility and closure reports be
printed directly from the respective AWARE data pages, which ensures date
correlation and eliminates discrepancies between the participant case file and
AWARE.

3. DOR Did Not Have Policies and Procedures for Staff to Report Data in
   Accordance With RSA Guidelines

DOR Response:
The DOR agrees with this finding related to (a) “Date of Individualized Plan for
Employment (IPE),” but disagrees with this finding related to (b) “Cost of
Purchased Services.”

(a) The DOR shall clarify through written guidance to all field staff, the IPE
   effective date as defined in title 34 Code of Federal Regulations, section
   361.45(d)(3) by December 31, 2015. The DOR will ensure, through this
   guidance and ongoing periodic review of consumer case records, including
   the 10 percent review of all case records of counselors with “approval
   authority,” that the required signatures are and have been obtained and that
   the AWARE case record reflects the appropriate IPE effective date.

(b) The total costs for purchased services in AWARE are calculated correctly
   during each applicable Federal Fiscal Year and are captured at the time of the
   year-end for RSA-911 report completion and submission. The DOR’s RSA-
   911 report is generated at a “point in time” and is accurate at the time of the
Mr. Hendren
September 25, 2015
Page 5 of 11

  report run. Each individual AWARE case record is accurate and complete as
  it captures all ongoing case service costs throughout the life of the case.

What was noted by the auditors is a discrepancy between the calculated, correct
costs in the AWARE system, which represent a point in time, and the current
case record which reflects true and accurate expenditures at the time of the
review, including changes necessitated by refunds or late invoices received and
paid by DOR after report submission.

In order to minimize the number of post-fiscal year and post-case closure
adjustments to fiscal information in consumer case records, DOR is including
more restrictive contractual language requiring that vendors submit invoices
within 60 days.

All vendor invoices are processed through each consumer’s record of services in
AWARE. This control ensures that no participant costs are captured outside of
the AWARE system. The DOR cannot pay an invoice without first having an
authorizing document in AWARE.

4. DOR’s Monitoring Did Not Provide Assurance That Data Were Accurate and
   Required Documentation Was Maintained in Participant Files

DOR Response:
The DOR agrees with this finding, however, as noted in the August 25, 2015 draft
audit report, DOR has already implemented a comprehensive case monitoring
program for its Rehabilitation Counselors (RC) with “approval authority” in
accordance with RAM 12050 and 12060. In July 2014, DOR implemented formal
policy and controls that requires an annual 10 percent caseload record of service
review for all RCs with “approval authority.” Team Managers continue to be
required to review and approve all casework for RCs without “approval authority.”
The review results are communicated to management for consideration and
determination of appropriate next steps.

OIG Recommendations

Office of the Inspector General (OIG) recommends that the Commissioner of
RSA require DOR to—
Mr. Hendren
September 25, 2015
Page 6 of 11


DOR Response:
As discussed above, the audit report period is from Federal Fiscal Year 2012-13
and DOR, as part of its continuous improvement efforts, has implemented
significant changes to processes that address many of the concerns raised.
Furthermore, DOR has a number of additional enhancements in progress.
Therefore, at this point in time, further requirements are unnecessary.

1.1   Implement system controls that prevent VR staff from altering closure dates
      in the AWARE system unless the case was mistakenly closed.

DOR Response:
The DOR agrees with this recommendation and has already taken steps to
ensure that these controls are in place and that they are effective. For the
Federal Fiscal Year 2014-15 (ending September 30, 2015), DOR will evaluate
the effectiveness of these controls through the review of the RSA-911 data report
to identity any case records closed and opened in multiple fiscal years. Those
cases will be reviewed to ensure that the noted case activity was appropriate and
correct. Any cases reopened incorrectly will be returned to the appropriate Team
Manager for review and correction. A summary report of the case closure record
review will be provided to the appropriate Assistant Deputy Director and Deputy
Directors of the Specialized Services and Vocational Rehabilitation Employment
Divisions by December 31, 2015.

1.2 Establish and implement controls that ensure that the dates for the notice of
    eligibility and closure reports are accurate in AWARE and that any
    discrepancies between the dates in participant case files and AWARE are
    fully explained.

DOR Response:
As noted above, DOR agrees with this recommendation and is taking additional
steps to further clarify instruction to staff, through the drafting and dissemination
of written memos and clear procedural guidance language in RAM. The
guidance and procedural clarification memo will be provided to all field staff by
December 31, 2015. Changes to the RAM will occur no later than June 30,
2016.
Mr. Hendren
September 25, 2015
Page 7 of 11

In April 2014, DOR implemented system controls that significantly enhanced data
integrity to address date discrepancies on eligibility and closure report dates in
participant case files and AWARE. The following forms have been automated in
AWARE version 5.17, which is scheduled to be implemented on or about
December, 2015:

•   DR 211 (Extension of Eligibility and Priority for Services)
•   DR 212 (Notice of Eligibility and Priority for Services)
•   DR 229A (Closure - Rehabilitated)
•   DR 229B (Closure - Other than Rehabilitated)

This enhanced AWARE control requires that eligibility and closure reports be
printed directly from the respective AWARE data pages, which will ensure date
correlation and eliminate discrepancies between the participant case file and
AWARE. Additionally, DOR is researching system enhancements to eliminate
the two-step process so that printing of forms is required before exiting the data
page. This research and proposed recommendations for changes will be
completed by June 30, 2016, and presented to the appropriate Assistant Deputy
Directors and Deputy Directors of the Specialized Services and Vocational
Rehabilitation Employment Divisions for approval.

1.3    Establish and implement controls to ensure that DOR staff record the
       correct date of the IPE (the latter of the counselor or participant signature
       date) in the AWARE system and account for all participant costs in
       AWARE.

DOR Response:
The DOR agrees to the first part of this recommendation. As noted above, DOR
will provide clear written guidance to all field staff on the proper date to use as
the IPE effective date as defined in title 34 Code of Federal Regulations section
361.45(d)(3). The DOR will ensure, through this guidance and periodic review of
consumer case records, that the required signatures are and have been obtained
and that the AWARE case record reflects the appropriate IPE effective date.
This written guidance will be provided to all field staff by December 31, 2015.
Mr. Hendren
September 25, 2015
Page 8 of 11

The DOR accounts for all participant costs in AWARE. The DOR will continue to
ensure that costs in the case record are true and accurate as required by
applicable state and federal regulations, policies, and requirements.

1.4   Establish and implement controls to ensure that closures are reviewed
      timely and case files contain all required documentation for the closure
      type.

DOR Response:
The DOR agrees to this recommendation, particularly in light of new
requirements resulting from changes due to WIOA. The DOR’s processes are
continuously examined, and closure policies and procedures revised accordingly,
to ensure alignment with the new WIOA “exit” requirements. Case file reviews
will continue to occur and case review summaries will continue to be reported to
central office staff twice annually. Additionally, by December 31, 2015, DOR will
institute and communicate a policy for timely Team Manager review of closures,
(e.g., within 15 days) that ensures the closure type is accurate and case
documentation supports the closure.

1.5   Provide evidence demonstrating that its recently implemented policies and
      procedures related to the date of application are effective in ensuring that
      team members are documenting when all requirements have been met to
      establish an accurate application date.

DOR Response:
AWARE requires staff to confirm that all three requirements for application are
met before entering the application date in the AWARE system, which is
inherently evidentiary. The attached screen print illustrates DOR’s internal
documentation process (see Attachment 02). As DOR considers the AWARE
electronic record as part of the record of services, DOR does not require
duplicative documentation to verify the three criteria for application have been
met. The DOR can provide a report on the implemented policies and procedures
described above.

1.6   Provide evidence demonstrating that its newly implemented policies and
      procedures related to manager monitoring of counselors with approval
      authority are effective in ensuring that managers will routinely identify
Mr. Hendren
September 25, 2015
Page 9 of 11

     missing documents and missing signatures or dates on documents,
     including notices of eligibility, IPEs, and closure reports.

DOR Response:
The DOR agrees with this recommendation and, if requested, will be able to
report on the newly implemented policies and procedures. In accordance with
RAM 12050 and 12060, DOR implemented formal policy and controls that
require an annual 10 percent caseload record of service review for all RCs with
“approval authority.” This policy is inherently evidentiary to ensure routine
monitoring of RC casework and a complete record of services exists. Team
Managers continue to be required to review and approve all casework for RCs
without “approval authority.”

The DOR’s Customer Service Unit completes and provides a summary report of
the record of services review to the appropriate Assistant Deputy Director and
Deputy Directors of the Specialized Services and Vocational Rehabilitation
Employment Divisions within 90 days of completion of these reviews.
Finding No. 2 – Performance Indicator Data Reported in DOR’s 2013
RSA-911 Report Were Not Supported

The employment data reported in DOR’s 2013 RSA-911 report for employment
start date, weekly earnings at closure, and hours worked in a week at closure
were not, in many instances, supported by documentation in participant case files
or AWARE. In addition, DOR did not have any procedures requiring supervisors
to review employment related data entered into participant records or any
underlying documentation supporting the AWARE employment screens.

DOR Response:
The DOR disagrees with Finding No. 2 and Recommendation 2.1.

In addition to DOR’s response to Finding 1.2, items 1 and 4, DOR considers the
data entered into the AWARE Employment Page as documentation to verify
requisite employment data elements. The AWARE Employment Page requires
staff to input the source of the participant’s employment data (see Attachment
03).

OIG Recommendation
Mr. Hendren
September 25, 2015
Page 10 of 11


OIG recommends that the Commissioner of RSA require DOR to—

DOR Response:
As discussed above, the audit report period is from the Federal Fiscal Year 2012-
13 and DOR, as part of its continuous improvement efforts, has implemented
significant changes to processes that address many of the concerns raised.
Furthermore, DOR has a number of additional enhancements in progress.
Therefore, at this point in time further direction is unnecessary.

2.1   Develop and implement policies and procedures that require VR team
      members to document the employment start date, weekly earnings at
      closure, and hours worked in a week at closure in applicable participants’
      case files or AWARE using third-party documentation or DOR staff case
      notes. The policies and procedures should also require supervisory review
      of the employment related data entered in the AWARE system and the
      supporting documentation.

DOR Response:
The DOR considers the data entered into the AWARE Employment Page as the
official documentation verifying employment and the details of that employment.
The AWARE Employment Page requires staff to document the source of the
information being recorded in the case record of the participant’s employment
data (see Attachment 03). This may include referencing “third-party” verification,
such as documentation received from a services provider, or information
received directly from a consumer either verbally or in writing. As with all other
closures, Team Manager’s review and approval of all case closures remains a
requirement.

Additionally, based on discussions with the Office of Inspector General, DOR
agrees that consistently providing a summary of the recorded employment data
in AWARE to all consumers at the time of case closure is an effective method of
further ensuring that correct information has been recorded. This information
would provide the consumer an opportunity to update the information on record
or to corroborate third-party reports. The DOR will contact the AWARE software
provider to determine the time and cost necessary to develop and implement this
enhancement by December 31, 2015. Provided resources are available, DOR
Mr. Hendren
September 25, 2015
Page 11 of 11

would implement this enhancement prior to the end of Federal Fiscal Year 2015-
16.

Thank you for this opportunity to provide comments on the August 25, 2015
draft audit report. Should you have any questions, please contact Suzanne
Chan, Operations and Accountability Officer at suzanne.chan@dor.ca.gov
or (916) 558-5797.

Sincerely,

Original signature on file

Juney S. Lee
Chief Deputy Director

Attachments

cc:   Theresa Correale, Deputy Director, Administrative Services, DOR
      Suzanne Chan, Operations and Accountability Officer, DOR