oversight

HUD's Oversight of Its Moving to Work Demonstration Program Needs Improvement

Published by the Department of Housing and Urban Development, Office of Inspector General on 2013-09-27.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

OFFICE OF AUDIT
REGION 3
PHILADELPHIA, PA




           U.S. Department of Housing and Urban
                       Development
                      Washington, DC

          Moving to Work Demonstration Program




2013-PH-0004                            SEPTEMBER 27, 2013
                                                        Issue Date: September 27, 2013

                                                        Audit Report Number: 2013-PH-0004




TO:            Dominique G. Blom, Deputy Assistant Secretary for Public Housing
                 Investments, PI
               //signed//
FROM:          John P. Buck, Regional Inspector General for Audit, Philadelphia Region, 3AGA


SUBJECT:       HUD’s Oversight of Its Moving to Work Demonstration Program Needs
               Improvement


    Attached is the U.S. Department of Housing and Urban Development (HUD), Office of
Inspector General’s (OIG) final results of our review of HUD’s monitoring and oversight of its
Moving to Work Demonstration program.

    HUD Handbook 2000.06, REV-4, sets specific timeframes for management decisions on
recommended corrective actions. For each recommendation without a management decision,
please respond and provide status reports in accordance with the HUD Handbook. Please furnish
us copies of any correspondence or directives issued because of the audit.

    The Inspector General Act, Title 5 United States Code, section 8L, requires that OIG post its
publicly available reports on the OIG Web site. Accordingly, this report will be posted at
http://www.hudoig.gov.

   If you have any questions or comments about this report, please do not hesitate to call me at
(215) 430-6729.
                                         September 27, 2013
                                         HUD’s Oversight of Its Moving to Work Demonstration
                                         Program Needs Improvement




Highlights
Audit Report 2013-PH-0004


 What We Audited and Why                  What We Found

We audited the U.S. Department of        HUD had implemented program monitoring
Housing and Urban Development’s          procedures; however, its program oversight was
(HUD) oversight of the Moving to         inadequate because it had not (1) implemented
Work Demonstration program based on      programwide performance indicators, (2) evaluated
the HUD Office of Inspector General’s    agencies’ programs according to its policy, (3)
(OIG) initiative to focus HUD            evaluated agencies’ compliance with key statutory
management’s attention on problem        requirements, (4) verified agencies’ self-reported
areas on which we and others have        performance data, and (5) performed required annual
reported over the years. Our objective   program risk assessments. As a result, HUD was
was to determine whether HUD had         unable to demonstrate program results. Also, HUD
implemented adequate program controls    had limited ability to assess agencies’ compliance with
to effectively monitor participant       statutory program goals and lacked assurance that
agencies’ performance and ensure that    agencies met key statutory requirements.
they met statutory program goals and
requirements.

 What We Recommend

We recommend that the Deputy
Assistant Secretary for Public Housing
Investments establish and implement
policies to improve HUD’s
administration of the program and
ensure that agencies meet statutory
goals and requirements before
expanding the program.
                          TABLE OF CONTENTS

Background and Objective                                            3

Results of Audit
      Finding: HUD Did Not Adequately Oversee Its Moving to Work
      Demonstration Program                                         4

Scope and Methodology                                              12

Internal Controls                                                  13

Follow-up on Prior Audits                                          14

Appendixes
A.    Auditee Comments and OIG’s Evaluation                        16




                                          2
                        BACKGROUND AND OBJECTIVE

The Omnibus Consolidated Rescissions and Appropriations Act of 1996, dated April 26, 1996,
authorized the Moving to Work Demonstration program. The program was established to give
public housing agencies and the U.S. Department of Housing and Urban Development (HUD)
flexibility to design and test innovative, locally designed strategies in pursuit of the national goal of
delivering rental assistance more efficiently. The program’s intent is to pursue three statutory
objectives: (1) reduce cost and achieve greater cost effectiveness in Federal expenditures; (2) give
incentives to families with children in which the head of household is working; seeking work; or
preparing by participating in job training, educational programs, or programs that assist people to
obtain employment and become economically self-sufficient; and (3) increase housing choices for
low-income families.

In pursuit of these objectives, Moving to Work housing agencies are granted exceptions to many
portions of the U.S. Housing Act of 1937 and combine funds received for the Housing Choice
Voucher program and public housing capital and operating funds to use interchangeably. Agencies
are required to

    •   Ensure that at least 75 percent of families admitted are very low income.
    •   Establish a reasonable rent policy designed to encourage self-sufficiency and employment.
    •   Continue to assist substantially the same number of eligible low-income families that they
        would have served had funding amounts not been combined.
    •   Maintain a mix of families comparable to those they would have served without the
        demonstration.
    •   Ensure that the housing provided meets HUD’s housing quality standards.

HUD’s Office of Public Housing Investments, within the Office of Public and Indian Housing at
HUD headquarters, is responsible for program oversight. As of July 2013, there were 35 agencies
participating in the program.

Our objective was to determine whether HUD had implemented adequate program controls to
effectively monitor participant agencies’ performance and ensure that they met statutory program
goals and requirements.




                                                    3
                                RESULTS OF AUDIT


Finding: HUD Did Not Adequately Oversee Its Moving to Work
Demonstration Program
HUD generally conducted periodic visits to program participants according to its policy and
maintained documentation to support its most recent admissions into the program. However, its
program oversight was inadequate because it had not (1) implemented programwide performance
indicators, (2) evaluated agencies’ programs according to its policy, (3) evaluated agencies’
compliance with key statutory program requirements, (4) verified agencies’ self-reported
performance data, and (5) performed required annual program risk assessments. These
conditions occurred because HUD believed that implementing performance indicators would be
difficult and contrary to the program’s purpose. Also, HUD cited a lack of funding needed to
perform critical evaluations and did not have adequate procedures to verify agencies’ compliance
with key statutory requirements. Additionally, HUD lacked procedures to verify agencies’ self-
reported performance data and was not aware of the requirement to perform annual program risk
assessments. As a result, it was unable to demonstrate program results. Also, HUD had limited
ability to assess agencies’ compliance with statutory program goals and lacked assurance that
agencies met key statutory requirements. Further, HUD lacked assurance that it had properly
identified and addressed risks that might prevent participating agencies from meeting program
goals and statutory requirements.




 HUD Conducted Periodic Site
 Visits According to Its Policy


              HUD’s Moving to Work desk guide states that it will conduct annual site visits to
              participant agencies to discuss their programs and provide technical assistance.
              HUD generally conducted annual site visits to agencies in accordance with its
              procedures. It prepared reports to document its annual site visits. We reviewed
              21 site visit reports for 4 agencies in Region 3, including the Philadelphia
              Housing Authority, Housing Authority of Baltimore City, District of Columbia
              Housing Authority, and Housing Authority of the City of Pittsburgh, for the
              period January 2006 through October 2012. During the site visits, HUD met with
              agency officials to discuss program updates, agencies’ short- and long-term
              program goals, current activities, and local issues. HUD provided technical
              assistance by providing verbal guidance on issues agencies encountered, such as
              entering data into HUD’s Public and Indian Housing Information Center database.
              HUD also toured housing developments to verify agencies’ progress.



                                               4
           Although HUD conducted annual site visits in accordance with its procedures, it
           did not validate the accuracy of agencies’ self-reported performance data during
           these visits. According to HUD officials, HUD was working on developing
           procedures to verify the accuracy of self-reported data, either through audits or
           during annual site visits.

HUD Maintained Documents
Supporting Its Most Recent
Admissions Into the Program


           Section 232 of the Consolidated Appropriations Act of 2010 allowed HUD to
           select four new participant agencies, which were designated as “high performing”
           under the Public Housing Assessment System and administered no more than
           5,000 aggregate housing vouchers and public housing units. Public and Indian
           Housing Notice 2012-16 required eligible agencies to submit an application
           containing an eligibility certification, evidence of capability, a Moving to Work
           plan, evidence of community support and involvement, and additional
           information regarding implementation of the Moving to Work plan. The notice
           also described HUD’s evaluation criteria for selecting the eligible agencies. The
           evaluation criteria required HUD to evaluate applications based on the following
           four rating factors: (1) demonstrated capability to effectively plan, implement,
           and administer the program; (2) feasibility of the Moving to Work plan; (3)
           degree of resident and community support and involvement; and (4) potential
           local and national impact of the Moving to Work plan. Each rating factor
           included a number of points attributed to the factor, with the possible number of
           points for all four factors totaling 100 points.

           HUD received applications from 12 agencies seeking to participate in the program
           and selected the 4 with the highest scores. We reviewed the applications and
           related evaluations for the four agencies and found that HUD maintained adequate
           documentation to show that it made the selections according to its policy.

HUD Lacked Programwide
Performance Indicators


           HUD had not established programwide performance indicators that would allow it
           to assess program results. The program was established to (1) reduce cost and
           achieve greater cost effectiveness in Federal expenditures, (2) incentivize families
           to become economically self-sufficient, and (3) increase housing choices for low-
           income families. The Government Performance and Results Modernization Act
           of 2010 requires Federal agencies to develop a performance plan for each
           program activity. The plan should establish a balanced set of performance
           indicators to be used in measuring or assessing progress toward each performance


                                            5
                   goal, including efficiency, output, and outcome indicators. HUD’s annual
                   performance plans for fiscal years 2011 and 2012 established agencywide
                   performance indicators; however, none of the indicators was specifically
                   connected to the program.

                   HUD officials stated that programwide performance indicators were not
                   implemented because the program lacked standard metrics related to program
                   performance. Based on discussions with HUD officials and HUD’s statements
                   during an April 2012 U.S. Government Accountability Office (GAO) audit of the
                   program, it appeared that HUD did not implement standard performance metrics
                   because it believed doing so could be difficult and might be contrary to the nature
                   of the program. Due to the lack of performance indicators, HUD was unable to
                   demonstrate program results or show that the three main program objectives were
                   met. We believe that HUD could establish policies to use the results from
                   agencies’ self-reported performance data to develop programwide performance
                   indicators for assessing program results.

                   The U.S. Office of Management and Budget (OMB) recently issued a
                   memorandum, 1 which noted the need to continually improve Federal program
                   performance by applying existing evidence on successes, generating new
                   knowledge, and using experimentation and innovation to test new approaches to
                   program delivery. According to the memorandum, evidence would include
                   evaluation results, performance measures, and other relevant data analytics and
                   research studies, with a preference for high-quality experimental and quasi-
                   experimental studies. For 2015 budget proposals, OMB urged agencies to (1) use
                   credible evidence in formulating 2015 budget proposals and performance plans and
                   (2) propose new ways to develop evidence to address policy challenges. OMB
                   stated that agencies’ budget requests would be more likely to be fully funded if they
                   showed a widespread commitment to evidence and innovation. Therefore, it is vital
                   that HUD develop programwide performance indicators to ensure that its future
                   budget requests are fully funded.

    HUD Failed To Evaluate
    Agencies’ Programs According
    to Its Policy


                   HUD did not evaluate agencies’ programs according to provisions in its standard
                   program agreement and as incorporated into its program monitoring policies.
                   According to the agreement, HUD would evaluate agencies’ programs during
                   fiscal years 2011 and 2014 to determine whether the agencies should continue in
                   the program. For the evaluations, HUD planned to use the following criteria: (1)
                   the agency complies with the standard program agreement; (2) the annual
                   program plans and reports have been satisfactorily completed and submitted in a

1
    OMB Memorandum 13-17, Next Steps in the Evidence and Innovation Agenda, dated July 26, 2013

                                                       6
                 timely manner, consistent with the standard program agreement; and (3) the
                 agency has demonstrated, through the annual program plan and report, that it used
                 its program designation in accordance with Section 204 of the 1996
                 Appropriations Act.

                 HUD planned to hire an outside research consultant to perform the evaluations
                 and use the results to determine whether the agencies would be allowed to
                 continue to participate in the program or have their participation terminated.
                 HUD preferred to hire an outside research consultant instead of performing the
                 evaluation in-house because it wanted a third-party opinion regarding the
                 program. However, HUD did not perform the 2011 evaluation and had not
                 performed one since. HUD officials informed us that they did not perform the
                 evaluation due to a lack of funding and the fiscal year 2014 evaluation might also
                 be jeopardized for the same reason. As a result, HUD lacked assurance that
                 agencies complied with the program agreement and whether the agencies should
                 continue to participate in the program. Based on the lack of funding for these
                 evaluations, HUD should not admit additional agencies into the program until the
                 evaluations are performed.

                 HUD officials responsible for the program informed us that they had been
                 requesting funds to perform the program evaluation through HUD’s
                 Transformation Initiative 2 since January 2010. HUD estimated a cost of
                 approximately $3 million to perform the evaluation. In February 2013, program
                 officials met with HUD’s Office of Policy Development and Research officials to
                 further discuss the need for the evaluation. The Office of Policy Development
                 and Research stated that it would investigate whether Transformation Initiative
                 funds could be allocated to perform the evaluation in fiscal year 2013 or 2014. In
                 July 2013, the Office of Policy Development and Research stated that it planned
                 to award an evaluation contract in fiscal year 2014. It also stated that some funds
                 had been reserved for the evaluation and that it planned to reserve additional
                 funding in fiscal year 2014. Since there are indications that Congress is
                 considering expanding the program to include more agencies, we believe HUD
                 needs to evaluate the program participants to determine whether they comply with
                 the program agreement before expanding the program.

    HUD Failed To Verify
    Agencies’ Compliance With
    Statutory Requirements


                 HUD had not implemented controls to verify whether agencies met statutory
                 requirements related to performance that were incorporated into the standard

2
  In 2010, Congress enacted the Transformation Initiative, which makes up to 1 percent of program funds available
for the following: (1) research, evaluation, and program metrics; (2) program demonstration; (3) technical
assistance; and (4) information technology.

                                                        7
                program agreement. The agreement required that agencies (1) ensure that at least
                75 percent of assisted families were very low-income families as defined in the
                Housing Act of 1937, (2) serve substantially the same number of families that
                would have been served if they had not participated in the program, and (3)
                maintain a mix of families comparable to those that would have been served
                without the program. As part of agencies’ annual report, the agencies would self-
                certify that they complied with the statutory requirements. Although the self-
                certification process served as a control measure to determine or ensure agencies’
                compliance with statutory requirements, the effectiveness of the control was
                limited because HUD had no policies or procedures to periodically verify the
                agencies’ self-certifications.

                We determined that HUD had taken steps to verify agencies’ compliance with the
                statutory requirements. For the second statutory requirement, HUD issued
                guidance in January 2013, which described how it would determine agencies’
                compliance with the requirement annually. 3 For the first and third statutory
                requirements, HUD had implemented reporting requirements instructing agencies
                to self-report data to support compliance. HUD officials informed us that they
                would verify the accuracy of the self-reported data through an audit or during
                annual site visits. However, HUD had not finalized policies or procedures to
                verify the self-reported information and thereby fully verify agencies’ compliance
                with the first and third statutory requirements.

    HUD Did Not Verify Agencies’
    Self-Reported Performance
    Data


                HUD lacked policies and procedures to verify the accuracy of key information
                that agencies self-report. Attachment B of each standard program agreement
                required agencies to report in their annual reports on benchmarks, metrics, and
                performance information, such as reducing the number of homeless persons
                without housing, reducing the processing time during the annual recertification
                process, and increasing the number of housing choices for low-income
                households. The information reported by agencies related to the outcomes of
                ongoing activities included the number of homeless persons provided with
                housing, the number of program staff hours saved annually through the rent
                simplification process, and the number of units under contract to low-income
                households.

                We reviewed 17 annual reports from the Moving to Work agencies located within
                Region 3’s jurisdiction for the period 2007 through 2012. The agencies reported
                benchmarks, metrics, and performance information. However, HUD did not

3
 Public and Indian Housing Notice 2013-02, Baseline Methodology for Moving to Work Public Housing Agencies,
dated January 10, 2013

                                                     8
                    verify the accuracy of the self-reported information. As stated above, HUD
                    officials stated that HUD planned to establish a means of verifying the accuracy
                    of self-reported information, either through audits or during annual site visits.
                    However, it had not finalized such procedures.

    HUD Failed To Perform
    Critical Annual Program Risk
    Assessments


                    HUD did not perform annual program risk assessments in accordance with its
                    own policy. HUD Handbook 1840.1 required program offices to perform an
                    annual risk assessment of their programs or administrative functions using a
                    standard risk-assessment worksheet. HUD officials informed us that they did not
                    perform annual program risk assessments because they were not aware of the
                    requirement. Because HUD did not perform annual risk assessments, it lacked
                    assurance that it properly identified and addressed risks that might prevent
                    participating agencies from meeting program goals and statutory requirements.

                    HUD had taken steps to perform a program risk assessment. It recently
                    performed an internal review of potential program risk areas. It planned to
                    develop a basic risk assessment based on the internal review. The basic risk
                    assessment would eventually feed into the Office of Public and Indian Housing’s
                    overall risk assessment. However, the basic risk assessment had not been
                    finalized.

    Prior GAO and OIG Reports
    Disclosed Similar Issues


                    Prior HUD Office of Inspector General (OIG) and GAO audit reports on the
                    program disclosed that HUD lacked (1) programwide performance measures to
                    determine whether overall program objectives were met and (2) policies and
                    procedures to verify the reliability of agencies’ self-reported data. GAO also had
                    previously reported that HUD failed to perform required program risk
                    assessments. 4 We determined that HUD was working on resolving audit
                    recommendations related to these findings. In conjunction with its audit
                    resolution efforts, HUD had implemented revised reporting requirements for
                    agencies as discussed in the paragraph below. Since some of our audit findings
                    were consistent with findings from prior reports, HUD should continue its efforts
                    to ensure that it has fully addressed all audit recommendations related to the
                    program.



4
    See the Follow-up on Prior Audits section of this report for details on prior HUD OIG and GAO audits.

                                                          9
    HUD Revised Its Standard
    Reporting Requirements


                 HUD Form 50900 specifies the elements that each agency must provide in its
                 annual plans and reports. 5 It requires information such as an overview of the
                 program goals and objectives for the year; planned capital expenditures by
                 development; proposed activities requesting HUD approval with baselines,
                 benchmarks, and metrics to assess outcomes; planned sources and uses of funds;
                 and a description of planned or ongoing agency-directed evaluations of the
                 demonstration. HUD uses the information to make determinations about the
                 program and respond to congressional and other inquiries regarding program
                 outcomes and positive practices learned throughout the program demonstration.

                 HUD revised Form 50900 in response to the prior GAO and HUD OIG reports.
                 In May 2013, OMB approved the revised form. The form includes standard
                 metrics for agencies to report on the progress of proposed and approved activities.
                 The standard metrics are organized to assist HUD in determining agencies’
                 compliance with the three statutory program objectives.

                 HUD’s updated reporting requirements will improve its program oversight.
                 However, it needs to implement a formal policy to develop programwide
                 performance indicators using results from the data it collects from agencies. It
                 also needs to complete its efforts to establish and implement policies to verify
                 agencies’ self-reported information to ensure the accuracy of data on which it
                 relies for various purposes.

    Conclusion

                 HUD needs to improve its program oversight to ensure that participant agencies
                 meet statutory program goals and requirements. Also, since there are indications
                 that Congress is considering expanding the program, HUD needs to ensure that it
                 has implemented effective evaluation processes before program expansion. We
                 included four of GAO’s recommendations in our audit because they are
                 significant to the success of this program. In about 14 years, HUD has not shown
                 that this demonstration program has met its objectives of achieving cost
                 effectiveness, incentivizing families to become self-sufficient, and increasing
                 housing choices for low-income families. GAO is reviewing HUD’s revised
                 reporting requirements in response to its prior report. HUD can provide us a copy
                 of its planned actions communicated to GAO rather than developing separate
                 management decisions for these four recommendations.



5
 Attachment B to the amended and restated Moving to Work agreement between HUD and Moving to Work
agencies

                                                  10
Recommendations

          We recommend that the Deputy Assistant Secretary for Public Housing
          Investments

          1A.     Develop programwide performance indicators based on the results from
                  agencies’ self-reported data.

          1B.     Ensure that the program is evaluated in accordance with the standard
                  program agreement and program monitoring policies.

          1C.     If Congress expands the program, delay adding participants to the program
                  until programwide performance indicators are in place and program
                  evaluation results warrant program expansion.

          1D.     Continue to develop and implement procedures to verify agencies’
                  compliance with key statutory requirements.

          1E.     Continue to develop and implement procedures to verify the accuracy of
                  agencies’ self-reported performance data.

          1F.     Ensure that annual risk assessments are completed as required.




                                          11
                         SCOPE AND METHODOLOGY

We conducted the audit from November 2012 through June 2013 primarily at our office located
in Philadelphia, PA. The audit covered the period October 2007 through September 2012 but
was expanded as necessary to accomplish our objective.

To accomplish our objective, we

   •   Reviewed relevant background information, including prior HUD OIG audit reports and
       GAO reports.
   •   Interviewed headquarters staff of HUD’s Office of Public Housing Investments and GAO
       personnel.
   •   Reviewed applicable HUD rules, handbooks, notices, and guidance.
   •   Reviewed HUD’s organizational chart related to the program.
   •   Reviewed program agreements, annual plans, and reports for four agencies in Region 3.
   •   Reviewed site visit reports for four agencies in Region 3.
   •   Reviewed HUD’s annual performance plans for fiscal years 2011 and 2012.
   •   Reviewed applications and related evaluations for the four agencies HUD recently
       selected to participate in the program.
   •   Reviewed proposed legislation to expand the program.

We nonstatistically selected as our sample four housing agencies within Region 3: the Housing
Authority of Baltimore City, the District of Columbia Housing Authority, the Philadelphia
Housing Authority, and the Housing Authority of the City of Pittsburgh.

We considered data in HUD’s Public and Indian Housing Information Center database. We used
the data obtained by HUD to determine whether agencies substantially served the same number
of families as if they had not participated in the program. The data did not materially affect our
results; thus, we considered the data adequate for our purposes.

We conducted the audit in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain sufficient, appropriate
evidence to provide a reasonable basis for our findings and conclusions based on our audit
objective. We believe that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objective.




                                               12
                              INTERNAL CONTROLS

Internal control is a process adopted by those charged with governance and management,
designed to provide reasonable assurance about the achievement of the organization’s mission,
goals, and objectives with regard to

   •   Effectiveness and efficiency of operations,
   •   Reliability of financial reporting, and
   •   Compliance with applicable laws and regulations.

Internal controls comprise the plans, policies, methods, and procedures used to meet the
organization’s mission, goals, and objectives. Internal controls include the processes and
procedures for planning, organizing, directing, and controlling program operations as well as the
systems for measuring, reporting, and monitoring program performance.


 Relevant Internal Controls

               We determined that the following internal controls were relevant to our audit
               objective:

               •      Policies and procedures – Controls that HUD has implemented to ensure that
                      housing agencies meet statutory goals and requirements.

               We assessed the relevant controls identified above.

               A deficiency in internal control exists when the design or operation of a control does
               not allow management or employees, in the normal course of performing their
               assigned functions, the reasonable opportunity to prevent, detect, or correct (1)
               impairments to effectiveness or efficiency of operations, (2) misstatements in
               financial or performance information, or (3) violations of laws and regulations on a
               timely basis.

 Significant Deficiency

               Based on our review, we believe that the following item is a significant deficiency:

               •      HUD lacked controls to allow it to determine whether housing agencies met
                      statutory goals and requirements.




                                                 13
                  FOLLOW-UP ON PRIOR AUDITS

Moving to Work
Demonstration: Opportunities
Exist to Improve Information
and Monitoring - Audit Report
GAO-12-490, Dated April 19,
2012

           GAO audited the program because Congress was considering expanding the
           program and asked GAO to examine the program’s success in addressing the
           three primary program objectives, HUD’s monitoring efforts, and the potential
           benefits of and concerns about expansion. GAO found that opportunities existed
           to improve HUD’s methods for evaluating the program. GAO recommended that
           HUD (1) improve its guidance on reporting performance information, (2) develop
           a plan for identifying and analyzing standard performance data, (3) establish
           programwide performance indicators, (4) systematically identify lessons learned,
           (5) clarify key terms, (6) implement a process for assessing compliance with
           statutory requirements, (7) perform annual assessments of program risks, and (8)
           verify the accuracy of self-reported data.

           HUD generally or partially agreed with seven of the recommendations. HUD
           disagreed with the recommendation that it establish programwide performance
           indicators. However, GAO believed performance indicators were critical to
           demonstrating program results and, thus, maintained its recommendation. In June
           2013, GAO testified to Congress that HUD had revised agency reporting
           requirements since it issued its report. GAO is reviewing the new requirements.

           HUD expects that its revised reporting requirements will address
           recommendations 1, 2, 5, 6, and 8. Although it initially disagreed with the
           recommendation to implement programwide performance indicators, HUD stated
           that its goal was to implement the recommendation by the end of fiscal year 2014.
           Also, as stated in the report, HUD had initiated steps to perform a program risk
           assessment and was working on determining an appropriate process to verify
           agencies’ self-reported performance data. However, as of the date of this report,
           all of the recommendations remained open.




                                           14
Additional Details To
Supplement Our Report on
HUD’s Fiscal Years 2012 and
2011 Financial Statements -
Audit Report 2013-FO-0003,
Dated November 15, 2012

           HUD OIG audited HUD’s consolidated financial statements in accordance with
           the Chief Financial Officers Act of 1990. Among the deficiencies found, HUD
           did not establish monitoring tools necessary to ensure that the Moving to Work
           statutory goals were met, finalize or implement performance measures and the
           methodology needed to evaluate the performance of agencies in achieving the
           program objectives, and establish adequate tools to verify the reliability of
           reported program data.

           HUD OIG made the following related recommendations: (1) develop, implement,
           and document methodologies to calculate and track performance measures to
           enable comparability of data among agencies and ensure the reliability of reported
           data; (2) develop, implement, and document standardized reporting requirements
           for the program data and results for all participant agencies; (3) update the
           program plan and report review procedures to include steps to verify the
           reliability of presented data against HUD systems and retain all supporting
           documentation as evidence of controls performed; and (4) ensure that the staffing
           and funding levels for the program office are adequate to provide proper oversight
           of the program.

           To resolve the audit recommendations, HUD proposed actions with target dates
           ranging from December 2013 to December 2014. HUD OIG agreed to HUD’s
           proposed actions and target dates. All of the recommendations were open as of
           the date of this report.




                                           15
Appendix A

        AUDITEE COMMENTS AND OIG’S EVALUATION


Ref to OIG Evaluation   Auditee Comments




Comment 1
Comment 2



Comment 3




Comment 4

Comment 1




                         16
Comment 5




Comment 6




Comment 7




Comment 8




            17
Comment 9




            18
Comment 10




Comments 1,
2, 3, & 4




              19
                         OIG Evaluation of Auditee Comments

Comment 1   We conducted the audit in accordance with generally accepted government
            auditing standards. Those standards require that we communicate the results of
            the audit. Accordingly, the report discusses the audit findings and related
            recommendations.

Comment 2   As stated in the section on prior audit reports, GAO made eight recommendations
            related to the program. Four of the recommendations are included in this report.
            These four recommendations include the recommendation related to the need for
            programwide performance indicators, on which HUD initially disagreed with
            GAO. As acknowledged in the report, HUD is working on resolving most of
            these recommendations. However, we believe it is important to make these same
            recommendations because they have not been completely resolved, and about 14
            years after the demonstration program was implemented to achieve cost
            effectiveness, incentivize families to become self-sufficient, and increase housing
            choices for low-income families, HUD has yet to demonstrate that the program
            has met the purposes for which it was established. To reduce the reporting
            burden, HUD can provide us a copy of its planned actions in place with GAO and
            we will track the resolution of these recommendations through GAO’s report. We
            have revised the conclusion in the finding to clarify this in the report.

Comment 3   We are encouraged that HUD agreed with the recommendations in OIG Audit
            Report 2013-FO-0003 and is working on resolving those recommendations. We
            are also pleased that HUD, for the most part, agrees with our audit
            recommendations and is only concerned that they appear duplicative. However,
            while the audit recommendations in this report appear to mirror recommendations
            from the prior audit referenced, the recommendations in this report are valid as
            explained in comments 5 and 9 below.

Comment 4   We have considered management’s comments and determined that all of the audit
            recommendations are appropriately directed to the Office of Public and Indian
            Housing. However, we have updated recommendation 1C, as explained in
            comment 7, to reflect reasonable action that HUD can take to address the related
            concern discussed in the audit finding.

Comment 5   The referenced recommendation from the prior OIG audit is not the same as the
            recommendation in this report or the recommendation GAO issued.
            Implementing performance measures to enable comparability of data among
            agencies and ensuring the reliability of reported data is not the same as
            implementing programwide performance indicators to determine whether the
            program as a whole is meeting the purposes for which was established. HUD’s
            action plan for implementing the recommendation from the prior OIG audit did
            not address the need for developing programwide performance indicators. HUD
            needs to establish programwide performance indicators that will allow it to
            determine whether the program has produced the intended results, particularly as

                                            20
                   Congress considers whether to expand the program. Therefore, the
                   recommendation in the prior OIG audit report should have a different
                   management decision and needs to be addressed separately from GAO’s
                   recommendation.

Comment 6          We acknowledged the Office of Public and Indian Housing’s efforts to obtain
                   funding for program evaluation. The Office of Public and Indian Housing is
                   responsible for administering the program. Also, it was the Office of Public and
                   Indian Housing that established, as a control, a plan to evaluate the program in
                   2011 and 2014 to determine whether agencies should continue in the program.
                   We believe this is a critical control that the Office of Pubic and Indian Housing
                   should implement. As stated in the report, the Office of Policy Development and
                   Research told OIG that it had reserved funds for a program evaluation in 2014 and
                   that it planned to reserve additional related funding in fiscal year 2014. Since it is
                   responsible for program administration, the Office of Public and Indian Housing
                   should coordinate with the Office of Policy Development and Research to ensure
                   that program evaluation occurs in accordance with its program monitoring
                   policies or determine another way to evaluate the program in fiscal year 2014.
                   We updated the recommendation to reflect that the Office of Public and Indian
                   Housing is responsible for ensuring that the program is evaluated.

Comment 7          We recognize that HUD does not have the authority to expand the program.
                   However, in a program report to Congress in August 2010, HUD recommended
                   that the program be expanded by up to twice its size, or from 35 to 60
                   participating agencies. 6 Also, in its fiscal year 2014 budget request presented to
                   Congress in April 2013, HUD proposed a substantial expansion of the program to
                   high-capacity agencies. We have updated the recommendation to emphasize that
                   the program must be evaluated before expansion and to delay adding program
                   participants until it has implemented programwide performance indicators and
                   determined that program evaluation results warrant program expansion.

Comment 8          We evaluated HUD’s revised Form 50900 during the audit. Although the form
                   includes standard metrics that were organized to assist HUD in determining
                   agencies’ compliance with the statutory requirements, agencies will continue to
                   self-report data to support compliance. However, HUD has not determined how it
                   will verify the data for accuracy as required by recommendation 1E. During the
                   audit, HUD officials stated that HUD planned to establish a means of verifying
                   the accuracy of self-reported data, either through audits or during annual site
                   visits. However, until HUD finalizes such procedures, the verification to
                   determine compliance will not be reliable. Therefore, the audit recommendation
                   will need to remain open until recommendation 1E is completed.

Comment 9          As stated above, we evaluated HUD’s revised Form 50900 during the audit and
                   determined that HUD needs to implement additional verification procedures.

6
    Moving to Work: Interim Policy Applications and the Future of the Demonstration, August 2010

                                                        21
             The referenced recommendation from the prior OIG audit asked HUD to
             implement steps to verify the reliability of presented data against its systems. Our
             recommendation and GAO’s recommendation did not ask for HUD to only
             implement a verification process for data it has in its systems. HUD still needs to
             address how it will verify self-reported information that it does not have in its
             systems. Therefore, the recommendation in the prior OIG audit report needs to be
             addressed separately from GAO’s recommendation. Although HUD officials
             stated that HUD planned to verify agencies’ self-reported data through audits or
             during annual site visits, HUD had not implemented or finalized such procedures.

Comment 10 In the GAO report, HUD agreed that it needed to conduct an annual program risk
           assessment and described its plans to develop a formal risk-based strategy for
           monitoring. During our audit, HUD officials stated that HUD had not performed
           annual program risk assessments. According to HUD officials, the Office of
           Public and Indian Housing recently performed an internal review of potential
           program risk areas and planned to develop a basic risk assessment based on the
           internal review.

             During the exit conference, HUD officials stated that the Office of Public and
             Indian Housing performed overall risk assessments that would cover the Moving
             to Work program. After we received HUD’s response to the draft report, we
             asked it to provide documentation on the joint assessments it stated it had
             performed. HUD provided a writeup on issues related to five participant agencies
             it had classified as “at risk” as of October 2012. This writeup did not address the
             requirement for an annual program risk assessment as outlined in HUD Handbook
             1840.1 and described in the report.

             During the audit, the HUD Office of the Chief Financial Officer confirmed that
             each program office is required to perform an annual risk assessment of its
             programs and maintain related supporting documentation. Given the specific
             statutory goals and requirements related to the program and the flexibility
             awarded to participant agencies, we agree with GAO’s recommendation and
             HUD’s initial response, which recognized that annual program risk assessments
             were necessary. Accordingly, HUD needs to complete the program risk
             assessment it said it had started and ensure that it completes annual program risk
             assessments.




                                              22