oversight

Moving to Work Demonstration: Opportunities Exist to Improve Information and Monitoring

Published by the Government Accountability Office on 2012-04-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          Report to Congressional Requesters




April 2012
             MOVING TO WORK
             DEMONSTRATION
             Opportunities Exist to
             Improve Information
             and Monitoring




GAO-12-490
                                               April 2012

                                               MOVING TO WORK DEMONSTRATION
                                               Opportunities Exist to Improve Information and
                                               Monitoring
Highlights of GAO-12-490, a report to
congressional requesters




Why GAO Did This Study                         What GAO Found
HUD’s MTW demonstration program                Public housing agencies (PHA) that participate in the Moving to Work (MTW)
gives participating PHAs the flexibility       program report annually on the performance of their activities, which include
to create innovative housing strategies        efforts to reduce administrative costs and encourage residents to work. But this
through their fiscal year 2018. MTW            performance information varies, and the Department of Housing and Urban
agencies must create activities linked         Development’s (HUD) guidance does not specify that it be quantifiable and
to three statutory purposes—reducing           outcome oriented. Further, HUD has not identified the performance data that
costs, providing incentives for self-          would be needed to assess the results of similar MTW activities or the program
sufficiency, and increasing housing            as a whole and has not established performance indicators for the program. The
choices—and meet five statutory
                                               shortage of such analyses and indicators has hindered comprehensive
requirements. Congress is considering
                                               evaluation efforts, although such evaluations are key to determining the success
expanding MTW and has asked GAO
to examine what is known about (1) the
                                               of any demonstration program. Further, while HUD has identified some lessons
program’s success in addressing the            learned from the program, it has no systematic process to identify them and thus
three purposes, (2) HUD’s monitoring           has relied primarily on ad hoc information. The absence of a systematic process
efforts, and (3) the potential benefits of     for identifying lessons learned limits HUD’s ability to promote useful practices
and concerns about expansion. GAO              that could be more broadly implemented to address the purposes of the program.
analyzed the most current annual
reports for 30 MTW agencies;                   HUD generally follows its MTW monitoring policies and procedures, but they could be
compared HUD’s monitoring efforts              strengthened. HUD staff review and approve each MTW agency’s annual plan to
with internal control standards; and           ensure that planned activities are linked to program purposes and visit each MTW
interviewed agency officials,                  agency annually to provide technical assistance. But HUD has not taken key
researchers, and industry officials.           monitoring steps set out in internal control standards, such as issuing guidance that
                                               defines program terms or assessing compliance with all of the requirements. Without
What GAO Recommends                            clarifying key terms and establishing a process for assessing compliance with
                                               statutory requirements, HUD lacks assurance that agencies are actually complying
GAO makes eight recommendations to             with the statute. Additionally, HUD has not done an annual assessment of program
HUD: that HUD improve its guidance             risks despite its own requirement to do so and has not developed risk-based
on reporting performance information,          monitoring procedures. Without taking these steps, HUD lacks assurance that it has
develop a plan for identifying and             identified all risks to the program. Finally, HUD does not have policies or procedures
analyzing standard performance data,           in place to verify the accuracy of key information that agencies self-report. For
establish performance indicators,              example, HUD staff do not verify self-reported performance information during their
systematically identify lessons learned,       reviews of annual reports or annual site visits. Without verifying at least some
clarify key terms, implement a process         information, HUD cannot be sure that self-reported information is accurate.
for assessing compliance with statutory
requirements, do annual assessments
                                               Expanding the MTW program may offer benefits but also raises questions.
of program risks, and verify the
                                               According to HUD, affordable housing advocates, and MTW agencies, expanding
accuracy of self-reported data. HUD
generally or in part agreed with seven
                                               MTW to additional PHAs would allow agencies to develop more activities tailored
of them. HUD disagreed with our                to local conditions and result in more lessons learned. However, data limitations
recommendation that it create overall          and monitoring weaknesses raise questions about expansion. HUD recently
performance indicators. GAO believes,          reported that expansion should occur only if newly admitted PHAs structured
however, that they are critical to             their programs to permit high-quality evaluations and ensure that lessons learned
demonstrating program results and              could be generalized. Until more complete information on the program’s
thus maintains its recommendation.             effectiveness and the extent to which agencies are adhering to program
                                               requirements is available, it will be difficult for Congress to know whether an
                                               expanded MTW would benefit additional agencies and the residents they serve.
                                               Some researchers and MTW agencies suggested alternatives to expansion,
                                               including implementing a program that was more limited in scope.
View GAO-12-490. For more information,
contact Mathew J. Scirè at (202) 512-8678 or
sciremj@gao.gov.

                                                                                         United States Government Accountability Office
Contents


Letter                                                                                    1
               Background                                                                 3
               HUD Has Not Identified Standard Performance Data and Indicators
                 Needed to Evaluate the Program                                           9
               HUD Generally Follows Its Monitoring Policies and Procedures,
                 but Could Strengthen Them                                              22
               Expanding MTW May Offer Benefits but Also Raises Questions               31
               Conclusions                                                              39
               Recommendations for Executive Action                                     41
               Agency Comments and Our Evaluation                                       42

Appendix I     Scope and Methodology                                                    46



Appendix II    Comments from the Department of Housing and Urban
               Development                                                              51



Appendix III   GAO Contact and Staff Acknowledgments                                    62



Table
               Table 1: Examples of MTW Activities, by Statutory Purpose                10


Figures
               Figure 1: Timeline of Public Housing Agencies Entering and Exiting
                        the MTW Demonstration Program                                     6
               Figure 2: Number of Reported MTW Activities Linked to Each of
                        the Statutory Purposes                                          12




               Page i                                GAO-12-490 Moving to Work Demonstration
Abbreviations

AHSSIA            Affordable Housing and Self-Sufficiency Improvement Act
                  of 2012
HUD               Department of Housing and Urban Development
MTW               Moving to Work
MTW-PIC           Moving to Work section of the Public and Indian Housing
                  Information Center
PHA               public housing agency




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                         GAO-12-490 Moving to Work Demonstration
United States Government Accountability Office
Washington, DC 20548




                                   April 19, 2012

                                   The Honorable Charles Grassley
                                   Ranking Member
                                   Committee on the Judiciary
                                   United States Senate

                                   The Honorable Judy Biggert
                                   Chairman
                                   Subcommittee on Insurance, Housing
                                     and Community Opportunity
                                   Committee on Financial Services
                                   House of Representatives

                                   The Honorable Shelley Moore Capito
                                   Chairman
                                   Subcommittee on Financial Institutions
                                     and Consumer Credit
                                   Committee on Financial Services
                                   House of Representatives

                                   The Department of Housing and Urban Development (HUD) provides
                                   over $25 billion each year to public housing agencies (PHA) to make
                                   rental housing affordable to about 3.2 million low-income households
                                   through a variety of programs. 1 As of January 2012, 35 of the
                                   approximately 3,300 PHAs operating in the United States were
                                   participating in the Moving to Work (MTW) demonstration program until
                                   the end of their fiscal year 2018. The MTW program has three statutory
                                   purposes: to reduce costs and achieve greater cost-effectiveness in
                                   federal housing expenditures, to give families with children incentives to
                                   obtain employment and become self-sufficient, and to increase housing
                                   choices for low-income families. It gives participating PHAs the flexibility
                                   to design and test innovative strategies for providing and administering
                                   housing assistance in their communities. The agencies (MTW agencies)



                                   1
                                    This amount includes funding for public housing capital and operating expenditures and
                                   tenant-based rental assistance. HUD provides separate funding for project-based rental
                                   assistance, which is not included in this dollar amount. A PHA is typically a local agency
                                   created under state law that manages housing for low-income residents at rents they can
                                   afford.




                                   Page 1                                          GAO-12-490 Moving to Work Demonstration
must implement activities that are linked to one or more of the program’s
statutory purposes but that are also designed specifically for the
communities the agencies serve. In fiscal year 2011, MTW agencies
received combined funding of $3.3 billion for housing activities. 2 This
represented more than 13 percent of HUD’s affordable housing
expenditures. Legislation has recently been proposed to expand the
number of MTW agencies. However, researchers and organizations that
advocate on behalf of residents, including legal aid groups, have raised
concerns about expanding it due to the lack of information on the effect of
the program as well as concerns about residents.

In response to your request for information on the effectiveness of the
MTW program, this report discusses (1) what is known about the extent to
which the MTW demonstration program is addressing the program’s
statutory purposes, (2) HUD’s monitoring of MTW agencies’ efforts to
address these purposes and meet statutory requirements, and (3) the
potential benefits of and concerns about expanding the number of PHAs
that can participate in the demonstration program.

To determine what is known about the extent to which the MTW program
is addressing statutory purposes, we analyzed the most recent annual
reports as of January 2012 for 30 agencies. 3 We assessed the reliability
of selected information in the reports by reviewing supporting
documentation for a sample of seven MTW agencies and interviewing
officials responsible for preparing and reviewing this information. (We
selected the sample of MTW agencies to provide diversity in factors such
as geography, agency size, and length of time participating in the
program.) We determined that the reports were sufficiently reliable for the
purposes of our review. We analyzed these reports and corresponding
annual plans to determine the number of ongoing activities and the
statutory purpose(s) to which each activity was linked. We compared
HUD’s guidance for MTW agencies on the type of performance
information they should report with the GPRA (Government Performance
and Results Act) Modernization Act of 2010. We also reviewed



2
 Two of the 35 MTW agencies did not begin participating in the MTW program until fiscal
year 2012 and are therefore not included in this dollar amount.
3
 According to HUD data, as of January 2012 35 agencies were participating in the MTW
program. Of these, 30 had submitted an annual report to HUD. The other five agencies
had not been in the program long enough to report on their accomplishments.




Page 2                                         GAO-12-490 Moving to Work Demonstration
             evaluations of the program and summarized the challenges that HUD and
             others have faced in conducting such evaluations. Finally, we assessed
             HUD’s past efforts to identify lessons learned from the MTW program.

             To assess HUD’s monitoring of MTW agencies’ efforts to address the
             program’s statutory purposes and meet requirements, we reviewed
             relevant policies and procedures, including HUD’s desk guide for the
             program, the Standard Agreement that HUD executed in 2008 to govern
             participation in the program, and documentation of the steps HUD had
             taken to help ensure compliance with its policies and procedures. We
             also examined the relevant internal control standards that apply to federal
             agencies and HUD’s guidance on internal controls and compared HUD’s
             monitoring policies and procedures with these standards. To discuss the
             potential benefits and concerns associated with expanding the number of
             PHAs that can participate in the program, we reviewed studies, reports,
             and testimonies by researchers, affordable housing advocates, and
             resident advocates. For all three objectives, we interviewed HUD officials,
             officials from our sample of seven MTW agencies, research
             organizations, affordable housing advocates, and organizations that
             advocate on behalf of residents of federally subsidized housing.

             We conducted this performance audit from July 2011 to April 2012 in
             accordance with generally accepted government auditing standards.
             Those standards require that we plan and perform the audit to obtain
             sufficient, appropriate evidence to provide a reasonable basis for our
             findings and conclusions based on our audit objectives. We believe that
             the evidence obtained provides a reasonable basis for our findings and
             conclusions based on our audit objectives. Appendix I contains additional
             information on our scope and methodology.


             The MTW demonstration program was authorized by the Omnibus
Background   Consolidated Rescissions and Appropriations Act of 1996. 4 The program
             is intended to give participating agencies the flexibility to design and test
             innovative strategies for providing and administering housing assistance.
             To implement such strategies, participating agencies may request
             waivers of certain provisions in the United States Housing Act of 1937, as




             4
             Pub. L. No. 104-134, 110 Stat. 1321, 1321-281.




             Page 3                                       GAO-12-490 Moving to Work Demonstration
amended. 5 For example, agencies may request to waive certain
provisions in order to combine the funding they are awarded annually
from different programs into a single, authoritywide funding source.
Requirements outside of the 1937 Housing Act, such as fair housing
rules, cannot be waived under MTW. In addition, certain sections of the
act, including those that cover labor requirements and the demolition and
disposition of public housing, cannot be waived. The 1996 act that
created the program requires participating agencies to address three
purposes and meet five requirements. Specifically, the three statutory
purposes are to (1) reduce costs and achieve greater cost-effectiveness
in federal housing expenditures, (2) give families with children incentives
to obtain employment and become self-sufficient, and (3) increase
housing choices for low-income families. For example, to reduce
administrative costs MTW agencies can reduce the frequency of income
verifications for households with fixed incomes. In making these changes,
MTW agencies must

1. serve substantially the same total number of eligible low-income
   families that they would have served had funding amounts not been
   combined;

2. maintain a mix of families (by family size) comparable to those they
   would have served without the demonstration;

3. ensure that at least 75 percent of households served are very low
   income; 6

4. establish a reasonable rent policy to encourage employment and self-
   sufficiency; and

5. assure that the housing they provide meets HUD’s housing quality
   standards.

The program’s ultimate goal is to identify successful approaches that can
be applied to PHAs nationwide.



5
42 U.S.C. 1437 et seq.
6
 Section 3 of the 1937 Housing Act, as amended, defines very low-income families as
families whose incomes do not exceed 50 percent of the median family income for the
area.




Page 4                                        GAO-12-490 Moving to Work Demonstration
The 1996 act authorized MTW for 30 agencies. HUD invited PHAs to
apply for the program and selected an initial cohort of 24 PHAs from
among the respondents. Six more were added as a result of the Jobs-
Plus initiative. 7 Since then, some agencies have opted not to participate,
and additional agencies have been added to replace them. Other
agencies have been added through specific appropriations language (see
fig. 1). In addition, some agencies have completed their participation and
exited the program. 8 As of January 2012, a total of 39 PHAs were
authorized to participate, and 35 were participating.




7
 The Jobs-Plus initiative was a welfare-to-work demonstration aimed at significantly
increasing employment and income among the residents of selected PHAs. HUD
considers Jobs-Plus a subset of the MTW demonstration.
8
 According to HUD, 3 of the original 24 MTW agencies completed their participation in
2004 and were terminated from the program based on the terms of their MTW
agreements.




Page 5                                          GAO-12-490 Moving to Work Demonstration
Figure 1: Timeline of Public Housing Agencies Entering and Exiting the MTW Demonstration Program

 MTW authorized by the                                                                                                                                       3 MTW agencies
 Omnibus Consolidated                                                                                                                                        authorized (the 2011
 Rescissions and Appropriations                                                                                         3 PHAs added by HUD                  Continuing Appropria-
 Act of 1996; the act authorized                                                                                        through a competitive                tions Act authorized
 30 MTW agencies (24 PHAs               HUD selected 6 MTW agencies                                                     selection process (the 2009          the addition of 3 more
 were selected and 6 slots were         through a competitive selection                                                 Appropriations Act authorized        agencies, expanding
 filled through HUD’s Jobs-Plus         process; 1 MTW agency added                     3 MTW agencies                  the addition of 3 more MTW           the total number of
 initiative)                            as authorized by 1996 statute                   completed participation         agencies, expanding the total        authorized agencies
                                                                                                                        number of authorized                 to 39)b
                                                                                                                        agencies to 33) a



      1996                           1999      2000                                2003       2004                                       2008      2009       2010      2011




                          MTW implemented; 5 of the 24                     6 (Jobs-Plus) selectees                1 MTW agency added as authorized by
                          PHAs originally selected opted                   completed participation                1996 statute (slot available from Jobs-Plus);
                          not to participate; 1 Jobs-Plus                                                         1 former MTW agency returned; 4 PHAs
                          selectee opted not to                                                                   specifically named and authorized to join in
                          participate; 2 agencies                                                                 the 2008 Appropriations Act
                          specifically named and
                          authorized to join in the 1999
                          Appropriations Act                                                                                   2 MTW agencies selected through a competitive
                                                                                                                               process (the 2010 Appropriations Act authorized
                                                                                                                               the addition of 3 more agencies, expanding the
                                                                                                                               total number of authorized agencies to 36) b


                                                                HUD = Department of Housing and Urban Development
                                                                MTW = Moving to Work
                                                                PHA = public housing agency
                                                                Source: HUD.

                                                            a
                                                             The act specified that the additional agencies must be considered “high performers” (as designated
                                                            by HUD’s Public Housing Assessment System) with current HOPE VI grants and no more than 5,000
                                                            units. HUD’s Public Housing Assessment System assigns a performance designation to each PHA
                                                            that is based on the physical condition of the properties, the agency’s financial condition and
                                                            management operations, and a survey of residents’ satisfaction with services provided. HOPE VI
                                                            grants fund the revitalization of severely distressed public housing.
                                                            b
                                                             The 2010 and 2011 Appropriations Acts specified that the additional agencies must be considered
                                                            “high performers” with no more than 5,000 units.

                                                            MTW agencies do not receive special funding allocations. Rather, they
                                                            receive funds from the three traditional primary funding sources (public
                                                            housing capital funds, public housing operating funds, and Housing
                                                            Choice Vouchers). 9 Traditionally, PHAs have been required to use the


                                                            9
                                                             For additional information on these funding sources, see Congressional Research
                                                            Service, Introduction to Public Housing, R41654 (Washington, D.C.: Feb. 24, 2011) and
                                                            Congressional Research Service, Section 8 Housing Choice Voucher Program: Issues
                                                            and Reform Proposals, RL34002 (Washington, D.C.: Jan. 6, 2011).




                                                            Page 6                                                          GAO-12-490 Moving to Work Demonstration
funds from each separate source only for specific purposes, but MTW
agencies may combine the money from the three sources and use them
for a variety of HUD-approved activities. This fungibility is intended to give
MTW agencies greater flexibility. For example, public housing operating
funds are traditionally used to make up the difference between the rents
charged for units and the actual cost of operating them. Capital funds are
used for modernization efforts and management improvements, while
voucher funds provide rental assistance in the private market. However,
by combining funds an MTW agency may use public housing capital
funds to issue additional vouchers or use voucher funds to develop more
public housing to better fit the needs of its community. MTW agencies
also have the authority to use their funds to implement innovative
activities that differ from traditional housing assistance. For instance, an
MTW agency can use funds to replace public housing with mixed-income
communities or reach special-needs populations, such as the homeless,
using vouchers paired with supportive services.

A Standard Agreement, executed in 2008 to replace individual contracts
between HUD and participating PHAs, governs the conditions of
participation in the program. 10 HUD enters into this agreement with each
MTW agency. HUD created the agreement to standardize the language in
its contracts and its reporting requirements and to help create a more
operationally sound program. The Standard Agreement includes a
termination date (the end of each agency’s 2018 fiscal year) and an
attachment that sets out reporting requirements (Attachment B). While
much of the Standard Agreement is the same for all MTW agencies, two
sections are tailored to individual agencies: a description of the formulas
for determining the amounts of funding each agency will receive and an
optional section that may include some agency-specific authorizations.

The agreement requires all MTW agencies to submit an annual MTW
plan to HUD for approval and an annual MTW report for acceptance. 11
While the format may vary, HUD has established a standard table of


10
  Earlier agreements varied across participants and had terms that ran from 5 to 7 years.
Amendments to extend the terms or add additional exceptions and flexibilities were
common, and over time the changes made the agreements difficult for HUD to monitor.
11
  MTW agencies with less than 10 percent of their housing stock in the MTW program
continue to submit the 5-Year and annual plans required by Section 5A of the 1937
Housing Act. Only information not included in these documents would need to be included
in a supplemental annual MTW plan.




Page 7                                          GAO-12-490 Moving to Work Demonstration
contents that lists the information that agencies must include in their
annual plans and reports. For example, the plan must include, among
other things, a description of how each planned activity relates to at least
one of the three purposes of the program; baselines, proposed
benchmarks, and proposed metrics for assessing the outcomes of each
activity; citations of the authorizations that give the agency the flexibility to
conduct the activity; and descriptions of required rent reform activities. 12
In addition, the plan must include a certification that the agency published
a notice of plans to hold a public hearing on the plan, made the agency’s
annual plan available for public inspection, and conducted a public
hearing to discuss the annual plan prior to its approval. Similarly, the
Standard Agreement outlines the information that MTW agencies are
required to include in annual MTW reports. These reports must include,
for example, detailed information on the impact of each activity, including
comparisons of actual outcomes to the benchmarks proposed in the
annual plan. If the agencies do not achieve the benchmarks or the
activities are determined to be ineffective, the MTW agency is required to
describe the challenges, and, if possible, identify alternative activities that
may be more effective. MTW agencies also are required to self-certify
that they are in compliance with three statutory requirements: assisting
substantially the same total number of eligible low-income families that
they would have served had funding amounts not been combined;
maintaining a mix of families (by family size) comparable to those they
would have served had funding amounts not been combined under the
demonstration; and ensuring that at least 75 percent of households
served are very low income.

As well as meeting the requirements in the Standard Agreement, MTW
agencies must submit tenant-related data into the Moving to Work section
of the Public and Indian Housing Information Center (MTW-PIC).
According to HUD officials, the MTW-PIC module was created because
the standard PIC system that non-MTW agencies use to report tenant
data could not accommodate some of the activities allowed under MTW,
such as less frequent tenant income recertifications and rent calculations
that vary from HUD’s standard calculations. The MTW-PIC module was
created in 2007, and most MTW agencies had transitioned to it by 2008.


12
  In training materials for the MTW program, HUD defines “baseline” as the performance
level that was being achieved prior to the implementation of the MTW activity,
“benchmark” as the anticipated outcome of a MTW activity, and “metric” as the specific
measure that quantifies the changes from an activity.




Page 8                                        GAO-12-490 Moving to Work Demonstration
                      The Office of Public Housing Investments within the Office of Public and
                      Indian Housing at HUD headquarters is the designated program office for
                      the MTW demonstration program. Within the Office of Public Housing
                      Investments is an MTW Office that includes a program director and four
                      coordinators who are each assigned to a specific group of MTW
                      agencies. The MTW Office is responsible for, among other things,
                      processing, reviewing, and approving all annual plans submitted by MTW
                      agencies; establishing guidelines for MTW agencies; monitoring approved
                      activities and accomplishments; and accepting annual reports. Individual
                      MTW coordinators facilitate the reviews of planned and implemented
                      activities and are responsible for coordinating with other HUD offices,
                      including local HUD field offices, to obtain additional input on MTW
                      agencies’ planned activities and accomplishments. In January 2011, the
                      Office of Public Housing Investments signed a Memorandum of
                      Understanding with HUD’s Office of Field Operations to increase
                      collaboration and formally describe the roles and responsibilities of the
                      MTW Office and HUD field office staff. Per this memorandum, field office
                      staff assist the MTW Office by reviewing and providing input on annual
                      plans and reports, ensuring that agencies are reporting tenant
                      information, and participating in annual site visits.


                      MTW agencies provide descriptions of their activities and performance
HUD Has Not           information in their annual reports to HUD. They show how the activities
Identified Standard   link to the program’s statutory purposes in their annual plans, as required,
                      and sometimes also in their annual reports. However, the type of
Performance Data      performance information they provide varies, and HUD has provided
and Indicators        limited guidance. While varied information on individual activities is
                      available, a comprehensive evaluation of the MTW program is lacking, in
Needed to Evaluate    part because HUD does not have a plan for identifying and analyzing
the Program           standard performance data and has not established performance
                      indicators for the MTW program as a whole. 13 Further, HUD does not
                      have a systematic process for identifying lessons learned by individual
                      MTW agencies that can be replicated at other PHAs.




                      13
                       For purposes of this report, we are defining “performance indicators” as programwide
                      measures of impact.




                      Page 9                                        GAO-12-490 Moving to Work Demonstration
Although Information on                    MTW agencies report information on specific activities, including
Activities Is Generally                    descriptions, in their annual reports. Agencies are required in their annual
Available at the Agency                    plans to link each of their proposed activities to one of the program’s
                                           three statutory purposes, and some agencies also show links between
Level, It Varies Due to                    ongoing activities and statutory purposes in their annual reports. The
HUD’s Limited Guidance                     three statutory purposes are to reduce costs and achieve greater cost-
                                           effectiveness in federal housing expenditures, give families with children
                                           incentives to obtain employment and become self-sufficient, and increase
                                           housing choices for low-income families. According to the most recently
                                           available annual reports, 30 agencies have over 360 ongoing activities,
                                           including rent reform initiatives and work requirements (see table 1). 14

Table 1: Examples of MTW Activities, by Statutory Purpose

Activity                                             Description (example)
Purpose 1: Reduce costs
Biennial recertifications                            Allow elderly and disabled residents, who have minimal income and experience
                                                     few changes in family composition, to be recertified every other year instead of
                                                     annually.
New inspections protocol                             Replace annual inspections of all properties in the tenant- and project-based
                                                     voucher programs with inspections of a certain percentage of participating
                                                     properties.
Standard deductions for medical expenses             Replace itemized medical and child care deductions with a standard deduction
(rent reform)                                        in determining rent payments.
Purpose 2: Encourage self-sufficiency
Minimum rents (rent reform)                          Require residents to pay at least a minimum rent (MTW agencies can exceed
                                                     the statutory limit of $50).
Work requirement                                     Require residents to work or engage in a learning or training activity, meet a
                                                     minimum income level to receive housing assistance, or enter and adhere to a
                                                     local welfare-to-work program. In some cases, residents may not be
                                                     unemployed for more than 90 days.
Enhanced supportive services                         Offer residents supportive services, such as on-site employment counseling
                                                     and skill building, classes in household budgeting and parenting, and child-care
                                                     and transportation services.
Purpose 3: Increase housing choices
Fund affordable housing development activities       Leverage funds to preserve affordable housing resources and create new
                                                     affordable housing opportunities.




                                           14
                                             Of the 35 participating agencies, 30 had issued an annual report for 2011 as of January
                                           2012. The other five agencies had not been in the program long enough to report on their
                                           accomplishments.




                                           Page 10                                         GAO-12-490 Moving to Work Demonstration
Activity                                     Description (example)
Sponsor-based voucher program                Provide vouchers to service providers who work with hard-to-house households
                                             to provide intensive supportive services.
Site-based waiting list                      Allow residents to apply for the waiting list for specific sites or communities
                                             where they want to live.
                                Source: Most recent MTW annual reports as of January 2012.

                                Note: The links between the activities and statutory purposes were determined by MTW agencies.

                                According to the most recent annual reports (and corresponding plans)
                                for 30 MTW agencies, agencies associated the largest percentage of
                                ongoing activities (41 percent) with the statutory purpose of reducing
                                costs and improving cost-effectiveness (see fig. 2). For example,
                                agencies associated changes in certification schedules, inspection
                                protocols, and medical deductions with reduced costs. The agencies
                                linked 30 percent of their ongoing activities to the statutory purpose of
                                increasing housing choices and 24 percent to encouraging self-
                                sufficiency. The agencies did not link 4 percent of their ongoing activities
                                to any purpose in either their most recent annual plan or report. 15




                                15
                                  According to HUD officials, once an activity is approved, agencies may not continue to
                                report its linkages to statutory purposes in subsequent plans or reports.




                                Page 11                                                      GAO-12-490 Moving to Work Demonstration
Figure 2: Number of Reported MTW Activities Linked to Each of the Statutory
Purposes




Note: Numbers do not add to 100 percent because of rounding.


In its Standard Agreement, HUD requires agencies to include in their
annual reports performance information on the impact of each
implemented activity, including describing the metrics used to assess
outcomes and comparing actual performance with proposed
benchmarks. 16 While HUD did not define these terms in the agreement, it
defined them in 2009 training materials. In these materials, HUD defined
a metric as the “unit of measure that quantifies the changes that might
occur as a result of the MTW activity” and a benchmark as the “projected
outcome of the MTW activity.” Further, in these 2009 training materials
HUD defined an outcome as the “actual, measured result of the
implemented activity.” As examples, the training materials stated that a
metric could be the hours of staff time saved, a benchmark could be the
number of anticipated staff hours saved, and the outcome could be staff
hours actually saved. HUD directs agencies to develop their own metrics


16
  For purposes of this report, we are defining “performance information” as information
that MTW agencies include in their annual reports on the impact of a specific MTW
activity.




Page 12                                            GAO-12-490 Moving to Work Demonstration
and benchmarks for each activity based on local and community
standards. 17 Our analysis of the most recent annual reports for 30 MTW
agencies showed that the agencies reported performance information for
91 percent of the ongoing activities included in the reports and used over
1,000 metrics to assess these activities. 18 MTW agencies collectively met
the benchmarks associated with 40 percent of these metrics and fell short
of meeting 17 percent of them. 19 For 30 percent of the metrics, it was too
soon to determine if the benchmarks had been met because the activities
were not yet completed. For the remaining 13 percent, information (either
the benchmark or performance data) was lacking to determine whether
the benchmark was met.

While MTW agencies are generally devising their own metrics for
activities and reporting performance information, the usefulness of this
information is limited because, in some cases, it is not outcome-oriented.
Our analysis of the most recent annual reports for 30 MTW agencies
showed that the type of information that agencies reported on the impact
of their activities varied. For example, for similar activities designed to
promote family self-sufficiency, one MTW agency reported only the
number of participants, which is generally considered an output, and
another did not provide any performance information. 20 In contrast, a third
agency reported on, among other things, the average income of program
graduates, which we consider an outcome.




17
  HUD has made some exceptions to its requirement for the reporting of impact
information. For example, some MTW agencies identified the use of funding flexibility as
an activity and noted that they were not required by HUD to report information on the
impact. According to the Director of the MTW Office, this exception applies in some
instances.
18
 Agencies often had more than one metric for a single activity.
19
  HUD does not penalize agencies for not meeting their self-determined benchmarks.
However, if the benchmarks were not achieved or if the activity was determined
ineffective, HUD requires agencies to describe the challenges and, if possible, identify
new strategies that might be more effective in their annual report. For our analysis, we
compared the actual, reported performance information with the proposed benchmark to
determine if the agency met its goal.
20
  For example, see GAO, Neighborhood Stabilization Program: HUD and Grantees Are
Taking Actions to Ensure Program Compliance but Data on Program Outputs Could be
Improved, GAO-11-48 (Washington, D.C.: Dec.17, 2010).




Page 13                                         GAO-12-490 Moving to Work Demonstration
Internal control standards state that good guidance (information and
communication) is a key component of a strong internal control
framework and that there is a need for clear documentation. 21 To be
consistent with the GPRA Modernization Act of 2010, HUD’s guidance on
reporting performance information should indicate the importance of
outcome-oriented information. 22 Specifically, the act states that an agency
should establish efficiency, output, and outcome indicators for each
program activity. Furthermore, Office of Management and Budget
guidance on implementation of the act states that quantitative and
outcome-focused measures are preferred. 23

At the time of our review, HUD’s guidance did not specify that agencies
should report quantifiable and outcome-oriented performance information.
According to the Director of the MTW Office, Attachment B of the
Standard Agreement is the most current guidance on the information that
agencies should report in their annual reports. It simply states that
agencies are to provide detailed information on the impact of the activity
and compare it against the proposed benchmarks to assess outcomes,
including whether an activity is on schedule. The attachment does not
define terms or set expectations for the type of information to be reported.
After the Standard Agreement was executed in 2008, HUD conducted
training for participating agencies. As previously discussed, the 2009
training materials defined key terms such as a metric and outcome and
outlined steps agencies could take to evaluate their activities. HUD also
encouraged the MTW agencies to use metrics and benchmarks that did
not focus on the number of individuals participating in an activity but
rather on the objectives of the activity and to report quantifiable
information. While HUD has posted the 2009 training materials on its
website, these materials have not been incorporated into Attachment B of
the Standard Agreement.

According to the Director of the MTW Office, HUD has not made its
guidance more specific because agencies are implementing a wide


21
  GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1
(Washington, D.C.: November 1999).
22
 Pub. L. No. 111-352, 124 Stat. 3866 (2011). The GPRA Modernization Act of 2010
amends GPRA, Pub. L. No. 103-62, 107 Stat. 285 (1993).
23
 OMB Circular A-11, Part 6 (“Preparation and Submission of Strategic Plans, Annual
Performance Plans, and Annual Program Performance Reports”), Section 220.7.IV.




Page 14                                       GAO-12-490 Moving to Work Demonstration
                           variety of activities and thus require some reporting flexibility. We
                           acknowledge the need for flexibility, but it is important that HUD require
                           agencies to report at least some outcome-oriented performance
                           information. Without more specific guidance on the reporting of
                           performance information, HUD cannot be assured of collecting
                           information that reflects the outcomes of individual activities. Such
                           information would help HUD assess the demonstration program and
                           whether the activities are furthering program purposes.


Design and Data            As we have previously reported, obtaining performance information from
Weaknesses Have            demonstration programs that are intended to test whether an approach
Prevented a                (or any of several approaches) can obtain positive results is critical. 24 This
                           information is needed to help determine whether the program has led to
Comprehensive Evaluation
                           improvements consistent with its purposes. HUD has sponsored three
of MTW                     broad reviews of the MTW program, but these studies are not
                           comprehensive evaluations because of data limitations, among other
                           things. 25 A 2004 Urban Institute evaluation of the MTW program found
                           that most MTW agencies reported modest benefits from activities related
                           to administrative streamlining and that these results were often not as
                           dramatic as the agencies had anticipated. 26 The study also noted the
                           difficulty in determining whether MTW activities related to employment




                           24
                            See GAO, Program Evaluation: Improving the Flow of Information to the Congress,
                           GAO/PEMD-95-1 (Washington, D.C.: Jan. 30, 1995).
                           25
                             We also identified five studies that looked at a specific MTW agency or activity. See
                           Thomas Boston and Linje Boston, Monitoring and Evaluating the Atlanta Housing
                           Authority’s MTW Program: Comprehensive Final Report Covering 2004, 2007 and 2010
                           (Atlanta, GA: Euquant, 2010); Quadel Consulting Corporation, Performance
                           Benchmarking of the Rent Simplification Initiative in the Cambridge Housing Authority’s
                           Moving to Work Program – Final Report (Cambridge, MA: Cambridge Housing Authority,
                           2008); Building Changes, The South King County Housing First Pilot Innovations and
                           Lessons Learned (Seattle, WA: Building Changes, May 2010); Sarah Delaney Rosendahl,
                           Housing Readiness Program Year One Evaluation for the Housing Authority of the County
                           of San Mateo (Oakland, CA: Mills College, May 2010); and Thomas D. Boston, Impact of
                           the Mixed-Income Revitalization of Grady Homes: Atlanta Housing Authority, 2011
                           (Atlanta, GA: Housing Authority of the City of Atlanta, July 2011).
                           26
                             Martin D. Abravanel, Robin E. Smith, Margery A. Turner, Elizabeth C. Cove, Laura E.
                           Harris, and Carlos A. Manjarrez, Housing Agency Responses to Federal Deregulation: An
                           Assessment of HUD’s “Moving to Work” Demonstration (Washington, D.C.: The Urban
                           Institute, 2004).




                           Page 15                                       GAO-12-490 Moving to Work Demonstration
and income had any independent effect and that MTW activities resulted
in both greater and more limited housing choice. 27

A second study, conducted in 2007 by Applied Real Estate Analysis, Inc.
and the Urban Institute, reviewed eight MTW agencies that had placed
limits on the length of time that residents could receive housing
assistance. 28 The study found that each of the approaches varied and
concluded that only limited information was available with which to
evaluate outcomes or establish cause-and-effect relationships between
agencies’ policies and recipients’ experiences. It noted that there were
significant limitations to what could be learned from these experiences
because no evaluative framework had been built into the program.
Finally, a 2010 HUD Report to Congress found that the effects of many
MTW activities, especially as they related to residents, could not be
conclusively identified because of the variety of and differences in the
activities and metrics that MTW agencies were implementing. However,
the report did identify some results concerning agencies’ ability to more
efficiently allocate resources and engage in strategic long-term
planning. 29 For instance, the study noted that some agencies had seen
positive effects from combining their traditional sources of funding and
streamlining their operations—for example, by simplifying their housing
quality inspections.

These three studies of the MTW program and our work have identified
several challenges that have hindered evaluation efforts. These
challenges include the way the program was initially designed and the
resulting lack of standard performance data as well as the lack of
performance indicators for the program as a whole.




27
  Because of the program’s design and lack of controls, the study’s authors had difficulty
isolating the effects of the MTW program from other factors that may affect outcomes.
28
  Robert Miller, Martin D. Abravanel, Helene Berlin, Elizabeth Cove, Maria-Alicia
Newsome, Carlos A. Manjarrez, Lipi Saikia, Robin E. Smith, and Maxine V. Mitchell, The
Experiences of Public Housing Agencies That Established Time Limits Policies Under the
MTW Demonstration (Applied Real Estate Analysis, Inc. and The Urban Institute, May
2007).
29
 HUD, Office of Public and Indian Housing and Office of Policy Research and
Development, Moving to Work: Interim Policy Applications and the Future of the
Demonstration, a report to Congress (Washington, D.C.: 2010).




Page 16                                         GAO-12-490 Moving to Work Demonstration
Lack of Program Evaluation   HUD has taken steps to address the problems noted with MTW’s initial
Design and Standard          design and the lack of standard data; however, it has not analyzed the
Performance Data             data it currently collects or determined whether these data are sufficient
                             to evaluate similar activities and the program as a whole. As we have
                             previously reported, comparable data are essential to a full analysis of
                             programs that incorporate a variety of activities. 30 We also noted that
                             obtaining performance information from demonstration programs that are
                             intended to test whether an approach (or any of several approaches) can
                             obtain positive results is critical. Finally, we have reported that agencies
                             need to identify any data that will be needed to assess the effectiveness
                             of program regulations. 31

                             Researchers and others have noted the limitations that the program’s
                             initial design posed to evaluation. In the early years of MTW, rigorous
                             evaluation strategies were not required, and the program lacked a
                             research design that would have helped in determining baseline
                             information. 32 The 2004 Urban Institute review of MTW concluded that
                             there were limits to what could be learned from its review for a variety of
                             reasons, such as the inability to separate individual components of
                             agencies’ MTW activities for analysis. 33 As a result, the report is mainly
                             descriptive and qualitative. 34 HUD’s 2010 Report to Congress noted that
                             because rent reform activities varied greatly and were not implemented
                             using a controlled experimental methodology, the authors were unable to
                             recommend specific reforms as best practices. 35




                             30
                               GAO/PEMD-95-1.
                             31
                               GAO, Reexamining Regulations: Opportunities Exist to Improve Effectiveness and
                             Transparency of Retrospective Reviews, GAO-07-791 (Washington, D.C.: July 16, 2007).
                             32
                               In its 2009 training materials, HUD defined a baseline as the performance level that was
                             being achieved prior to the implementation of the MTW activity.
                             33
                               Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).
                             34
                               Researchers from the Urban Institute told us that if very specific reporting requirements
                             were established and reliable data were available, the program could be rigorously
                             assessed.
                             35
                                HUD, Moving to Work (2010). A controlled experimental methodology is a research
                             design that randomly assigns participants to treatment and control groups in order to
                             rigorously analyze the effects of the studied activity.




                             Page 17                                          GAO-12-490 Moving to Work Demonstration
To help evaluate aspects of the MTW program moving forward, MTW
officials have added requirements for new agencies. According to HUD’s
2010 Report to Congress, the three agencies admitted to the MTW
program in 2009 had strong evaluation components. 36 Two of these
agencies have commitments from local universities to evaluate their
programs. Additionally, HUD has required the two newest agencies to
participate in a controlled rent reform study. However, these
improvements will not help evaluate the program as a whole or the
activities implemented by the 30 other MTW agencies.

Likely due to the absence of an evaluation framework for the MTW
program, researchers have noted the lack of standard performance data
needed to evaluate similar activities and the program as a whole. The
2004 Urban Institute study noted that the lack of consistent data on
resident characteristics, incomes, and rent payments prevented the
authors from being able to determine whether individual agencies were
able to achieve the goal of increasing self-sufficiency. 37 Similarly, the
HUD Inspector General reported in 2005 that HUD lacked the empirical
data needed to assess the program as a whole. 38

Since these reports, HUD has started collecting some additional data
from MTW agencies, but it has not yet analyzed the data. HUD created
the MTW-PIC module to collect tenant characteristics such as household
size, income, and educational attainment. However, according to MTW
officials, HUD has not used these data to analyze the program’s effects,
such as changes in resident income. In addition, HUD’s Standard
Agreement has required agencies since 2009 to provide information in
their annual reports on the impact of activities, including benchmarks and
metrics. While these reports are informative, they do not lend themselves
to quantitative analysis because the reporting requirements do not call for
standardized data, such as the number of residents that found
employment.




36
 HUD, Moving to Work (2010).
37
 Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).
38
  HUD, Office of Inspector General, Design and Implementation of the Public
Housing/Section 8 Moving to Work Demonstration Program, 2005-SE-001 (Seattle, WA:
Apr. 12, 2005).




Page 18                                     GAO-12-490 Moving to Work Demonstration
                                 In addition, whether these data are sufficient to assess similar activities
                                 and the program as a whole is not clear, and HUD has not identified the
                                 data it would need for such an assessment. For example, neither MTW-
                                 PIC nor annual reports capture standard data on implemented activities.
                                 Further, according to the Director of the MTW Office, MTW-PIC does not
                                 include information on individuals who receive nontraditional services
                                 from an MTW agency, such as homeless assistance or case
                                 management. Representatives from MTW agencies have suggested that
                                 HUD should collect some standard data for similar activities. For
                                 example, they noted that if HUD required all agencies that implemented
                                 rent reform activities to report standard data, the results of these efforts
                                 could be analyzed even if the specific activities varied. The Director of the
                                 MTW Office also noted that MTW-PIC was a potential tool for collecting
                                 and analyzing standard demographic data.

                                 The MTW Office has recently developed a statement of work for an
                                 evaluation of the program, but HUD has not allocated funding for the
                                 study, according to the Director of the MTW Office. Among other things,
                                 the proposed evaluation is intended to assess the current state of the
                                 MTW demonstration and determine the extent to which the three statutory
                                 purposes have been addressed. The study is also expected to include an
                                 analysis of outcomes associated with specific activities and the
                                 demonstration as a whole to identify which MTW activities are appropriate
                                 for expansion to all PHAs. However, the approach envisioned may be
                                 limited because it would primarily rely on existing data sources. Until HUD
                                 develops and implements a plan (that includes the identification of
                                 standard data) to quantitatively assess similar activities and the MTW
                                 program as a whole, HUD cannot determine their effectiveness. While
                                 such analyses may be challenging, they would enhance HUD’s ability to
                                 rigorously assess the demonstration.

Lack of Performance Indicators   HUD has not established performance indicators for the MTW program as
                                 a whole. The GPRA Modernization Act of 2010 requires that federal
                                 agencies establish efficiency, output, and outcome indicators for each
                                 program activity as appropriate. 39 Internal control standards also require
                                 the establishment of performance indicators. 40 In addition, we have
                                 previously reported that successful performance indicators demonstrate


                                 39
                                  Pub. L. No. 111-352, 124 Stat. 3866 (2011).
                                 40
                                  GAO/AIMD-00-21.3.1.




                                 Page 19                                        GAO-12-490 Moving to Work Demonstration
                          results and provide useful information for decision making in order to
                          track how programs and activities can contribute to attaining an
                          organization’s goals and mission, among other things. 41

                          As previously discussed, MTW agencies set their own performance
                          metrics for activities, but HUD has not established performance indicators
                          for the program as a whole. HUD’s Fiscal Year 2011 Annual Performance
                          Plan established agencywide performance indicators but did not explicitly
                          connect the MTW program to any of them. 42 Although there are no
                          specific targets for the program, the Director of the MTW Office noted that
                          the program’s activities support some of the agencywide indicators.
                          Specific performance indicators for the MTW program could be based on
                          the statutory purposes. For example, agencies could report on the
                          savings achieved (reducing costs) and the number of additional
                          households served (increasing housing choices). Without performance
                          indicators for the MTW program, however, HUD cannot demonstrate the
                          results of the program as a whole.


HUD Lacks a Systematic    While HUD has identified some lessons learned on an ad hoc basis, it
Process for Identifying   does not have a systematic process in place for identifying such lessons.
Lessons Learned           We have previously reported that obtaining impact information from
                          demonstration programs that are intended to test whether an approach
                          (or any of several approaches) can obtain positive results is critical. 43 This
                          information should be gathered to help determine whether programs have
                          led to improvements consistent with their purposes.

                          Since 2000, HUD has identified some activities that could potentially be
                          replicated by other PHAs. A HUD-sponsored contractor developed five
                          case studies from 2000 to 2002 that were intended to describe some of
                          the issues, successes, and challenges involved in implementing the MTW
                          demonstration. Four of the case studies were developed around specific
                          MTW sites, while the fifth provided an overview of the block grant
                          approach. Further, from June 2008 to April 2009 HUD identified practices
                          that, with statutory or regulatory changes, could be replicated at other


                          41
                           GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season
                          Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 2002).
                          42
                           HUD, Fiscal Year 2011 Annual Performance Plan (Washington, D.C.: 2011).
                          43
                           GAO/PEMD-95-1.




                          Page 20                                      GAO-12-490 Moving to Work Demonstration
PHAs nationwide. These practices, which are posted on HUD’s website,
included implementing savings accounts for public housing and voucher
recipients to promote resident savings. Most recently, HUD’s 2010 Report
to Congress described promising policies, practices, and concerns. 44 In
addition, officials from some of the MTW agencies we interviewed noted
that HUD officials had shared information on activities that had shown
positive effects during site visits, quarterly phone calls, newsletters, and
annual conferences. Finally, HUD’s statement of work for its proposed
evaluation of the MTW program includes the creation of five case studies
that would review MTW flexibilities.

However, these efforts have shortcomings. In most cases, the practices
chosen were based on the opinions of HUD or contracted staff and
largely involved anecdotal (or qualitative) data rather than quantitative
data. The lack of standard performance data has affected HUD’s ability to
systematically identify lessons learned. In its 2005 report on the MTW
program, the HUD Inspector General noted that the lack of data on the
program made it difficult to identify activities that could be considered
models for addressing the three statutory purposes or that could be used
to show the importance of individual policy changes. 45 Further, HUD has
not established criteria, such as demonstrated performance, for
identifying lessons learned. Finally, HUD has not made regular efforts to
review and identify lessons learned. Because HUD does not currently
have a systematic process for identifying lessons learned, it is limited in
its ability to promote useful practices that could be implemented more
broadly.




44
 HUD, Moving to Work (2010).
45
 HUD, Office of Inspector General, 2005-SE-001.




Page 21                                     GAO-12-490 Moving to Work Demonstration
                           HUD has policies and procedures in place to monitor MTW agencies.
HUD Generally              First, HUD requires program staff to review and comment on agencies’
Follows Its                annual plans and reports. Second, staff review tenant data submitted by
                           MTW agencies. Third, program staff conduct annual site visits at each
Monitoring Policies        participating agency to provide technical assistance and program
and Procedures, but        updates. HUD generally follows these policies and procedures, which
Could Strengthen           focus on technical assistance rather than compliance. Due in part to this
                           focus, HUD’s policies and procedures have several key weaknesses.
Them                       Specifically, HUD has not clarified program terminology, ensured that
                           each MTW agency is meeting statutory requirements, performed an
                           annual risk assessment, or developed policies and procedures to verify
                           the accuracy of key information that MTW agencies self-report.


HUD Reviews Agency         HUD’s monitoring policies and procedures for the MTW program are
Information and Conducts   contained in a desk guide, which describes the roles and responsibilities
Annual Site Visits         of HUD staff in reviewing annual plans and reports and data submissions,
                           making site visits, and performing other monitoring activities. In January
                           2011, HUD’s Office of Public Housing Investments and Office of Field
                           Operations signed a Memorandum of Understanding documenting the
                           framework for headquarters and field staff to follow in overseeing MTW
                           agencies. According to the memorandum, the MTW Office (within the
                           Office of Public Housing Investments) is responsible for oversight of the
                           MTW program. In many cases, the MTW Office works with field offices to
                           jointly develop responses to MTW agency issues. Further, a MTW
                           Working Group—consisting of representatives from Public and Indian
                           Housing programs, the Real Estate Assessment Center, and the Office of
                           Policy Development and Research—was established to assist with the
                           annual plan and report review process. 46 As part of the memorandum, the
                           Offices of Public Housing Investments and Field Operations agreed to the
                           protocols set forth in the desk guide.

                           HUD staff from the MTW Office and field offices and the MTW Working
                           Group share responsibility for reviewing and commenting on participating
                           agencies’ annual plans and reports. The Standard Agreement



                           46
                              HUD’s Real Estate Assessment Center administers physical inspections of HUD’s public
                           and multifamily housing properties. The Office of Policy Development and Research is
                           responsible for maintaining current information on housing needs, market conditions, and
                           existing programs, as well as for conducting research on priority housing and community
                           development issues.




                           Page 22                                       GAO-12-490 Moving to Work Demonstration
(Attachment B) outlines the requirements for annual plans and reports
that agencies must submit. MTW coordinators, who are each responsible
for a specific number of MTW agencies, have the lead role in reviewing
annual plans and reports to determine if they meet the requirements of
Attachment B and obtaining input from other HUD staff, including field
offices and the MTW Working Group. Field offices are required to review
the annual plans and reports submitted by the MTW agencies in their
jurisdictions and provide their assessment to the MTW coordinator.
Similarly, the MTW Working Group reviews and provides comments to
the MTW coordinator. Coordinators summarize the comments from the
field offices and MTW Working Group and send them to the agencies.
The coordinators and field office staff work with MTW agencies to resolve
any outstanding issues. Once such issues have been resolved, the MTW
Office approves annual plans and accepts annual reports. 47

In 2009, the MTW Office developed review procedures and checklists for
the coordinators and field office staff to use when reviewing annual plans
and reports. The checklists provide a framework for MTW coordinators to
determine whether the annual plans and reports are complete and
consistent with the requirements of Attachment B. More specifically,
coordinators are to review plans to ensure that they include how
proposed activities relate to at least one of the statutory purposes and
show the anticipated impact on the related purpose; corresponding
baselines, benchmarks, and metrics; and information on rent reform
initiatives, among other items. In addition, they are to ensure that the
plans include a certification that the agency published a notice of plans to
hold a public hearing to discuss its annual plan, made the agency’s
annual plan available for public inspection, and conducted a public
hearing to discuss the annual plan prior to its approval, among other
things. 48 Further, coordinators are to review reports to ensure that they
describe the impact of each implemented activity, explain any
benchmarks that were not achieved and any revised benchmarks or
metrics, and include the agency’s certification that it has met three of the



47
  According to HUD officials, HUD staff “accept” rather than approve annual reports
because, unlike the annual plans, the reports do not include a request to conduct certain
activities.
48
  Soliciting public comments on the plan is important because, unlike other PHAs, MTW
agencies are not subject to the Public Housing Assessment System, which includes a
customer satisfaction survey that promotes resident participation.




Page 23                                         GAO-12-490 Moving to Work Demonstration
five statutory requirements. 49 However, these procedures do not require
MTW coordinators to verify each agency’s certification that it has met the
three statutory requirements.

Interviews with MTW coordinators and field staff and documentation for
our sample of seven MTW agencies indicated that HUD generally
followed these procedures. Documentation we reviewed for the agencies
in our sample showed that the coordinators generally completed
checklists while reviewing annual plans and reports. For example,
coordinators verified that all ongoing activities were reported, ensured the
agency included its certification that it had met three of the five statutory
requirements, and made certain the agency certified that it had held a
public hearing on its annual plan, among other requirements.
Coordinators also provided comments to agency staff on annual plans
and reports. 50 Once it had determined that the agency had addressed all
of the comments, the MTW Office notified the agency in writing that its
plan had been approved and report accepted.

HUD field staff also monitor MTW agencies’ compliance with data
submission requirements, including the requirement to report information
on resident characteristics in MTW-PIC. HUD requires MTW agencies to
report timely, accurate, and complete data on at least 95 percent of the
families receiving housing assistance. 51 To monitor compliance, field staff
review monthly reports from MTW-PIC that summarize each participating
agency’s tenant data reporting rates. The reports we reviewed indicated
that agencies were complying with the reporting requirements. For


49
  MTW agencies certify that they are assisting substantially the same total number of
eligible low-income families that they would have served in the absence of MTW;
maintaining a mix of families (by family size) comparable to those they would have served
without the demonstration; and ensuring that at least 75 percent of households served are
very low income. MTW agencies are not required to certify that they meet the other two
requirements—that MTW agencies establish a reasonable rent policy and assure that
housing provided meets HUD’s quality standards.
50
  Field office staff that we interviewed said that they reviewed annual plans and reports
and sent their comments and concerns to MTW coordinators.
51
  MTW agencies are subject to a variety of other reporting requirements. For example,
MTW agencies are required to report voucher utilization in the Voucher Management
System. They also must procure a public accountant to perform an Office of Management
and Budget Circular A-133 compliance audit and submit unaudited financial statements. In
addition, they are subject to HUD physical and management inspections of public housing
and on-site monitoring reviews related to voucher reporting.




Page 24                                          GAO-12-490 Moving to Work Demonstration
example, in January 2012, MTW agencies overall achieved a 100-percent
tenant data submission rate.

In addition, HUD conducts annual site visits to provide technical
assistance to each MTW agency. The MTW Office and the local field
office conduct these visits jointly. 52 The MTW Office (in particular the
coordinator assigned to the agency) takes the lead role in conducting the
visit, including preparing the agenda, coordinating with the local HUD field
office, and working with the MTW agency to select properties to visit.
According to HUD officials, the primary objective of the site visit is to
provide technical assistance and build a working relationship with the
participating agencies, not to assess compliance with statutory
requirements. However, HUD officials stated that if compliance issues
with statutory purposes are found, HUD staff address these issues during
the site visit, and coordinators often develop timelines for the agency to
come into compliance.

Our analysis of documentation of site visits to participating agencies
indicated that MTW Office and field staff generally followed HUD’s annual
site visit procedures. Specifically, analysis of site visit reports indicated
that HUD officials generally discussed the effectiveness of activities and
helped resolve any outstanding issues. For example, as a result of site
visits, HUD staff recommended that an agency include cost-saving
measures in its annual plan, requested clarification of output measures,
and encouraged one agency to submit articles to the MTW newsletter to
share its experiences on how rent reform encouraged self-sufficiency.

Interviews with our sample of MTW agencies and corresponding field
office officials also indicated that HUD was following its policies and
procedures for annual site visits. MTW agency officials we spoke with
indicated that the site visits were generally beneficial because they
provided an opportunity for in-person discussions that helped facilitate
communication with HUD. HUD’s field office staff noted their active
involvement over the years, which had become more defined with the
issuance of the desk guide in 2011. According to the Director of the MTW
Office, the office is considering conducting future site visits using a risk-


52
  According to HUD officials, the nature of MTW site visits has changed since the
implementation of the Standard Agreement in 2008. Prior to implementation of the
Standard Agreement, HUD hired contractors to conduct site visits and provided monitoring
guidance.




Page 25                                       GAO-12-490 Moving to Work Demonstration
                            based approach. Using this approach, HUD would conduct site visits less
                            frequently but would focus on larger agencies that had implemented a
                            wide range of complex activities and newly admitted agencies that were
                            implementing new activities.

                            To foster information sharing across agencies and provide technical
                            assistance, HUD employs a number of additional strategies. For example,
                            HUD hosts annual conferences to share information with MTW agencies
                            and facilitate information sharing among agencies. The conferences
                            cover a variety of topics, and all participating MTW agencies are invited to
                            attend. For example, the 2011 conference focused on effectively
                            managing funds in a challenging budgetary environment. HUD also has
                            engaged participating agencies in quarterly conference calls and other
                            training related to program changes such as the conversion from PIC to
                            MTW-PIC and the transition to the Standard Agreement. Further, HUD
                            issues notices on various topics, such as MTW reporting requirements,
                            and publishes quarterly newsletters that highlight activities relating to
                            each statutory purpose, among other topics. HUD also publishes each
                            agency’s annual report and researchers’ evaluations of MTW activities on
                            its website.


HUD’s Monitoring Policies   Although HUD follows the policies and procedures that it has in place, it
and Procedures Have         could do more to ensure that MTW agencies are demonstrating
Several Key Weaknesses      compliance with statutory requirements and to identify possible risks
                            relating to activities implemented by each agency, among other things.
                            First, HUD has not issued guidance to participating agencies clarifying
                            key program terms, including definitions of the purposes and statutory
                            requirements of the MTW program. Internal control standards require the
                            establishment of clear, consistent goals and objectives. 53 As previously
                            noted, MTW authorizing legislation established three purposes for the
                            program, and agencies must link each of their activities to one of these
                            purposes. However, HUD has not clearly defined what the language in
                            some of these purposes means, such as “increasing housing choices for
                            low-income families.” MTW agencies have linked activities to this purpose
                            that range from using block grant funding to support homeownership
                            programs to requiring applicants to complete a renter education program
                            to establishing a prisoner re-entry housing program. HUD noted the lack



                            53
                             GAO/AIMD-00-21.3.1.




                            Page 26                                 GAO-12-490 Moving to Work Demonstration
of a clear definition in its 2010 Report to Congress but continued to
require that MTW agencies link activities to this purpose. 54 According to
MTW officials, they have not defined what is meant by “increasing
housing choices” so that agencies have the ability to define this term in a
manner that fits their local needs. In addition, HUD has not clarified what
is meant by “serving a comparable mix of families” but also requires
agencies to comply with this requirement. MTW agencies we spoke with
described varying interpretations of this requirement. For example,
officials from one agency told us that they observed how family sizes
changed in their community and compared those changes to changes in
families within the MTW program, using community survey data and data
from the agency’s internal system. Officials from another agency we
spoke with said that over time it had become increasingly difficult to
determine compliance with this statutory requirement.

HUD has recently taken steps to clarify some terminology, explaining how
agencies can certify that at least 75 percent of the families they serve
have very low incomes and that they are serving substantially the same
number of households under MTW as they did before the program. In
addition, HUD is revising its reporting requirements for MTW agencies. As
part of this process, HUD officials told us that they plan to update their
guidance to more completely collect information related to the program’s
statutory purposes and requirements. They acknowledged that the
guidance could be strengthened to require MTW agencies to provide their
agency-specific definition for the three statutory purposes. As a first step,
they noted that they planned to require agencies to define “self-
sufficiency” by either choosing one of the definitions provided by HUD or
creating their own. Similarly, the officials stated that they would consider
requiring MTW agencies to choose between using HUD’s definition of
increasing housing choices or creating their own definition. Although a
step in the right direction, allowing MTW agencies to create their own
definitions of key terms would make it difficult to assess the effectiveness
of efforts to address statutory purposes. HUD officials also said that the
revised guidance would provide standardized tables for agencies to report
data related to the requirement to serve a comparable mix of families.
Until HUD clearly defines what is meant by all of the statutory purposes
and requirements of the MTW program, HUD cannot effectively determine



54
  This requirement is contained in Attachment B of the Standard Agreement. As
previously noted, an agency can associate an activity with more than one purpose.




Page 27                                        GAO-12-490 Moving to Work Demonstration
whether agencies are addressing these purposes and meeting
requirements.

Second, HUD has only recently assessed agencies’ compliance with two
self-certified requirements and has not assessed compliance with the
third. Internal control standards require control activities to be in place to
address program risks. 55 In addressing these risks, internal control
guidance states that management should formulate an approach for
assessing compliance with program requirements. 56 While HUD has
recently made efforts to assess agencies’ compliance with two of the
three self-certified requirements, it does not have a process in place to
systematically review compliance with all three requirements. In 2011,
HUD for the first time assessed participating agencies’ compliance with
the requirement to assist “substantially the same” number of eligible
families that would have been served in the absence of MTW. HUD
collected data from MTW-PIC, the Voucher Management System, and
each participating agency’s most recent annual report on the number of
public housing units occupied, vouchers utilized, and other families
housed and used a formula to compare these data with similar data
reported before MTW. 57 HUD and MTW agency staff we interviewed told
us that they worked together to discuss discrepancies in the calculations.
According to the Director of the MTW Office, agencies were in
compliance with this requirement if they were serving at least 95 percent
of the number of families in their baseline figure. HUD’s recent review of
each agency’s baseline calculation indicated that all but one of the
agencies were in compliance. Also in 2011, HUD reviewed MTW-PIC
data for the first time to determine agencies’ compliance with the
requirement that at least 75 percent of assisted residents be very low
income. HUD’s analysis of MTW-PIC data showed that, as of September
2011, 91 percent of the residents served by MTW agencies fell into this
category.

While HUD has taken steps to assess compliance with these two
statutory requirements, it has not yet developed a methodology for


55
 GAO/AIMD-00-21.3.1.
56
  GAO, Internal Control Management and Evaluation Tool, GAO-01-1008G (Washington,
D.C.: August 2001).
57
  The baseline calculation also adjusts for any incremental vouchers that the agency may
have been awarded over time.




Page 28                                        GAO-12-490 Moving to Work Demonstration
assessing agencies’ compliance with the requirement to maintain a
comparable mix of families. The Director of the MTW Office
acknowledged that self-certifications were not the best means of ensuring
compliance and told us that the planned revisions to the reporting
requirements for MTW agencies would help assess compliance with the
requirements to maintain a comparable mix of families and ensure that at
least 75 percent of families assisted are very low income. Without a
process for systematically assessing compliance with statutory
requirements, HUD lacks assurance that agencies are complying with
them.

Third, HUD has not performed an annual assessment of program risks.
Internal control standards state that an agency should have a risk
assessment plan that considers internal and external risk factors and
establishes a control structure to address those risks. 58 The standards
also state that managers should focus on control activities to address
risks that may involve verifications, performance reviews, and
documentation, among other things. HUD’s own internal control
standards also require its program offices to perform an annual risk
assessment of their programs or administrative functions using a HUD
risk-assessment worksheet. 59 These standards also stress the
importance of performing a risk assessment when there are significant
program changes. According to the Director of the MTW Office, the office
has not performed an annual risk assessment for the MTW program
because it was not aware of this requirement.

In addition, HUD’s procedures for monitoring the MTW program are not
risk-based, meaning that HUD has not tailored its monitoring efforts to
reflect the perceived risk of each MTW agency. 60 MTW agencies, unlike
other PHAs, are exempt from receiving performance scores that reflect


58
 GAO/AIMD-00-21.3.1.
59
 HUD, Office of the Chief Financial Officer, Departmental Management Control Program,
Handbook 1840.1 Rev-3 (Washington, D.C.: 1999).
60
  MTW agencies are exempt from scoring in the Public Housing Assessment System and
the Section 8 Management Assessment System. However, MTW agencies are subject to
physical inspections conducted by the Real Estate Assessment Center under HUD
guidelines and issued a score. This score is entered into the Public Housing Assessment
System and can be viewed by MTW staff at any time. A score of 22 or below is flagged by
the Real Estate Assessment Center (the maximum score is 30) and reported to the
appropriate field office.




Page 29                                       GAO-12-490 Moving to Work Demonstration
their perceived level of risk. While monitoring procedures are not risk-
based, the Director of the MTW Office stated that his office would become
aware of risks from HUD’s field office staff, which have routine
responsibility for reviewing financial audits and Office of Management and
Budget compliance audits. As previously discussed, HUD is considering
moving toward conducting risk-based site visits. 61 In addition, according
to HUD officials, the office is considering other methods to more
rigorously analyze MTW agency risk factors. By not performing an annual
risk assessment and implementing a risk-based approach to monitoring
MTW agencies, HUD lacks assurance that it has properly identified and
addressed risks that may prevent participating agencies from addressing
program purposes and meeting statutory requirements. HUD also lacks
assurance that it is efficiently using its limited monitoring resources.

Finally, HUD does not have policies or procedures in place to verify the
accuracy of key information that MTW agencies self-report. Internal
control standards and guidance emphasize the need for federal agencies
to have control activities in place to help ensure that program participants
report information accurately. 62 HUD requires agencies to report on
benchmarks, metrics, and performance information in their annual reports
but does not have policies or procedures in place to verify the accuracy of
this self-reported data. Further, MTW coordinators do not verify this
information during their annual site visits or during their annual report
review process, preventing HUD from efficiently using its limited
monitoring resources. As previously noted, HUD relies on the
participating agency to submit accurate information. We requested
documentation from our sample of MTW agencies that supported the
outcomes they reported in their most recent annual report for a selected
activity and received various types of support. While this information was
sufficient for our own review, HUD has not taken similar steps to collect
evidence to substantiate agencies’ self-reported information. According to
HUD officials, the MTW Office is considering ways to verify the accuracy
of key information that MTW agencies self-report. Although verifying all of
the self-reported information may be challenging, GAO guidance on data
reliability recommends tracing a sample of data records to source
documents to determine whether the data accurately and completely


61
  Under this approach, the MTW Office would focus first on agencies that were
implementing new activities or significant changes and visit these sites.
62
 GAO/AIMD-00-21.3.1 and GAO-01-1008G.




Page 30                                       GAO-12-490 Moving to Work Demonstration
                        reflect the source documents. 63 Because HUD does not verify the
                        accuracy of any reported performance information, it lacks assurance that
                        this information is accurate. To the extent that HUD relies on this
                        information to assess program compliance with statutory purposes and
                        requirements, its analyses are limited.


                        Legislation has been proposed to expand the number of PHAs that can
Expanding MTW May       participate in the MTW program, and a recent HUD report recommended
Offer Benefits but      expanding the program up to twice its size. As of March 2012, a
                        maximum of 39 PHAs could participate in the program, but a 2011 Senate
Also Raises Questions   bill would direct HUD to increase that number up to 250. 64 In addition,
                        legislation has been drafted that would establish MTW as a permanent
                        program and eliminate the current restrictions on the number of agencies
                        that can participate. 65 HUD’s 2010 Report to Congress recommends
                        increasing the number of participating agencies to about 60. 66 HUD and
                        some stakeholders believe that expansion could provide the needed
                        information on the effect of the MTW program and allow more PHAs to
                        test innovative ideas, but questions remain about the lack of performance
                        information on current MTW activities. In addition, alternatives to
                        expansion exist, including implementing a more narrowly focused
                        program.




                        63
                          GAO, Assessing the Reliability of Computer-Processed Data, GAO-09-680G
                        (Washington, D.C.: July 2009).
                        64
                         Moving to Work Charter Program Act of 2011 (S. 117).
                        65
                          Moving to Work Improvement, Expansion, and Permanency Act of 2011 (draft
                        legislation); Affordable Housing and Self-Sufficiency Improvement Act of 2012 (AHSSIA)
                        (draft legislation).
                        66
                         HUD, Moving to Work (2010).




                        Page 31                                       GAO-12-490 Moving to Work Demonstration
Current MTW Participants   According to HUD, some affordable housing advocates, and MTW
and Affordable Housing     agencies we interviewed, expanding the MTW program could help
Advocates Indicated that   demonstrate the program’s effect and increase the number of lessons
                           that can be learned from the program. HUD has reported that doubling
Expanding MTW Could        the number of MTW agencies with the use of strategic criteria and
Help Demonstrate the       program implementation could help demonstrate the effects of MTW on a
Program’s Effect and       broader scale and enable the housing industry to learn even more from
Increase Innovation        the demonstration. 67 For example, expansion could provide more
                           information on how MTW flexibilities would affect a broader group of
                           PHAs. The Director of the MTW Office noted that some MTW activities,
                           specifically those related to administrative streamlining, had influenced
                           the draft Affordable Housing and Self-Sufficiency Improvement Act of
                           2012 (AHSSIA). Some affordable housing advocates that we met with
                           emphasized the value of the changes, such as decreases in concentrated
                           poverty, that have occurred in some of the communities affected by the
                           MTW program and indicated that expansion could enable more PHAs to
                           address local needs and therefore benefit additional communities.
                           Similarly, officials from MTW agencies that we contacted stated that
                           expansion of the program would provide a broader testing ground for new
                           approaches and best practices.

                           In addition, information from a private research organization, affordable
                           housing advocates, and MTW agencies that we met with suggests that
                           allowing additional PHAs to participate in the program could result in
                           additional opportunities for PHAs to test innovative ideas, in part by using
                           the flexibilities provided to tailor their housing programs and activities to
                           meet local conditions. In 2004, the Urban Institute reported that the local
                           flexibility and independence permitted under MTW appeared to allow
                           strong, creative PHAs to experiment with innovative solutions to local
                           challenges. 68 It noted that these PHAs were able to be more responsive
                           to local conditions and priorities because they were not as limited by
                           program requirements as they might have been before MTW. For
                           example, one agency used MTW to increase the proportion of project-
                           based units relative to tenant-based assistance. Through MTW, this
                           adjustment was done to an extent that would not have been permissible
                           under standard rules. The Urban Institute concluded that further
                           deregulation of PHAs could yield benefits in terms of design and


                           67
                            HUD, Moving to Work (2010).
                           68
                            Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).




                           Page 32                                    GAO-12-490 Moving to Work Demonstration
implementation innovations. Officials from an organization that advocates
on behalf of large PHAs and supports expansion noted that affordable
housing needs varied by locality and that the MTW program enabled
participating agencies to design effective approaches based on local
needs. Similarly, another affordable housing advocacy organization told
us that they supported expanding MTW not only because it enabled
participating agencies to tailor activities to local needs but also because it
involved local communities in the process.

Officials from several of the MTW agencies we interviewed also noted
that the MTW program had empowered them to create and implement
strategies that addressed local issues and said that expanding the
program would give more PHAs the same flexibility. For example, in one
northeastern state where the housing stock was relatively old, the MTW
agency was able to focus on developing new affordable housing. Another
MTW agency in a western state with mostly newer housing stock chose to
reduce the frequency of inspections of its properties and focus its efforts
on administrative streamlining and the disposition of its older units.
Further, several MTW agencies that we interviewed described how they
implemented the requirement to establish a rent policy that encouraged
employment and self sufficiency. For example, officials from one MTW
agency told us that they believed the traditional requirement that
residents pay 30 percent of their adjusted income in rent was a
disincentive to work, because as resident income increases so would the
payment toward rent. To encourage residents to seek work, this agency
implemented work requirements and a minimum rent. Additionally, some
agencies have used their MTW status to establish programs that focus on
specific populations, including working families with children, the elderly
and disabled, and the homeless.

Some proponents of expansion that we interviewed also noted that
expanding the MTW program could provide more PHAs with the ability to
use funding from different sources more flexibly than possible without
MTW status. Agencies without MTW status have to implement their
activities while adhering to the regulations associated with three different
funding streams, evidence of the fragmented nature of housing




Page 33                                  GAO-12-490 Moving to Work Demonstration
                         assistance. 69 As we have seen, MTW agencies may request waivers of
                         certain provisions of the 1937 Housing Act in order to combine annual
                         funding from separate sources into a single authoritywide funding source.
                         HUD field office staff with responsibility for monitoring MTW agencies
                         observed that the single-fund flexibility was beneficial because it enabled
                         participating agencies to develop supportive service programs, such as
                         job training or educational programs, which help move families toward
                         self-sufficiency. One HUD field office official stated that this flexibility
                         would be a significant benefit for other PHAs. An affordable housing
                         advocate we met with also noted that this ability to use different kinds of
                         funds interchangeably was beneficial because it enabled MTW agencies
                         to shift funds based on local priorities. Further, officials from the MTW
                         agencies we interviewed agreed that this flexibility was beneficial. For
                         example, officials from one MTW agency stated it had been able to use
                         the single fund to organize itself as a business organization, develop a
                         strategic plan based on the housing needs of low-income families in the
                         community, leverage public funds and public and private partnerships,
                         and develop mixed-income communities. Two of the MTW agencies that
                         we interviewed also stated that the single-fund flexibility had enabled
                         them to fund programs that encouraged self-sufficiency among residents.
                         For example, officials explained that they had used funding for coaching
                         and counseling services, job training support, and education programs.
                         Finally, officials from three of the MTW agencies we interviewed noted a
                         related benefit of participation. They said that their MTW status had
                         enabled them to respond more quickly to real estate opportunities
                         because they do not have to wait for HUD approvals to purchase
                         properties.


Lack of Performance      A lack of performance information, limited HUD oversight, and concerns
Information and Other    about the program’s impact on residents raise questions about expanding
Issues Raise Questions   the MTW program. As we noted previously, conclusive information about
                         the effectiveness of the MTW program is limited in part because HUD
about Expansion          does not have a plan for identifying and analyzing standard performance
                         data, has not established performance indicators for the program as



                         69
                           We recently reported that there were 160 different programs, tax expenditures, and
                         other tools that supported homeownership and rental housing in fiscal year 2010,
                         including multiple programs that provided funding for PHAs. See GAO, 2012 Annual
                         Report: Opportunities to Reduce Duplication, Overlap and Fragmentation, Achieve
                         Savings, and Enhance Revenue, GAO-12-342SP (Washington, D.C.: Feb. 28, 2012).




                         Page 34                                        GAO-12-490 Moving to Work Demonstration
whole, and does not have a systematic process for identifying lessons
learned.

HUD’s 2010 Report to Congress noted that the conclusive impacts of
many MTW activities, particularly as they relate to residents, could not yet
be known. 70 For example, the report noted that the rent reforms
implemented under MTW varied greatly and were not implemented using
a controlled experimental methodology. As a result, which aspects of rent
reforms should be recommended for all PHAs were not clear. The report
also noted the limitations that exist when evaluating the outcomes of
MTW—limitations that stem from the weak initial reporting requirements
and lack of a research design. The report concluded that, given these
limitations, expansion should occur only if newly admitted PHAs structure
their programs for high-quality evaluations that permit lessons learned to
be generalized beyond a single PHA experience.

Similarly, affordable housing advocates and legal aid organizations that
we interviewed stated that because lessons had not been learned from
MTW, there was no basis for expanding the program. For example,
officials from a national affordable housing advocacy organization stated
that some MTW agencies have used their flexibility to establish limits on
the length of time someone can live in assisted housing, but there is little
research on the effect of such efforts. The officials stated that there was
no evidence that this policy had helped anyone become self-sufficient and
move out of public housing. The officials added that data were not
available on the extent to which MTW agencies have provided incentives
for residents to become self-sufficient or have increased housing choices.
Similarly, an official from a national housing law advocacy organization
stated that data were not available to determine the effect of the MTW
program, particularly at the national level.

In addition, our own work, some research organizations, and affordable
housing advocates question HUD’s ability to effectively manage an
expanded MTW program. As previously noted, HUD’s current monitoring
procedures have several key weaknesses, including the lack of a
systematic process for assessing agencies’ compliance with statutory
requirements and an assessment of program risks. Some research
organizations also have questioned HUD’s capacity to oversee additional



70
 HUD, Moving to Work (2010).




Page 35                                 GAO-12-490 Moving to Work Demonstration
MTW agencies. For example, the Urban Institute reported that the
approval process that HUD was using at the time of the institute’s 2004
review would not be feasible for an expanded program because of the
administrative burden involved. 71 At the time of the 2004 study as well as
our review, HUD reviewed each individual request to waive specific
provisions of the 1937 Housing Act before approving annual plans. Staff
from another research organization questioned whether HUD has the
capacity to oversee additional agencies. Similarly, one affordable housing
advocate that we interviewed stated that HUD’s capacity to oversee an
expanded program is not clear, in part because current monitoring
activities are not transparent. At the time of our review, HUD had four full-
time MTW coordinators, who each managed from 6 to 10 MTW
agencies. 72 According to the Director of the MTW Office, it takes more
resources for HUD to oversee MTW agencies than non-MTW agencies.
Thus, if additional agencies were added under the current program
design, HUD would likely need additional resources.

Researchers and several of the affordable housing advocates and legal
aid agencies that we met with also raised concerns that the current
program, and therefore also an expanded program, could negatively
affect residents of MTW agencies. For example, two research
organizations have stated residents could be negatively affected by MTW
agencies that implement voucher policies that reduce portability—that is,
residents’ ability to use their vouchers in an area outside of the area
where they received it. One of these research organizations stated that
the differences in the way voucher programs were implemented across
MTW agencies could reduce residents’ ability to use vouchers outside of
the area where they received the assistance. Officials from the other
organization noted that some MTW agencies had instituted policies that
prohibited vouchers from being transported out of the originating
jurisdictions, thereby limiting housing choices. According to HUD officials,
MTW agencies with policies that limit portability can make exceptions. For
example, these agencies have made exceptions for residents seeking
employment opportunities.




71
     Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).
72
  There are three additional coordinators who are each responsible for a single MTW
agency as designated in the agency’s MTW agreement. These coordinators have other
responsibilities within HUD and are not assigned to the MTW Office.




Page 36                                        GAO-12-490 Moving to Work Demonstration
                        Legal aid organizations that have worked with residents of MTW agencies
                        as well as affordable housing advocates told us some of the requirements
                        that MTW agencies have implemented, such as work requirements, were
                        potentially harmful to residents. 73 For example, legal aid representatives
                        from one community told us that the work requirement was not
                        consistently enforced across various mixed-income properties that
                        included public housing as well as market-rate units. According to these
                        officials, they have had clients who have been evicted for not working,
                        even though the client was in school or disabled—both exceptions to the
                        work requirement. These officials also stated that property managers in
                        the city’s various mixed-income developments did not implement MTW
                        policies consistently. For example, the officials stated that residents have
                        been told by property managers that they would be in compliance with the
                        work requirement if they were in school or another training program, only
                        to have the MTW agency determine that they were not in compliance.
                        According to HUD officials, inconsistent enforcement of policies is not
                        unique to MTW agencies, and residents would have recourse. Legal aid
                        representatives that worked with residents of another MTW agency also
                        told us that the work requirement was a punitive policy that negatively
                        affected the poorest residents. The officials stated that there were better
                        methods for encouraging work and self-sufficiency, such as job training.
                        Officials from a national affordable housing advocacy organization agreed
                        that work requirements are punitive and stated that they disagreed with a
                        policy of making housing assistance contingent on other factors, such as
                        having a job. In their view, housing assistance should be a stable form of
                        assistance for low-income households.


Alternatives Exist to   Alternatives to expansion include implementing a program that is targeted
Expanding the MTW       more to specific activities and waiving some regulations for all PHAs as
Program                 described in proposed legislation. According to the Urban Institute, an
                        alternative to expanding MTW could be to systematically test a limited
                        number of programmatic alternatives—such as flat rents, time limits, or
                        debt financing of capital improvements. 74 This approach would not allow
                        individual agencies as much discretion to design combinations of reforms
                        around local conditions and priorities. However, this approach could yield
                        more systematic evidence about the costs and benefits of particular


                        73
                         Work requirements can be met by attending job training programs as well as working.
                        74
                         Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).




                        Page 37                                      GAO-12-490 Moving to Work Demonstration
program reforms if it included a rigorous evaluation design and mandatory
data collection on key outcomes, such as the number and characteristics
of participating households. Collection of such information in a
standardized format would need to be a minimum requirement for
participation if the point was to learn from the experiences of those testing
activities. In addition, an official from an affordable housing advocacy
organization that we met with stated that testing the effectiveness of
discrete activities on a smaller scale would be useful. HUD also noted
that altering the scope of the demonstration for new participants could
improve what was learned from specific activities. For example, its 2010
Report to Congress stated that data on MTW could be strengthened if the
scope of the demonstration were altered for new participants by selecting
agencies committed to testing a particular activity, such as rent reform,
and requiring rigorous evaluation. 75 In December 2011, HUD issued a
request for proposals for a demonstration that would test alternatives to
the current rent structure in the voucher program. According to the
proposal, the demonstration would most likely be undertaken at select
MTW agencies. In addition, HUD’s 2012 appropriations act authorized a
Rental Assistance Demonstration that would enable HUD to authorize
and evaluate new approaches to preserving affordable rental housing,
including converting public housing to project-based rental assistance.
AHSSIA includes authorization for a revised version of the Rental
Assistance Demonstration.

In addition, according to researchers, PHAs could be allowed some level
of deregulation, so that they could implement the same sort of
administrative streamlining activities that MTW agencies implement. The
Urban Institute has reported that further deregulation of PHAs may yield
benefits in terms of program design and implementation innovations but
that such deregulation could entail risks and should be evaluated. 76 Some
of the MTW agencies that we met with were supportive of the
administrative streamlining authorities allowed by MTW and described in
proposed legislation such as AHSSIA. As proposed, AHSSIA would allow
PHAs to reduce the frequency of income certifications for the elderly and
disabled to save on administrative costs, among other things. According
to the Director of the MTW Office, the experiences of MTW agencies
informed this policy proposal. In addition, HUD’s 2010 Report to


75
 HUD, Moving to Work (2010).
76
 Abravanel and others, An Assessment of HUD’s Moving to Work Demonstration (2004).




Page 38                                    GAO-12-490 Moving to Work Demonstration
              Congress supported allowing more PHAs to participate in the program.
              Finally, we recently reported on cost savings that could be realized from
              allowing additional housing authorities to implement some of the reforms
              MTW agencies have tested. 77


              The MTW demonstration is designed to provide participating agencies
Conclusions   with the flexibility to develop and test activities that achieve cost-
              efficiency, encourage residents with children to obtain employment and
              become self-sufficient, and increase housing choices for low-income
              families. While this flexibility has allowed participating agencies to
              implement hundreds of activities, HUD has not done all that it can to
              evaluate the program’s effectiveness, identify successful approaches that
              could be applied to public housing agencies more broadly, or ensure that
              MTW agencies comply with program requirements. Because Congress is
              considering expanding the program to many more PHAs, the absence of
              information needed to conduct a comprehensive program evaluation and
              compliance reviews is significant. HUD has recognized the importance of
              rigorous evaluation by requiring newly admitted agencies to have strong
              evaluation components. However, these improvements will not help
              evaluate the program as a whole. Without more complete knowledge of
              the program’s effectiveness and the extent to which agencies are
              adhering to program requirements, it is difficult for Congress to know
              whether an expanded MTW will benefit additional agencies and the
              residents they serve.

              Recognizing that it needed to do more to improve what was known about
              the program’s effectiveness, HUD started requiring MTW agencies to
              describe the impact of each implemented activity in their annual reports
              beginning in 2009. However, the information that MTW agencies reported
              did not always reflect outcomes, and HUD’s guidance does not require
              that information on activities be quantifiable and outcome-oriented to the
              extent possible. Without more specific guidance on reporting performance
              information, HUD cannot be assured of collecting data that reflects the
              outcomes of activities.




              77
               GAO, Housing Choice Vouchers: Options Exist to Increase Program Efficiencies,
              GAO-12-300 (Washington, D.C.: Mar. 19, 2012).




              Page 39                                      GAO-12-490 Moving to Work Demonstration
Further, challenges such as the lack of analysis of standard data and the
absence of performance indicators have prevented comprehensive
evaluations of similar activities and the overall MTW program. HUD has
recently started collecting additional information through MTW-PIC and
annual reports, but has not yet analyzed the data. Further, whether the
data collected are sufficient to assess similar activities and the program
as a whole is not clear, and HUD has not identified the performance data
it needs to undertake such analysis. Until HUD has a plan (that includes
the identification of standard data) to quantitatively assess similar
activities and the MTW program as a whole, HUD cannot determine their
effectiveness. Additionally, HUD has not established performance
indicators specific to MTW. Indicators linked to the statutory purposes of
reducing costs, encouraging self-sufficiency, and increasing housing
choices would help HUD demonstrate that the program has produced
desired results.

Similarly, HUD does not have a systematic process in place to identify
lessons learned from the MTW demonstration. Identifying activities that
could be replicated more broadly is a goal of the MTW program and could
be aided by the analysis of some standard performance data. The
absence of a criteria-based, regular process for identifying lessons
learned complicates efforts to determine which MTW activities are most
effective and should be replicated more broadly.

At the same time, HUD’s monitoring efforts are not as strong as they
could be. First, because HUD has not clarified key terms related to the
three statutory purposes and five requirements, it cannot effectively
determine whether MTW agencies are actually addressing these
purposes and meeting requirements. Second, HUD does not have a
process in place to systematically review compliance with all
requirements. Such a review is especially important to a program like
MTW that allows participants to self-certify their compliance with some
program requirements. HUD has begun assessing compliance with two of
the MTW requirements that call for self-certification, but not the third, and
thus lacks assurance that agencies are complying with all three.

Moreover, HUD’s procedures for monitoring MTW agencies are not risk-
based. It does not conduct an annual assessment of risks and provides
the same level of monitoring for all agencies, even though some may
pose greater risks than others. A risk-based approach to monitoring
would provide greater assurance that HUD has addressed all risks,
particularly those that may prevent participating agencies from addressing
program purposes and meeting statutory requirements. Further, unless it


Page 40                                  GAO-12-490 Moving to Work Demonstration
                      implements a risk-based approach (such as that currently being
                      considered for annual site visits) to monitoring MTW agencies, HUD
                      cannot be assured that it is using its limited monitoring resources most
                      efficiently. Finally, just as HUD does not assess compliance with all three
                      self-certified requirements, it does not verify the accuracy of key
                      information that agencies self-report, including information on the impact
                      of MTW activities. Annual site visits have been used primarily to provide
                      technical assistance rather than to assess self-reported information. By
                      not verifying the accuracy of any performance information, HUD lacks
                      assurance that this information is accurate.


                      To improve what is known about the effectiveness of the MTW program,
Recommendations for   we recommend that the Secretary of the Department of Housing and
Executive Action      Urban Development

                      •   improve HUD’s guidance to MTW agencies on providing performance
                          information in their annual reports by requiring that such information
                          be quantifiable and outcome-oriented to the extent possible;

                      •   develop and implement a plan for quantitatively assessing the
                          effectiveness of similar activities and the program as a whole
                          including the identification of standard performance data needed; and

                      •   establish performance indicators for the MTW program as a whole.

                      To enhance the ability to identify MTW practices that could be applied
                      more broadly, we recommend that the Secretary of the Department of
                      Housing and Urban Development create a process to systematically
                      identify lessons learned.

                      To improve HUD’s oversight of the MTW program, we recommend that
                      the Secretary of the Department of Housing and Urban Development

                      •   issue guidance that clarifies key program terms, such as the three
                          statutory purposes of the program and the five statutory requirements
                          that MTW agencies must meet;

                      •   develop and implement a systematic process for assessing
                          compliance with statutory requirements;




                      Page 41                                 GAO-12-490 Moving to Work Demonstration
                     •   conduct an annual risk assessment for the MTW program and
                         implement risk-based monitoring policies and procedures such as
                         those currently being considered for site visits; and

                     •   implement control activities designed to verify the accuracy of a
                         sample of the performance information that MTW agencies self-report.

                     We provided a draft of this report to HUD. The Assistant Secretary for
Agency Comments      Public and Indian Housing provided written comments, which are
and Our Evaluation   reprinted in appendix II. HUD disagreed with our recommendation that the
                     agency develop performance indicators for the MTW program as a whole,
                     was in partial agreement with four recommendations, and generally
                     agreed with three. The agency said that developing programwide
                     performance measures could be difficult and might be contrary to the
                     nature of the demonstration. In addition, HUD emphasized the
                     improvements that it had made to its reporting requirements in order to
                     collect more consistent, outcome-oriented data. We acknowledged these
                     improvements in the draft report, but as our recommendations indicated,
                     we saw opportunities for additional improvements. HUD also noted that
                     some of our recommendations might be a good fit for the existing
                     program but that others might be more appropriate for a future expanded
                     demonstration.

                     In disagreeing with our recommendation that it establish performance
                     indicators for the MTW program as a whole, HUD emphasized the
                     difficulty of measuring all activities against the same standard. The
                     agency noted that because each MTW agency had implemented a unique
                     combination of activities, developing programwide performance measures
                     would make determining the impacts of specific activities unclear and
                     prevent the identification of individual policies that could be applied more
                     broadly. However, the purpose of programwide indicators would not be to
                     isolate the impact of individual activities but to demonstrate programwide
                     results—including showing the extent to which the program was
                     addressing its statutory purposes of achieving greater cost-effectiveness
                     in federal housing expenditures, giving families with children incentives to
                     obtain employment and become self-sufficient, and increasing housing
                     choices for low-income families. HUD also stated that applying
                     programwide performance measures would be complicated by the fact
                     that activities that advance one statutory purpose might conflict with other
                     purposes. We agree that it is important to evaluate similar activities and
                     have a separate recommendation addressing this need. But, the purpose
                     of programwide assessment is to demonstrate whether the provision of



                     Page 42                                 GAO-12-490 Moving to Work Demonstration
flexibility in itself results in the intended benefits of the MTW program,
such as cost savings or increased family self-sufficiency. Demonstrating
that the increased flexibility the program offers has produced the intended
results is critical, particularly as Congress considers whether to expand
the program. We continue to believe in the importance of demonstrating
program results and therefore continue to recommend that HUD develop
performance indicators for the MTW program as a whole.

HUD was in partial agreement with four recommendations.

•   First, HUD said that proposed revisions to the reporting requirements
    for MTW agencies had addressed our recommendation that the
    agency improve its guidance to MTW agencies on providing
    performance information in annual reports. HUD’s draft guidance is in
    line with our recommendation that HUD require agencies to report
    quantifiable and outcome-oriented information. However, because
    these proposed revisions have yet to be finalized, we did not revise
    our recommendation.

•   Second, HUD agreed that quantitatively assessing the effectiveness
    of similar activities was an important step but noted the difficulties
    associated with assessing the effectiveness of the program as a
    whole. However, as noted above, we continue to believe in the
    importance of demonstrating program results. Consequently, we did
    not revise our recommendation.

•   Third, HUD stated that providing a menu of standard metrics may be
    the best way to clarify the program’s statutory purposes and that it
    had made progress in recent years in addressing our
    recommendation that it issue guidance that clarifies the statutory
    requirements. HUD also noted that the proposed revisions to the
    reporting requirements would provide additional clarification on the
    statutory requirements. These efforts, which were acknowledged in
    the draft report, are a step in the right direction, and we encourage
    HUD to continue finalizing this guidance. As noted above, because
    these proposed revisions have yet to be finalized, we did not revise
    our recommendation.

•   Fourth, HUD described recent efforts to assess compliance with two
    statutory requirements and analysis that it could conduct once
    proposed revisions to reporting requirements for MTW agencies were
    finalized. Because the process used to assess compliance with one of
    the requirements has not been formalized in policy and the proposed
    revisions have not been finalized, we did not revise our


Page 43                                 GAO-12-490 Moving to Work Demonstration
     recommendation that HUD develop and implement a systematic
     process for assessing compliance with statutory requirements.

HUD generally agreed with the three remaining recommendations. For
example, HUD agreed that it should proactively identify lessons learned
and described some of its recent efforts to do so. We acknowledged
these efforts in our draft report but noted the absence of a criteria-based,
regular process for identifying lessons learned. HUD also described plans
to develop a formal risk-based strategy for monitoring and, when we
asked for further clarification, stated that it agreed with our
recommendation to conduct an annual risk assessment for the MTW
program. Finally, HUD discussed potential strategies for verifying the
information that MTW agencies report using existing or planned HUD
systems.

HUD also requested that we consolidate four separate recommendations
into two, but we continue to believe that maintaining distinctions between
the separate recommendations is important. First, HUD requested that we
combine two recommendations: that it create a plan to quantitatively
assess the effectiveness of similar activities and the program as a whole
(including identifying the standard performance data needed), and that it
establish performance indicators for the program as a whole. Although
related, the two recommendations are distinct because the first focuses
on the need for program evaluation and the second on performance
measurement. Program evaluations typically examine a broad range of
information on program performance, while performance measurement
generates outcomes that show whether a program has achieved specific
objectives. 78 As a result, we did not combine the recommendations.

Second, HUD requested that we combine the recommendation that it
issue guidance clarifying key program terms (such as the three statutory
purposes and five statutory requirements) with the recommendation that it
implement a systematic process for assessing compliance with statutory
requirements. However, defining program requirements and assessing
them are separate and distinct activities. Therefore, we did not combine
the recommendations.




78
 GAO, Performance Measurement and Evaluation: Definitions and Relationships,
GAO-11-646SP (Washington, D.C.: May 2011).




Page 44                                     GAO-12-490 Moving to Work Demonstration
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies to the Secretary of Housing
and Urban Development and other interested committees. In addition, the
report will be available at no charge on the GAO website at
http://www.gao.gov.

If you or your staff have any questions concerning this report, please
contact me at (202) 512-8678 or sciremj@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are listed in
appendix III.




Mathew J. Scirè
Director, Financial Markets
   and Community Investment




Page 45                                 GAO-12-490 Moving to Work Demonstration
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             Our objectives were to examine (1) what is known about the extent to
             which the Moving to Work (MTW) demonstration program is addressing
             the program’s statutory purposes, (2) the Department of Housing and
             Urban Development’s (HUD) monitoring of MTW agencies’ efforts to
             address these purposes and meet statutory requirements, and (3)
             potential benefits of and concerns about expanding the number of public
             housing agencies (PHA) that can participate in the demonstration
             program.

             To evaluate what is known about the extent to which the MTW program is
             addressing the program’s statutory purposes, we reviewed the most
             recent annual reports as of January 2012 for 30 MTW agencies. 1 We
             reviewed these annual reports and the corresponding annual plans to
             identify the ongoing activities that the agencies were implementing,
             determine the extent to which these activities were linked with one or
             more of the program’s statutory purposes, and assess the performance
             information provided for each activity. 2 To assess the performance
             information provided by MTW agencies, we examined HUD’s reporting
             guidance and compared it with internal control standards for federal
             agencies. 3 We assessed the reliability of selected information in the
             reports by reviewing supporting documentation from a sample of seven
             MTW agencies and interviewing the officials responsible for preparing
             and reviewing this information. These seven agencies were

             •   Cambridge Housing Authority (Cambridge, Massachusetts),

             •   Chicago Housing Authority (Chicago, Illinois),

             •   Housing Authority of the City of Atlanta (Atlanta, Georgia),

             •   Housing Authority of the City of Pittsburgh (Pittsburgh, Pennsylvania),


             1
              Of the 35 participating agencies, 30 had issued an annual report for 2011 as of January
             2012. The other five agencies had not been in the program long enough to report on their
             accomplishments.
             2
              We reviewed the annual reports to identify ongoing activities. For these activities, we
             reviewed the reports to determine whether the activity was linked to a statutory purpose
             and the corresponding annual plan if that information was not available in the report. The
             results from the two analyses were combined.
             3
              GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1
             (Washington, D.C.: November 1999).




             Page 46                                         GAO-12-490 Moving to Work Demonstration
Appendix I: Scope and Methodology




•   Housing Authority of the County of Santa Clara/Housing Authority of
    the City of San Jose (Santa Clara County and San Jose, California),

•   Lawrence-Douglas County Housing Authority (Lawrence, Kansas),
    and

•   Vancouver Housing Authority (Vancouver, Washington). 4

We selected these agencies to provide diversity in geography, agency
size, and length of time participating in the program. We determined that
the reports were sufficiently reliable for the purposes of our review.

Through interviews and a literature search, we identified three studies of
the MTW program as a whole. We reviewed these studies to identify
information on the program’s effectiveness and any challenges
associated with assessing it. We determined that these studies were
methodologically sound and reliable for our purposes. We examined
HUD’s recent efforts to collect data from MTW agencies, including
documentation on the reporting requirements for MTW agencies. In
addition, we reviewed HUD’s fiscal year 2010-15 strategic plan and Fiscal
Year 2011 Annual Performance Plan for any performance indicators for
the MTW program. We also reviewed the GPRA (Government
Performance and Results Act) Modernization Act of 2010, Office of
Management and Budget guidance, internal control standards, and a
GAO report on attributes of successful performance measures. 5 Further,
we identified five studies of specific MTW agencies or activities identified
by HUD and representatives of the sample of MTW agencies we
interviewed. Finally, we reviewed published reports and HUD’s website
for information on HUD’s efforts to identify lessons learned.

To assess HUD’s monitoring of MTW agencies’ efforts to address the
program’s statutory purposes and meet requirements, we obtained and
reviewed documentation of monitoring policies and procedures, including
the Standard Agreement that HUD executed with MTW agencies in 2008,


4
 The Housing Authority of the County of Santa Clara and the Housing Authority of the City
of San Jose are consolidated. We considered them as one MTW agency in our sample, as
they report the information to HUD jointly.
5
 See GAO/AIMD-00-21.3.1 and GAO, Tax Administration: IRS Needs to Further Refine Its
Tax Filing Season Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22,
2002).




Page 47                                        GAO-12-490 Moving to Work Demonstration
Appendix I: Scope and Methodology




the MTW Desk Guide, a 2011 Memorandum of Understanding between
HUD’s Office of Public Housing Investments and Office of Field
Operations, and other HUD guidance. Based on these documents and
interviews with HUD staff, we identified three key monitoring processes:
the review of annual plans and reports, reviews of data entered into the
Moving to Work section of the Public and Indian Housing Information
Center (MTW-PIC), and annual site visits to each MTW agency.

To assess the extent to which HUD staff were following these monitoring
policies and procedures, we reviewed documentation of monitoring
activities for our sample of seven MTW agencies. For example, to verify
the steps HUD had taken to review annual plans and reports, we
reviewed the checklists that the MTW coordinators used to document
their review of these plans and reports. We also reviewed HUD’s
comment letters for fiscal year 2011. To verify the steps HUD had taken
to review data submitted into the MTW-PIC system, we reviewed monthly
reports that showed the degree to which MTW agencies overall complied
with reporting requirements from August 2011 through January 2012.
Finally, to verify that both headquarters and field office staff made site
visits and the extent to which they made annual visits, we reviewed the
most recently available site visit reports completed by the MTW Office for
all agencies as of October 2011. In addition, we interviewed the MTW
agencies in our sample and the corresponding HUD field office officials to
discuss the annual site visits.

We also compared HUD’s monitoring policies and procedures to internal
control standards for the federal government and HUD’s own program
management guidance. 6 As a part of this analysis, we compared HUD’s
guidance to MTW agencies with the internal control requirement for clear
goals and objectives. We also reviewed information on HUD’s efforts to
clarify how agencies could certify compliance with the requirement to
assist “substantially the same” number of eligible families that would have
been served in the absence of MTW. In addition, we compared HUD’s
efforts to assess agencies’ compliance with statutory requirements with
the internal control standard related to assessing compliance with
program requirements. Further, we reviewed internal control standards for


6
 See GAO/AIMD-00-21.3.1; GAO, Internal Control Management and Evaluation Tool,
GAO-01-1008G (Washington, D.C.: August 2001); and HUD, Office of the Chief Financial
Officer, Departmental Management Control Program, Handbook 1840.1 Rev-3
(Washington, D.C.: 1999).




Page 48                                      GAO-12-490 Moving to Work Demonstration
Appendix I: Scope and Methodology




the federal government and HUD’s own internal control standards and
identified the requirement that programs have an annual risk assessment.
We interviewed HUD regarding any risk assessment performed for the
MTW program. Finally, we interviewed HUD officials to determine
whether any of the performance information that MTW agencies reported
had been verified. We compared HUD’s lack of verification with the
internal control standards and guidance that emphasized the need for
control activities to ensure that program participants report information
accurately.

To discuss the potential benefits and concerns associated with expanding
the number of PHAs that can participate in the program, we reviewed
studies, reports, and testimonies by researchers, affordable housing
advocates, resident advocates, and the HUD Office of Inspector General.
For all three objectives, we interviewed officials from the seven MTW
agencies in our sample and representatives from affordable housing
advocacy organizations such as the Council of Large Public Housing
Agencies, the National Association of Housing and Redevelopment
Officials, the National Leased Housing Association, and the Public
Housing Authorities Directors Association. We spoke with resident
advocacy organizations such as the National Low-Income Housing
Coalition, the National Housing Law Project, and legal aid agencies that
represented residents serviced by five of our sample MTW agencies. 7 We
also interviewed staff from the Center for Budget Policy and Priorities, a
research organization that has studied and written about the MTW
program; researchers who had evaluated the MTW program; and HUD
officials from the MTW office and the field offices that corresponded to our
sample of agencies. During our interviews, we discussed the potential
benefits of expansion and the concerns of these organizations. Based on
our review of available studies and reports and interviews with the above
mentioned stakeholders, we identified key benefits and concerns. We
also made observations based on our findings related to the availability of
performance information for the program and HUD’s monitoring efforts.

We conducted this performance audit from July 2011 to April 2012 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our



7
Two legal aid agencies did not respond to our requests for interviews.




Page 49                                        GAO-12-490 Moving to Work Demonstration
Appendix I: Scope and Methodology




findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 50                                GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
             Appendix II: Comments from the Department
             of Housing and Urban Development



of Housing and Urban Development




             Page 51                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 52                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 53                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 54                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 55                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 56                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 57                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 58                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 59                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 60                                     GAO-12-490 Moving to Work Demonstration
Appendix II: Comments from the Department
of Housing and Urban Development




Page 61                                     GAO-12-490 Moving to Work Demonstration
Appendix III: GAO Contact and Staff
                  Appendix III: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Mathew J. Scirè, (202) 512-8678 or sciremj@gao.gov
GAO Contact
                  In addition to the contact named above, Paige Smith (Assistant Director),
Staff             Anna Carbino, Emily Chalmers, John McGrail, Marc Molino, Lisa Moore,
Acknowledgments   Daniel Newman, Lauren Nunnally, and Andrew Stavisky made key
                  contributions to this report.




(250626)
                  Page 62                                GAO-12-490 Moving to Work Demonstration
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.