United States General Accounting Office GAO Report to the Chairman, Subcommittee on Housing and Community Opportunity, Committee on Banking and Financial Services, House of Representatives January 1997 PUBLIC HOUSING HUD Should Improve the Usefulness and Accuracy of Its Management Assessment Program GAO/RCED-97-27 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-275648 January 29, 1997 The Honorable Rick Lazio Chairman, Subcommittee on Housing and Community Opportunity Committee on Banking and Financial Services House of Representatives Dear Mr. Chairman: This report responds to your request that we review the Department of Housing and Urban Development’s (HUD) use of its Public Housing Management Assessment Program (PHMAP). Specifically, the report discusses (1) HUD’s use and implementation of the program at its field offices, (2) public housing authorities’ (PHA) PHMAP scores over the first 4 years of the program, and (3) limits on any additional uses for the program. We are sending copies of this report to the Secretary of Housing and Urban Development and will make copies available to others upon request. Major contributors to this report are listed in appendix VI. Please call me at (202) 512-7631 if you or your staff have any questions. Sincerely yours, Judy A. England-Joseph Director, Housing and Community Development Issues Executive Summary At a current annual cost of $5.4 billion, the Department of Housing and Purpose Urban Development (HUD) subsidizes the operation, maintenance, and modernization of the nation’s public housing, a $90 billion investment that provides homes to 3 million people. Because HUD provides this subsidy to more than 3,000 independent, state-chartered public housing authorities, the Congress holds HUD responsible for ensuring that these authorities efficiently provide safe and decent housing and protect the federal investment in their properties. However, the Public Housing Management Assessment Program (PHMAP)—HUD’s primary tool for measuring housing authorities’ performance—has been criticized as unreliable, inaccurate, and at times conflicting with good property management practices. Nevertheless, because no other measurement tool exists, the Congress and HUD have proposed at different times using the program as a basis for deregulating or rewarding high-scoring housing authorities. Stressing the need for HUD to hold housing authorities accountable while making better use of the data PHMAP produces, the Chairman of the Subcommittee on Housing and Community Opportunity, House Committee on Banking and Financial Services, asked GAO to review HUD’s implementation and use of PHMAP. Specifically, the Chairman asked GAO to determine • whether HUD’s field offices are using PHMAP and complying with the program’s statutory and regulatory requirements to monitor and provide technical assistance to housing authorities; • whether PHMAP scores have increased and how HUD uses the program to inform HUD’s Secretary and the Congress about the performance of housing authorities; and • whether PHMAP scores are consistently accurate and can be considered a generally accepted measure of good property management. The National Affordable Housing Act of 1990 directed HUD to use certain Background indicators, including vacancy rates and the percentage of rents uncollected, to assess the management performance of local housing authorities. The act also directed HUD to determine the cause of an authority’s management problems, commit both HUD and the housing authority to a specific course of corrective action, and document agreed upon corrective actions in memorandums of agreement. To meet the act’s requirements, HUD developed PHMAP to annually obtain data from each housing authority on 12 basic indicators of management performance, such as vacancy rates and operating expenses. On the basis of aggregate Page 2 GAO/RCED-97-27 Public Housing Executive Summary performance against these indicators, HUD calculates a score from 0 to 100 for each authority and assigns one of the following three designations: “troubled performer” for a score less than 60, “standard performer” for a score between 60 and less than 90, and “high performer” for a score 90 or above. HUD’s field offices have nearly 800 staff devoted to oversight of housing authorities and implementation of the full range of the Department’s public housing programs, including PHMAP. Because HUD’s field offices are responsible for implementing PHMAP, HUD expects them to ensure housing authorities meet PHMAP’s requirements and provide technical assistance when an authority has problems doing so. Technical assistance can cover a wide variety of activities, such as a focused review of an authority’s compliance with HUD’s rules and regulations, discussions over the telephone, on-site reviews of HUD-funded modernization work, or suggestions for improving an authority’s occupancy rate and rent collection procedures. Field offices depend on each authority to submit and certify to the accuracy of about half the data that lead to the overall PHMAP score; the balance of the information HUD uses comes from its existing information system for tracking expenditures from major grants. With each troubled authority, the act requires HUD to perform an independent management assessment to determine the causes of an authority’s problems and then enter into a binding memorandum of agreement stipulating the problems that authority needs to address and an approach and a timetable to resolve them. Also, when a troubled authority’s new PHMAP score would cause HUD to remove its troubled designation, HUD’s policy has been to require its field offices to verify the accuracy and completeness of the data the housing authority submitted. Standard- and high-performing authorities that fail any of the 12 indicators must submit a plan for improving their performance in that indicator. While HUD’s primary use of PHMAP has been to identify troubled housing authorities and target technical assistance to them, the Congress and HUD have proposed to use PHMAP for other purposes. In 1994, the Senate Committee on Banking, Housing, and Urban Affairs proposed additional flexibility for housing authorities that had achieved PHMAP scores over 90. In its fiscal year 1997 budget request, HUD proposed to give high-performing housing authorities bonuses based in part on their PHMAP scores. Page 3 GAO/RCED-97-27 Public Housing Executive Summary Most of HUD’s field offices are using PHMAP to identify troubled housing Results in Brief authorities and target HUD’s limited technical assistance resources. However, the field offices have not been systematically using the assessment program, as required by statutes and regulations, to monitor housing authorities’ progress in improving their performance and target technical assistance to them. For example, the field offices have generally not been meeting the act’s requirement to enter into memorandums of agreement with troubled authorities, nor have the field offices consistently met HUD’s requirement that housing authorities document plans to correct low scores in individual performance indicators. Also, the impact of a 1995 reorganization of the field offices’ functions and current departmental downsizing continue to influence some offices’ ability to provide technical assistance. Performance scores generally have increased during the first 4 full years of the program. With average scores increasing, the total number of troubled housing authorities has decreased, and the greatest proportion of those that are troubled are the smallest authorities—those managing fewer than 100 units. The proportion of high-performing authorities has increased steadily from about 33 percent in 1992 to over 50 percent in 1995. High-performing authorities manage nearly 50 percent of all public housing units. Periodically, HUD officials provide the Secretary and the Congress information on the performance of all housing authorities as well as the number of troubled authorities. HUD’s confirmatory reviews of the information underlying assessment scores have shown the scores to be inaccurate in half the instances when such reviews were performed. Regardless of the scores’ accuracy, HUD and public housing industry officials do not believe that the management assessment program comprehensively assesses how well local housing authorities manage their properties. This is because the assessment program does not include indicators to specifically measure overall housing quality or the quality of maintenance. Principal Findings Field Offices’ Use of GAO visited 5 of HUD’s 49 field offices and sent a survey to all 49 offices PHMAP and Related asking about their use of PHMAP. Officials in these offices generally found PHMAP useful to identify troubled housing authorities and target limited Oversight Tools Has Been technical assistance and oversight resources. However, field offices Limited Page 4 GAO/RCED-97-27 Public Housing Executive Summary reported that they have not been systematically complying with PHMAP’s statutory and regulatory follow-up requirements for all housing authorities. The extent to which HUD’s field offices used PHMAP to provide technical assistance to housing authorities varies widely. In part, this difference stems from how different field offices have interpreted their role in helping authorities improve performance on PHMAP indicators as well as their overall operations. Some field offices told GAO that they interpret their role in providing oversight and technical assistance narrowly, generally limiting their assistance to advice, information on complying with federal rules and regulations, and suggestions for solving management problems. Other offices were more willing to get involved in a housing authority’s operations. For example, staff from one field office spent several days at a troubled authority to help it set up proper tenant rent records and waiting lists. Regarding noncompliance with follow-up requirements, GAO found the following: • In 1992, HUD’s field offices entered into 29 percent of the required memorandums of agreement; by 1995, just 18 percent of the housing authorities that should have been operating under a memorandum of agreement actually were. Primarily, field offices said the reason they did not enter into these required agreements with troubled housing authorities is that the housing authorities had already corrected or were in the process of correcting their management deficiencies. • Even though HUD requires an improvement plan to address each performance indicator an authority fails unless that authority can correct the deficiency within 90 days, 31 percent of HUD’s field offices had not ensured that housing authorities had developed such plans. • Field offices generally did not meet HUD’s requirement to confirm the accuracy of all PHMAP scores that were high enough to remove the troubled designation from a housing authority. In fiscal year 1995, HUD’s field offices confirmed fewer than 30 percent of the scores that should have been confirmed. While 13 offices performed none of the required confirmatory reviews, some of these same offices performed confirmatory reviews of other standard- or high-performing authorities. Although no minimum level of activity is required, in 1995 HUD confirmed just over 6 percent of all scores, with some field offices performing no confirmatory reviews and others performing 10 or more. Page 5 GAO/RCED-97-27 Public Housing Executive Summary • At the five field offices GAO visited, limited use was made of the independent financial and compliance audits conducted annually at each authority. Over a year ago, HUD began to require that these audits certify that the PHMAP data from each authority were accurate and complete; HUD added this requirement because it does not have the resources to confirm each score every year. Nonetheless, few HUD staff in the field offices GAO visited were aware of this requirement or used the audits to better focus their oversight and technical assistance. The technical assistance HUD staff find most effective at improving the performance of housing authorities is often the type of help least frequently provided. HUD’s field offices told GAO that technical assistance and oversight are most effective when they provide it on-site at the local authority. For example, while most staff in field offices said confirmatory reviews, which must take place on-site, were one of the most effective ways to provide technical assistance, discussions over the telephone were the most common form of technical assistance. Some field offices cited resource constraints—a lack of staff, travel funds, or expertise—as the main reason for not meeting follow-up requirements or visiting housing authorities more often; others opted not to enforce requirements when they believed the authorities were already addressing their problems. Scores Are Increasing, but Average PHMAP scores have increased over the life of the program, rising HUD Recognizes Database from an average of 83 in 1992 to 86 in 1995, the last year of complete data. Flaws Limit Program’s Use In addition, HUD’s database of PHMAP scores indicates the following two trends: • The number of high-performing authorities grew each year, from almost a third in 1992 (33 percent) to over half in 1995 (57 percent). • The number of troubled housing authorities has declined from 130 in 1992 to 83 in 1995. However, by 1995, the smallest housing authorities—those managing fewer than 100 units—accounted for a greater share of those designated as troubled than when the program began. In 1995, half of all housing authorities HUD designated as troubled were small. GAO found missing, inaccurate, and inconsistent data in HUD’s primary database for storing PHMAP scores. Nevertheless, HUD makes regular, periodic use of the database to provide information to the Secretary and the Congress on all housing authorities’ scores and the number of troubled authorities at any given time. However, before providing this information to others, HUD first manually verifies much of the data it draws from this Page 6 GAO/RCED-97-27 Public Housing Executive Summary system. Senior HUD officials acknowledged these problems with the database and added that they are currently working to address data accuracy and reliability problems as well as improve their ability to correct errors sooner. The Questionable PHMAP scores are often changed after HUD confirms the data used to Accuracy and Validity of support the scores. In commenting on this report, HUD said that most PHMAP Scores Limit the confirmatory reviews are conducted of high-risk housing authorities whose data are most susceptible to being found inaccurate. Over Program’s Usefulness half—58 percent—of the changes HUD made to PHMAP scores resulted in HUD’s lowering the score by an average of 14 points; 42 percent of the changes resulted in HUD’s raising the score by an average of 8 points. Typically, HUD changes a PHMAP score after such a review for several reasons, including the housing authority’s failure to report correctly the required data or its failure to maintain documentation to support its data. PHMAP scores are not a generally accepted measure of good property management. HUD officials, as well as representatives of public housing industry associations and professional property management consultants, told GAO that the PHMAP indicators do not assess all major aspects of a housing authority’s performance. For example, PHMAP does not include an independent on-site inspection of the condition of an authority’s housing, so it does not adequately assess the quality of modernization work or routine maintenance. These same HUD officials and industry representatives also told GAO that PHMAP does not always allow for extenuating circumstances that can lead to decisions inconsistent with good property management. For example, a housing authority can improve its PHMAP score on the tenants accounts receivable indicator by writing off as uncollectible past due rents from vacated tenants, but PHMAP would not measure how diligent an effort the authority had undertaken to collect the rent. GAO recommends that the Secretary of HUD Recommendations • provide guidance to its field offices that clearly (1) articulates their minimally acceptable roles regarding oversight and assistance to housing authorities and (2) emphasizes the importance of using the results of the independent audits to better target the Department’s limited technical assistance resources. Page 7 GAO/RCED-97-27 Public Housing Executive Summary Furthermore, because scores are not consistently accurate and PHMAP does not measure all aspects of property management, GAO recommends that HUD • not consider additional uses for PHMAP, including using scores as criteria for funding bonuses, until it determines that PHMAP meets an acceptable level of accuracy and more comprehensively measures property management performance and • require its field offices to confirm the PHMAP scores of housing authorities with scores low enough that the authorities are at risk of being designated troubled. GAO provided a draft of this report to HUD for its review and comment. HUD Agency Comments agreed with GAO’s recommendations and described the steps that the Department has begun taking to implement them. However, HUD expressed concern that the draft report (1) used potentially inaccurate data from HUD’s PHMAP database, (2) incorrectly assumed that PHMAP was intended to be an all-encompassing system that measures both management performance and physical housing conditions, (3) neglected to place PHMAP in a historical perspective by discussing HUD’s previous systems for assessing and identifying troubled housing authorities, and (4) reached incorrect conclusions regarding the reliability of all PHMAP scores on the basis of the results of confirmatory reviews of the high-risk authorities most susceptible to discrepancies in their PHMAP data. GAO used the best data available for this review. GAO recognizes and discusses in the report the inaccuracies in the PHMAP data. Where HUD was able to provide more accurate data than its PHMAP database reports, GAO used that data in this report. GAO did not assume that PHMAP should be a complete measure of both performance and physical conditions. This report describes the current uses of PHMAP data and addresses how the program’s limitations affect its suitability for additional purposes. GAO did not provide a historical perspective on the program because discussing HUD’s previous systems for assessing and identifying troubled housing authorities did not contribute to the review’s objectives of evaluating HUD’s use of PHMAP, trends in scores, or limits on additional uses for the program. Finally, GAO did not reach a conclusion about the reliability of all housing authorities’ PHMAP scores. The report discusses only the reliability of PHMAP scores for those housing authorities whose scores are so low that they Page 8 GAO/RCED-97-27 Public Housing Executive Summary may be at risk of being designated troubled. Statements have been added to the report to clarify this point. HUD’s written comments are presented in appendix V and GAO’s responses are discussed at the end of each chapter as appropriate. Page 9 GAO/RCED-97-27 Public Housing Contents Executive Summary 2 Chapter 1 12 Indicator Grades Determine the PHMAP Score, Performance 14 Introduction Designation, Required Follow-Up, and Incentives HUD’s Field Offices Implement PHMAP 16 Objectives, Scope, and Methodology 18 Chapter 2 21 HUD Uses PHMAP to Identify Troubled Housing Authorities, but 21 Although Field Offices Technical Assistance Varies Use PHMAP to HUD’s Infrequent Use of Some Oversight Tools May Not 24 Adequately Improve the Performance of PHAs or Target Identify Troubled Technical Assistance PHAs, Compliance Conclusions 31 With Statutory and Recommendation 31 Agency Comments 31 Agency Follow-Up Requirements Has Been Limited Chapter 3 33 Most PHMAP Scores Are Increasing and Fewer Housing 33 Although PHMAP Authorities Are Troubled Scores Have Risen, HUD Recognizes Database Flaws and Plans Corrections 37 Agency Comments 39 HUD Recognizes That Flaws in the Program’s Database Limit Its Use Chapter 4 40 Accuracy of Scores and Validity as a Management Assessment 40 The Questionable Tool Limit Uses for PHMAP Accuracy of PHMAP’s The Congress and HUD Have Proposed to Use PHMAP as a Basis 43 for Deregulation and Funding Bonuses Scores and the Conclusions 45 Program’s Validity Recommendations 45 Limit Its Usefulness Agency Comments 46 Page 10 GAO/RCED-97-27 Public Housing Contents Appendixes Appendix I: Average PHMAP Score by Geographic Region, Fiscal 48 Years 1992-95 Appendix II: Number of Troubled PHAs by Geographic Region, 50 Fiscal Years 1992-95 Appendix III: Number of Standard-Performing PHAs by 52 Geographic Region, Fiscal Years 1992-95 Appendix IV: Number of High-Performing PHAs by Geographic 54 Region, Fiscal Years 1992-95 Appendix V: Comments From the Department of Housing and 56 Urban Development Appendix VI: Major Contributors to This Report 61 Tables Table 1.1: Twelve PHMAP Indicators 13 Table 2.1: PHAs Receiving a Confirmatory Review From a HUD 28 Field Office, Fiscal Years 1992-95 Table 3.1: Average PHMAP Score by PHA Size Category, Fiscal 34 Years 1992-95 Table 3.2: Number of PHAs by PHMAP Performance Category, 35 Fiscal Years 1992-95 Table 3.3: Failed Indicators by Percentage of PHAs, Fiscal Years 36 1992-95 Table 4.1: Changes in PHMAP Scores After HUD’s Field Offices 41 Performed Confirmatory Reviews in Fiscal Year 1995 Figures Figure 2.1: Percentage of Troubled PHAs Operating Under a 26 MOA, Fiscal Years 1992-95 Figure 2.2: Number of Confirmatory Reviews Performed by HUD 29 Field Offices, Fiscal Year 1995 Figure 3.1: Number of Troubled PHAs by Size, Fiscal Years 37 1992-95 Abbreviations GAO General Accounting Office HUD Department of Housing and Urban Development IG Office of Inspector General MOA memorandum of agreement PHA public housing authority PHMAP Public Housing Management Assessment Program SMIRPH System for Management Information Retrieval-Public Housing Page 11 GAO/RCED-97-27 Public Housing Chapter 1 Introduction Because public housing represents a $90 billion investment on the part of the federal government since the program’s inception in 1937 and because the Department of Housing and Urban Development (HUD) currently spends $5.4 billion a year on operating subsidies and modernization grants for this housing, interest remains keen in knowing how well local public housing authorities (PHA) are managing their properties. The PHAs, through which HUD provides these subsidies and grants, house 3 million low-income people, many of whom are elderly or disabled. The Congress holds HUD responsible for ensuring that the authorities provide safe and decent housing, operate their developments efficiently, and protect the federal investment in their properties. The National Affordable Housing Act of 1990 required HUD to develop indicators to assess the management performance of PHAs.1 This law became the framework through which HUD developed one of its primary oversight tools for housing authorities, the Public Housing Management Assessment Program (PHMAP). Primarily, PHMAP establishes objective standards for HUD to evaluate and monitor the management operations of all PHAs to identify those that are troubled. According to HUD, PHMAP also allows the Department to identify ways to reward high-performing PHAs as well as improve the management practices of troubled PHAs. The program also allows PHAs’ governing bodies, management officials, residents, and the local community to better understand and identify specific program areas needing improvement. To help improve public housing management, the National Affordable Housing Act of 1990, as amended (the act), required HUD to develop indicators to assess the performance of PHAs in all the major aspects of their management operations. The act required HUD to use certain indicators as well as provided discretion for the Secretary of HUD to develop up to five additional indicators that the Department deemed appropriate. HUD implemented PHMAP by using the 12 indicators listed in table 1.1, the first seven of which are those required by statute. 1 P.L. 101-625, Section 502(a), as amended by the departments of Veterans Affairs and Housing and Urban Development, and Independent Agencies Appropriations Act of 1992. Page 12 GAO/RCED-97-27 Public Housing Chapter 1 Introduction Table 1.1: Twelve PHMAP Indicators PHMAP indicator Measurement 1.Vacancy number and percentage Number and percentage of vacancies, including progress made within the previous 3 years to reduce vacancies 2.Modernization Amount and percentage of funds unexpended after 3 years 3.Rents uncollected Balance uncollected as a percentage of total rents to be collected 4.Energy consumption Increase in annual consumption 5.Unit turnaround Average time required to repair and reoccupy vacant units 6.Outstanding workorders Proportion of maintenance workorders outstanding 7.Annual inspection and condition of units Percentage of units and systems not and systems inspected to determine preventive maintenance or modernization needs 8.Tenants accounts receivable Percentage of monies owed to the authority by current residents 9.Operating reserves Percentage of operating reserves maintained by the authority 10.Routine operating expenses Level of operating expenses compared to operating income and federal subsidy 11.Resident initiatives Existence of a partnership between residents and the authority to promote opportunities for self-sufficiency and other programs 12.Development Ability to develop additional public housing units Because some indicators are more important than others in measuring management performance, HUD assigns them added weight in determining the overall score.2 HUD considers the indicators for vacancies, rents uncollected, annual inspection and condition of units and systems, and resident initiatives most indicative of good property management and delivery of services to residents, so each one has a greater weight than other indicators. After reviewing existing procedures and extensively consulting with a group of PHAs, public housing industry groups, private management firms, resident groups, and HUD staff in field offices, HUD has significantly revised the PHMAP indicators. HUD’s revisions to PHMAP, published December 30, 1996, eliminated three indicators; consolidated four other indicators into 2 According to a HUD official, these weights reflect the Department’s determination that they are the most important indicators of good property management. Page 13 GAO/RCED-97-27 Public Housing Chapter 1 Introduction two; and added one new indicator, security.3 These revisions primarily address the performance indicators on which housing authorities report data, not HUD’s use of PHMAP data. Annually, PHAs receive a grade of “A” through “F” for each of the twelve Indicator Grades indicators that apply to their operations. HUD uses a formula that reflects Determine the the weights assigned to each indicator, converts indicator grades into PHMAP Score, points, totals each PHA’s points, and divides that total by the maximum total the PHA could have achieved to arrive at a percentage. That Performance percentage, a number between 0 and 100, is the PHMAP score. Designation, Required draws data on the performance of a housing authority from two Follow-Up, and HUD sources to determine the authority’s PHMAP score. First, the housing Incentives authority submits data to HUD for about half of the PHMAP indicators and certifies that this information is accurate and complete.4 HUD assigns grades to each of these indicators according to a comparison of the authority’s data and HUD’s criteria for grades “A” through “F.” The balance of the information HUD uses comes from its own information system for tracking expenditures from major grants. This system contains the financial and other types of data the field offices need to grade the remaining indicators for which the PHAs do not provide data. The field offices use this data and the PHA-certified data to determine indicator scores, the PHMAP score, and the PHA’s performance designation. The PHMAP score is HUD’s starting point for both the performance designation it assigns to a PHA and, depending on that designation, the extent of follow-up required of the PHA to correct deficiencies identified during the PHMAP assessment. Generally, HUD uses three designations to describe the performance of PHAs: • troubled performers are those scoring less than 60 percent;5 • standard performers are those scoring between 60 and less than 90 percent; and • high performers are those scoring 90 percent or more. 3 HUD eliminated the tenants accounts receivable, routine operating expenses, and development indicators. HUD also consolidated unit turnaround into the vacancy indicator and energy consumption into the financial management (formerly operating reserves) indicator. 4 PHAs certify the data for the following indicators: vacancy number and percentage, rents uncollected, unit turnaround, outstanding work orders, annual inspection and condition of units and systems, and resident initiatives. 5 PHMAP includes an additional designation, modernization troubled (mod-troubled), which can apply to any PHA that scores less than 60 percent on the modernization indicator. Page 14 GAO/RCED-97-27 Public Housing Chapter 1 Introduction HUD has the discretion to withhold the troubled designation or award the high performer designation if a PHA’s score is within 10 points of the threshold for either designation and HUD determines that its score results from the physical condition and/or neighborhood environment of that authority’s units rather than from the PHA’s poor management practices. If a housing authority is designated as troubled, it faces several mandatory follow-up activities and/or corrective actions to improve performance and remove the troubled designation. Specifically, the act requires HUD to perform an independent management assessment of the troubled PHA’s overall operations to identify the causes of the deficiencies that led to its poor PHMAP score. HUD uses private contractors to perform these independent assessments. HUD expects the independent assessments to form the basis for the second requirement for troubled PHAs—the memorandum of agreement (MOA). A memorandum of agreement is a binding contract between HUD and a troubled PHA to identify solutions to its management problems and pursue those solutions in a way that is significant, expeditious, and lasting. Among other things, HUD requires that the MOA address the specific responsibilities of HUD and the PHA, the resources each will commit to resolving the authority’s problems, the annual and quarterly performance targets for improving its performance on PHMAP indicators, and the incentives for it to meet its performance targets as well as sanctions for failing to do so. A PHA’s initial MOA generally lasts 18 months so that it can complete a second-year agreement with HUD, if necessary, before the first expires.6 HUD’s regulations for implementing PHMAP require standard- and high-performing PHAs to develop improvement plans for every PHMAP indicator on which the PHA received an “F,” unless the PHA can correct the deficiency within 90 days;7 HUD may also choose to require these plans for indicators receiving scores of “D” or “E” when failure to raise the grade might pose significant added risk. An improvement plan documents how and when the PHA plans to correct deficiencies. Although similar in content and scope to a memorandum of agreement, improvement plans differ in that (1) PHAs develop and submit them to HUD for approval rather than 6 Second and subsequent year agreements are necessary only as long as the PHA remains troubled. 7 Mandatory improvement plans would not apply to the modernization indicator because any PHA receiving an F on this would be designated mod-troubled. Page 15 GAO/RCED-97-27 Public Housing Chapter 1 Introduction negotiate them with HUD officials and (2) they are not a binding contractual commitment between the PHA and HUD. When HUD first implemented PHMAP, it offered high-performers a variety of incentives, primarily regulatory relief from various reporting requirements. These incentives included less frequent reviews of changes to a PHA’s operating budget and, for those performing well on the modernization indicator, no prior HUD review for architects’ or engineers’ contracts. In addition to regulatory relief, high-performing PHAs receive a HUD certificate of commendation and public recognition for their performance. In its fiscal year 1997 budget request, HUD proposed an additional PHMAP-based incentive for high-performing PHAs when it sought to create a $500 million capital bonus fund (as part of the $3.2 billion it sought for its public housing capital fund).8 To be eligible for a bonus, a PHA would have to be a PHMAP high performer and have undertaken substantive efforts to obtain education and job training for its residents. However, the Congress chose not to fund the bonus proposal for public housing or any of HUD’s other major programs, in part because of concerns about HUD’s ability to accurately and reliably track the performance of bonus recipients. With nearly 800 staff devoted to oversight of housing authorities and HUD’s Field Offices implementation of the full range of HUD’s public housing programs, its field Implement PHMAP offices have the bulk of the Department’s responsibility for the day-to-day implementation of PHMAP. Field offices’ PHMAP responsibilities include determining the indicator grades and PHMAP scores, negotiating memorandums of agreement, approving PHAs’ improvement plans, and monitoring their progress in meeting the goals the MOA or improvement plan set forth. To determine a housing authority’s PHMAP score, a field office relies on that PHA to provide about half the data that leads to the overall PHMAP score and certify the data’s accuracy. As a result, the overall PHMAP score and everything it influences—from incentives for high performers to sanctions for troubled PHAs—are very much a joint effort and a shared responsibility. A PHA may also request to exclude or modify the data HUD should consider in computing its PHMAP score. An exclusion means that the indicator (or one or more of its components) is entirely excluded from calculations to determine the PHMAP score. For example, PHAs with no ongoing 8 The public housing capital fund would have consolidated public housing modernization, development, and several other capital repair and replacement programs. Page 16 GAO/RCED-97-27 Public Housing Chapter 1 Introduction modernization or development programs are automatically excluded from being assessed on those indicators. Modifying the data for an indicator allows HUD to consider unique or unusual circumstances by exempting some of the data HUD usually requires the PHA to consider. The PHA still receives a score for the indicator, but the score would not reflect the data associated with the PHA’s unique or unusual circumstances. For example, a PHA operating under a court order not to collect tenants’ rent at specific developments until it corrects deficiencies the court had identified can seek to exempt those units in its developments from being considered in its indicator score for rents uncollected. A PHA always has the right to appeal a field office’s decision about modifications, exclusions, indicator scores, or the performance designation. However, after those appeals have been exhausted, the field office certifies the PHA’s PHMAP score, assigns a final performance designation, and proceeds with any required improvement plans, MOAs, or other necessary follow-up. When a troubled authority’s new PHMAP score is high enough to cause HUD to remove its troubled designation, HUD’s policy is to require the field office to verify the accuracy and completeness of the new data submitted by the housing authority. HUD also requires the field office to conduct a confirmatory review to verify the data the PHA had certified as well as the accuracy of the data HUD had obtained from its own information system. HUD’s guidance for implementing PHMAP stipulates that a confirmatory review must take place on-site at the PHA and cannot be accomplished through remote monitoring. HUD’s field offices may choose to conduct some confirmatory reviews of standard- and high-performing PHAs’ PHMAP certifications. HUD expects its field offices to choose these PHAs according to the risk they pose and focus on those with the highest potential for fraud, waste, mismanagement, or poor performance. Some of the factors HUD field offices may consider in analyzing the risk associated with a PHA’s PHMAP certification include size (number of units), borderline troubled designation (5 percent above or below the percentage for the designation), and negative trends in overall or individual indicator scores over several years. In May 1995, HUD expanded the scope of the annual independent audit each PHA receives in order to improve the Department’s ability to determine whether PHA-certified data are accurate. The annual audit, conducted pursuant to the requirements of the Single Audit Act, examines the housing authority’s financial statements, internal controls, and compliance with HUD’s rules and regulations. Housing authorities are Page 17 GAO/RCED-97-27 Public Housing Chapter 1 Introduction responsible for selecting their own auditors and submitting the results of the audits to their HUD field office. Field offices are responsible for reviewing the audits to ensure they meet all of HUD’s requirements and, when they have approved the audit, reimbursing housing authorities for them. In fiscal year 1995, these independent audits cost HUD about $8 million for all housing authorities. HUD now requires the independent auditors to determine whether a housing authority has adequate documentation for the data it submits to HUD for its PHMAP certification. According to HUD officials, because the Department’s resources are too limited to conduct annual confirmatory reviews of most housing authorities, they expected to use the results of these audits to better focus HUD’s attention, oversight, and technical assistance. In addition to paying for the audits, HUD expects its field offices to use the results as part of a risk assessment to determine which housing authorities should get the most sustained attention and technical assistance. Stressing the need for HUD to hold housing authorities accountable while Objectives, Scope, making better use of the data that PHMAP produces, the Chairman of the and Methodology Subcommittee on Housing and Community Opportunity, House Committee on Banking and Financial Services, asked GAO to review HUD’s use and implementation of PHMAP. As agreed with the Chairman’s office, we reviewed • whether HUD’s field offices are using PHMAP and complying with the program’s statutory and regulatory requirements to monitor and provide technical assistance to housing authorities, • whether PHMAP scores have increased and how HUD uses the program to inform HUD’s Secretary and the Congress about the performance of housing authorities, and • whether PHMAP scores are consistently accurate and can be considered a generally accepted measure of good property management. We developed information from several different sources to address questions concerning the usefulness of PHMAP to HUD and other uses for which PHMAP may not be appropriate. To determine PHMAP’s usefulness to HUD, we interviewed officials and collected information on technical assistance activities at both the Department’s headquarters and field offices. At HUD’s headquarters, we analyzed a variety of documents pertaining to PHMAP and discussed the program’s use as a basis for Page 18 GAO/RCED-97-27 Public Housing Chapter 1 Introduction technical assistance with the Offices of the Deputy Assistant Secretaries under HUD’s Assistant Secretary for Public and Indian Housing. At HUD’s field offices, our approach was twofold. First, we surveyed them via fax questionnaire to obtain data on the use of PHMAP, such as the number of confirmatory reviews each field office performs and how useful such program tools as improvement plans have been.9 This data reflect responses from all of HUD’s public housing field offices.10 Second, we visited five HUD field offices to review their use of PHMAP in more depth and to supplement the information we had gathered in our survey.11 We judgmentally selected the five field offices because of their geographic distribution, variations in the number of HUD staff in each office as well as the number of PHAs each oversees, and variations in average PHMAP scores for the PHAs reporting to those offices. To provide information on PHAs’ PHMAP scores, we relied on existing data from HUD sources, including HUD’s primary public housing database, the System for Management Information Retrieval-Public Housing (SMIRPH). From this database, we extracted the module containing housing authorities’ PHMAP data, including the PHMAP scores and individual indicator grades. Our analysis covers federal fiscal years 1992 through 1995 because the first fiscal year in which the rules governing PHMAP took effect was 1992 and the most recent year for which all PHMAP scores were complete at the time of our review was 1995. We did not systematically verify the accuracy of HUD’s data or conduct a reliability assessment of HUD’s database. In performing our analysis we found erroneous and incomplete information for a few PHAs, ranging from 1 to 3 percent of the total. We confirmed this with HUD officials, who attributed the errors to mistakes in data input or the field office’s having entered incomplete scores. However, because we used these data in context with additional evidence we obtained directly from HUD’s field offices and we did not focus on the scores of specific PHAs or small groups 9 HUD headquarters does not collect or centrally maintain the information for which we surveyed the field offices. 10 Although 51 of HUD’s 79 field offices have public housing oversight responsibilities, 2 of those 51 have assumed the workload of another field office due to attrition or temporary vacancies in the public housing division. In these cases, the field office handling the workload provided a combined response reflecting data for both office’s jurisdictions. As a result, our data is drawn from 49 responses, but it does reflect the PHMAP-related activities of all of HUD’s public housing field offices. 11 We visited HUD’s field offices in Birmingham, Alabama; Kansas City, Missouri; Minneapolis, Minnesota; San Antonio, Texas; and San Francisco, California. Page 19 GAO/RCED-97-27 Public Housing Chapter 1 Introduction of PHAs, we believe our conclusions about overall trends in scores are valid. Throughout the course of our work, because the number of PHAs reporting PHMAP scores is too great for us to visit a representative sample, we consulted with several prominent groups representing the public housing industry to discuss HUD’s uses for PHMAP as well as their perspectives on the program’s ability to measure the performance of public housing authorities. These groups include the Council of Large Public Housing Authorities, the National Association of Housing and Redevelopment Officials, and the Public Housing Authorities Directors Association. We provided a draft of this report to HUD for review and comment. HUD’s comments appear in appendix V and are addressed at the end of each applicable chapter. We performed our work from January through December 1996 in accordance with generally accepted government auditing standards. Page 20 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited HUD’s field offices use PHMAP scores for their primary intended purposes: as a standard, objective means to identify troubled housing authorities; to compare performance among PHAs; and to identify when, where, and how to target HUD’s limited resources for technical assistance. However, beyond identifying troubled authorities and what they need, the amounts and kinds of technical assistance HUD provides varies because its field offices interpret their responsibilities differently—some choose to be actively involved while others adopt a hands-off approach. Furthermore, HUD’s 1995 reorganization of its field offices adversely affected some offices’ ability to provide technical assistance while others adapted to changed expectations and resumed providing as much assistance as they did before the reorganization. As part of HUD’s oversight of public housing, the PHMAP score is an HUD Uses PHMAP to important tool for identifying troubled authorities so HUD can focus Identify Troubled technical assistance and monitoring on them. The most common types of Housing Authorities, technical assistance that HUD’s 49 public housing field offices provided all PHAs were telephone consultations, training, and participation in but Technical conferences. However, we found differences in how field offices defined Assistance Varies their roles in providing PHAs technical assistance as well as some innovations in how others provided that assistance. For example, some field offices have encouraged high-performing PHAs to provide “peer assistance” to lower performers. Many of the differences in assistance were due to variations in field offices’ interpretations of their roles and the impact of HUD’s 1995 reorganization of its field offices. HUD headquarters officials believe that more training for all field staff and leadership from field office managers would help achieve more quality and consistency among field offices in providing technical assistance. HUD Uses PHMAP to Officials in 40 of HUD’s 49 field offices rated PHMAP as being of “utmost” or Target PHAs for Technical “major” importance in identifying which housing authorities need the most Assistance technical assistance. According to field office staff, PHMAP provides standard indicators to objectively measure an authority’s performance. In addition, some staff said that because PHAs have a strong aversion to failing performance scores and try to avoid failure, they are confident that when PHAs report information that results in low scores or failing grades, the data and the resulting scores are accurate. Because an accumulation of low or failing scores results in a PHA’s being designated troubled, HUD staff are confident that those PHAs PHMAP identifies as the worst-performing housing authorities are accurately designated as troubled performers. Page 21 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited Some field office staff also use declining PHMAP scores to provide an early warning of management problems and to identify which PHAs could need additional technical assistance. In addition, the staff use PHMAP’s 12 individual indicator grades to better focus their limited technical assistance resources and thereby maximize the benefits PHAs receive from HUD’s assistance. For example, one field office developed a package of technical assistance for the “resident initiatives” indicator because many PHAs failed this indicator. The package of assistance included sample policies and procedures for operating resident programs. Another field office developed assistance specifically for small housing authorities because many of them were having trouble renting their units when they became vacant (thus failing PHMAP’s unit turnaround indicator). Among other things, that field office provided its small PHAs an extensive list of suggestions on how and where to better market their units. Most technical assistance from HUD’s field offices consisted of telephone consultations, training sessions, and industry conferences. HUD also provided assistance—although limited because of time constraints—at the time of a PHMAP confirmatory review. During telephone consultations, several offices we visited answered questions from housing authority staff and helped the executive directors of new housing authorities better understand public housing regulations and operations. Training sessions covered these and other topics and provided more details than telephone discussions. In addition, to increase the amount of personal contact they have with housing authority staff and to provide technical assistance, field office staff said they regularly participate in conferences hosted by public housing industry associations. Field Offices’ Field offices’ interpretations of their obligation to improve the Interpretations of Their performance of housing authorities influences the type of technical Role and Their Recent assistance they provide. For example, officials in one field office did not believe that it was HUD’s role to manage PHAs’ operations. Instead, they Reorganization Influence believed that the role of their field office should be limited to providing the Level and Types of information on compliance with federal rules and regulations and to Technical Assistance suggesting solutions to management problems. This field office avoids showing PHAs how to manage their developments because the staff believe that they do not have sufficient expertise and that the housing authorities would view this advice as intrusive. In contrast, staff at other field offices that we visited believed they are obligated to tell PHAs what must be done to correct management Page 22 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited deficiencies because HUD is responsible for ensuring that PHAs use federal funds efficiently and effectively to provide safe, decent housing. For example, staff from one field office spent several days at a troubled authority to help it set up proper tenant rent records and waiting lists. In addition to differences in how they view their role to directly assist PHAs, we found differences in the extent to which field offices use outside resources to help their housing authorities. Some field offices told us that to compensate for a shortage of resources from HUD, they help PHAs in their jurisdiction by encouraging technical assistance from other PHAs rather than providing it themselves. For example, some of the field offices arranged for high-performing PHAs to provide peer assistance to authorities with management problems. One field office persuaded staff from a high-performing PHA to temporarily manage a small authority that unexpectedly lost its executive director. Another field office recruited a high-performing PHA to help another one develop a system for inspecting its housing units. In 1995, HUD reorganized the field offices and changed the responsibilities of the staff who oversee and assist PHAs. Before the reorganization, most field office staff were generalists and broadly understood federal housing regulations and PHA operations. After the reorganization, however, the responsibilities of individual field office staff became more specialized to focus on the rules and regulations of specific public housing operations.1 This specialization confused some staff in field offices and housing authorities as well as impaired the ability of some field offices to provide technical assistance. For example, field office staff we visited said that some specialists do not have the skills needed to do their jobs because many of them did not have the work experience or requisite training for the specialists’ positions; the staff also noted that HUD had not provided sufficient training for the staff to understand the reorganization and their new responsibilities. The staff also said that the reorganization was a source of confusion for PHAs. Before the reorganization, a housing authority could call one employee at HUD’s field office to answer all its questions; afterward, a housing authority generally needed to call several different staff at HUD’s field office to answer questions. Adjusting to the reorganization differed across field offices. At one field office, staff resisted the reorganization because they did not want to become specialists and they recognized that technical assistance to the 1 The five areas in which field office staff now specialize are finance and budget; marketing, leasing, and management; facilities management; community relations and involvement; and organization, management, and personnel. Page 23 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited PHAs suffered as a result. For example, the staff now disagree over who is responsible for overseeing certain PHA operations. They also have resisted working together to provide technical assistance and have not been sharing PHMAP information to develop the best plan for correcting management deficiencies. Other field offices we visited adapted to the reorganization. Staff in these field offices worked cooperatively to build on the skills of the experienced staff. For example, one field office continues to assign each housing authority to only one staff member who provides or coordinates all technical assistance to that authority. The responsible staff member, however, belongs to a team of staff from all operational areas who work together to solve each PHA’s problems. Officials at HUD headquarters, including the Deputy Assistant Secretary for Public and Assisted Housing Operations, acknowledged that some field offices had difficulty adjusting to the reorganization. They stated that although adequate training was crucial to the reorganization’s success, some field offices either did not seek it or did not take the need for it seriously, despite the availability of training funds for field staff. HUD officials continue to emphasize the importance and availability of training and expect field office management to assess the staff’s skills and expertise and request the appropriate training. These officials believe that because of limited staff resources, now and in the future, the reorganization is the best way for field offices to provide effective oversight and technical assistance to PHAs. Furthermore, they believe that managers of the field offices must take a more active leadership role in directing their staff to work together. The act and HUD’s requirements for how field offices use PHMAP provide for HUD’s Infrequent Use several tools to guide improvements in a housing authority’s performance of Some Oversight and thereby raise its indicator grades and PHMAP score. These tools include Tools May Not the memorandums of agreement (MOA), improvement plans, confirmatory reviews, and the annual independent audits. While such tools as MOAs and Adequately Improve improvement plans generally apply to PHAs designated as troubled or the Performance of failing specific indicators, a confirmatory review is mandatory for any PHA coming off HUD’s troubled list and an independent audit is mandatory for PHAs or Target all PHAs. Nonetheless, we found that the compliance of field offices with Technical Assistance statutory requirements and HUD’s guidance for using these tools has been inadequate and infrequent. Furthermore, HUD has not determined whether these statutory or agency requirements are effective, adequately improve housing authority Page 24 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited performance, or help the field offices better target limited technical assistance resources. As a result, HUD has little information to determine which of these tools best improve a PHA’s performance and which tools its field offices can use most effectively to offset their declining resources. Field Offices Make Limited Over 90 percent of the field offices we surveyed reported that on-site visits Use of Oversight Tools to the housing authorities were one of the most effective means to ensure compliance with PHMAP requirements and provide technical assistance. Officials at one field office responded that PHAs under its jurisdiction believed that on-site visits from HUD staff to provide technical assistance were essential to maintaining effective operations. Yet, most field office staff we visited made fewer personal visits to housing authorities than they felt were necessary because of limited staff resources and travel funds. Field office staff told us, for example, that their workload has increased because their offices have been unable to replace staff who have left the agency. With less time available for on-site visits, direct monitoring of the PHAs’ performance has occurred less frequently. In addition, some field office staff said that they could rarely justify to their management using limited staff and travel resources to visit a PHA that is more than a 1-day trip from the office unless that authority’s PHMAP score was below 60. Memorandums of Although HUD is required by law to enter into MOAs with troubled housing Agreement authorities to improve management performance, few field offices have done so. Figure 2.1 shows that the percentage of troubled PHAs operating under an MOA has been decreasing since 1992. Furthermore, in fiscal year 1995, only 3 of HUD’s 32 field offices that had troubled PHAs were fully in compliance with the requirement to enter into an MOA with each troubled authority.2 2 For fiscal year 1995, the 49 field offices responded that a total of 71 housing authorities were troubled, but only 13 had MOAs. However, HUD’s PHMAP database reports 150 housing authorities as troubled in fiscal year 1995. HUD headquarters officials told us that our survey information was probably more accurate than its database. Page 25 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited Figure 2.1: Percentage of Troubled PHAs Operating Under a MOA, Fiscal 100 Percentage of troubled housing authorities Years 1992-95 90 80 70 60 50 40 29 30 23 20 20 18 10 0 1992 1993 1994 1995 Fiscal year Source: HUD field offices’ responses to GAO’s questionnaire. The primary reason HUD’s field offices told us that they did not enter into these required agreements with troubled housing authorities is that the PHAs had already corrected or were in the process of correcting their management deficiencies. However, HUD headquarters officials told us they did not accept this as a valid reason for not meeting the requirement and questioned how the field offices could be sure the housing authorities were no longer troubled. Improvement Plans When a PHA fails any of PHMAP’s 12 performance indicators, HUD requires the responsible HUD field office to obtain a plan from that PHA for improving its performance and to track its progress against the plan. However, we found that nearly a third—31 percent—of HUD’s field offices had not ensured that local housing authorities had developed these plans. We also found examples of PHAs’ plans lacking specific strategies and time frames for correcting management deficiencies. For example, one PHA’s plan for a failing “rents uncollected” indicator simply stated that the housing authority would start collecting rent. Although field office staff acknowledged that the PHA also needed to update its standard tenant lease Page 26 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited and develop a rent collection policy to improve this indicator grade, they said that they had not yet had the time to contact the PHA to revise its plan. HUD requires its field offices to monitor the progress of housing authorities in implementing improvement plans to ensure PHAs meet the quarterly and annual performance targets in their plans. However, four of the five field offices we visited told us they do not follow up with the PHAs to determine the status of improvement plans or whether the plans had corrected the management deficiencies. Field office staff said that they did not have time to track the effectiveness of the plans because their workloads have been increasing due to decreasing numbers of staff. HUD headquarters officials confirmed that systematic tracking of the field offices’ success in obtaining improvement plans or executing MOAs has not been done. They emphasized that responsibility for implementing PHMAP rests with the field offices and said that limited efforts were underway to ensure field offices do more to use these tools and measure their effectiveness. However, they could not tell us whether troubled PHAs without MOAs had improved their scores and left the troubled list without such oversight, nor could they tell us whether improvement plans are instrumental in improving indicator scores. Field Offices Confirm Few When a troubled housing authority receives a new PHMAP score that is high PHMAP Scores enough to remove that designation, HUD requires that the field office confirm the score’s accuracy by verifying that the PHA’s improvements have been effective before removing the troubled designation. However, we found most field offices are not meeting this requirement. In 1995, for example, HUD’s field offices confirmed less than 30 percent of the scores that should have been confirmed. HUD officials acknowledged that the infrequency of confirmatory reviews by its field offices hampers the program’s credibility and integrity. Because it has done so few confirmatory reviews, HUD cannot say that most scores are accurate, nor can it say that most troubled PHAs that raised their scores above 60 really are no longer troubled. The HUD Inspector General (IG) recently noted that without more confirmatory reviews, the self-reporting nature of PHMAP creates a temptation for PHAs to manipulate data to raise their scores.3 In fiscal year 1995, 24 of the 49 field offices had housing authorities with PHMAP scores high enough to remove them from HUD’s troubled list, but only 11 of the 24 field offices performed all or some of the required 3 Limited Review of the Public Housing Management Assessment Program (Audit Related Memorandum No. 96-PH-101-0801), July 1996, HUD Office of Inspector General. Page 27 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited confirmatory reviews.4 The remaining 13 offices performed none of the required confirmatory reviews. Nonetheless, some of these same 13 field offices performed discretionary confirmatory reviews of other housing authorities that had not been classified as troubled. In one case, a field office had just one housing authority whose new PHMAP score was high enough to remove its troubled designation. Although the field office did not perform a confirmatory review for that authority until the next fiscal year, it did complete nine confirmatory reviews of standard- or high-performing housing authorities. HUD headquarters officials told us that although they encourage the field offices to do as many additional, discretionary confirmatory reviews as possible, they expect field offices to complete the mandatory reviews first. They also told us that limited resources kept them from monitoring the performance of field offices on these reviews. In addition to the field offices’ lack of compliance with HUD’s requirement for performing confirmatory reviews, few offices are performing discretionary confirmatory reviews. Over the life of the program, HUD has confirmed 6.7 percent of all PHMAP scores. Table 2.1 shows that since the program began in 1992, HUD has confirmed no more than 8 percent of all PHMAP scores in any given year (see table 2.1). Table 2.1: PHAs Receiving a Confirmatory Review From a HUD 1992 1993 1994 1995 Field Office, Fiscal Years 1992-95 Number of PHAs 2,372 3,071 3,071 3,077 submitting PHMAP certifications Number of 137 225 241 200 confirmatory reviews completed Percentage of 5% 7% 8% 6% PHAs receiving a confirmatory review 4 In fiscal year 1995, a total of 58 PHAs reported new PHMAP scores that would have taken them off HUD’s troubled list. HUD’s field offices confirmed 15 (26 percent) of these 58 PHMAP scores. Page 28 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited To expand on fiscal year 1995 data, nine field offices performed no confirmatory reviews, over two thirds performed five or fewer, and 4 offices performed 10 or more confirmatory reviews (see fig. 2.2).5 Figure 2.2: Number of Confirmatory Reviews Performed by HUD Field Offices, Fiscal Year 1995 8% >10 Reviews (4 Field offices) 0 Reviews (9 Field offices) • 18% • 53% • 1-5 Reviews (26 Field offices) 20% • 6-10 Reviews (10 Field offices) Source: HUD field offices’ responses to GAO’s questionnaire. Recognizing that PHMAP scores may not be as accurate as they could or should be to give the program integrity and credibility, HUD has added new requirements and begun initiatives to improve the accuracy of the scores and strengthen the program. HUD currently requires its field offices to confirm the PHMAP scores of housing authorities whose scores have risen to 60 or above, thereby removing them from the troubled list. Recently, HUD formed a team of “expert” field office staff to develop review guidelines and to perform confirmatory reviews at selected housing 5 HUD’s Oklahoma City Field Office reported having one staff member and performing no confirmatory reviews in fiscal year 1995. We did not include that office’s response in our analysis, however, because its responsibilities for overseeing public housing had been transferred to HUD’s Ft. Worth office after the bombing of the Oklahoma City Federal Building. Page 29 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited authorities whose new PHMAP scores meet HUD’s criteria for a mandatory confirmatory review. HUD officials expect this team to perform as many as 12 confirmatory reviews in 1 year, during which they will focus primarily on large, high-risk housing authorities. Field Offices Are Not Using In May 1995, HUD expanded the scope of the mandatory annual financial Independent Audits to Verify audits of PHAs to require that auditors review the records underlying a Data Provided by PHAs PHA’s self-reported PHMAP data.6 HUD expects the financial audits to verify that the PHAs’ data are accurate and complete and that the PHAs have adequate documentation to support their submissions. HUD adopted this requirement because the field offices do not have sufficient resources to confirm each PHA’s score every year. Moreover, HUD officials told us that further departmental downsizing will limit its field offices’ ability to provide meaningful technical assistance, including confirmatory reviews. As a result, HUD expects that the PHMAP review in the annual audit can help ensure the integrity of housing authorities’ PHMAP data and should be a valuable tool for aiding the field offices to identify those housing authorities most needing technical assistance. HUD does not consider the auditors’ analysis to be a confirmatory review because the auditors do not verify the information HUD maintains in its information system. Furthermore, even though the auditors certify that a housing authority has documentation to support the data it submitted to HUD, they do not verify that some of the activities reflected in that data were actually performed by that authority. For example, while the auditors verify that a PHA has data indicating it has met the requirements for the indicator on conducting annual inspections of all of its housing units and major systems (e.g., heating, plumbing, and electrical), the auditors do not verify that those inspections actually took place. Although the independent audit requirement has been in place since May 1995, few of the staff in the five field offices we visited were aware of it. Before field offices authorize payment for an annual audit, HUD headquarters officials said that they expect field offices to review the audits for quality and completeness and verify that the audits addressed all appropriate areas of the PHAs’ operations, including the PHMAP. However, field office staff said that they had not seen an audit of a housing authority 6 Annually, HUD requires and pays for a single financial and compliance audit of every public housing authority. Page 30 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited that tested the reliability of its PHMAP submission.7 HUD also expects the field offices to consider significant audit findings in deciding which PHAs need additional oversight or assistance. HUD officials acknowledged, however, that the independent auditors may need training to better understand HUD’s expectations of them, regulations, and PHMAP system as well as the operations of PHAs. Similarly, these officials noted that staff in HUD’s field offices need training and guidance in how to better use the annual independent audit. One of the key challenges HUD faces in the coming years is effectively Conclusions downsizing the Department while maintaining the needed level of oversight at public housing authorities. However, HUD is currently not maintaining a consistent, minimally acceptable level of oversight at all housing authorities because of the variance in how field offices interpret their roles to provide that oversight as well as their lack of systematic compliance with follow-up requirements. Furthermore, because field offices are not making enough use of the independent audits’ verification of PHMAP data to target their technical assistance, HUD is not using the resources it has to effectively determine which housing authorities’ scores are most likely to be inaccurate. As a result, HUD is not ensuring that the housing authorities most in need of oversight and assistance are receiving it and thereby improving their performance. Continued departmental downsizing likely will cause HUD to leverage its existing resources to achieve a minimally acceptable level of oversight. This oversight is needed for HUD to be reasonably confident that all housing authorities are using federal funds appropriately, managing and maintaining their developments properly, and reporting accurately their performance information. To make better use of the limited resources it has to devote to the Recommendation oversight of public housing, we recommend that HUD provide guidance to its field offices that clearly (1) articulates their minimally acceptable roles regarding oversight and assistance to housing authorities and (2) emphasizes the importance of using the results of the independent audits to better target HUD’s limited technical assistance resources. HUD agreed with our findings regarding oversight of public housing Agency Comments authorities and stated that it has begun taking steps to address this 7 HUD’s IG evaluated some housing authorities’ annual independent audits that included a PHMAP review. The IG found that the auditors had problems confirming the reliability of the PHMAP submissions because the PHAs lacked documentation to support the self-reported indicator grades. Page 31 GAO/RCED-97-27 Public Housing Chapter 2 Although Field Offices Use PHMAP to Identify Troubled PHAs, Compliance With Statutory and Agency Follow-Up Requirements Has Been Limited recommendation. These steps include a wide variety of training and other activities to (1) explain the revisions HUD is making to PHMAP; (2) reemphasize the need for and importance of statutory and agency follow-up requirements, such as memorandums of agreement, improvement plans, and confirmatory reviews; and (3) update HUD’s guidance to its field offices regarding their PHMAP and other oversight responsibilities. Page 32 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use According to a HUD database of PHMAP scores, average PHMAP scores have increased over the life of the program from an average of 83 in 1992 to 86 in 1995 (the last year of complete data). The number of high-performing housing authorities increased, with more than half of all authorities designated high performers in 1995, and the number of troubled authorities decreased. However, the smallest housing authorities—those with fewer than 100 units—now make up a greater proportion of those designated troubled than when the program began. During our analysis of this database, we found omissions of key data, such as the number of units under a PHA’s management and its performance designation. We also found inconsistencies between PHMAP scores and the assigned performance designations. Notwithstanding these weaknesses, the database represents the most complete data available on PHA performance over time. Nationwide, average PHMAP scores generally increased over the 4 years of Most PHMAP Scores the program for which we analyzed data. By 1995, over half of all public Are Increasing and housing authorities were high performers. Subsequent analysis showed Fewer Housing little regional variation in how well they scored on PHMAP. While the overall increases in PHMAP scores held true for all sizes of PHAs, the largest ones Authorities Are had scores consistently lower than the national average. With average Troubled scores increasing, the number of PHAs with scores low enough for HUD to designate them as troubled also decreased. The number of troubled authorities reached 83 in 1995, with half of that total consisting of the smallest housing authorities (those managing fewer than 100 units). Average PHMAP Scores The average PHMAP score for all housing authorities rose from about 83 in Increased 1992 to 86 in 1995. This increase held true for PHAs of all sizes, although large PHAs—those with more than 1,250 units—consistently scored lower than the national average (see table 3.1). In fiscal year 1995, 151 large PHAs accounted for approximately 5 percent of all PHAs reporting PHMAP scores, but they operated nearly 60 percent of all public housing units. Consequently, while more PHAs had higher scores, more units were under the control of PHAs with somewhat lower scores. Page 33 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use Table 3.1: Average PHMAP Score by PHA Size Category, Fiscal Years 1992-95 PHA size category 1992 1993 1994 1995 (number of Number of PHMAP Number of PHMAP Number of PHMAP Number of PHMAP units)a PHAs score PHAs score PHAs score PHAs score No size datab 4 54 4 43 3 41 7 31 1-99 1,453 83 1,471 84 1,481 87 1,488 87 100-499 1,241 83 1,262 85 1,266 88 1,269 87 500-1,249 243 83 242 84 243 86 242 86 1,250 or more 149 78 151 80 151 81 151 83 All sizes 3,090 83 3,130 84 3,144 87 3,157 86 a All size categories were calculated on the number of managed units for fiscal year 1995. HUD does not maintain information on the number of units managed in previous years. To the extent that specific PHAs reduced or increased the number of units under their jurisdiction in previous years, the current size categories may not represent previous years. b The database did not contain size information for these PHAs. Source: GAO’s analysis of data from HUD’s System for Management Information Retrieval-Public Housing (SMIRPH) database. Appendix I provides average PHMAP scores for PHAs for all of HUD’s field offices for fiscal years 1992 through 1995. The Majority of PHAs Were By fiscal year 1995, more than half—about 57 percent—of all public High Performers housing authorities were designated as high performers. As shown in table 3.2, the number of high performing authorities grew each year, rising from 1,033 (33 percent) in 1992 to 1,791 (57 percent) in 1995. Also, by 1995, nearly 50 percent of all public housing units were under the management of high-performing authorities. Page 34 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use Table 3.2: Number of PHAs by PHMAP Performance Category, Fiscal Years Performance category Units (in Number of PHAs 1992-95 (score) thousands)a 1992 1993 1994 1995 Troubled (<60) 214 130 118 101 83 Standard 443 1,927 1,719 1,358 1,216 (60-<90) High (90-100) 656 1,033 1,293 1,685 1,791 All 1,313 3,090 3,130 3,144 3,090 performance categories a HUD’s SMIRPH database contains the number of units for only fiscal year 1995. Source: GAO’s analysis of data from HUD’s SMIRPH database. Little Variation Among Our analysis showed little regional variation in PHMAP scores. The regional Regions differences we found were slightly greater than those associated with the size of housing authorities, but no region was significantly below the national average. Likewise, there was little variation among the regions in the percentage of troubled PHAs under their jurisdiction. For example, in fiscal year 1995, 5 percent of all PHAs nationwide were troubled, but within the 10 regions we analyzed, the percentage of troubled housing authorities ranged from 2 to 9 percent. Appendixes I-IV provide detailed information on average PHMAP scores as well as the number of troubled, standard- and high-performing PHAs, respectively, for each HUD field office. PHAs Consistently Failed Despite some improvement in overall scores, some indicators were more Some Indicators problematic for PHAs than others. As shown in table 3.3, with the exception of 1 year, PHAs consistently had the most difficulty with the energy consumption indicator—which had the highest failure rate for 1992, 1994, and 1995.1 Similarly, the indicators for unit turnaround, tenants accounts receivable, and operating expenses proved troublesome, with 10 percent or more of all PHAs failing them in 1995. 1 This indicator measures the annual increase in the housing authority’s energy consumption. Housing authorities with no increase from year to year receive an A for the indicator; those whose consumption increases receive lower grades. In commenting on this report, HUD officials noted that in some cases failing this indicator was related less to PHA performance than to such conditions as regional weather variations or an inappropriate baseline to measure this indicator. Page 35 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use Table 3.3: Failed Indicators by Percentage of PHAs, Fiscal Years Percentage of PHAs that failed 1992-95 PHMAP indicator 1992 1993 1994 1995 1. Vacancy 5.5 4.2 3.6 2.9 number and percentage 2. Modernization 1.9 1.4 1.4 1.5 3. Rents 4.1 3.1 3.0 2.2 uncollected 4. Energy 18.0 12.4 14.3 15.3 consumption 5. Unit turnaround 16.4 13.6 12.9 11.2 6. Outstanding 3.6 2.1 2.4 1.7 workorders 7. Annual 2.2 2.3 2.1 1.9 inspection and condition of units and systems 8. Tenants 16.9 15.2 14.2 12.8 accounts receivable 9. Operating 6.3 5.8 4.8 7.4 reserves 10. Routine 11.2 10.8 10.1 10.0 operating expenses 11. Resident 13.4 26.7 11.0 5.9 initiatives 12. Development 8.5 6.2 6.2 3.0 Source: GAO’s analysis of data from HUD’s SMIRPH database. A HUD official explained that the high failure rate in 1993 for the indicator measuring resident initiatives occurred because the PHAs were not paying attention to this indicator. In 1992, all PHAs received an automatic “C” for this indicator because HUD had not provided enough information on the requirements for grades “A” through “F” until after the assessment period started. This official said that many PHAs assumed they would receive an automatic “C” the next year as well, even though HUD had stated in 1992 that the automatic grade was a one-time occurrence. This official added that most field offices followed up by providing technical assistance to the PHAs with failing grades and were able to resolve the problems in the following year. This appears to be supported by the decline of the failure rate over the following 2 years to less than 6 percent in 1995. Page 36 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use Smaller PHAs Were More While the total number of troubled housing authorities declined—130 were Likely to Be Troubled troubled in 1992 compared to 83 in 1995—more of those PHAs were concentrated among the smallest housing authorities than when the program began. The percentage of troubled PHAs that were small—managing fewer than 100 units each—grew from 32 percent of all troubled authorities in 1992 to 49 percent in 1995 (see fig. 3.1). Figure 3.1: Number of Troubled PHAs by Size, Fiscal Years 1992-95 70 Number of troubled PHAs 65 60 55 50 45 40 35 30 25 20 15 10 5 0 1992 1993 1994 1995 1-99 units 100-499 units 500-1,249 units 1,250 or more units Note: For each fiscal year, the figures exclude six or fewer PHAs for which there was no information on size in HUD’s database. Source: HUD’s SMIRPH database. We found missing, inaccurate, and inconsistent data in HUD’s SMIRPH HUD Recognizes database, the primary database for storing PHMAP scores. A HUD official Database Flaws and attributed these problems to data input problems at the field offices. Plans Corrections Although HUD headquarters makes regular, periodic use of this database, it Page 37 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use must also manually verify much of the information before providing it to HUD’s Secretary, Members of Congress, and others. HUD’s General Deputy Assistant Secretary for Public and Indian Housing acknowledged that the SMIRPH database, as currently implemented, does not produce a complete, accurate list of troubled PHAs and that HUD is in the process of making it more reliable and useful. We found that the number of troubled authorities (150) for fiscal year 1995 that we derived from the database was inaccurate when we compared it to the number reported (83) as of December 20, 1995, by HUD’s Management Assessment Coordinator. We also found performance designations that were inconsistent with PHMAP scores. In 1995, for the 150 PHAs we found to be troubled, HUD had designated 42 as high performers, 7 as standard, and 51 had no designation. Among high-performing PHAs in 1995, of the 1,791 PHAs that we found that had PHMAP scores of 90 or higher, HUD had designated one as troubled, 43 as standard, and 325 had no performance designation. We also found some omissions in the database. Data, such as the number of units and performance designations, had not been entered for all PHAs. For example, we found that the database did not have size information on 18 PHAs from fiscal years 1992 through 1995. We also found that no designations had been entered for 132 PHAs with scores less than 60 and 1,037 PHAs with scores 90 or higher. HUD’s Management Assessment Coordinator stated that these problems with missing, inaccurate, and inconsistent data occurred because field offices either (1) did not enter the information at all or (2) entered it incorrectly. These instances of inconsistent or missing data suggest that basic system safeguards do not exist to prevent field offices from making these data entry errors or omitting essential PHMAP data. While HUD officials who oversee PHMAP and the Department’s field offices acknowledged problems with the database, they added that the program’s redesign includes changes that will address the problems with data accuracy and reliability. HUD officials told us they plan to change procedures for entering information on PHAs into the database to allow field offices to update PHA data on a real-time basis and to make immediate corrections when they find errors or omissions. These procedural changes will also enable HUD headquarters staff to access field office data directly and allow ongoing reviews of the information for accuracy and completeness. HUD officials also believe that the changes will increase control over the information from the field offices and help ensure that the information in the SMIRPH database is accurate. Page 38 GAO/RCED-97-27 Public Housing Chapter 3 Although PHMAP Scores Have Risen, HUD Recognizes That Flaws in the Program’s Database Limit Its Use HUD expressed concern that our draft report used data from the SMIRPH Agency Comments database that HUD had not verified for accuracy. HUD noted that it is making changes to the database that will improve headquarters’ ability to find and correct data errors that have been entered by staff at its field offices. To address HUD’s concern that we used inaccurate, unverified data from its database to analyze PHMAP data on housing authorities’ scores by size and region, we recalculated the number of troubled housing authorities by size category for 1995 using data HUD verified with its field offices; we also modified this report to reflect a more accurate and lower number of troubled housing authorities in 1995. Recalculating the number of troubled authorities by size did not change our conclusion that a greater proportion of the authorities that HUD verified as being troubled are those with fewer than 100 units. In fact, while HUD’s database indicates that 44 percent of troubled authorities in 1995 were small, HUD’s verified list of troubled authorities indicates 49 percent were small. Furthermore, although HUD officials told us that a manually-verified list of troubled authorities for 1992 was not available, they agreed with our conclusion that the smallest housing authorities make up a greater proportion of troubled housing authorities in 1995 than in 1992. Because our draft report presented no analysis of data on a regional basis (only data as drawn from HUD’s database) and because we draw no conclusions in that regard in this report, we have retained appendixes I-IV, which show average PHMAP scores and the number of troubled, standard, and high-performers in HUD’s regions. Where HUD provided us with manually verified data—particularly in appendix II showing troubled authorities—we have modified the appendixes to reflect the more accurate data. Page 39 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness Our review and those of others indicate that PHMAP scores are often inaccurate, imprecise, and must be changed when HUD verifies the data that public housing authorities have submitted to support their scores. Furthermore, professional property managers and others in the public housing industry question whether PHMAP can capture all aspects of management operations. Although HUD has taken some steps to help ensure that future scores are more accurate than they have been over the program’s first 4 years, these steps will be resource-intensive and do not address all of the program’s limitations. In the past, both HUD and the Congress have proposed additional uses for PHMAP, such as deregulating and awarding bonuses to PHAs with high PHMAP scores. However, until greater confidence exists that individual scores are accurate and HUD brings greater validity to PHMAP as a comprehensive measure of management operations, such additional uses for the program may not be appropriate. After performing on-site reviews of selected PHAs to confirm the accuracy Accuracy of Scores of their PHMAP scores, HUD’s field offices changed half of the scores. In and Validity as a commenting on this report, HUD indicated that most confirmatory reviews Management involved high-risk PHAs, whose PHMAP data have been most susceptible to being found inaccurate. In similar reviews, HUD’s independent assessment Assessment Tool contractors as well as HUD’s IG found that many scores or grades for Limit Uses for PHMAP specific indicators were inaccurate. To better identify PHAs that need oversight and technical assistance, HUD staff often supplement their decision-making with other measures of management problems to get a more complete picture of an authority’s performance. Professional property managers and industry representatives agreed that more information is needed than PHMAP provides to give a complete picture of how well a PHA’s management is performing. After Confirmatory After performing confirmatory reviews of 200 PHAs in fiscal year 1995, Reviews, PHMAP Scores HUD’s 49 field offices changed 98 PHMAP scores (see table 4.1). Change Significantly Page 40 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness Table 4.1: Changes in PHMAP Scores After HUD’s Field Offices Performed Number of housing Confirmatory Reviews in Fiscal Year Change in authorities Average change 1995 PHMAP score (percentage of total) in points No change 96 (49) 0 Scores lowered 57 (29) –14 Scores raised 41 (21) +8 Note: The field offices had not reported the final PHMAP scores for 6 out of the 200 confirmatory reviews because the reviews’ results were being finalized at the time the offices responded to our questionnaire. As a result, this table reflects results from 194 of the 200 confirmatory reviews performed in fiscal year 1995. In several cases, the changes HUD made to PHMAP scores also meant HUD would have to change the performance designation of those PHAs. For example, HUD • lowered the scores of 14 PHAs enough to designate them as troubled, • raised the scores of 4 troubled PHAs to 60 points or higher, and • raised the scores of 10 standard-performing PHAs to 90 or higher. Both of HUD’s independent assessment contractors as well as HUD’s IG have reviewed PHMAP data to confirm the accuracy of PHAs’ scores. For example, in 1993, the IG confirmed the scores of 12 housing authorities. As a result of this review, the IG concluded that the PHMAP scores for 9 of the 12 PHAs should be lowered because 3 of them fell below 60, a score which should have warranted the troubled designation. In a second report on PHMAP, the IG reported that six of HUD’s field offices reduced over half of the scores they reviewed. Similarly, one of HUD’s independent assessment contractors reported that for the 30 assessments it has performed at troubled housing authorities, it found 21 indicator grades and/or PHMAP scores that were inaccurate. Over 50 percent of the contractor’s assessments resulted in lowering the indicator grades to an “F.” The contractor most often lowered the indicators used to measure outstanding workorders and annual inspections of housing conditions and systems. Several reasons explain why HUD and others changed so many PHMAP scores after performing a confirmatory review. Some field office staff said these scores changed because the PHAs did not understand all the requirements of PHMAP and therefore misreported their data. They also told us that PHMAP is particularly difficult for smaller housing authorities whose limited staff can find HUD’s paperwork requirements overwhelming. Page 41 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness HUD staff do not believe many PHAs intentionally try to deceive the Department by reporting false PHMAP information. Instead, they, as well as the contractor staff, said that the PHAs often have insufficient documentation to support the data they must submit to the field offices or do not understand how HUD wants them to report the information. For example, while a PHA may report the average number of days their housing units have been vacant, the PHA may not have the tenant files to document when the previous tenants moved out and when the new tenants’ leases took effect. Without supporting documentation or evidence of a system to track unit turnaround, HUD assigns an “F” to this indicator. Similarly, a PHA may be providing support programs for its residents, but fail to understand that its board of commissioners must approve those programs to receive a passing grade on PHMAP’s indicator for resident initiatives. Typically, when HUD’s field office staff find examples, such as these, during a confirmatory review, they use the correct data to recalculate the housing authority’s grade for each of the affected indicators. HUD and Industry HUD’s field office staff did not use PHMAP alone to assess the management Professionals Supplement performance of its public housing authorities. Although they agreed that PHMAP accurately identifies troubled authorities, several staff said that they PHMAP With Additional consider other factors besides PHMAP indicators to supplement their Factors to Evaluate decision-making for the other authorities they oversee. They said that Management Performance some PHAs with scores over 90 have management problems that the program’s indicators do not measure. Other factors used by some HUD staff to identify the potential for management problems at standard- and high-performing authorities include • the failure of a PHA to implement consistent and effective operating policies and procedures, • the frequency of changes in the executive leadership and the continued interference into a PHA’s daily operations by its board of commissioners, • the number and the type of telephone calls received from a PHA’s residents and staff, and • any adverse news stories about a PHA. Staff at the five field offices we visited said that they believed some housing authorities with high PHMAP scores were not operating their housing programs efficiently or effectively. These field offices differed, however, in how they treated those PHAs. Staff at two field offices told us that although they use the scores to determine which PHAs need on-site Page 42 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness reviews, they would not let a high score prevent them from visiting an authority they believed had serious management problems. The HUD IG also questioned whether or not PHMAP scores accurately measure the management performance of public housing authorities. The IG’s reviews of high- and standard-performing PHAs found instances of fraud and program abuse. For example, the IG reported that the executive director of a high-performing PHA had charged over $62,000 in ineligible expenses, including excessive compensatory time, unsupported travel costs, and health and insurance benefits for his divorced spouse. Another PHA executive director falsified PHMAP data to obtain a high-performing designation. After reviewing the operations of a standard-performing PHA, the IG also cited numerous program abuses and mismanagement. The IG concluded that although PHMAP could be a useful tool to assess PHAs, the program was too unreliable for HUD to make oversight decisions. Other public housing professionals—property managers and those representing industry associations—agreed that more information is needed than PHMAP provides to give a complete picture of how well a PHA is managed. For example, they noted that PHMAP does not automatically include an on-site observation and inspection of a PHA’s housing developments. One association noted that while a PHA could improve its PHMAP score by simply writing off more past due rents from former tenants as uncollectible to improve its grade on the indicator for rents uncollected, its PHMAP score would not measure how diligent an effort it had undertaken to collect the rent. Another industry association official knew of several examples of PHAs that were making good property management decisions, such as choosing to perform deferred maintenance when a unit became vacant rather than rent it immediately, that ironically led to lower PHMAP scores. Citing a similar situation, HUD has agreed that occasionally the best decision for a PHA is to take an action that yields a lower PHMAP score, and that the score should not be the sole driving force influencing a PHA’s decisions. While HUD’s primary use of PHMAP has been to identify troubled housing The Congress and authorities and target technical assistance to them, the Congress and HUD HUD Have Proposed have proposed to use this program for other purposes. In 1994, the Senate to Use PHMAP as a Committee on Banking, Housing, and Urban Affairs proposed some deregulation and additional flexibility for those authorities that had Basis for Deregulation achieved PHMAP scores of 90 or above. In addition, in its fiscal year 1997 and Funding Bonuses budget request, HUD proposed to give high-performing PHAs bonuses based Page 43 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness in part on their PHMAP scores. Because PHMAP scores do not always measure the true management performance of the PHAs, the benefits of these proposals need to be weighed against the possibility of granting undeserved flexibility and awards. To encourage individual PHAs to be more innovative, the Banking Committee proposed limited deregulation and additional flexibility for high-performing PHAs in two ways. First, it proposed permitting a PHA that generates income over a certain level to exclude that income from calculations of its need for a subsidy from HUD to operate and manage its properties.1 At that time, each dollar of extra income that a PHA generated reduced its subsidy by a dollar, thereby creating a disincentive to generate additional income from sources other than rent. Second, the Committee proposed to waive all but a few key regulations—such as nondiscrimination, equal opportunity, and tenant income eligibility—so high-performing PHAs could have more flexibility to bring innovative solutions to local problems and achieve more efficient operations. In its fiscal year 1997 budget request, HUD proposed to award $500 million to high-performing PHAs as bonuses based, in part, on their PHMAP scores. As we reported in our testimony in June 1996 and as we found in the course of our work on this report, HUD does not confirm the scores of high performers and generally accepts them.2 In our June 1996 testimony, we recommended that the Congress consider not appropriating the bonus funding until HUD develops adequate performance measures and supporting information systems. The HUD appropriations bill which the Congress approved and the President signed did not contain funding for performance bonuses. The three associations representing the public housing industry and the professional property managers that we interviewed all opposed or had strong reservations about using PHMAP scores for purposes other than identifying troubled housing authorities and targeting technical assistance to them. They also believed that other uses would be inappropriate because of the limited number of confirmatory reviews the field offices perform and the proportion of PHMAP scores that have been changed after a review. Two of the associations did not believe that PHMAP scores 1 Housing authorities receive operating subsidies from HUD each year to make up the difference between the rent they are allowed to charge their tenants and the expected costs of operating their developments. 2 Housing and Urban Development: Comments on HUD’s FY 1997 Budget Request (GAO/T-RCED-96-205, June 17, 1996). Page 44 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness adequately measured the management performance of housing authorities because they thought some PHAs that received high scores did not provide their residents with decent, safe housing. The professional property management firm that independently verified some scores also agreed that the usefulness of these scores is limited. Because this firm has recommended lowering many scores after an independent assessment, the firm lacks confidence in the scores’ accuracy and does not believe that the program provides enough information about the management performance of PHAs for HUD to make effective funding decisions. In recent years, both the Congress and HUD have proposed additional uses Conclusions for PHMAP, such as bonuses to reward those housing authorities with the highest scores. While PHMAP has provided a quantifiable means to assess the management performance of housing authorities, the scores are not sufficiently accurate for detailed comparisons of performance. Although HUD is currently working to enhance the accuracy of these scores, they do not yet provide a comprehensive, generally accepted way to assess the performance of PHAs. To be useful for other purposes, not only would these scores have to be more accurate, but the program would have to be expanded to provide a more comprehensive measure of public housing authorities’ management operations. Because HUD does not frequently confirm most scores—confirmatory reviews have focused on troubled PHAs—HUD does not know how many authorities are not receiving the proper designation. When HUD does confirm scores, it changes half of them—and more than half of these changes result in HUD’s lowering the score. We found that when HUD lowers a PHMAP score, it does so by an average of 14 points. If this average change held true for housing authorities in general, then HUD may not be properly designating as troubled those authorities currently scoring between 60 and the low 70s whose scores should be lower. As a result, those authorities are not receiving the oversight and technical assistance HUD should be providing to improve their performance. We recommend that until it establishes a cost-effective means to ensure Recommendations consistently accurate scores, HUD should • not consider additional uses for PHMAP, including using its scores as criteria for funding bonuses, until it determines that PHMAP meets an Page 45 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness acceptable level of accuracy and more comprehensively measures property management performance and • require its field offices to confirm the PHMAP scores of housing authorities with scores low enough that they are at risk of being designated troubled. HUD agreed with our findings and recommendations. When we met with Agency Comments HUD officials, including the General Deputy Assistant Secretary for Public and Indian Housing, to discuss a draft of this report, they told us that the Department is no longer considering additional uses for PHMAP, such as using scores as criteria for funding bonuses. Even in the absence of using PHMAP for such purposes, we believe that it is important that HUD works to ensure scores are more consistently accurate and have, therefore, retained this recommendation. HUD has begun taking steps to address our recommendation that it confirm PHMAP scores of those housing authorities that are at risk of being designated troubled but expressed concern that it may not have sufficient resources to fully implement this recommendation. HUD expressed three concerns relating to the information and conclusions presented in this chapter of our report. HUD believed that this chapter (1) assumes that PHMAP was intended to be an all-inclusive assessment system for property management, (2) does not place PHMAP in a historical perspective, and (3) reaches incorrect conclusions regarding the overall reliability of PHMAP scores. We do not believe that we characterize PHMAP’s purpose as being an all-inclusive measure of property management. Our discussion of the program does not state that this is the purpose of PHMAP. Rather, the report discusses how the program’s limitations—including its intentional design not to be a complete performance measure—affect its suitability for additional purposes, such as those proposed in recent years by HUD and the Congress. HUD agreed that there is a perception that PHMAP is an all-encompassing system to assess the performance of PHAs and stated it is taking steps to address this misperception. Seeking to clarify the program’s purpose, HUD added language to its recently revised interim PHMAP rule (published in the December 30, 1996, Federal Register), that the program’s indicators reflect performance in only specific areas. HUD correctly states that this report does not provide a historical perspective of PHMAP by discussing previous HUD systems for assessing and identifying troubled housing authorities. We believe that such information Page 46 GAO/RCED-97-27 Public Housing Chapter 4 The Questionable Accuracy of PHMAP’s Scores and the Program’s Validity Limit Its Usefulness would not contribute substantially to our report’s three objectives to evaluate HUD’s use of the current program, provide trends in PHMAP scores from fiscal years 1992 through 1995, and discuss limitations in the program’s design and implementation that affect its usefulness for purposes other than identifying troubled housing authorities and targeting assistance to them. Therefore, we have not added the historical information HUD suggested to the report. Finally, HUD is concerned that we have incorrectly reached conclusions about the reliability of all PHMAP scores based on the results of confirmatory reviews of high-risk authorities. HUD noted that the accuracy of the scores of these PHAs does not necessarily represent the accuracy of all PHMAP scores because the data provided by these PHAs are most susceptible to being inaccurate. Our report did not reach a conclusion about the reliability of all housing authorities’ scores because of the changes that resulted from confirmatory reviews. This report discusses the reliability of PHMAP scores for housing authorities whose scores are low enough that they may be at risk of being designated troubled. We have added language to the report to clarify this point. Page 47 GAO/RCED-97-27 Public Housing Appendix I Average PHMAP Score by Geographic Region, Fiscal Years 1992-95 Average PHMAP score Region 1992 1993 1994 1995 Great Plains 82.2 83.1 86.9 82.9 Des Moines 84.4 82.3 88.9 92.1 Kansas City 81.7 81.5 86.9 88.3 Omaha 84.0 88.4 88.3 90.4 St. Louis 78.9 78.7 83.6 53.1 Mid-Atlantic 80.1 81.3 83.1 85.7 Baltimore 81.8 84.3 82.8 84.8 Charleston 76.0 79.7 82.2 86.1 Philadelphia 80.9 81.4 85.9 87.4 Pittsburgh 80.1 80.5 81.3 83.2 Richmond 84.5 84.7 84.2 88.7 District of Columbia 72.5 71.7 73.9 76.3 Midwest 83.6 86.1 88.8 88.9 Chicago 75.4 77.2 82.9 83.9 Cincinnati 77.6 78.8 84.8 87.1 Cleveland 77.9 80.8 83.9 85.5 Columbus 80.7 85.1 86.3 90.7 Detroit 82.4 84.0 87.6 87.9 Grand Rapids 85.5 86.3 91.2 90.2 Indianapolis 82.6 87.3 89.6 89.5 Milwaukee 90.2 92.5 93.1 91.5 Minneapolis 86.5 90.1 90.6 90.7 Northwest/Alaska 89.9 92.9 92.9 92.6 Anchorage 77.8 86.1 94.9 99.2 Portland 90.6 93.9 94.1 95.4 Seattle 89.6 92.0 91.4 89.4 New York/New Jersey 81.8 84.9 88.2 89.7 Buffalo 84.6 86.9 90.0 91.2 New York 79.6 85.0 83.4 90.5 Newark 80.9 83.7 88.9 88.4 New England 83.4 85.3 89.3 89.2 Boston 83.6 82.9 88.0 90.4 Hartford 74.4 76.9 83.7 78.3 Manchester 91.9 94.1 95.1 94.0 Providence 79.9 87.3 89.9 92.4 (continued) Page 48 GAO/RCED-97-27 Public Housing Appendix I Average PHMAP Score by Geographic Region, Fiscal Years 1992-95 Average PHMAP score Region 1992 1993 1994 1995 Pacific/Hawaii 85.4 85.4 86.9 87.2 Honolulu 81.5 68.9 73.8 70.6 Los Angeles 89.1 90.0 90.4 91.0 Phoenix 85.4 83.3 88.2 83.8 Sacramento 86.1 81.9 75.5 85.5 San Francisco 82.9 84.9 87.4 87.7 Rocky Mountains 87.8 87.9 86.8 91.5 Denver 87.8 87.9 86.8 91.5 Southeast/Caribbean 82.9 85.4 87.9 84.6 Atlanta 81.7 83.7 85.5 87.9 Birmingham 85.8 87.3 89.7 90.8 Columbia 84.2 86.5 91.7 94.0 Greensboro 82.5 85.6 88.8 89.9 Jackson 82.6 86.6 86.4 87.4 Jacksonville 79.6 83.3 86.0 83.3 Louisville 86.9 88.4 91.0 60.6 Knoxville 85.4 87.7 91.9 90.1 Nashville 77.1 82.4 85.4 85.3 Caribbean 44.5 36.7 48.2 32.0 Southwest 80.3 80.8 84.0 85.7 Albuquerque 74.8 70.4 72.7 83.7 Beaumont 79.1 78.4 83.7 84.1 Ft. Worth 80.5 79.5 82.9 85.9 Houston 80.1 78.3 80.8 82.7 Little Rock 86.3 90.5 91.6 91.5 New Orleans 76.5 78.6 81.3 79.5 Oklahoma City 80.0 81.1 84.0 86.1 San Antonio 79.8 79.9 85.5 87.0 Source: GAO’s analysis of data from the Department of Housing and Urban Development’s (HUD) System for Management Information Retrieval-Public Housing (SMIRPH) database. Page 49 GAO/RCED-97-27 Public Housing Appendix II Number of Troubled PHAs by Geographic Region, Fiscal Years 1992-95 Number of troubled PHAs Region 1992 1993 1994 1995 Great Plains 10 13 12 7 Des Moines 1 3 1 1 Kansas City 1 3 5 2 Omaha 0 1 1 1 St. Louis 8 6 5 3 Mid-Atlantic 7 7 11 6 Baltimore 0 0 1 0 Charleston 1 0 0 0 Philadelphia 2 2 4 3 Pittsburgh 2 3 4 2 Richmond 1 1 1 0 District of Columbia 1 1 1 1 Midwest 24 16 12 12 Chicago 12 10 7 5 Cincinnati 1 0 0 0 Cleveland 2 0 1 1 Columbus 0 0 0 0 Detroit 4 5 3 2 Grand Rapids 1 0 0 0 Indianapolis 2 0 0 2 Milwaukee 0 0 0 1 Minneapolis 2 1 1 1 Northwest/Alaska 0 0 0 1 Anchorage 0 0 0 0 Portland 0 0 0 0 Seattle 0 0 0 1 New York/New Jersey 10 5 6 3 Buffalo 0 1 1 0 New York 4 2 3 1 Newark 6 2 2 2 New England 12 5 2 5 Boston 2 0 0 0 Hartford 7 5 2 4 Manchester 0 0 0 1 Providence 3 0 0 0 (continued) Page 50 GAO/RCED-97-27 Public Housing Appendix II Number of Troubled PHAs by Geographic Region, Fiscal Years 1992-95 Number of troubled PHAs Region 1992 1993 1994 1995 Pacific/Hawaii 2 2 2 4 Honolulu 0 0 0 0 Los Angeles 0 0 0 0 Phoenix 1 1 1 2 Sacramento 0 0 0 0 San Francisco 1 1 1 2 Rocky Mountains 2 4 8 0 Denver 2 4 8 0 Southeast/Caribbean 30 30 19 58 Atlanta 9 12 7 5 Birmingham 1 1 0 0 Columbia 2 2 0 0 Greensboro 3 3 0 1 Jackson 2 0 1 1 Jacksonville 2 4 3 3 Louisville 2 1 2 1 Knoxville 0 0 0 1 Nashville 7 4 4 2 Caribbean 2 1 1 1 Southwest 33 36 29 28 Albuquerque 4 9 8 3 Beaumont 3 4 0 1 Ft. Worth 7 7 8 6 Houston 1 1 0 1 Little Rock 3 1 0 1 New Orleans 12 7 9 12 Oklahoma City 1 1 0 0 San Antonio 2 6 4 4 All regions 130 118 101 150 Source: GAO’s analysis of data from HUD’s SMIRPH database. Page 51 GAO/RCED-97-27 Public Housing Appendix III Number of Standard-Performing PHAs by Geographic Region, Fiscal Years 1992-95 Number of standard-performing PHAs Region 1992 1993 1994 1995 Great Plains 235 213 157 141 Des Moines 30 31 18 12 Kansas City 99 95 62 60 Omaha 67 47 40 34 St. Louis 39 40 37 35 Mid-Atlantic 121 115 89 78 Baltimore 15 13 12 13 Charleston 25 25 22 20 Philadelphia 36 33 17 11 Pittsburgh 26 27 20 18 Richmond 13 12 13 11 District of Columbia 6 5 5 5 Midwest 304 246 215 205 Chicago 73 67 61 61 Cincinnati 6 6 6 4 Cleveland 12 13 9 9 Columbus 22 12 13 9 Detroit 24 19 19 17 Grand Rapids 44 40 23 28 Indianapolis 27 20 17 13 Milwaukee 36 30 24 24 Minneapolis 60 39 43 40 Northwest/Alaska 24 13 10 10 Anchorage 1 1 0 0 Portland 11 5 4 3 Seattle 12 7 6 7 New York/New Jersey 107 93 67 54 Buffalo 31 28 19 15 New York 18 15 14 6 Newark 58 50 34 33 New England 88 84 61 51 Boston 40 49 31 22 Hartford 21 17 15 18 Manchester 10 5 4 4 Providence 17 13 11 7 (continued) Page 52 GAO/RCED-97-27 Public Housing Appendix III Number of Standard-Performing PHAs by Geographic Region, Fiscal Years 1992-95 Number of standard-performing PHAs Region 1992 1993 1994 1995 Pacific/Hawaii 42 46 31 27 Honolulu 2 2 2 2 Los Angeles 8 9 9 8 Phoenix 8 8 4 3 Sacramento 4 7 6 3 San Francisco 20 20 10 11 Rocky Mountains 49 44 31 35 Denver 49 44 31 35 Southeast/Caribbean 459 421 334 310 Atlanta 110 110 103 94 Birmingham 80 73 60 49 Columbia 19 19 9 7 Greensboro 63 58 43 31 Jackson 30 27 25 25 Jacksonville 49 37 36 47 Louisville 52 46 23 20 Knoxville 22 18 9 9 Nashville 34 32 25 27 Caribbean 0 1 1 1 Southwest 497 445 363 331 Albuquerque 26 18 17 18 Beaumont 56 55 48 42 Ft. Worth 130 128 97 86 Houston 16 16 16 11 Little Rock 56 39 28 31 New Orleans 69 62 54 53 Oklahoma City 78 74 65 51 San Antonio 66 53 38 39 All regions 1,927 1,719 1,358 1,242 Source: GAO’s analysis of data from HUD’s SMIRPH database. Page 53 GAO/RCED-97-27 Public Housing Appendix IV Number of High-Performing PHAs by Geographic Region, Fiscal Years 1992-95 Number of high-performing PHAs Region 1992 1993 1994 1995 Great Plains 102 122 179 200 Des Moines 18 15 30 36 Kansas City 36 39 70 75 Omaha 31 50 57 63 St. Louis 17 18 22 26 Mid-Atlantic 42 48 74 90 Baltimore 3 5 6 6 Charleston 6 7 10 12 Philadelphia 12 15 30 37 Pittsburgh 8 6 13 17 Richmond 12 14 13 16 District of Columbia 1 1 2 2 Midwest 200 266 303 316 Chicago 13 21 30 34 Cincinnati 2 3 3 5 Cleveland 5 6 9 9 Columbus 2 12 11 15‘ Detroit 18 23 25 28 Grand Rapids 31‘ 36 53 48 Indianapolis 12 21 24 26 Milwaukee 59 65 71 71 Minneapolis 58 79 77 80 Northwest/Alaska 34 45 48 47 Anchorage 0 0 1 1 Portland 18 24 25 26 Seattle 16 21 22 20 New York/New Jersey 46 65 90 106 Buffalo 20 22 31 36 New York 10 15 15 25 Newark 16 28 44 45 New England 67 78 102 112 Boston 24 17 34 44 Hartford 5 11 15 12 Manchester 33 38 39 38 Providence 5 12 14 18 (continued) Page 54 GAO/RCED-97-27 Public Housing Appendix IV Number of High-Performing PHAs by Geographic Region, Fiscal Years 1992-95 Number of high-performing PHAs Region 1992 1993 1994 1995 Pacific/Hawaii 35 31 46 49 Honolulu 0 0 0 0 Los Angeles 14 13 13 14 Phoenix 6 6 10 11 Sacramento 3 0 1 4 San Francisco 12 12 22 20 Rocky Mountains 64 67 77 84 Denver 64 67 77 84 Southeast/Caribbean 281 355 459 487 Atlanta 51 79 91 101 Birmingham 63 70 85 96 Columbia 19 19 31 34 Greensboro 31 36 54 65 Jackson 19 24 27 27 Jacksonville 22 35 41 30 Louisville 52 59 81 85 Knoxville 10 14 23 23 Nashville 14 19 26 26 Caribbean 0 0 0 0 Southwest 162 216 307 342 Albuquerque 6 9 13 16 Beaumont 11 11 22 27 Ft. Worth 36 38 68 83 Houston 3 3 4 8 Little Rock 51 70 82 78 New Orleans 16 29 34 34 Oklahoma City 23 27 37 51 San Antonio 16 29 47 45 All regions 1,033 1,293 1,685 1,833 Source: GAO’s analysis of data from HUD’s SMIRPH database. Page 55 GAO/RCED-97-27 Public Housing Appendix V Comments From the Department of Housing and Urban Development Page 56 GAO/RCED-97-27 Public Housing Appendix V Comments From the Department of Housing and Urban Development Page 57 GAO/RCED-97-27 Public Housing Appendix V Comments From the Department of Housing and Urban Development Page 58 GAO/RCED-97-27 Public Housing Appendix V Comments From the Department of Housing and Urban Development Page 59 GAO/RCED-97-27 Public Housing Appendix V Comments From the Department of Housing and Urban Development Page 60 GAO/RCED-97-27 Public Housing Appendix VI Major Contributors to This Report Lawrence J. Dyckman, Associate Director Resources, Eric Marts, Assistant Director Community, and Carol Anderson-Guthrie Economic Curtis Groves Bill MacBlane Development Division Luann Moy Terri Russell (385602) Page 61 GAO/RCED-97-27 Public Housing Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Public Housing: HUD Should Improve the Usefulness and Accuracy of Its Management Assessment Program
Published by the Government Accountability Office on 1997-01-29.
Below is a raw (and likely hideous) rendition of the original report. (PDF)