oversight

HUD's Office of Single Family Housing Could Improve the Reliability of Its Process for Reporting Performance Measure Results

Published by the Department of Housing and Urban Development, Office of Inspector General on 2008-06-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                             Issue Date
                                                                      June 24, 2008
                                                             Audit Report Number
                                                                      2008-KC-0004




TO:         Brian D. Montgomery, Assistant Secretary for Housing – Federal Housing
               Commissioner, H

            //signed//
FROM:       Ronald J. Hosking, Regional Inspector General for Audit, 7AGA

SUBJECT: HUD’s Office of Single Family Housing Could Improve the Reliability of Its
           Process for Reporting Performance Measure Results


                                   HIGHLIGHTS

 What We Audited and Why

             We audited the U.S. Department of Housing and Urban Development’s (HUD)
             Office of Single Family Housing (Single Family) to determine whether Single
             Family implemented a reliable process to ensure accurate reporting of its
             performance measure results.

 What We Found

             Single Family could improve the reliability of its process for reporting
             performance measure results. Single Family has a performance measurement
             process in place; however, it could make the process more reliable if it routinely
             evaluated data used for performance measure results and formally documented its
             structure and process for developing, monitoring, and reporting on performance
             measures.




                                              1
What We Recommend


           We recommend that HUD establish and implement effective written policies and
           procedures for routinely evaluating the data used to report performance measure
           results to ensure that the data are the most accurate and appropriate data available.
           We also recommend that HUD establish and implement effective written policies
           and procedures for developing, monitoring, and reporting on performance
           measures.

           For each recommendation without a management decision, please respond and
           provide status reports in accordance with HUD Handbook 2000.06, REV-3.
           Please furnish us copies of any correspondence or directives issued because of the
           audit.

Auditee’s Response

           We provided the report to HUD on April 25, 2008, and asked for comments by
           May 26, 2008. Single Family requested, and we granted, an extension to
           May 29, 2008. On May 29, 2008, Single Family asked for another extension to
           June 4, 2008, which we granted. On June 4, 2008, we received draft comments.
           We continued to seek official comments until July 19, 2008, when we received
           Single Family’s official comments. In its response, HUD provided additional
           clarification of its processes and agreed to implement our recommendations.

           The complete text of the auditee’s response, along with our evaluation of that
           response, can be found in appendix A of this report.




                                             2
                            TABLE OF CONTENTS

Background and Objectives                                                        4

Results of Audit
      Finding: Single Family Could Improve the Reliability of Its Process for    5
                Reporting Performance Measure Results

Scope and Methodology                                                            9

Internal Controls                                                               10

Appendixes
   A. Auditee Comments and OIG’s Evaluation                                     11




                                             3
                     BACKGROUND AND OBJECTIVES

The U.S. Congress passed the Government Performance Results Act of 1993 (Performance Act)
because it found, in part, that federal managers were seriously disadvantaged in their efforts to
improve program efficiency and effectiveness because of insufficient articulation of program
goals and inadequate information on program performance. It also found that congressional
policy making, spending decisions, and program oversight were seriously handicapped by
insufficient attention to program performance and results. The U.S. Congress passed the
Performance Act to alleviate these problems by requiring federal agencies to incorporate
strategic planning and performance measurement into agency management.

The Performance Act is intended, in part, to improve congressional decision making by giving
the U.S. Congress comprehensive and reliable information on the extent to which federal
programs are fulfilling their statutory intent. Managers should use performance information to
continuously improve organizational processes, identify performance gaps, and set improvement
goals. Decision makers are to routinely receive the performance and cost information needed to
assess their programs and make informed decisions.

In its Executive Guide for Effectively Implementing the Government Performance and Results
Act, the U.S. Government Accountability Office identifies key steps that agencies need to take
toward implementing Performance Act requirements, along with a set of practices that can help
ensure that agencies generate the information congressional and executive branch decision
makers need in considering measures to improve government performance and reduce costs.
One of the recommended practices is that agencies collect sufficiently complete, accurate, and
consistent data. The guide points out that agencies need to ensure that the collected data are
complete, accurate, and consistent enough to document performance and support decision
making at various organizational levels.

The U.S. Department of Housing and Urban Development (HUD) reports its performance
measure results to the U.S Congress in HUD’s annual performance and accountability report.
HUD’s Office of Single Family Housing (Single Family) is responsible for developing its
performance data for reporting its accomplishments in HUD’s performance and accountability
report. Single Family reported the results of 12 performance measures in HUD’s 2006
performance accountability report and 13 results in its 2007 report.

The objective of our audit was to determine whether Single Family implemented a reliable
process to ensure accurate reporting of its performance measure results.




                                                4
                                 RESULTS OF AUDIT

Finding: Single Family Could Improve the Reliability of Its Process for
            Reporting Performance Measure Results
Single Family has a performance measurement process in place; however, it could make the
process more reliable if it strengthened its process controls. Single Family had not considered
the potential benefits of a more structured process and how it could improve its performance
measurement process. By strengthening its controls, Single Family could better ensure the
accuracy of reported accomplishments and that it used the most appropriate data available,
thereby avoiding unintended effects on funding and program decisions.


Single Family has a performance measurement process in place that it has used for many years.
However, it could improve its controls over the process if it routinely evaluated data used for
performance measure results and formally documented its structure and process for developing,
monitoring, and reporting on performance measures.

 Accuracy and Appropriateness
 of Changes to Performance
 Data Not Fully Evaluated
               Single Family could not show that it had evaluated the impact of changes made to
               some of the data routinely used for its performance measure results. It could not
               show that it had evaluated the data to ensure that the data were the most accurate and
               appropriate available for reporting performance. U.S. Office of Management and
               Budget Circular A-136 requires agencies to ensure that performance data reported in
               annual performance reports are reliable and complete. In addition, the U.S.
               Government Accountability Office’s guidelines on internal controls state that
               agencies need to establish controls to monitor performance measures and indicators
               and that these controls should be aimed at validating the propriety and integrity of
               performance measures and indicators.

               Single Family reported its performance results for 12 performance measures in
               HUD’s fiscal year 2006 performance accountability report and 13 measures in the
               2007 report. We reviewed the reported results for 14 of the 25 performance
               measures and identified two examples of changes made to the methods Single
               Family used to develop performance data and report performance results in fiscal
               year 2007 as compared with fiscal year 2006. However, Single Family could not
               show that it had evaluated the new processes to ensure that the changes provided the
               most accurate and appropriate results for reporting performance.

               In the first example, a Single Family contractor used a different data system report
               when developing performance data and reporting fiscal year 2007 performance
               measure results than it had used for 2006. However, Single Family could not
               demonstrate that it had performed an analysis to determine whether the change made
                                                 5
            by the contractor was appropriate or that the report used by the contractor contained
            the most appropriate data available to report accurate and complete results in HUD’s
            performance accountability report. Further, Single Family could not show that it had
            approved or was even aware of the change made by the contractor.

            In the second example, Single Family changed its method for developing housing
            counseling program performance measures and reporting the performance measure
            results between fiscal years 2006 and 2007. For fiscal year 2006, HUD reported its
            annual performance by estimating what it thought it had accomplished in 2006 based
            on actual accomplishments from fiscal year 2005. For fiscal year 2007, HUD
            changed its method and reported its performance using only three fiscal year
            quarters of actual accomplishments data. HUD changed its method based on
            concerns reported by the Office of Inspector General (OIG) in 2006. However,
            Single Family could not show whether it had evaluated the data used for reporting
            housing counseling program performance measure results to ensure that the data
            were the most accurate and appropriate data available.

Performance Measurement
Process Could Benefit from a
More Structured Approach


            While we recognize that Single Family has had a performance measurement
            process in place for years, it had not documented its structure and processes for
            developing, monitoring, and reporting on performance measures. The U.S.
            Government Accountability Office’s guidelines on internal controls state that
            internal controls and all transactions and other significant events need to be
            clearly documented and readily available for examination. The documentation
            should appear in management directives, administrative policies, or operating
            manuals.

            Single Family officials told us that although they did not have a fully documented
            structure or process, senior officials held meetings to review historical
            performance data and analyses of future business volume in their efforts to
            address governmental performance measurement requirements. However, Single
            Family could not show to what extent that its managers evaluated business data or
            reached decisions and made plans for how Single Family would develop, monitor,
            and report on its performance measures.

            In addition, Single Family routinely employed contractors to obtain performance
            measure results from reports generated by Single Family data systems and to
            report the results in HUD’s Integrated Performance and Reporting system.
            However, it had not developed detailed policies and procedures to identify the
            data fields to use from the reports, the calculations to perform, or the step-by-step
            process for entering the data into the performance reporting system. In order to
            perform their duties the current contractor employees relied on on-the-job training
            and informal information obtained from contractor employees that had performed
            the tasks in the past.
                                              6
Single Family Managers Had
Not Assessed the Benefits of
Added Controls to Improve the
Process

           Single Family had not considered whether a more structured process could
           improve controls over its performance measurement process. More specifically, it
           was not aware of its technical responsibilities and had not considered the positive
           impact that strengthening controls could have on performance and reporting
           performance measure results. The U.S. Government Accountability Office’s
           guidelines on internal controls state that management is responsible for
           developing detailed policies, procedures, and practices to ensure that they are
           built into and an integral part of operations.

           Single Family officials told us that the Single Family performance measures used
           in fiscal years 2006 and 2007 had been in place for many years. They also stated
           that the HUD data systems used to support the reported performance results had
           changed very little or not at all in recent years. Therefore, they felt it was not
           necessary to reevaluate the data used to support performance results to ensure that
           the data were the most accurate and appropriate data available. However, Single
           Family officials agreed that having a documented structure and processes in place
           would be beneficial.

Reported Performance Measure
Results Could Be More Reliable


           By strengthening its controls, Single Family could better ensure the accuracy of
           reported accomplishments and the use of the most appropriate data available.
           Therefore, congressional and HUD decision makers would have more reliable
           performance data with which to make more informed funding and program
           decisions.

           The U.S. Congress passed the Performance Act to improve federal program
           efficiency and effectiveness, congressional policy making, spending decisions,
           and program oversight. It expected federal agencies to annually report
           comprehensive and reliable information on their accomplishments, which it could
           then use to make informed decisions on funding federal programs and whether to
           make changes to the individual programs. Further, it expected agency managers
           to use the performance information to continuously improve program
           performance and reduce costs.

           Without improvements to its performance measure controls, decision makers are
           at risk of not having the most accurate information on which to base funding and
           program decisions. In addition, Single Family increases its risk of not being able

                                            7
          to manage its programs with maximum efficiency and effectiveness, or to manage
          its programs to ensure that they are fulfilling their statutory intent.

          Therefore, having the most reliable performance data available is critical to
          decision makers to ensure that they are providing scarce federal resources to
          programs that are accomplishing their statutory intent, and reevaluating and
          making changes to programs that are not adequately assisting those intended to be
          served.

Recommendations

          We recommend that HUD’s Assistant Secretary for Housing

          1A. Establish and implement effective written policies and procedures for
              routinely evaluating the data used to report performance measure results to
              ensure that the data are the most accurate and appropriate data available.

          1B. Establish and implement effective written policies and procedures for
              developing, monitoring, and reporting on performance measures.




                                          8
                         SCOPE AND METHODOLOGY

Our review period was from October 1, 2005, to September 31, 2007. We expanded the period as
needed to evaluate historical and current information pertinent to our review. We limited the review
to Single Family’s developing, monitoring, and reporting on performance measures.

To achieve our objectives, we interviewed HUD staff from its Office of Housing, Office of
Single Family Housing, and Office of Departmental Operations and Coordination to gain an
understanding of the process involved in developing, monitoring, and reporting on performance
measures. We also reviewed Government Accountability Office and HUD OIG reports, data
obtained from HUD’s computer systems, and applicable federal requirements.

In addition, we selected a sample of the Single Family performance measures for fiscal years
2006 and 2007. Single Family had 12 performance measures in 2006 and 13 in 2007. We
selected 14 performance measures to review:

   •   Five measures related to grant-based activities (two from fiscal year 2006 and three from
       fiscal year 2007) and
   •   Nine measures that were not related to grant-based activities (five from fiscal year 2006
       and four from fiscal year 2007).

We compared the performance reported for these measures in HUD’s performance
accountability report to reports generated by HUD’s data systems. We did not assess the
accuracy or the validity of the data. We reviewed reports from the systems only to understand
and evaluate the process used by Single Family in developing, monitoring, and reporting on its
performance measures. We did not rely on the data to reach our conclusions and, therefore, did
not assess the accuracy or validity of the data.

We performed on-site work from October 2007 to March 2008 at HUD headquarters located at
451 7th Street, SW, Washington, DC.

We conducted this performance audit in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions
based on our audit objectives. We believe that the evidence obtained provides a reasonable basis
for our findings and conclusions based on our audit objectives.




                                                 9
                             INTERNAL CONTROLS

Internal control is an integral component of an organization’s management that provides
reasonable assurance that the following objectives are being achieved:

   •   Effectiveness and efficiency of operations,
   •   Reliability of financial reporting, and
   •   Compliance with applicable laws and regulations.

Internal controls relate to management’s plans, methods, and procedures used to meet its
mission, goals, and objectives. Internal controls include the processes and procedures for
planning, organizing, directing, and controlling program operations. They include the systems
for measuring, reporting, and monitoring program performance.



 Relevant Internal Controls


              We determined the following internal controls were relevant to our audit objectives:

              •   Controls over Single Family’s developing, monitoring, and reporting on its
                  performance measures.

              We assessed the relevant controls identified above.

              A significant weakness exists if management controls do not provide reasonable
              assurance that the process for planning, organizing, directing, and controlling
              program operations will meet the organization’s objectives.


 Significant Weakness


              We did not identify any significant weaknesses.




                                               10
                        APPENDIXES

Appendix A

        AUDITEE COMMENTS AND OIG’S EVALUATION


Ref to OIG Evaluation      Auditee Comments




Comment 1


Comment 2




                            11
Comment 3

Comment 4


Comment 3




Comment 5




            12
Comment 6




            13
                 OIG Evaluation of Auditee Comments


Comment 1   We revised the report to clarify the status of contractors assisting with
            performance measures in 2006 and 2007.

Comment 2   Single Family officials told us that it was possible that the contractor
            presented the change in reports to Single Family staff but any decisions or
            approval regarding the change would have been verbal and not likely
            documented. We requested that Single Family provide documentation of
            its involvement with the decision to make the change but Single Family
            did not provide any documentation or explanation of what took place with
            the change in reports.

Comment 3   As explained in comment 2, Single Family did not provide documentation
            or an explanation regarding its involvement in the change in reports. As
            stated in the report, Single Family officials told us that senior managers
            held meetings to review historical performance data and analyses of future
            business volume but they could not show to what extent that managers
            evaluated the data.

            In addition, based on our understanding of the review performed by the
            Office of the FHA Comptroller for the financial statement audit process,
            the Office of the FHA Comptroller did not validate the data to ensure that
            it is the most accurate and appropriate data available. The Office of the
            FHA Comptroller reviewed the reports used by the Single Family
            contractor when the contractor reported Single Family’s performance
            measures and compared the performance measures identified and
            calculated from these reports to the performance measures reported in the
            FHA financial statements. This process is simply a second check to make
            sure that the reports support the performance measures reported in the
            financial statements. Therefore, the data review performed by the Office
            of the FHA Comptroller does not constitute an analysis of the accuracy
            and appropriateness of the data.

Comment 4   We agree that the result of the change in reports was minimal in the
            example given in the report. However, Single Family could not show
            whether the change made by the contractor was appropriate or that the
            report used contained the most appropriate data available to report
            accurate and complete results. If Single Family allows contractors to
            make decisions regarding how to obtain data to report on performance
            measures, without appropriate and documented input from Single Family,
            this could result in incorrect reporting and could make a more significant
            difference in another situation.

Comment 5   We agree that as a result of an OIG report Single Family made positive
            changes in its process for obtaining and reporting on performance measures
            for the housing counseling program when it agreed to use actual data rather
                                      14
            than estimating results based on historical performance. We also recognize
            that the data collection instrument and the data element did not change.

            Single Family could not show whether it had evaluated the data collection
            instrument and the data elements used for reporting performance measure
            results to ensure that the data were the most accurate and appropriate data
            available. Using the same data collection instrument and the same data
            elements as in the past does not always ensure that this produces the most
            accurate and appropriate results. We changed the statement in the report
            to better characterize the conclusion.

Comment 6   HUD’s annual performance plan provides general, overall information
            pertaining to each of HUD’s performance measures. The plan does not
            provide the level of information needed to show that Single Family
            adequately ensures the accuracy of reported accomplishments and the use of
            the most appropriate data available.

            For each performance measure, the annual performance plan has four
            catagories:
                • Indicator background and context,
                • Data source
                • Limitations/advantages of the data, and
                • Validation, verification, and improvement of measure.

            The indicator background and context explains why this information is
            included as a performance measure but does not provide details of the data
            source or validation efforts performed by the program offices. The
            remaining three categories are also general and do not provide details
            showing what efforts Single Family makes to analyze the accuracy of
            reported accomplishments and appropriateness of data used. While the
            annual performance plan provides general information about each
            performance measure, it does not constitute sufficient evidence of a reliable
            process for reporting performance measure results.




                                      15