Early Warning System Stress Model AOO-02 Office of Inspector General 150-1 Farm Credit Drive McLean, Virginia 22102-5090 (703) 883-4030 June 9, 2000 Honorable Michael M. Reyna Chairman and Chief Executive Officer Farm Credit Administration McLean, Virginia Dear Mr. Reyna: We have completed our audit of the Office of Examination's (OE) Early'Narning System Stress Model (EWSSM). Our objective was to evaluate the EWSSM for its reliability in forecasting which Farm Credit System (FCS) institutions will develop weaknesses that threaten their safety and soundness as well as the effectiveness of special supervision in causing correction of such weaknesses. We documented the components and functions ofOE's Early Warning System and compared the EWSSM forecasts for FCS institutions with actual results. We also benchmarked OE's stress model with those of other Federal financial regulatory agencies. OE's Early Warning System is designed to identify existing and prospective risk in Fes institutions. The EWSSM component of the Early Warning System has been reliable in forecasting emerging risk in the limited number of projections it has made so far. Similarly, special supervision has generally been effective in causing correction of weaknesses in those institutions before they actually became unsafe or unsound. The administration of the EWSSM could be improved by formally tracking the accuracy of its projections over time and by exempting FCS institutions from EWSSM testing whenever separate analysis has identified them as low risk. We conducted this audit in accordance with Government Auditing Standards issued by the Comptroller General for audits of Federal organizations, programs, activities, and functions. We conducted our fieldwork from December 1999 to March 2000 at FCA headquarters in McLean, Virginia. An entrance conference was held on December 7, 1999. We provided a final draft of this report to management on May 22, 2000 and have included their written response. Re~y, .1/ ~~?£i h./ Eldon W. Stoehr tf.?l Inspector General Table of Contents Background 1 EWSSM is only one part ofOE's multicomponent Early Warning System 1 Objective, Scope and Methodology 2 Findings, Conclusions, and Recommendations OE's EWSSM has been reliable in forecasting emerging risk in FCS institutions .2 OE's special supervision program has generally been effective 3 OE does not formally track the accuracy of the EWSSM 3 OE's EWSSM is unique among peer financial regulatory agencies 4 Management's Response FCA Office of Inspector General AOO-02: OE's Early Warning System Stress Model BACKGROUND The Farm Credit Administration (FCA or Agency) is an independent Federal financial regulatory agency of the United States Government. It has regulatory, examination, and supervisory responsibilities for the Farm Credit System (FCS or System) banks, associations, and related institutions chartered under the Farm Credit Act of 1971, as amended. FCA's ability to project which FCS institutions are vulnerable to stress before unsafe or unsound conditions actually develop allows more time for the Agency to cause corrective actions, and is, therefore, an important tool for an effective supervision program. FCS institutions, which develop unsafe or unsound conditions, threaten the equity of individual shareholders and represent potential claims against the FCS Insurance Fund. The losses incurred in the FCS in the early 80's (when FCA had no effective Early Warning System) show the effects of corrective action taken too late. In 1998, the Office of Examination (OE) developed and implemented an Early Warning System Stress Model (EWSSM) to prospectively view the effects that stress in the environment may have on individual direct lending FCS institutions over a 12 to 24 month period. The EWSSM is part ofOE's overall Early Warning System for the effective supervision of the System. EWSSM is only one component of OE's comprehensive Early Warning System. OE has several components in its Early Warning System to identify and address emerging risk in FCS institutions. The following is a summary listing and brief description of the individual components that collectively make up OE's Early Warning System: • EWSSM - Projects the effect of stress on individual FCS institutions over the next 12 to 24 months. Current call report information provides the baseline over which examination managers overlay their assumptions to develop a "stress" forecast for individual institutions. • Benchmark Ratings and Trends Report - Analyzes 18 months of trends of ratios and benchmark ratings to monitor and track changes and emerging risks. • New Money Refinancing and Rollover Trends in FCS Institutions - Analyzes four years of trends to identify increased use of refinancing and rollovers in direct lender associations. • Benchmark Variance Report - Identifies differences between average benchmark ratings and OE assigned rating for causes of these differences. • Credit Quality in the FCS - Identifies institutions with deterioration in asset quality over the past year and evaluates levels of risk funds available to cover risk. • Delinquencies Report - Analyzes loans past due 30-89 days as a percent of total loans and compares to previous periods by institution type. 06/09/00 Page 1 of5 FCA Office of Inspector General A 00-02 Office of Examination's (OE) Early Warning System Stress Model • FIRS Ratings and Trends by Asset Size of FCS Institutions - Distribution of assets by FIRS ratings. • FIRS Predictor Report - Logistic regression analysis used to establish relationship between ratings and ratios. • Special Supervision - Oversight program designed to correct conditions in individual institutions that are serious but do not necessarily impair the safety and soundness of the institution. OBJECTIVE, SCOPE, AND METHODOLOGY The objective of this audit was to evaluate the EWSSM for its reliability in projecting which FCS institutions are likely to develop weaknesses that threaten their safety and soundness as well as the effectiveness of special supervision in causing correction of such weaknesses. The scope of the audit included documenting the components and functions of OE's Early Warning System, and comparing OE's EWSSM forecasts for FCS institutions with actual results. We also benchmarked OE's stress model with those of other Federal financial regulatory agencies. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS OE's EWSSM has been reliable in forecasting emerging risk in FCS institutions. The first EWSSM report assessing prospective risk in FCS institutions over the next 12 and 24 month periods was produced in June of 1998. Semiannual reports have been produced since that time. Two of these reports were available to us for comparing projections to actual conditions in FCS institutions. During the 12 months covered by these two reports, only one of the 162 FCS institutions developed "unsatisfactory" risk that was not projected to do so by the "most likely" prediction of EWSSM. The two other FCS institutions that developed "unsatisfactory" levels of risk during that period were projected to do so by the EWSSM. Two other institutions projected by the EWSSM to reach an "unsatisfactory" level of risk were placed under "special supervision" and had not reached "unsatisfactory" levels of risk. The first EWSSM report projected that several other institutions would develop "unsatisfactory" risk that did not. The projections in the initial report used a more conservative risk measure that caused this to occur. The subsequent EWSSM report increased the threshold for "unsatisfactory" risk from 60 percent adverse assets to risk funds to 90 percent adverse assets to risk funds. Consequently, only one institution was projected to reach an "unsatisfactory" risk level in the second EWSSM report but had not actually reached that level by the end of the 12-month period. Future comparisons of projections with actual results will further test whether this remains a suitable measure of risk. 06/09/00 Page 2 of5 FCA Office of Inspector General A 00-02 Office of Examination's (OE) Early Warning System Stress Model OE's special supervision program has generally been effective. OE initiated a special supervision program in September of 1998 and, since that time, nine institutions have been placed under "special supervision." Three of these nine institutions were just placed under "special supervision" in October of 1999 and there has been insufficient time for corrective actions to actually produce improvement in these institutions. Two of the other six institutions were under the program for a year and improved enough to be removed from the program. All but one of the remaining four institutions have been under the program for a year or less and each have made progress in correcting the conditions that led to their need for "special supervision." The only institution under "special supervision" for more than a year has made sufficient progress to return to "normal supervision" but its supervisory status has not yet been changed. OE does not formally track the accuracy of the EWSSM. OE's EWSSM projections of "unsatisfactory" risk developing in individual FCS institutions are relied upon throughout the Agency and select portions of these projections are shared with FCS institutions. OE stated in the June 1999 EWSSM report that projections of the first two reports (June 98 and Dec. 98) were fairly consistent with actual results, however, there is no analysis available to support the accuracy of projections reported by the EWSSM. Formal tracking of the projections with actual results would contribute to the credibility of EWSSM projections and provide a basis for measuring trends. Other financial regulators track the accuracy of predictions made by their models according to "Type I" and "Type II" errors. Type I errors occur when an institution develops "unsatisfactory" risk that was not projected to occur. Type II errors occur when an institution is projected to develop "unsatisfactory risk" but does not. Based on our analysis, FCA's Type I errors have been low. Only one institution developed "unsatisfactory" risk that was not projected in the first two EWSSM reports. FCA's Type II errors have been high, although 11 of the 12 errors occurred in projections made by the first EWSSM report, before the threshold for "unsatisfactory" risk level was increased from 60 percent to 90 percent adverse assets to risk funds. Other fmancial regulators have experienced similar frequencies of Type I and Type II errors--Type I errors are low and Type II errors are high. The effectiveness of OE special supervision efforts would also increase Type II errors. However, given that only two EWSSM reports have aged sufficiently to compare projections to actual, the accuracy rate of the EWSSM results is not yet firmly established. With the December 1999 EWSSM report, OE began tracking some aspects of prior EWSSM projections. Agreed Upon Action 1. DE will formally track the accuracy ofEWSSM projections. 06/09/00 Page 3 of 5 FCA Office of Inspector General A 00-02 Office of Examination's (OE) Early Warning System Stress Model OE's EWSSM is unique among peer financial regulatory agencies. The EWSSM projects risk in each FCS direct lender institution based on current call report data and then overlaying examiners' assumptions about changes in adversely classified assets, non accrual assets, and loan growth for each scenario. Based on that analysis, EWSSM reports identify "unsatisfactory" risk exposure in FCS institutions. FCA currently defines "unsatisfactory" risk as any institution with adversely classified assets greater than 90 percent of its risk funds. Projections are developed for "most likely" and "worst case" scenarios with 12 month and 24-month time horizons, respectively. The results are then put into EWSSM reports and integrated into OE resource planning and communicated in varying degrees to outside parties. This analysis is now performed for all direct lender institutions. It may be possible to reduce the resource requirements for this activity without diminishing its usefulness by exempting" 1" rated institutions. A similar effect could be gained by exempting those institutions that could withstand a prescribed credit shock; i.e., a 5% increase in adverse assets. The initial part of the current process is labor intensive, as examiners have to derive this information from a number of sources and institution specific analysis. Modifying the current approach would be consistent with a "risk based" examination approach. The models used by the Office of the Comptroller of the Currency and the Federal Reserve Board (FRB) predict banks with an increased probability of failure. Their models use call report data, historical bank failure information, and general assumptions of future economic conditions projected over five and two year time horizons, respectively. Reports produced from these models are then used to identify institutions with potential problems for added attention by examination staff. Additionally, the FRB and the Federal Deposit Insurance Corporation have models that predict the current quarter's CAMEL rating based on call report data and the most recent examination reports. When reports produced from these models identify an institution with a significant increase in its risk profile, those regulators perform a manual review of such institutions. This review validates the model's assessment of the institution's model-driven CAMEL rating and dictates whether further action is necessary. These bank failure models and the current CAMEL models used by other regulators only involve examiners' input and review for those institutions identified by the models. In contrast, FCA' s model requires manual resources to derive institution specific data on all institutions that the model uses. Then the model's report is used to highlight projected problem institutions, which in tum is integrated into resource planning. OE is developing a Loan Portfolio Stress Testing Model that will help confirm OE's EWSSM results. It uses institution-specific loan portfolio data and then applies stress factors (commodity prices, input prices) to that data which should make it a very reliable predictor model. OE plans to use the loan portfolio stress model as a situational examination tool to be used on a limited basis because it is too resource intensive to use on a widespread periodic basis akin to the current EWSSM. 06/09/00 Page 4 of5 FCA Office of Inspector General A 00-02 Office of Examination's (OE) Early Warning System Stress Model Recommendation 2. OE should exempt FCS institutions from EWSSM testing whenever separate analysis has identified them as low risk. 06/09/00 Page 5 of 5
Early Warning System Stress Model
Published by the Farm Credit Administration, Office of Inspector General on 2000-06-09.
Below is a raw (and likely hideous) rendition of the original report. (PDF)