oversight

Early Warning System Stress Model

Published by the Farm Credit Administration, Office of Inspector General on 2000-06-09.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

Early Warning System
        Stress Model

             AOO-02

                                                       Office of Inspector General
                                                       150-1 Farm Credit Drive
                                                       McLean, Virginia 22102-5090
                                                       (703) 883-4030




June 9, 2000


Honorable Michael M. Reyna
Chairman and Chief Executive Officer
Farm Credit Administration
McLean, Virginia

Dear Mr. Reyna:

We have completed our audit of the Office of Examination's (OE) Early'Narning System Stress
Model (EWSSM). Our objective was to evaluate the EWSSM for its reliability in forecasting
which Farm Credit System (FCS) institutions will develop weaknesses that threaten their safety
and soundness as well as the effectiveness of special supervision in causing correction of such
weaknesses. We documented the components and functions ofOE's Early Warning System and
compared the EWSSM forecasts for FCS institutions with actual results. We also benchmarked
OE's stress model with those of other Federal financial regulatory agencies.

OE's Early Warning System is designed to identify existing and prospective risk in Fes
institutions. The EWSSM component of the Early Warning System has been reliable in
forecasting emerging risk in the limited number of projections it has made so far. Similarly,
special supervision has generally been effective in causing correction of weaknesses in those
institutions before they actually became unsafe or unsound. The administration of the EWSSM
could be improved by formally tracking the accuracy of its projections over time and by
exempting FCS institutions from EWSSM testing whenever separate analysis has identified them
as low risk.

We conducted this audit in accordance with Government Auditing Standards issued by the
Comptroller General for audits of Federal organizations, programs, activities, and functions. We
conducted our fieldwork from December 1999 to March 2000 at FCA headquarters in McLean,
Virginia. An entrance conference was held on December 7, 1999. We provided a final draft of
this report to management on May 22, 2000 and have included their written response.

Re~y,               .1/
~~?£i
 h./
Eldon W. Stoehr
                    tf.?l
Inspector General
                                   Table of Contents




Background                                                    1


      EWSSM is only one part ofOE's multicomponent

      Early Warning System                                    1


Objective, Scope and Methodology                             2


Findings, Conclusions, and Recommendations


      OE's EWSSM has been reliable in forecasting

      emerging risk in FCS institutions                      .2


      OE's special supervision program has generally

      been effective                                         3


      OE does not formally track the accuracy of the EWSSM   3


      OE's EWSSM is unique among peer financial

      regulatory agencies                                    4



Management's Response
                                   FCA Office of Inspector General

                           AOO-02: OE's Early Warning System Stress Model





BACKGROUND

The Farm Credit Administration (FCA or Agency) is an independent Federal financial regulatory
agency of the United States Government. It has regulatory, examination, and supervisory
responsibilities for the Farm Credit System (FCS or System) banks, associations, and related
institutions chartered under the Farm Credit Act of 1971, as amended.

FCA's ability to project which FCS institutions are vulnerable to stress before unsafe or unsound
conditions actually develop allows more time for the Agency to cause corrective actions, and is,
therefore, an important tool for an effective supervision program. FCS institutions, which
develop unsafe or unsound conditions, threaten the equity of individual shareholders and
represent potential claims against the FCS Insurance Fund. The losses incurred in the FCS in the
early 80's (when FCA had no effective Early Warning System) show the effects of corrective
action taken too late.

In 1998, the Office of Examination (OE) developed and implemented an Early Warning System
Stress Model (EWSSM) to prospectively view the effects that stress in the environment may have
on individual direct lending FCS institutions over a 12 to 24 month period. The EWSSM is part
ofOE's overall Early Warning System for the effective supervision of the System.

EWSSM is only one component of OE's comprehensive Early Warning System.

OE has several components in its Early Warning System to identify and address emerging risk in
FCS institutions. The following is a summary listing and brief description of the individual
components that collectively make up OE's Early Warning System:

•	 EWSSM - Projects the effect of stress on individual FCS institutions over the next 12 to 24
   months. Current call report information provides the baseline over which examination
   managers overlay their assumptions to develop a "stress" forecast for individual institutions.

•	 Benchmark Ratings and Trends Report - Analyzes 18 months of trends of ratios and
   benchmark ratings to monitor and track changes and emerging risks.

•	 New Money Refinancing and Rollover Trends in FCS Institutions - Analyzes four years of
   trends to identify increased use of refinancing and rollovers in direct lender associations.

•	 Benchmark Variance Report - Identifies differences between average benchmark ratings and
   OE assigned rating for causes of these differences.

•	 Credit Quality in the FCS - Identifies institutions with deterioration in asset quality over the
   past year and evaluates levels of risk funds available to cover risk.

•	 Delinquencies Report - Analyzes loans past due 30-89 days as a percent of total loans and
   compares to previous periods by institution type.

06/09/00	                                                                               Page 1 of5
                                     FCA Office of Inspector General

                A 00-02 Office of Examination's (OE) Early Warning System Stress Model




•	 FIRS Ratings and Trends by Asset Size of FCS Institutions - Distribution of assets by FIRS
   ratings.

•	 FIRS Predictor Report - Logistic regression analysis used to establish relationship between
   ratings and ratios.

•	 Special Supervision - Oversight program designed to correct conditions in individual
   institutions that are serious but do not necessarily impair the safety and soundness of the
   institution.


OBJECTIVE, SCOPE, AND METHODOLOGY

The objective of this audit was to evaluate the EWSSM for its reliability in projecting which FCS
institutions are likely to develop weaknesses that threaten their safety and soundness as well as
the effectiveness of special supervision in causing correction of such weaknesses. The scope of
the audit included documenting the components and functions of OE's Early Warning System,
and comparing OE's EWSSM forecasts for FCS institutions with actual results. We also
benchmarked OE's stress model with those of other Federal financial regulatory agencies.


FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS

OE's EWSSM has been reliable in forecasting emerging risk in FCS institutions.

The first EWSSM report assessing prospective risk in FCS institutions over the next 12 and 24­
month periods was produced in June of 1998. Semiannual reports have been produced since that
time. Two of these reports were available to us for comparing projections to actual conditions in
FCS institutions. During the 12 months covered by these two reports, only one of the 162 FCS
institutions developed "unsatisfactory" risk that was not projected to do so by the "most likely"
prediction of EWSSM. The two other FCS institutions that developed "unsatisfactory" levels of
risk during that period were projected to do so by the EWSSM. Two other institutions projected
by the EWSSM to reach an "unsatisfactory" level of risk were placed under "special supervision"
and had not reached "unsatisfactory" levels of risk.

The first EWSSM report projected that several other institutions would develop "unsatisfactory"
risk that did not. The projections in the initial report used a more conservative risk measure that
caused this to occur. The subsequent EWSSM report increased the threshold for "unsatisfactory"
risk from 60 percent adverse assets to risk funds to 90 percent adverse assets to risk funds.
Consequently, only one institution was projected to reach an "unsatisfactory" risk level in the
second EWSSM report but had not actually reached that level by the end of the 12-month period.
Future comparisons of projections with actual results will further test whether this remains a
suitable measure of risk.


06/09/00	                                                                                 Page 2 of5
                                       FCA Office of Inspector General

                  A 00-02 Office of Examination's (OE) Early Warning System Stress Model




OE's special supervision program has generally been effective.

OE initiated a special supervision program in September of 1998 and, since that time, nine
institutions have been placed under "special supervision." Three of these nine institutions were
just placed under "special supervision" in October of 1999 and there has been insufficient time for
corrective actions to actually produce improvement in these institutions. Two of the other six
institutions were under the program for a year and improved enough to be removed from the
program. All but one of the remaining four institutions have been under the program for a year or
less and each have made progress in correcting the conditions that led to their need for "special
supervision." The only institution under "special supervision" for more than a year has made
sufficient progress to return to "normal supervision" but its supervisory status has not yet been
changed.

OE does not formally track the accuracy of the EWSSM.

OE's EWSSM projections of "unsatisfactory" risk developing in individual FCS institutions are
relied upon throughout the Agency and select portions of these projections are shared with FCS
institutions. OE stated in the June 1999 EWSSM report that projections of the first two reports
(June 98 and Dec. 98) were fairly consistent with actual results, however, there is no analysis
available to support the accuracy of projections reported by the EWSSM. Formal tracking of the
projections with actual results would contribute to the credibility of EWSSM projections and
provide a basis for measuring trends.

Other financial regulators track the accuracy of predictions made by their models according to
"Type I" and "Type II" errors. Type I errors occur when an institution develops "unsatisfactory"
risk that was not projected to occur. Type II errors occur when an institution is projected to
develop "unsatisfactory risk" but does not.

Based on our analysis, FCA's Type I errors have been low. Only one institution developed
"unsatisfactory" risk that was not projected in the first two EWSSM reports. FCA's Type II
errors have been high, although 11 of the 12 errors occurred in projections made by the first
EWSSM report, before the threshold for "unsatisfactory" risk level was increased from 60 percent
to 90 percent adverse assets to risk funds. Other fmancial regulators have experienced similar
frequencies of Type I and Type II errors--Type I errors are low and Type II errors are high. The
effectiveness of OE special supervision efforts would also increase Type II errors. However,
given that only two EWSSM reports have aged sufficiently to compare projections to actual, the
accuracy rate of the EWSSM results is not yet firmly established. With the December 1999
EWSSM report, OE began tracking some aspects of prior EWSSM projections.

Agreed Upon Action

1.         DE will formally track the accuracy ofEWSSM projections.




06/09/00                                                                                    Page 3 of 5
                                     FCA Office of Inspector General

                A 00-02 Office of Examination's (OE) Early Warning System Stress Model




OE's EWSSM is unique among peer financial regulatory agencies.

The EWSSM projects risk in each FCS direct lender institution based on current call report data
and then overlaying examiners' assumptions about changes in adversely classified assets, non­
accrual assets, and loan growth for each scenario. Based on that analysis, EWSSM reports
identify "unsatisfactory" risk exposure in FCS institutions.               FCA currently defines
"unsatisfactory" risk as any institution with adversely classified assets greater than 90 percent of
its risk funds. Projections are developed for "most likely" and "worst case" scenarios with 12­
month and 24-month time horizons, respectively. The results are then put into EWSSM reports
and integrated into OE resource planning and communicated in varying degrees to outside parties.

This analysis is now performed for all direct lender institutions. It may be possible to reduce the
resource requirements for this activity without diminishing its usefulness by exempting" 1" rated
institutions. A similar effect could be gained by exempting those institutions that could withstand
a prescribed credit shock; i.e., a 5% increase in adverse assets. The initial part of the current
process is labor intensive, as examiners have to derive this information from a number of sources
and institution specific analysis. Modifying the current approach would be consistent with a "risk
based" examination approach.

The models used by the Office of the Comptroller of the Currency and the Federal Reserve Board
(FRB) predict banks with an increased probability of failure. Their models use call report data,
historical bank failure information, and general assumptions of future economic conditions
projected over five and two year time horizons, respectively. Reports produced from these
models are then used to identify institutions with potential problems for added attention by
examination staff. Additionally, the FRB and the Federal Deposit Insurance Corporation have
models that predict the current quarter's CAMEL rating based on call report data and the most
recent examination reports. When reports produced from these models identify an institution with
a significant increase in its risk profile, those regulators perform a manual review of such
institutions. This review validates the model's assessment of the institution's model-driven
CAMEL rating and dictates whether further action is necessary.

These bank failure models and the current CAMEL models used by other regulators only involve
examiners' input and review for those institutions identified by the models. In contrast, FCA' s
model requires manual resources to derive institution specific data on all institutions that the
model uses. Then the model's report is used to highlight projected problem institutions, which in
tum is integrated into resource planning.

OE is developing a Loan Portfolio Stress Testing Model that will help confirm OE's EWSSM
results. It uses institution-specific loan portfolio data and then applies stress factors (commodity
prices, input prices) to that data which should make it a very reliable predictor model. OE plans
to use the loan portfolio stress model as a situational examination tool to be used on a limited
basis because it is too resource intensive to use on a widespread periodic basis akin to the current
EWSSM.



06/09/00                                                                                  Page 4 of5
                                    FCA Office of Inspector General

               A 00-02 Office of Examination's (OE) Early Warning System Stress Model


Recommendation

2.	     OE should exempt FCS institutions from EWSSM testing whenever separate analysis
        has identified them as low risk.




06/09/00	                                                                                Page 5 of 5