oversight

Review Results for Hotline Case Number HL-09-0756 Kaibab Band of Paiute Indians, Pipe Spring, AZ

Published by the Department of Housing and Urban Development, Office of Inspector General on 2010-01-25.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                     U.S. Department of Housing and Urban Development
                                                                   Office of Inspector General
                                                                                                Region IX
                                                                        611 West Sixth Street, Suite 1160
                                                                      Los Angeles, California 90017-3101
                                                                                    Voice (213) 894-8016
                                                                                     Fax (213) 894-8115




                                                                  Issue Date

                                                                          January 25, 2010
                                                                  Audit Memorandum Number

                                                                           2010-LA-0803


MEMORANDUM FOR:              Deborah Lalancette, Director, Office of Grants Management, PNP



FROM:                        Joan S. Hobbs, Regional Inspector General for Audit, Region IX,
                             9DGA


SUBJECT:                     Review Results for Hotline Case Number HL-09-0756
                             Kaibab Band of Paiute Indians, Pipe Spring, AZ


                                     INTRODUCTION

In response to an April 3, 2009, hotline request from the Kaibab Band of Paiute Indians, Pipe
Spring, AZ (Kaibab), the U.S. Department of Housing and Urban Development’s (HUD) Office
of Inspector General (OIG) reviewed HUD’s evaluation of Kaibab’s application for an Indian
Community Development Block Grant (Indian Block Grant) under HUD’s 2008 notice of
funding availability. The review objective was to evaluate the merits of the complainant’s
allegation that the HUD Southwest Office of Native American Programs (Southwest Office)
treated Kaibab’s application prejudicially without providing for a fair review.

For each recommendation without a management decision, please respond and provide status
reports in accordance with HUD Handbook 2000.06, REV-3. Please furnish us copies of any
correspondence or directives issued because of the audit.

                             SCOPE AND METHODOLOGY

Our review was limited to an examination of the Southwest Office’s evaluation of Kaibab’s 2008
Indian Block Grant application, as well as elements of Kaibab’s 2007 grant application and
scoring that were relevant to the 2008 application. The complainant also submitted information
regarding how the Southwest Office rated grant applications from other tribes for various years.
However, this memorandum does not discuss other tribal applications because that information
did not have an effect on the outcome of the 2008 decision related to the subject application.

To conduct our review, we interviewed the complainant’s representative for the 2008 application
and HUD officials in the Office of Native American Programs including the Grants Management
Director, the Southwest Office Administrator, and the individuals who evaluated Kaibab’s 2007
and 2008 applications. We reviewed the relevant parts of HUD’s 2008 notice of funding
availability (2008 notice); HUD laws and regulations for Indian Block Grants; and Office of
Native American Programs guidance for the evaluation, rating, and ranking of Indian Block
Grant applications. We also reviewed documentation submitted by Kaibab for its 2007 and 2008
Indian Block Grant applications, along with the evaluations prepared by HUD review officials.
Finally, we reviewed letters from the complainant and its representative that detailed the
allegations and provided arguments regarding why it believed the support in the application
warranted passing the threshold criteria.

Our review of Kaibab’s grant applications was limited to an evaluation of whether the Southwest
Office’s review of the applications complied with applicable requirements and guidance and
provided a fair review. To accomplish this objective, we considered the following questions:

        Were the review processes consistent with Indian Block Grant program requirements?

        Was the reviewer’s determination that the proposal failed to meet certain threshold
        requirements reasonably supported?

        Was the reviewer’s overall conclusion consistent with the applicable threshold criteria
        for economic development projects?

We limited our review to work necessary to address the complaint's central issue. Accordingly,
we did not conduct the review in accordance with generally accepted government auditing
standards but obtained sufficient evidence to provide a reasonable basis for our conclusions.

                                       BACKGROUND

The initial complaint regarding the Kaibab 2008 Indian Block Grant application was made to
OIG by the consultant/developer that represented Kaibab for the 2008 grant application process
and assisted with the preparation of the grant application. The OIG hotline accepted the
complaint after the Kaibab tribe followed up with its own signed complaint letter.

Kaibab, a small Native American tribe located in Northwest Arizona, submitted an application to
the Southwest Office for an Indian Block Grant economic development grant of $605,000 for
competition under HUD’s 2008 notice. The project was described as an RV park campground to
be located on tribal land donated by Kaibab. Southwest Office grants management reviewer(s)
rated the application acceptable under the initial application screening criteria and passed it
under the general threshold requirements of 24 CFR (Code of Federal Regulations) 1003.301(a).
In accordance with the 2008 notice requirements, reviewers then evaluated whether the


                                                2
application met the project-specific threshold requirements for economic development projects.1
The reviewers determined that the Kaibab 2008 Indian Block Grant application did not meet the
following two of the five project-specific threshold requirements:

        Criterion (1): In accordance with 24 CFR 1003.302, for economic development
        assistance, the application must include a financial analysis. The financial analysis must
        demonstrate that the project is financially feasible and the project has a reasonable chance
        of success. The analysis must also demonstrate the public benefit resulting from the
        ICDBG [Indian Block Grant] assistance. The more funds the application requests, the
        greater the public benefit that must be demonstrated.

        Criterion (4): Not more than a reasonable rate of return on investment is provided to the
        owner.

In accordance with the application review procedures, because it did not meet the threshold
screening requirements, the Kaibab 2008 Indian Block Grant application was ineligible to be
further considered under the rating and ranking review process. The Southwest Office proceeded
to issue a letter, dated November 24, 2008, notifying Kaibab that its 2008 application did not
meet threshold requirements and would not receive further funding consideration.

According to a letter, dated January 16, 2009, from HUD’s Deputy Assistant Secretary for
Native American Programs to the chairperson of the Kaibab Band of Paiute Indians, HUD
denied a request for reconsideration of the funding determination. HUD’s basis for the denial
was section V(B)(10) of the 2008 notice, which stated that judgments made within the provisions
of the notice and program regulations are not subject to claims of error.

Kaibab had also submitted an Indian Block Grant application in 2007 for basically the same
project, which Southwest Office officials also declined to rate or rank based upon failure to meet
two project-specific threshold criteria. The complainants updated the 2007 proposal and felt that
the 2008 application adequately addressed the reasons why the 2007 application had failed.
Therefore, the complainant believed that the 2008 application should have passed the project-
specific threshold criteria and at least have been rated and ranked.

                                        RESULTS OF REVIEW

                                                  Summary

Our review did not find evidence that the Southwest Office treated Kaibab’s 2008 Indian Block
Grant economic development grant application with prejudice or failed to provide a fair review.

            The grant application review procedures—as related to project-specific threshold
            screening—were performed in compliance with program requirements. However, we
            noted that the review processes were not standardized in a way that easily precluded

1
 Fiscal Year 2008 SuperNOFA [notice of funding availability] for HUD’s Discretionary Programs; Community
Development Block Grant Program for Indian Tribes and Alaska Native Villages; Section III.(C)( 3)( c) – Economic
Development Project Thresholds


                                                       3
           any perception of unfairness. Specifically, the Southwest Office assigned
           applications to grants management specialists without regard to ensuring impartial
           treatment in appearance and in fact. Although economic development proposals
           tended to be uncommon and more complicated than other Indian Block Grant
           proposals, application reviewers did not receive training or guidance regarding the
           evaluation of such proposals. Finally, the Southwest Office did not require reviewers
           to clearly and thoroughly document the reasons for their determination that an
           application failed to meet threshold requirements and, therefore, would not be rated or
           ranked.

           We determined that the Southwest Office reviewers’ determinations regarding
           specific threshold criteria were generally defensible and made in good faith. We
           noted that, according to the 2008 notice, if an application failed even one threshold
           criterion, it could not proceed to rating and ranking. Because we noted that the
           reviewers’ written comments were unclear, we also relied on interviews and
           consulted program criteria to evaluate the reasonableness of the reviewers’
           conclusions for Kaibab’s 2007 and 2008 applications. We agreed that some of the
           concerns raised by the reviewers regarding criteria for financial feasibility, reasonable
           chance of success, and public benefit were defensible. We also identified significant
           weaknesses in the proposals that were not directly addressed by the reviewers.

           We concluded that the Southwest Office’s overall decision that Kaibab’s application
           did not meet all threshold requirements in 2008 (and in 2007) was within the
           parameters of applicable program criteria. We noted that the Indian Block Grant
           project-specific threshold criteria and related guidance were nonspecific regarding
           key requirements for economic development proposals. As a result, an economic
           development proposal had to be persuasive in all respects to pass threshold screening.
           The Kaibab application did not include an independent evaluation of the feasibility
           study, which is listed in the 2008 notice as an appropriate document to include. As a
           result of this omission, we concluded that the Kaibab application was less persuasive
           to reviewers, who were not expected to be underwriters or industry experts. In
           addition, program guidance was unclear regarding the depth of review required under
           the threshold determination for financial feasibility because the 2008 notice also
           required reviewers to consider financial feasibility under one of the rating and ranking
           factors. Accordingly, reviewers had substantial discretion regarding their approach to
           evaluating financial feasibility for the threshold screening. Finally, economic
           development proposals represented less than 7 percent (in 2008) of the Indian Block
           Grant applications to the Southwest Office, and the projects were generally perceived
           to entail greater risk of failure to succeed in the long run. As a result, all economic
           development proposals—not just the Kaibab proposal—faced similar barriers to
           approval and funding.

We recommend that the Office of Native American Programs Director of Grants Management
require the Southwest Office Administrator to establish a consistent process for assignment of
grant applications to reviewers, develop standards to ensure that written review comments are




                                                 4
clear and complete, and develop a consistent evaluation approach for certain nonspecific project
eligibility criteria.

                                           Discussion

The Southwest Office Grants Management’s Indian Block Grant Application Review Processes
Had Weaknesses

Although the procedures for screening, rating, and ranking applications complied with program
requirements and were well documented, the Southwest Office practices were not standardized
regarding (1) assignment and reassignment of applications to grants management specialists, (2)
application of certain nonspecific threshold criteria, and (3) documentation of review
evaluations/comments. As a result, the application review process was not transparent to
outsiders, and allegations of unequal treatment were not easily dispelled.

       Assignment of Reviewers Was Not Consistent
       In 2007 and 2008, the Southwest Office generally assigned grants management
       specialists to review applications from the tribes for which they regularly tracked grants.
       According to Southwest Office officials, the grants management supervisor reassigned
       applications to other reviewers in some cases when the designated specialist was unable
       to complete the review within required timeframes due to workload, absence, or other
       reasons. There was apparently no protocol for application reassignments. Likewise,
       although each application also received a secondary review (usually by a ―team‖
       member), there was no protocol for designating who would perform the secondary
       review. In 2007 and 2008, the grants management supervisor reassigned the Kaibab
       application before the Kaibab specialist completed the review. However, we found no
       indication that this reassignment improperly influenced the review outcome.

       According to officials, other offices under the Office of Native American Programs used
       a panel approach for rating and ranking applications. However, the Grants Management
       Director of the Office of Native American Programs stated that the panel approach was
       preferable but not required. We noted that use of a panel or some pre-established rotation
       of application assignments to reviewers would preclude the impression that a particular
       application was directed to a certain reviewer to facilitate favorable or unfavorable
       treatment.

       Guidance for Evaluation of Nonspecific Criteria Was Lacking
       The Southwest Office had not provided training or other guidance to specialists that
       would address the unique features of economic development applications (which
       comprised less than 4 percent of the Indian Block Grant applications to the Southwest
       Office). Because the project-specific threshold criteria provided little guidance regarding
       how to assess financial feasibility or chance of success, each reviewer applied his or her
       own level of knowledge and experience when evaluating these criteria for economic
       development projects. As a result, different reviewers could reasonably apply very
       different benchmarks to the financial projections in the applications and, thus, could
       provide different feedback to applicants with similar proposals. In the case of the 2007



                                                5
       and 2008 Kaibab applications that we reviewed, the reviewers had very different levels of
       training and experience on these issues, yet they arrived at similar conclusions.

       Written Review Comments Were Not Clear
       The reviewers’ written comments related to the Kaibab application in 2008 were
       generally unclear and unconvincing. For example, the comments consisted largely of
       quotes from various parts of the applications; however, it was not always clear whether
       the quote was meant to demonstrate that the application met the requirement or fell short.
       Also, some quotations from the application seemed to have been taken out of context
       without an explanation of why other material was deemed less relevant. The comments
       for the 2007 application were also primarily quotations, but the reviewer’s conclusions
       were in italics. Nevertheless, the distinction between quotations and comments was
       unclear until the 2007 reviewer explained this during our interview. Because the written
       comments constituted the primary documentation to support a negative threshold decision
       that disqualified the application from further review, we believe that the rationale should
       have been carefully documented, even if it required a longer explanation. Further, if it is
       the Southwest Office’s or Office of Native American Programs’ policy to provide the
       comments to the applicant, we question why the comments were written in an
       abbreviated manner that made it difficult for an outsider to interpret.

Reviewers’ Overall Conclusions Were Defensible Based Upon Information in the Applications

The reviewers of record generally based their conclusions on information contained in the
applications for both the 2007 and 2008 Kaibab grant applications. Further, given the non-
specificity of some guidance (discussed in the next section of this memorandum), we concluded
that each reviewer arrived at a defensible conclusion in good faith. It should be noted that the
grant application must pass every element of the five project-specific threshold criteria for
economic development; therefore, even one deficiency will cause the application to fail the
overall threshold review. Further, the first project-specific threshold criterion for economic
development entailed five separate tests. We acknowledge that the written review comments
were unclear and did not completely support all of the reviewers’ conclusions. However, we
held discussions with the reviewers and evaluated relevant portions of the grant applications to
clarify the basis on which each application did not meet particular threshold criteria.

       Criterion (1)
       In both years, reviewers concluded that the applications did not adequately demonstrate
       that the project was financially feasible or had a reasonable chance of success—two of
       the five tests under the first criterion. As discussed in the next section, the program
       guidance for determining financial feasibility was general. As a result, much was left to
       the discretion of the reviewers who had to rely solely on the information provided in the
       application. Both reviewers stated that the applications failed to convince them of the
       project’s financial feasibility and reasonable chance of success. We agreed that some of
       the reviewers’ concerns were defensible and questioned whether applications complied
       with the Office of Native American Programs’ guidelines regarding the public benefit
       criterion.




                                                6
                 The 2008 reviewer explained that the primary concerns were that the application
                 showed negative projected net income in the first 5 years2 and did not
                 demonstrate how the project would provide management with enough experience
                 to successfully operate the proposed business. In addition, she questioned the
                 marketing plan and thought that the application had insufficient industry data to
                 make up for the absence of historical financial data. We determined that the
                 reviewer’s concerns were defensible regarding the proposal’s (1) negative net
                 income for 4 to 5 years and (2) lack of a specific plan to ensure management
                 expertise. The reviewer considered these to be significant indicators of the
                 likelihood of success, based upon her training and experience.

                 The 2007 reviewer noted that the application contained conflicting opinions
                 regarding the financial feasibility of the project and questioned the long payback
                 period and the marketing plan. In 2007, the reviewer also concluded in the
                 written comments that the application did not demonstrate an adequate public
                 benefit—another test under the first criterion. Our review of the application in
                 this regard also concluded that this was a clear weakness in the 2007 proposal.
                 We also noted that, because the application had to pass all of the threshold
                 criteria, the 2007 reviewer may have curtailed further discussion of the proposal’s
                 financial feasibility once it was determined that the proposal did not meet the
                 public benefit requirements.

        The 2007 application’s deficiencies related to public benefit were addressed to some
        extent in the 2008 proposal, and the 2008 reviewer did not note any concerns regarding
        public benefit. However, we considered the limited support for demonstrable public
        benefit to be a deficiency in the application both years. We reviewed Office of Native
        American Programs guidance that stated ―when considering the sufficiency of the
        proposed public benefit [for economic development project threshold], the reviewer
        should consider the common indicator used in the CDBG [Community Development
        Block Grant] program, that no more than $35,000 should be spent for each job
        created.‖ 3,4 In 2007, the application proposed two jobs ($605,000/2 = $302,500 per job),
        and in 2008, the application proposed five jobs ($605,000/5 = $121,000 per job). We
        questioned why Southwest Office reviewers did not discuss (either in written comments
        or during interviews) how this indicator did or did not apply to the Kaibab proposal each
        year.




2
  The 2008 written comments stated, ―Year 15 reflects only a net profit of $10,989.‖ We examined the referenced
page in the application and determined that the statement was correct; however, further scrutiny of the financial
projections revealed that the correct net profit for year 15 should have been $53,964. It is possible that the error
would have been found by the Southwest Office if the more in-depth review of the financials during the ranking and
rating process had been performed. However, the review in question was performed at threshold, and the reviewer
relied on the data without recomputation.
3
  Business Process Guide, chapter 5
4
  The CDBG program is the Community Development Block Grant program under the HUD Office of Community
Planning and Development.


                                                         7
       Criterion (4)
       In both years, the reviewers concluded that the proposals did not meet the criterion that
       the activity should not provide more than a reasonable rate of return on investment. For
       both applications, reviewer comments did not specifically state that the rate of return on
       investment appeared too high. Comments from both reviewers indicated confusion or
       imprecision regarding the intent of criterion (4). In both 2007 and 2008, the reviewer’s
       comments discussed profitability, payback period, and ―not more than a reasonable ROI
       [return on investment]‖ as if these terms were interchangeable. Further, the
       complainant’s representative stated several times that Southwest Office officials would
       not say whether the return on investment was too high or too low. To be fair, both
       reviewers’ written comments regarding criterion (4) focused on the apparent
       insufficiency of the return on investment data provided in the applications. However,
       given the reviewers’ apparent lack of understanding/clarity about the intent of criterion
       (4), if the applications had failed on this criterion alone, it could be appropriate to
       consider whether they were improperly failed at threshold. However, in both cases, the
       applications also failed under the first criterion.

              The 2008 reviewer stated that the return on investment criterion was meant to
              screen out projects for which the developer reaped large profits from a public
              investment. The reviewer for the 2008 application had also reversed her initial
              conclusion regarding return on investment after a briefing with the applicant.
              Apparently the reviewer had failed the application on this point because the
              projected rate of return was so low in the first few years. However, this response
              seemed more applicable to the first criterion regarding financial feasibility.

              Regarding the return on investment criterion, the 2007 reviewer stated that the
              payback period of 182 years was not reasonable. This response raised the
              question of whether the reviewer thought that a long payback period, which might
              indicate that the project was not financially feasible, also meant that the return on
              investment was too high. We noted that, because a long payback period would
              more likely be accompanied by a low return on investment, the statement did not
              support the criterion that the rate of return was not more than reasonable.

Indian Block Grant Economic Development Criteria and Guidance Were General

The project-specific threshold criteria for Indian Block Grant economic development projects
provided no measures or standards and were not supplemented by more specific guidance. The
threshold criteria language was almost identical among the Office of Native American Programs
review forms, the applicable notice, the regulations, and the United States Code (42 U.S.C.
5305). Office of Native American Programs officials stated that there was no guidance available
to supplement this language, which stated, ―The financial analysis must demonstrate that the
project is financially feasible and the project has a reasonable chance of success.‖ Our own
research identified minimal guidance in HUD training materials. Again, the specific guidance
primarily repeated the general language in the notice.




                                                8
The Application Must Be Persuasive
Because the criteria were imprecise, each economic development application had to
supply the HUD reviewer with persuasive information showing that the proposed project
would succeed. Accordingly, an application must demonstrate that the project will be
financially feasible by furnishing its own financial estimates and industry standards for
comparison. Because Southwest Office officials received no supplemental training or
guidance regarding how to evaluate a project’s financial feasibility or whether it had a
reasonable chance of success, they stated that the information in the application had to be
―convincing.‖ The complainant noted that the Kaibab application contained a detailed
feasibility study that included the financial statements and projections. However, we
noted that the application did not contain any evaluations or opinions of the feasibility
study by an independent underwriter or other expert on financial projections for
economic development projects (the 2008 application contained an independent cost
certification). The feasibility study was prepared by the tribal consultant/developer that
submitted the application and was also a complainant. According to information in the
application, this individual appeared to be well qualified. Nevertheless, the 2008 notice
stated that the application should appropriately include a ―Financial analysis and
feasibility study no more than two years old which indicates how the proposed business
will capture a fair share of the market, and which has been conducted by an independent
entity‖ (rating factor 3: 5.C.(1)(d)).

In this instance of an economic development proposal for a very specialized industry, we
believe that the absence of an independent review of the feasibility study was a
significant omission from the application. Without an outside expert’s opinion to rely on,
the HUD reviewers—who were not expected to be underwriters or industry experts—
appropriately relied on their own judgments and experience to assess the adequacy of
items such as projected revenues and resulting net incomes. The complainant has
requested an independent outside review of the application, yet we noted that the original
application could have included a review or certification of the feasibility study and
financial projections by an independent professional.

The 2008 Notice Rating Factors Also Addressed Financial Feasibility
According to the 2008 notice, the financial feasibility of an economic development
proposal was to be evaluated under both the threshold screening phase and the rating and
ranking phase. However, there was no guidance regarding how in-depth the threshold
screening for financial feasibility should be, as compared to the rating and ranking review
for the same item. For example, the 2008 notice’s rating and ranking requirements
covered five rating factors including one that addressed the ―soundness of the approach‖
(number 3). Under this, subfactor 5, Commitment to Sustain Activities, also required
analysis of the financial viability for an economic development proposal. The
requirements outlined somewhat specific criteria, including the statement: ―(d) Does the
business plan or cash flow analysis indicate that cash flow will be positive within the first
year?‖ Office of Native American Programs guidance for rating this notice requirement
further suggested that if the business plan or cash flow analysis indicated that the cash
flow will be positive in 2 to 3 years, the application could still be awarded points under
this subfactor. Neither reviewer referred to this criterion for their threshold evaluation.



                                          9
       Economic Development Projects Are Viewed as Having More Risk Than Other Indian
       Block Grant Activities
       Economic development projects are generally considered by SWONAP staff to be more
       high-risk ventures when compared to other Indian Block Grant project activities. We
       could not find any language in the laws or regulations indicating that a higher-risk level
       should be afforded to the economic development projects. The 2008 notice appeared to
       address the risk by requiring financial feasibility and other related tests for these projects.
       In contrast, the affordable housing projects had no similar ―financial feasibility‖ threshold
       tests. As a result, it was more difficult for economic development proposals to persuade
       review officials that the project would be as viable over time as the more traditional
       housing projects. Also, a small percentage of competitive Indian Block Grant
       applications are for economic development projects (4 percent of 2007 and 2008
       applications combined). Because Southwest Office staff had limited experience with the
       track record of such projects, it had not developed its own formal or informal indicators
       or standards to predict the success of economic development proposals. We concluded
       that all economic development proposals would have faced similar barriers to approval
       and funding.

                                    RECOMMENDATIONS

We recommend that the Office of Native American Programs Director of Grants Management
require the Southwest Office Administrator to

       1A. Develop and implement an application review assignment process to ensure that all
           applications receive equal consideration for threshold screening in appearance and in
           fact.

       1B. Develop protocols for written review comments that require reviewers to clearly
           state the basis of their conclusions. The comments should be understandable to an
           outsider or grantee.

       1C. Provide training and develop internal guidelines for evaluation of economic
           development grants that will improve consistency in the evaluation approach for key
           nonspecific eligibility criteria.

                                    AUDITEE’S RESPONSE

We provided a discussion draft memorandum report to the auditee on November 17, 2009, and
held an exit conference with its staff on December 2, 2009. The auditee provided written
comments on January 4, 2010. It generally agreed with our results, but had some disagreements
with two of the recommendations.

The complete text of the auditee’s response, along with our evaluation of that response, can be
found in appendix A of this report.




                                                 10
Appendix A

             AUDITEE COMMENTS




Comment 1




Comment 2




                    11
Comment 3




Comment 4




Comment 5




Comment 6




            12
Comment 7



Comment 8




            13
                          OIG Evaluation of Auditee Comments

Ref to OIG Evaluation                      Auditee Comments

Comment 1    We agree that Rating Factor 3, subfactor 5(c) of the 2008 Notice states that an
             independent financial analysis and feasibility study is an appropriate document to
             include, but that the project threshold provision does not specify the financial
             analysis must be independent. However, the SWONAP reviewer—who was the
             grants management supervisor at the time—clearly deemed the lack of
             independence on the part of the individual who prepared the proposal’s feasibility
             study to be a major deficiency for the threshold determination. If SWONAP
             believes this criterion was improperly applied at threshold, then the reviewer’s
             argument for questioning the feasibility study’s conclusion is significantly
             weakened. The reviewer was very clear that she relied on the proposal to be
             convincing, and that the feasibility study’s lack of independence diminished its
             credibility to her. The supervisor’s written comments demonstrated this by noting
             that the feasibility study was prepared by an entity that was ―qualified but not
             independent since also the application grant writer.‖

Comment 2    As the auditee response states, the report recognized that reviewers must rely on
             the arguments presented in the application to determine the likelihood of project
             success. During interviews, both reviewers were adamant that the information in
             the application had to convince them. Although, as acknowledged under
             comment 1, an independent feasibility study was not required by the project
             threshold determination criteria, it is reasonable to conclude that a feasibility
             study prepared by a qualified independent entity would be more convincing than
             one prepared by an individual representing the applicant. Accordingly, we stand
             by our conclusion that the lack of independence on the part of the feasibility study
             author, in the absence of an independent review or certification, significantly
             impaired the credibility of the feasibility study conclusions presented in the
             proposal. As a result, in both application years, reviewers disregarded the study’s
             conclusion (that the project was financially feasible), and relied on their own
             judgment/analyses regarding whether the feasibility study data supported a
             conclusion that the project was financially feasible and likely to succeed.

Comment 3    We revised the report to state that panels are used for rating and ranking
             applications. We cited this practice as an example of how ONAP already has
             some operating procedures in place that could help to reduce the effect of any
             individual reviewer bias (in appearance or in fact) and facilitate equal
             consideration of applications. Furthermore, the Indian Block Grant regulations do
             not preclude the use of panels for the threshold determinations. The report also
             suggested that other practices such as pre-established rotation of reviewer
             assignments would also help to reduce any impression of unequal treatment from
             different reviewers.




                                              14
Comment 4   We agree the proper comparison should be to the other Indian Block Grant
            (ICDBG) projects, and revised the report accordingly.

Comment 5   The auditee’s response does not address the recommendation that the review
            assignment process should ensure that all applications receive equal consideration
            for threshold screening in appearance and in fact. The report did not find that
            SWONAP’s processing of the 2008 Indian Block Grant application was
            inconsistent with ONAP requirements. Rather, the report stated that because the
            process to assign applications to [different] reviewers was not standardized, the
            application review process was not transparent to outsiders, and allegations of
            unequal treatment were not easily dispelled. If SWONAP continues to arbitrarily
            assign (and change assignments for) applications to reviewers with varying
            qualifications and experience, it will perpetuate the appearance of unequal
            consideration among applicants. As a result, applicants who are denied funds will
            submit further complaints and FOIA requests. A more transparent review process
            would likely reduce such complaints.

Comment 6   We acknowledge the auditee’s commitment to improve written documentation of
            reviewers’ decisions. We changed the recommendation wording to state
            ―protocols‖ in place of ―standards‖ to avoid the use of a term already employed in
            formal ONAP guidance. However, during our review, SWONAP reviewers could
            not identify any writing protocols or guidance that they used. Further, the only
            specific writing protocol we could find in the ONAP business process (BP) guide
            pertained to the rating and ranking process:

                   It is important for GM Specialists to clearly document their decision-
                   making process when rating applications. It is not enough to simply
                   indicate a page number on the rating sheet. Instead, GM Specialists must
                   provide some level of factual discussion on the rating sheet so that a clear
                   path can be shown between the rating of the information in the application
                   and the ultimate decision that is made. GM Specialists should be aware
                   that applicants could request to see the rating sheet during the debriefing
                   process.

            Our recommendation pertains to documentation of the reasons an application was
            failed at the project threshold stage. We are recommending that SWONAP
            require its reviewers to clearly state the basis of their conclusions in a manner that
            is understandable to an outsider or grantee.

Comment 7   The auditee’s response does not address the recommendation that SWONAP
            develop internal guidelines that will improve consistency in the evaluation
            approach for certain criteria. The report noted that the threshold criteria language
            for economic development activities was almost identical among the Office of
            Native American Programs review forms, the applicable notice, the regulations,
            and the United States Code (42 U.S.C. 5305). The detailed discussion and review
            checklist in the ONAP business process protocols that the auditee refers to is



                                              15
            nearly identical to the language in the applicable law and regulations. As a result,
            the reviewers must apply their own judgment regarding benchmarks used by the
            application to conclude the feasibility study demonstrated the proposal’s financial
            feasibility and reasonable chance of success. We further note that, according to
            the 2008 NOFA, ―Applicants within an Area ONAP’s geographic jurisdiction
            compete only against each other for that Area ONAP’s allocation of funds‖.
            Therefore SWONAP could appropriately augment ONAP business protocols with
            its own internal guidelines to address significant inconsistencies in evaluation
            approaches taken by its reviewers.

Comment 8   We acknowledge the auditee’s commitment to staff training regarding the ICDBG
            NOFA requirements. However, the auditee’s response does not address specific
            training for evaluation of economic development grants that will improve
            consistency among SWONAP reviewers. As we noted under comment 7,
            SWONAP’s funding allocation is dedicated to a defined group of potential
            applicants. Therefore SWONAP could appropriately provide its staff with
            additional training to promote consistency in the application of ONAP business
            protocols regarding the nonspecific criteria for economic development activities.




                                             16